prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What background do they have?
Context 1: Introduction
In June 2015, the operators of the online discussion site Reddit banned several communities under new anti-harassment rules  BIBREF0 used this opportunity to combine rich online data with computational methods to study a current question: Does eliminating these “echo chambers” diminish the amount of hate speech overall? Exciting opportunities like these, at the intersection of “thick” cultural and societal questions on the one hand, and the computational analysis of rich textual data on larger-than-human scales on the other, are becoming increasingly common 
Indeed, computational analysis is opening new possibilities for exploring challenging questions at the heart of some of the most pressing contemporary cultural and social issues  While a human reader is better equipped to make logical inferences, resolve ambiguities, and apply cultural knowledge than a computer, human time and attention are limited  Moreover, many patterns are not obvious in any specific context, but only stand out in the aggregate  For example, in a landmark study, BIBREF1 analyzed the authorship of The Federalist Papers using a statistical text analysis by focusing on style, based on the distribution of function words, rather than content  As another example, BIBREF2 studied what defines English haiku and showed how computational analysis and close reading can complement each other  Computational approaches are valuable precisely because they help us identify patterns that would not otherwise be discernible 
Yet these approaches are not a panacea  Examining thick social and cultural questions using computational text analysis carries significant challenges  For one, texts are culturally and socially situated  They reflect the ideas, values and beliefs of both their authors and their target audiences, and such subtleties of meaning and interpretation are difficult to incorporate in computational approaches  For another, many of the social and cultural concepts we seek to examine are highly contested — hate speech is just one such example  Choices regarding how to operationalize and analyze these concepts can raise serious concerns about conceptual validity and may lead to shallow or obvious conclusions, rather than findings that reflect the depth of the questions we seek to address 
These are just a small sample of the many opportunities and challenges faced in computational analyses of textual data  New possibilities and frustrating obstacles emerge at every stage of research, from identification of the research question to interpretation of the results  In this article, we take the reader through a typical research process that involves measuring social or cultural concepts using computational methods, discussing both the opportunities and complications that often arise  In the Reddit case, for example, hate speech is measured, however imperfectly, by the presence of particular words semi-automatically extracted from a machine learning algorithm  Operationalizations are never perfect translations, and are often refined over the course of an investigation, but they are crucial 
We begin our exploration with the identification of research questions, proceed through data selection, conceptualization, and operationalization, and end with analysis and the interpretation of results  The research process sounds more or less linear this way, but each of these phases overlaps, and in some instances turns back upon itself  The analysis phase, for example, often feeds back into the original research questions, which may continue to evolve for much of the project  At each stage, our discussion is critically informed by insights from the humanities and social sciences, fields that have focused on, and worked to tackle, the challenges of textual analysis—albeit at smaller scales—since their inception 
In describing our experiences with computational text analysis, we hope to achieve three primary goals  First, we aim to shed light on thorny issues not always at the forefront of discussions about computational text analysis methods  Second, we hope to provide a set of best practices for working with thick social and cultural concepts  Our guidance is based on our own experiences and is therefore inherently imperfect  Still, given our diversity of disciplinary backgrounds and research practices, we hope to capture a range of ideas and identify commonalities that will resonate for many  And this leads to our final goal: to help promote interdisciplinary collaborations  Interdisciplinary insights and partnerships are essential for realizing the full potential of any computational text analysis that involves social and cultural concepts, and the more we are able to bridge these divides, the more fruitful we believe our work will be 
Research questions
We typically start by identifying the questions we wish to explore  Can text analysis provide a new perspective on a “big question” that has been attracting interest for years? Or can we raise new questions that have only recently emerged, for example about social media? For social scientists working in computational analysis, the questions are often grounded in theory, asking: How can we explain what we observe? These questions are also influenced by the availability and accessibility of data sources  For example, the choice to work with data from a particular social media platform may be partly determined by the fact that it is freely available, and this will in turn shape the kinds of questions that can be asked  A key output of this phase are the concepts to measure, for example: influence; copying and reproduction; the creation of patterns of language use; hate speech  Computational analysis of text motivated by these questions is insight driven: we aim to describe a phenomenon or explain how it came about  For example, what can we learn about how and why hate speech is used or how this changes over time? Is hate speech one thing, or does it comprise multiple forms of expression? Is there a clear boundary between hate speech and other types of speech, and what features make it more or less ambiguous? In these cases, it is critical to communicate high-level patterns in terms that are recognizable 
This contrasts with much of the work in computational text analysis, which tends to focus on automating tasks that humans perform inefficiently  These tasks range from core linguistically motivated tasks that constitute the backbone of natural language processing, such as part-of-speech tagging and parsing, to filtering spam and detecting sentiment  Many tasks are motivated by applications, for example to automatically block online trolls  Success, then, is often measured by performance, and communicating why a certain prediction was made—for example, why a document was labeled as positive sentiment, or why a word was classified as a noun—is less important than the accuracy of the prediction itself  The approaches we use and what we mean by `success' are thus guided by our research questions 
Domain experts and fellow researchers can provide feedback on questions and help with dynamically revising them  For example, they may say “we already think we know that”, “that's too naïve”, “that doesn't reflect social reality” (negative); “two major camps in the field would give different answers to that question” (neutral); “we tried to look at that back in the 1960s, but we didn't have the technology” (positive); and “that sounds like something that people who made that archive would love”, “that's a really fundamental question” (very positive) 
Sometimes we also hope to connect to multiple disciplines  For example, while focusing on the humanistic concerns of an archive, we could also ask social questions such as “is this archive more about collaborative processes, culture-building or norm creation?” or “how well does this archive reflect the society in which it is embedded?" BIBREF3 used quantitative methods to tell a story about Darwin's intellectual development—an essential biographical question for a key figure in the history of science  At the same time, their methods connected Darwin's development to the changing landscape of Victorian scientific culture, allowing them to contrast Darwin's “foraging” in the scientific literature of his time to the ways in which that literature was itself produced  Finally, their methods provided a case study, and validation of technical approaches, for cognitive scientists who are interested in how people explore and exploit sources of knowledge 
Questions about potential “dual use” may also arise  Returning to our introductory example, BIBREF0 started with a deceptively simple question: if an internet platform eliminates forums for hate speech, does this impact hate speech in other forums? The research was motivated by the belief that a rising tide of online hate speech was (and is) making the internet increasingly unfriendly for disempowered groups, including minorities, women, and LBGTQ individuals  Yet the possibility of dual use troubled the researchers from the onset  Could the methodology be adopted to target the speech of groups like Black Lives Matter? Could it be adopted by repressive governments to minimize online dissent? While these concerns remained, they concluded that hypothetical dual use scenarios did not outweigh the tangible contribution this research could offer towards making the online environment more equal and just 
Data
The next step involves deciding on the data sources, collecting and compiling the dataset, and inspecting its metadata 
Data acquisition
Many scholars in the humanities and the social sciences work with sources that are not available in digital form, and indeed may never be digitized  Others work with both analogue and digitized materials, and the increasing digitization of archives has opened opportunities to study these archives in new ways  We can go to the canonical archive or open up something that nobody has studied before  For example, we might focus on major historical moments (French Revolution, post-Milosevic Serbia) or critical epochs (Britain entering the Victorian era, the transition from Latin to proto-Romance)  Or, we could look for records of how people conducted science, wrote and consumed literature, and worked out their philosophies 
A growing number of researchers work with born-digital sources or data  Born-digital data, e g , from social media, generally do not involve direct elicitation from participants and therefore enable unobtrusive measurements BIBREF5 , BIBREF6   In contrast, methods like surveys sometimes elicit altered responses from participants, who might adapt their responses to what they think is expected  Moreover, born-digital data is often massive, enabling large-scale studies of language and behavior in a variety of social contexts 
Still, many scholars in the social sciences and humanities work with multiple data sources  The variety of sources typically used means that more than one data collection method is often required  For example, a project examining coverage of a UK General Election, could draw data from traditional media, web archives, Twitter and Facebook, campaign manifestos, etc  and might combine textual analysis of these materials with surveys, laboratory experiments, or field observations offline  In contrast, many computational studies based on born-digital data have focused on one specific source, such as Twitter 
The use of born-digital data raises ethical concerns  Although early studies often treated privacy as a binary construct, many now acknowledge its complexity BIBREF7   Conversations on private matters can be posted online, visible for all, but social norms regarding what should be considered public information may differ from the data's explicit visibility settings  Often no informed consent has been obtained, raising concerns and challenges regarding publishing content and potentially harmful secondary uses BIBREF8 , BIBREF4  
Recently, concerns about potential harms stemming from secondary uses have led a number of digital service providers to restrict access to born-digital data  Facebook and Twitter, for example, have reduced or eliminated public access to their application programming interfaces (APIs) and expressed hesitation about allowing academic researchers to use data from their platforms to examine certain sensitive or controversial topics  Despite the seeming abundance of born-digital data, we therefore cannot take its availability for granted 
Working with data that someone else has acquired presents additional problems related to provenance and contextualisation  It may not always be possible to determine the criteria applied during the creation process  For example, why were certain newspapers digitized but not others, and what does this say about the collection? Similar questions arise with the use of born-digital data  For instance, when using the Internet Archive’s Wayback Machine to gather data from archived web pages, we need to consider what pages were captured, which are likely missing, and why 
We must often repurpose born-digital data (e g , Twitter was not designed to measure public opinion), but data biases may lead to spurious results and limit justification for generalization  In particular, data collected via black box APIs designed for commercial, not research, purposes are likely to introduce biases into the inferences we draw, and the closed nature of these APIs means we rarely know what biases are introduced, let alone how severely they might impact our research BIBREF10   These, however, are not new problems  Historians, for example, have always understood that their sources were produced within particular contexts and for particular purposes, which are not always apparent to us 
Non-representative data can still be useful for making comparisons within a sample  In the introductory example on hate speech BIBREF0 , the Reddit forums do not present a comprehensive or balanced picture of hate speech: the writing is almost exclusively in English, the targets of hate speech are mainly restricted (e g , to black people, or women), and the population of writers is shaped by Reddit's demographics, which skew towards young white men  These biases limit the generalizability of the findings, which cannot be extrapolated to other languages, other types of hate speech, and other demographic groups  However, because the findings are based on measurements on the same sort of hate speech and the same population of writers, as long as the collected data are representative of this specific population, these biases do not pose an intractable validity problem if claims are properly restricted 
The size of many newly available datasets is one of their most appealing characteristics  Bigger datasets often make statistics more robust  The size needed for a computational text analysis depends on the research goal: When it involves studying rare events, bigger datasets are needed  However, larger is not always better  Some very large archives are “secretly” collections of multiple and distinct processes that no in-field scholar would consider related  For example, Google Books is frequently used to study cultural patterns, but the over-representation of scientific articles in Google books can be problematic BIBREF11   Even very large born-digital datasets usually cover limited timespans compared to, e g , the Gutenberg archive of British novels 
This stage of the research also raises important questions about fairness  Are marginalized groups, for example, represented in the tweets we have collected? If not, what types of biases might result from analyses relying on those tweets?
Local experts and “informants” can help navigate the data  They can help understand the role an archive plays in the time and place  They might tell us: Is this the central archive, or a peripheral one? What makes it unusual? Or they might tell us how certain underrepresented communities use a social media platform and advise us on strategies for ensuring our data collection includes their perspectives 
However, when it is practically infeasible to navigate the data in this way—for instance, when we cannot determine what is missing from Twitter's Streaming API or what webpages are left out of the Internet Archive—we should be open about the limitations of our analyses, acknowledging the flaws in our data and drawing cautious and reasonable conclusions from them  In all cases, we should report the choices we have made when creating or re-using any dataset 
Compiling data
After identifying the data source(s), the next step is compiling the data  This step is fundamental: if the sources cannot support a convincing result, no result will be convincing  In many cases, this involves defining a “core" set of documents and a “comparison" set  We often have a specific set of documents in mind: an author's work, a particular journal, a time period  But if we want to say that this “core" set has some distinctive property, we need a “comparison" set  Expanding the collection beyond the documents that we would immediately think of has the beneficial effect of increasing our sample size  Having more sources increases the chance that we will notice something consistent across many individually varying contexts 
Comparing sets of documents can sometimes support causal inference, presented as a contrast between a treatment group and a control  In BIBREF0 , the treatment consisted of the text written in the two forums that were eventually closed by Reddit  However, identifying a control group required a considerable amount of time and effort  Reddit is a diverse platform, with a wide variety of interactional and linguistic styles; it would be pointless to compare hate speech forums against forums dedicated to, say, pictures of wrecked bicycles  Chandrasekharan et al  used a matching design, populating the control group with forums that were as similar as possible to the treatment group, but were not banned from Reddit  The goal is to estimate the counterfactual scenario: in this case, what would have happened had the site not taken action against these specific forums? An ideal control would make it possible to distinguish the effect of the treatment — closing the forums — from other idiosyncratic properties of texts that were treated 
We also look for categories of documents that might not be useful  We might remove documents that are meta-discourse, like introductions and notes, or documents that are in a language that is not the primary language of the collection, or duplicates when we are working with archived web pages  However, we need to carefully consider the potential consequences of information we remove  Does its removal alter the data, or the interpretation of the data, we are analyzing? Are we losing anything that might be valuable at a later stage?
Labels and metadata
Sometimes all we have is documents, but often we want to look at documents in the context of some additional information, or metadata  This additional information could tell us about the creation of documents (date, author, forum), or about the reception of documents (flagged as hate speech, helpful review)  Information about text segments can be extremely valuable, but it is also prone to errors, inconsistencies, bias, and missing information  Examining metadata is a good way to check a collection's balance and representativeness  Are sources disproportionately of one form? Is the collection missing a specific time window? This type of curation can be extremely time consuming as it may require expert labeling, but it often leads to the most compelling results  Sometimes metadata are also used as target labels to develop machine learning models  But using them as a “ground truth” requires caution  Labels sometimes mean something different than we expect  For example, a down vote for a social media post could indicate that the content is offensive, or that the voter simply disagreed with the expressed view 
Conceptualization
A core step in many analyses is translating social and cultural concepts (such as hate speech, rumor, or conversion) into measurable quantities  Before we can develop measurements for these concepts (the operationalization step, or the “implementation” step as denoted by BIBREF12 ), we need to define them  In the conceptualization phase we often start with questions such as: who are the domain experts, and how have they approached the topic? We are looking for a definition of the concept that is flexible enough to apply on our dataset, yet formal enough for computational research  For example, our introductory study on hate speech BIBREF0 used a statement on hate speech produced by the European Union Court of Human Rights  The goal was not to implement this definition directly in software but to use it as a reference point to anchor subsequent analyses 
If we want to move beyond the use of ad hoc definitions, it can be useful to distinguish between what political scientists Adcock and Collier call the “background concept” and the “systematized concept” BIBREF13   The background concept comprises the full and diverse set of meanings that might be associated with a particular term  This involves delving into theoretical, conceptual, and empirical studies to assess how a concept has been defined by other scholars and, most importantly, to determine which definition is most appropriate for the particular research question and the theoretical framework in which it is situated  That definition, in turn, represents the systematized concept: the formulation that is adopted for the study 
It is important to consider that for social and cultural concepts there is no absolute ground truth  There are often multiple valid definitions for a concept (the “background” concept in the terms of Adcock and Collier), and definitions might be contested over time  This may be uncomfortable for computer scientists, whose primary measure of success is often based on comparing a model's output against “ground truth” or a “gold standard”, e g , by comparing a sentiment classifier's output against manual annotations  However, the notion of ground truth is uncommon in the humanities and the social sciences and it is often taken too far in machine learning  BIBREF14 notes that in literary criticism and the digital humanities more broadly “interpretation, ambiguity, and argumentation are prized far above ground truth and definitive conclusions"  BIBREF15 draw attention to the different attitudes of literary scholars and computational linguists towards ambiguity, stating that “In Computational Linguistics [  ] ambiguity is almost uniformly treated as a problem to be solved; the focus is on disambiguation, with the assumption that one true, correct interpretation exists " The latter is probably true for tasks such as spam filtering, but in the social sciences and the humanities many relevant concepts are fundamentally unobservable, such as latent traits of political actors BIBREF16 or cultural fit in organizations BIBREF17 , leading to validation challenges  Moreover, when the ground truth comes from people, it may be influenced by ideological priors, priming, simple differences of opinion or perspective, and many other factors BIBREF18   We return to this issue in our discussions on validation and analysis 
Operationalization
In this phase we develop measures (or, “operationalizations”, or “indicators”) for the concepts of interest, a process called “operationalization”  Regardless of whether we are working with computers, the output produced coincides with Adcock and Collier's “scores”—the concrete translation and output of the systematized concept into numbers or labels BIBREF13   Choices made during this phase are always tied to the question “Are we measuring what we intend to measure?” Does our operationalization match our conceptual definition? To ensure validity we must recognize gaps between what is important and what is easy to measure  We first discuss modeling considerations  Next, we describe several frequently used computational approaches and their limitations and strengths 
Modeling considerations
The variables (both predictors and outcomes) are rarely simply binary or categorical  For example, a study on language use and age could focus on chronological age (instead of, e g , social age BIBREF19 )  However, even then, age can be modeled in different ways  Discretization can make the modeling easier and various NLP studies have modeled age as a categorical variable BIBREF20   But any discretization raises questions: How many categories? Where to place the boundaries? Fine distinctions might not always be meaningful for the analysis we are interested in, but categories that are too broad can threaten validity  Other interesting variables include time, space, and even the social network position of the author  It is often preferable to keep the variable in its most precise form  For example, BIBREF21 perform exploration in the context of hypothesis testing by using latitude and longitude coordinates — the original metadata attached to geotagged social media such as tweets — rather than aggregating into administrative units such as counties or cities  This is necessary when such administrative units are unlikely to be related to the target concept, as is the case in their analysis of dialect differences  Focusing on precise geographical coordinates also makes it possible to recognize fine-grained effects, such as language variation across the geography of a city 
Using a particular classification scheme means deciding which variations are visible, and which ones are hidden BIBREF22   We are looking for a categorization scheme for which it is feasible to collect a large enough labeled document collection (e g , to train supervised models), but which is also fine-grained enough for our purposes  Classification schemes rarely exhibit the ideal properties, i e , that they are consistent, their categories are mutually exclusive, and that the system is complete BIBREF22   Borderline cases are challenging, especially with social and cultural concepts, where the boundaries are often not clear-cut  The choice of scheme can also have ethical implications BIBREF22   For example, gender is usually represented as a binary variable in NLP and computational models tend to learn gender-stereotypical patterns  The operationalization of gender in NLP has been challenged only recently BIBREF23 , BIBREF24 , BIBREF25  
Supervised and unsupervised learning are the most common approaches to learning from data  With supervised learning, a model learns from labeled data (e g , social media messages labeled by sentiment) to infer (or predict) these labels from unlabeled texts  In contrast, unsupervised learning uses unlabeled data  Supervised approaches are especially suitable when we have a clear definition of the concept of interest and when labels are available (either annotated or native to the data)  Unsupervised approaches, such as topic models, are especially useful for exploration  In this setting, conceptualization and operationalization may occur simultaneously, with theory emerging from the data BIBREF26   Unsupervised approaches are also used when there is a clear way of measuring a concept, often based on strong assumptions  For example, BIBREF3 measure “surprise” in an analysis of Darwin's reading decisions based on the divergence between two probability distributions 
From an analysis perspective, the unit of text that we are labeling (or annotating, or coding), either automatic or manual, can sometimes be different than one's final unit of analysis  For example, if in a study on media frames in news stories, the theoretical framework and research question point toward frames at the story level (e g , what is the overall causal analysis of the news article?), the story must be the unit of analysis  Yet it is often difficult to validly and reliably code a single frame at the story level  Multiple perspectives are likely to sit side-by-side in a story  Thus, an article on income inequality might point to multiple causes, such as globalization, education, and tax policies  Coding at the sentence level would detect each of these causal explanations individually, but this information would need to be somehow aggregated to determine the overall story-level frame  Sometimes scholars solve this problem by only examining headlines and lead paragraphs, arguing that based on journalistic convention, the most important information can be found at the beginning of a story  However, this leads to a return to a shorter, less nuanced analysis 
From a computational perspective, the unit of text can also make a huge difference, especially when we are using bag-of-words models, where word order within a unit does not matter  Small segments, like tweets, sometimes do not have enough information to make their semantic context clear  In contrast, larger segments, like novels, have too much variation, making it difficult to train focused models  Finding a good segmentation sometimes means combining short documents and subdividing long documents  The word “document" can therefore be misleading  But it is so ingrained in the common NLP lexicon that we use it anyway in this article 
For insight-driven text analysis, it is often critical that high-level patterns can be communicated  Furthermore, interpretable models make it easier to find spurious features, to do error analysis, and to support interpretation of results  Some approaches are effective for prediction, but harder to interpret  The value we place on interpretability can therefore influence the approach we choose  There is an increasing interest in developing interpretable or transparent models in the NLP and machine learning communities 
Annotation
Many studies involve human coders  Sometimes the goal is to fully code the data, but in a computational analysis we often use the labels (or annotations) to train machine learning models to automatically recognize them, and to identify language patterns that are associated with these labels  For example, for a project analyzing rumors online BIBREF27 , conversation threads were annotated along different dimensions, including rumor versus non-rumor and stance towards a rumor 
The collection of annotation choices make up an annotation scheme (or “codebook”)  Existing schemes and annotations can be useful as starting points  Usually settling on an annotation scheme requires several iterations, in which the guidelines are updated and annotation examples are added  For example, a political scientist could use a mixed deductive-inductive strategy for developing a codebook  She starts by laying out a set of theory-driven deductive coding rules, which means that the broad principles of the coding rules are laid out without examining examples first  These are then tested (and possibly adjusted) based on a sample of the data  In line with Adcock and Collier's notion of “content validity” BIBREF13 , the goal is to assess whether the codebook adequately captures the systematized concept  By looking at the data themselves, she gains a better sense of whether some things have been left out of the coding rules and whether anything is superfluous, misleading, or confusing  Adjustments are made and the process is repeated, often with another researcher involved 
The final annotations can be collected using a crowdsourcing platform, a smaller number of highly-trained annotators, or a group of experts  Which type of annotator to use should be informed by the complexity and specificity of the concept  For more complex concepts, highly-trained or expert annotators tend to produce more reliable results  However, complex concepts can sometimes be broken down into micro-tasks that can be performed independently in parallel by crowdsourced annotators  Concepts from highly specialized domains may require expert annotators  In all cases, however, some training will be required, and the training phase should involve continual checks of inter-annotator agreement (i e  intercoder reliability) or checks against a gold standard (e g  quizzes in crowdsourcing platforms) 
We also need to decide how inter-annotator agreement will be measured and what an acceptable level of agreement would be  Krippendorff's alpha is frequently used in the social sciences, but the right measure depends on the type of data and task  For manual coding, we can continually check inter-annotator agreement and begin introducing checks of intra-annotator agreement, too  For most communication scholars using only manual content analysis, an acceptable rate of agreement is achieved when Krippendorf's alpha reaches 0 80 or above  When human-coded data are used to validate machine learning algorithms, the reliability of the human-coded data is even more important  Disagreement between annotators can signal weaknesses of the annotation scheme, or highlight the inherent ambiguity in what we are trying to measure  Disagreement itself can be meaningful and can be integrated in subsequent analyses BIBREF28 , BIBREF29  
Data pre-processing
Preparing the data can be a complex and time-consuming process, often involving working with partially or wholly unstructured data  The pre-processing steps have a big impact on the operationalizations, subsequent analyses and reproducibility efforts BIBREF30 , and they are usually tightly linked to what we intend to measure  Unfortunately, these steps tend to be underreported, but documenting the pre-processing choices made is essential and is analogous to recording the decisions taken during the production of a scholarly edition or protocols in biomedical research  Data may also vary enormously in quality, depending on how it has been generated  Many historians, for example, work with text produced from an analogue original using Optical Character Recognition (OCR)  Often, there will be limited information available regarding the accuracy of the OCR, and the degree of accuracy may even vary within a single corpus (e g  where digitized text has been produced over a period of years, and the software has gradually improved)  The first step, then, is to try to correct for common OCR errors  These will vary depending on the type of text, the date at which the `original' was produced, and the nature of the font and typesetting 
One step that almost everyone takes is to tokenize the original character sequence into the words and word-like units  Tokenization is a more subtle and more powerful process than people expect  It is often done using regular expressions or scripts that have been circulating within the NLP community  Tokenization heuristics, however, can be badly confused by emoticons, creative orthography (e g , U$A, sh!t), and missing whitespace  Multi-word terms are also challenging  Treating them as a single unit can dramatically alter the patterns in text  Many words that are individually ambiguous have clear, unmistakable meanings as terms, like “black hole" or “European Union"  However, deciding what constitutes a multi-word term is a difficult problem  In writing systems like Chinese, tokenization is a research problem in its own right 
Beyond tokenization, common steps include lowercasing, removing punctuation, stemming (removing suffixes), lemmatization (converting inflections to a base lemma), and normalization, which has never been clearly defined, but often includes grouping abbreviations like “U S A " and “USA", ordinals like “1st" and “first", and variant spellings like “noooooo"  The main goal of these steps is to improve the ratio of tokens (individual occurrences) to types (the distinct things in a corpus)  Each step requires making additional assumptions about which distinctions are relevant: is “apple” different from “Apple”? Is “burnt” different from “burned”? Is “cool" different from “coooool"? Sometimes these steps can actively hide useful patterns, like social meaning BIBREF32   Some of us therefore try do as little modification as possible 
From a multilingual perspective, English and Chinese have an unusually simple inflectional system, and so it is statistically reasonable to treat each inflection as a unique word type  Romance languages have considerably more inflections than English; many indigenous North American languages have still more  For these languages, unseen data is far more likely to include previously-unseen inflections, and therefore, dealing with inflections is more important  On the other hand, the resources for handling inflections vary greatly by language, with European languages dominating the attention of the computational linguistics community thus far 
We sometimes also remove words that are not relevant to our goals, for example by calculating vocabulary frequencies  We construct a “stoplist” of words that we are not interested in  If we are looking for semantic themes we might remove function words like determiners and prepositions  If we are looking for author-specific styles, we might remove all words except function words  Some words are generally meaningful but too frequent to be useful within a specific collection  We sometimes also remove very infrequent words  Their occurrences are too low for robust patterns and removing them helps reducing the vocabulary size 
The choice of processing steps can be guided by theory or knowledge about the domain as well as experimental investigation  When we have labels, predictive accuracy of a model is a way to assess the effect of the processing steps  In unsupervised settings, it is more challenging to understand the effects of different steps  Inferences drawn from unsupervised settings can be sensitive to pre-processing choices BIBREF33   Stemming has been found to provide little measurable benefits for topic modeling and can sometimes even be harmful BIBREF34   All in all, this again highlights the need to document these steps 
Finally, we can also mark up the data, e g , by identifying entities (people, places, organizations, etc ) or parts of speech  Although many NLP tools are available for such tasks, they are often challenged by linguistic variation, such as orthographic variation in historical texts BIBREF35 and social media BIBREF32   Moreover, the performance of NLP tools often drops when applying them outside the training domain, such as applying tools developed on newswire texts to texts written by younger authors BIBREF36   Problems (e g , disambiguation in named entity recognition) are sometimes resolved using considerable manual intervention  This combination of the automated and the manual, however, becomes more difficult as the scale of the data increases, and the `certainty' brought by the latter may have to be abandoned 
Dictionary-based approaches
Dictionaries are frequently used to code texts in content analyses BIBREF37   Dictionaries consist of one or more categories (i e  word lists)  Sometimes the output is simply the number of category occurrences (e g , positive sentiment), thus weighting words within a category equally  In some other cases, words are assigned continuous scores  The high transparency of dictionaries makes them sometimes more suitable than supervised machine learning models  However, dictionaries should only be used if the scores assigned to words match how the words are used in the data (see BIBREF38 for a detailed discussion on limitations)  There are many off-the-shelf dictionaries available (e g , LIWC BIBREF39 )  These are often well-validated, but applying them on a new domain may not be appropriate without additional validation  Corpus- or domain-specific dictionaries can overcome limitations of general-purpose dictionaries 
The dictionaries are often manually compiled, but increasingly they are constructed semi-automatically (e g , BIBREF40 )  When we semi-automatically create a word list, we use automation to identify an initial word list, and human insight to filter it  By automatically generating the initial words lists, words can be identified that human annotators might have difficulty intuiting  By manually filtering the lists, we use our theoretical understanding of the target concept to remove spurious features 
In the introduction study, SAGE BIBREF41 was used to obtain a list of words that distinguished the text in the treatment group (subreddits that were closed by Reddit) from text in the control group (similar subreddits that were not closed)  The researchers then returned to the hate speech definition provided by the European Court of Human Rights, and manually filtered the top SAGE words based on this definition  Not all identified words fitted the definition  The others included: the names of the subreddits themselves, names of related subreddits, community-specific jargon that was not directly related to hate speech, and terms such as IQ and welfare, which were frequently used in discourses of hate speech, but had significant other uses  The word lists provided the measurement instrument for their main result, which is that the use of hate speech throughout Reddit declined after the two treatment subreddits were closed 
Supervised models
Supervised learning is frequently used to scale up analyses  For example, BIBREF42 wanted to analyze the motivations of Movember campaign participants  By developing a classifier based on a small set of annotations, they were able to expand the analysis to over 90k participants 
The choice of supervised learning model is often guided by the task definition and the label types  For example, to identify stance towards rumors based on sequential annotations, an algorithm for learning from sequential BIBREF43 or time series data BIBREF44 could be used  The features (sometimes called variables or predictors) are used by the model to make the predictions  They may vary from content-based features such as single words, sequences of words, or information about their syntactic structure, to meta-information such as user or network information  Deciding on the features requires experimentation and expert insight and is often called feature engineering  For insight-driven analysis, we are often interested in why a prediction has been made and features that can be interpreted by humans may be preferred  Recent neural network approaches often use simple features as input (such as word embeddings or character sequences), which requires less feature engineering but make interpretation more difficult 
Supervised models are powerful, but they can latch on to spurious features of the dataset  This is particularly true for datasets that are not well-balanced, and for annotations that are noisy  In our introductory example on hate speech in Reddit BIBREF0 , the annotations are automatically derived from the forum in which each post appears, and indeed, many of the posts in the forums (subreddits) that were banned by Reddit would be perceived by many as hate speech  But even in banned subreddits, not all of the content is hate speech (e g , some of the top features were self-referential like the name of the subreddit) but a classifier would learn a high weight for these features 
Even when expert annotations are available on the level of individual posts, spurious features may remain  BIBREF45 produced expert annotations of hate speech on Twitter  They found that one of the strongest features for sexism is the name of an Australian TV show, because people like to post sexist comments about the contestants  If we are trying to make claims about what inhibits or encourages hate speech, we would not want those claims to be tied to the TV show's popularity  Such problems are inevitable when datasets are not well-balanced over time, across genres, topics, etc  Especially with social media data, we lack a clear and objective definition of `balance' at this time 
The risk of supervised models latching on to spurious features reinforces the need for interpretability  Although the development of supervised models is usually performance driven, placing more emphasis on interpretability could increase the adoption of these models in insight-driven analyses  One way would be to only use models that are already somewhat interpretable, for example models that use a small number of human-interpretable features  Rather than imposing such restrictions, there is also work on generating post-hoc explanations for individual predictions (e g , BIBREF46 ), even when the underlying model itself is very complex 
Topic modeling
Topic models (e g , LDA BIBREF47 ) are usually unsupervised and therefore less biased towards human-defined categories  They are especially suited for insight-driven analysis, because they are constrained in ways that make their output interpretable  Although there is no guarantee that a “topic” will correspond to a recognizable theme or event or discourse, they often do so in ways that other methods do not  Their easy applicability without supervision and ready interpretability make topic models good for exploration  Topic models are less successful for many performance-driven applications  Raw word features are almost always better than topics for search and document classification  LSTMs and other neural network models are better as language models  Continuous word embeddings have more expressive power to represent fine-grained semantic similarities between words 
A topic model provides a different perspective on a collection  It creates a set of probability distributions over the vocabulary of the collection, which, when combined together in different proportions, best match the content of the collection  We can sort the words in each of these distributions in descending order by probability, take some arbitrary number of most-probable words, and get a sense of what (if anything) the topic is “about”  Each of the text segments also has its own distribution over the topics, and we can sort these segments by their probability within a given topic to get a sense of how that topic is used 
One of the most common questions about topic models is how many topics to use, usually with the implicit assumption that there is a “right” number that is inherent in the collection  We prefer to think of this parameter as more like the scale of a map or the magnification of a microscope  The “right” number is determined by the needs of the user, not by the collection  If the analyst is looking for a broad overview, a relatively small number of topics may be best  If the analyst is looking for fine-grained phenomena, a larger number is better 
After fitting the model, it may be necessary to circle back to an earlier phase  Topic models find consistent patterns  When authors repeatedly use a particular theme or discourse, that repetition creates a consistent pattern  But other factors can also create similar patterns, which look as good to the algorithm  We might notice a topic that has highest probability on French stopwords, indicating that we need to do a better job of filtering by language  We might notice a topic of word fragments, such as “ing”, “tion”, “inter”, indicating that we are not handling end-of-line hyphenation correctly  We may need to add to our stoplist or change how we curate multi-word terms 
Validation
The output of our measurement procedures (in the social sciences often called the “scores”) must now be assessed in terms of their reliability and validity with regard to the (systemized) concept  Reliability aims to capture repeatability, i e  the extent to which a given tool provides consistent results 
Validity assesses the extent to which a given measurement tool measures what it is supposed to measure  In NLP and machine learning, most models are primarily evaluated by comparing the machine-generated labels against an annotated sample  This approach presumes that the human output is the “gold standard" against which performance should be tested  In contrast, when the reliability is measured based on the output of different annotators, no coder is taken as the standard and the likelihood of coders reaching agreement by chance (rather than because they are “correct") is factored into the resulting statistic  Comparing against a “gold standard” suggests that the threshold for human inter- and intra-coder reliability should be particularly high 
Accuracy, as well as other measures such as precision, recall and F-score, are sometimes presented as a measure of validity, but if we do not have a genuinely objective determination of what something is supposed measure—as is often the case in text analysis—then accuracy is perhaps a better indication of reliability than of validity  In that case, validity needs to be assessed based on other techniques like those we discuss later in this section  It is also worth asking what level of accuracy is sufficient for our analysis and to what extent there may be an upper bound, especially when the labels are native to the data or when the notion of a “gold standard” is not appropriate 
For some in the humanities, validation takes the form of close reading, not designed to confirm whether the model output is correct, but to present what BIBREF48 refers to as a form of “further discovery in two directions”  Model outputs tell us something about the texts, while a close reading of the texts alongside those outputs tells us something about the models that can be used for more effective model building  Applying this circular, iterative process to 450 18th-century novels written in three languages, Piper was able to uncover a new form of “conversional novel” that was not previously captured in “literary history's received critical categories” BIBREF48  
Along similar lines, we can subject both the machine-generated output and the human annotations to another round of content validation  That is, take a stratified random sample, selecting observations from the full range of scores, and ask: Do these make sense in light of the systematized concept? If not, what seems to be missing? Or is something extraneous being captured? This is primarily a qualitative process that requires returning to theory and interrogating the systematized concept, indicators, and scores together  This type of validation is rarely done in NLP, but it is especially important when it is difficult to assess what drives a given machine learning model  If there is a mismatch between the scores and systematized concept at this stage, the codebook may need to be adjusted, human coders retrained, more training data prepared, algorithms adjusted, or in some instances, even a new analytical method adopted 
Other types of validation are also possible, such as comparing with other approaches that aim to capture the same concept, or comparing the output with external measures (e g , public opinion polls, the occurrence of future events)  We can also go beyond only evaluating the labels (or point estimates)  BIBREF16 used human judgments to not only assess the positional estimates from a scaling method of latent political traits but also to assess uncertainty intervals  Using different types of validation can increase our confidence in the approach, especially when there is no clear notion of ground truth 
Besides focusing on rather abstract evaluation measures, we could also assess the models in task-based settings using human experts  Furthermore, for insight-driven analyses, it can be more useful to focus on improving explanatory power than making small improvements in predictive performance 
Analysis
In this phase, we use our models to explore or answer our research questions  For example, given a topic model we can look at the connection between topics and metadata elements  Tags such as “hate speech" or metadata information imply a certain way of organizing the collection  Computational models provide another organization, which may differ in ways that provide more insight into how these categories manifest themselves, or fail to do so 
Moreover, when using a supervised approach, the “errors”, i e  disagreement between the system output and human-provided labels, can point towards interesting cases for closer analysis and help us reflect on our conceptualizations  In the words of BIBREF2 , they can be “opportunities for interpretation”  Other types of “failures” can be insightful as well  Sometimes there is a “dog that didn't bark” BIBREF49 –i e , something that everyone thinks we should have found, but we did not  Or, sometimes the failures are telling us about the existence of something in the data that nobody noticed, or thought important, until then (e g , the large number of travel journals in Darwin's reading lists) 
Computational text analysis is not a replacement for but rather an addition to the approaches one can take to analyze social and cultural phenomena using textual data  By moving back and forth between large-scale computational analyses and small-scale qualitative analyses, we can combine their strengths so that we can identify large-scale and long-term trends, but also tell individual stories  For example, the Reddit study on hate speech BIBREF0 raised various follow-up questions: Can we distinguish hate speech from people talking about hate speech? Did people find new ways to express hate speech? If so, did the total amount of online hate speech decrease after all? As possible next steps, a qualitative discourse analyst might examine a smaller corpus to investigate whether commenters were indeed expressing hate speech in new ways; a specialist in interview methodologies might reach out to commenters to better understand the role of online hate speech in their lives  Computational text analysis represents a step towards better understanding social and cultural phenomena, and it is in many cases better suited towards opening questions rather than closing them 
Conclusion
Insight-driven computational analysis of text is becoming increasingly common  It not only helps us see more broadly, it helps us see subtle patterns more clearly and allows us to explore radical new questions about culture and society  In this article we have consolidated our experiences, as scholars from very different disciplines, in analyzing text as social and cultural data and described how the research process often unfolds  Each of the steps in the process is time-consuming and labor-intensive  Each presents challenges  And especially when working across disciplines, the research often involves a fair amount of discussion—even negotiation—about what means of operationalization and approaches to analysis are appropriate and feasible  And yet, with a bit of perseverance and mutual understanding, conceptually sound and meaningful work results so that we can truly make use of the exciting opportunities rich textual data offers 
Acknowledgements
This work was supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1  Dong Nguyen is supported with an Alan Turing Institute Fellowship (TU/A/000006)  Maria Liakata is a Turing fellow at 40%  We would also like to thank the participants of the “Bridging disciplines in analysing text as social and cultural data” workshop held at the Turing Institute (2017) for insightful discussions  The workshop was funded by a Turing Institute seed funding award to Nguyen and Liakata 

ground_truth: Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does their ensemble method work?
Context 2: Introduction
Since humans amass more and more generally available data in the form of unstructured text it would be very useful to teach machines to read and comprehend such data and then use this understanding to answer our questions  A significant amount of research has recently focused on answering one particular kind of questions the answer to which depends on understanding a context document  These are cloze-style questions BIBREF0 which require the reader to fill in a missing word in a sentence  An important advantage of such questions is that they can be generated automatically from a suitable text corpus which allows us to produce a practically unlimited amount of them  That opens the task to notoriously data-hungry deep-learning techniques which now seem to outperform all alternative approaches 
Two such large-scale datasets have recently been proposed by researchers from Google DeepMind and Facebook AI: the CNN/Daily Mail dataset BIBREF1 and the Children's Book Test (CBT) BIBREF2 respectively  These have attracted a lot of attention from the research community BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 with a new state-of-the-art model coming out every few weeks 
However if our goal is a production-level system actually capable of helping humans, we want the model to use all available resources as efficiently as possible  Given that
we believe that if the community is striving to bring the performance as far as possible, it should move its work to larger data 
This thinking goes in line with recent developments in the area of language modelling  For a long time models were being compared on several "standard" datasets with publications often presenting minuscule improvements in performance  Then the large-scale One Billion Word corpus dataset appeared BIBREF15 and it allowed Jozefowicz et al  to train much larger LSTM models BIBREF16 that almost halved the state-of-the-art perplexity on this dataset 
We think it is time to make a similar step in the area of text comprehension  Hence we are introducing the BookTest, a new dataset very similar to the Children's Book test but more than 60 times larger to enable training larger models even in the domain of text comprehension  Furthermore the methodology used to create our data can later be used to create even larger datasets when the need arises thanks to further technological progress 
We show that if we evaluate a model trained on the new dataset on the now standard Children's Book Test dataset, we see an improvement in accuracy much larger than other research groups achieved by enhancing the model architecture itself (while still using the original CBT training data)  By training on the new dataset, we reduce the prediction error by almost one third  On the named-entity version of CBT this brings the ensemble of our models to the level of human baseline as reported by Facebook BIBREF2   However in the final section we show in our own human study that there is still room for improvement on the CBT beyond the performance of our model 
Task Description
A natural way of testing a reader's comprehension of a text is to ask her a question the answer to which can be deduced from the text  Hence the task we are trying to solve consists of answering a cloze-style question, the answer to which depends on the understanding of a context document provided with the question  The model is also provided with a set of possible answers from which the correct one is to be selected  This can be formalized as follows:
The training data consist of tuples INLINEFORM0 , where INLINEFORM1 is a question, INLINEFORM2 is a document that contains the answer to question INLINEFORM3 , INLINEFORM4 is a set of possible answers and INLINEFORM5 is the ground-truth answer  Both INLINEFORM6 and INLINEFORM7 are sequences of words from vocabulary INLINEFORM8   We also assume that all possible answers are words from the vocabulary, that is INLINEFORM9   In the CBT and CNN/Daily Mail datasets it is also true that the ground-truth answer INLINEFORM10 appears in the document  This is exploited by many machine learning models BIBREF2 , BIBREF4 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF10 , BIBREF11 , BIBREF12 , however some do not explicitly depend on this property BIBREF1 , BIBREF3 , BIBREF5 , BIBREF9
Current Landscape
We will now briefly review what datasets for text comprehension have been published up to date and look at models which have been recently applied to solving the task we have just described 
Datasets
A crucial condition for applying deep-learning techniques is to have a huge amount of data available for training  For question answering this specifically means having a large number of document-question-answer triples available  While there is an unlimited amount of text available, coming up with relevant questions and the corresponding answers can be extremely labour-intensive if done by human annotators  There were efforts to provide such human-generated datasets, e g  Microsoft's MCTest BIBREF17 , however their scale is not suitable for deep learning without pre-training on other data BIBREF18 (such as using pre-trained word embedding vectors) 
Google DeepMind managed to avoid this scale issue with their way of generating document-question-answer triples automatically, closely followed by Facebook with a similar method  Let us now briefly introduce the two resulting datasets whose properties are summarized in Table TABREF8  
These two datasets BIBREF1 exploit a useful feature of online news articles – many articles include a short summarizing sentence near the top of the page  Since all information in the summary sentence is also presented in the article body, we get a nice cloze-style question about the article contents by removing a word from the short summary 
The dataset's authors also replaced all named entities in the dataset by anonymous tokens which are further shuffled for each new batch  This forces the model to rely solely on information from the context document, not being able to transfer any meaning of the named entities between documents 
This restricts the task to one specific aspect of context-dependent question answering which may be useful however it moves the task further from the real application scenario, where we would like the model to use all information available to answer questions  Furthermore Chen et al  BIBREF5 have suggested that this can make about 17% of the questions unanswerable even by humans  They also claim that more than a half of the question sentences are mere paraphrases or exact matches of a single sentence from the context document  This raises a question to what extent the dataset can test deeper understanding of the articles 
The Children's Book Test BIBREF2 uses a different source - books freely available thanks to Project Gutenberg  Since no summary is available, each example consists of a context document formed from 20 consecutive sentences from the story together with a question formed from the subsequent sentence 
The dataset comes in four flavours depending on what type of word is omitted from the question sentence  Based on human evaluation done in BIBREF2 it seems that NE and CN are more context dependent than the other two types – prepositions and verbs  Therefore we (and all of the recent publications) focus only on these two word types 
Several new datasets related to the (now almost standard) ones above emerged recently  We will now briefly present them and explain how the dataset we are introducing in this article differs from them 
The LAMBADA dataset BIBREF19 is designed to measure progress in understanding common-sense questions about short stories that can be easily answered by humans but cannot be answered by current standard machine-learning models (e g  plain LSTM language models)  This dataset is useful for measuring the gap between humans and machine learning algorithms  However, by contrast to our BookTest dataset, it will not allow us to track progress towards the performance of the baseline systems or on examples where machine learning may show super-human performance  Also LAMBADA is just a diagnostic dataset and does not provide ready-to-use question-answering training data, just a plain-text corpus which may moreover include copyrighted books making its use potentially problematic for some purposes  We are providing ready training data consisting of copyright-free books only 
The SQuAD dataset BIBREF20 based on Wikipedia and the Who-did-What dataset BIBREF21 based on Gigaword news articles are factoid question-answering datasets where a multi-word answer should be extracted from a context document  This is in contrast to the previous datasets, including CNN/DM, CBT, LAMBADA and our new dataset, which require only single-word answers  Both these datasets however provide less than 130,000 training questions, two orders of magnitude less than our dataset does 
The Story Cloze Test BIBREF22 provides a crowd-sourced corpus of 49,255 commonsense stories for training and 3,744 testing stories with right and wrong endings  Hence the dataset is again rather small  Similarly to LAMBADA, the Story Cloze Test was designed to be easily answerable by humans 
In the WikiReading BIBREF23 dataset the context document is formed from a Wikipedia article and the question-answer pair is taken from the corresponding WikiData page  For each entity (e g  Hillary Clinton), WikiData contain a number of property-value pairs (e g  place of birth: Chicago) which form the datasets's question-answer pairs  The dataset is certainly relevant to the community, however the questions are of very limited variety with only 20 properties (and hence unique questions) covering INLINEFORM0 of the dataset  Furthermore many of the frequent properties are mentioned at a set spot within the article (e g  the date of birth is almost always in brackets behind the name of a person) which may make the task easier for machines  We are trying to provide a more varied dataset 
Although there are several datasets related to task we are aiming to solve, they differ sufficiently for our dataset to bring new value to the community  Its biggest advantage is its size which can furthermore be easily upscaled without expensive human annotation  Finally while we are emphasizing the differences, models could certainly benefit from as diverse a collection of datasets as possible 
Machine Learning Models
A first major work applying deep-learning techniques to text comprehension was Hermann et al  BIBREF1   This work was followed by the application of Memory Networks to the same task BIBREF2   Later three models emerged around the same time BIBREF3 , BIBREF4 , BIBREF5 including our psr model BIBREF4   The AS Reader inspired several subsequent models that use it as a sub-component in a diverse ensemble BIBREF8 ; extend it with a hierarchical structure BIBREF6 , BIBREF24 , BIBREF7 ; compute attention over the context document for every word in the query BIBREF10 or use two-way context-query attention mechanism for every word in the context and the query BIBREF11 that is similar in its spirit to models recently proposed in different domains, e g  BIBREF25 in information retrieval  Other neural approaches to text comprehension are explored in BIBREF9 , BIBREF12  
Possible Directions for Improvements
Accuracy in any machine learning tasks can be enhanced either by improving a machine learning model or by using more in-domain training data  Current state of the art models BIBREF6 , BIBREF7 , BIBREF8 , BIBREF11 improve over AS Reader's accuracy on CBT NE and CN datasets by 1-2 percent absolute  This suggests that with current techniques there is only limited room for improvement on the algorithmic side 
The other possibility to improve performance is simply to use more training data  The importance of training data was highlighted by the frequently quoted Mercer's statement that “There is no data like more data ” The observation that having more data is often more important than having better algorithms has been frequently stressed since then BIBREF13 , BIBREF14  
As a step in the direction of exploiting the potential of more data in the domain of text comprehension, we created a new dataset called BookTest similar to, but much larger than the widely used CBT and CNN/DM datasets 
BookTest
Similarly to the CBT, our BookTest dataset is derived from books available through project Gutenberg  We used 3555 copyright-free books to extract CN examples and 10507 books for NE examples, for comparison the CBT dataset was extracted from just 108 books 
When creating our dataset we follow the same procedure as was used to create the CBT dataset BIBREF2   That is, we detect whether each sentence contains either a named entity or a common noun that already appeared in one of the preceding twenty sentences  This word is then replaced by a gap tag (XXXXX) in this sentence which is hence turned into a cloze-style question  The preceding 20 sentences are used as the context document  For common noun and named entity detection we use the Stanford POS tagger BIBREF27 and Stanford NER BIBREF28  
The training dataset consists of the original CBT NE and CN data extended with new NE and CN examples  The new BookTest dataset hence contains INLINEFORM0 training examples and INLINEFORM1 tokens 
The validation dataset consists of INLINEFORM0 NE and INLINEFORM1 CN questions  We have one test set for NE and one for CN, each containing INLINEFORM2 examples  The training, validation and test sets were generated from non-overlapping sets of books 
When generating the dataset we removed all editions of books used to create CBT validation and test sets from our training dataset  Therefore the models trained on the BookTest corpus can be evaluated on the original CBT data and they can be compared with recent text-comprehension models utilizing this dataset BIBREF2 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11  
Baselines
We will now use our psr model to evaluate the performance gain from increasing the dataset size 
AS Reader
In BIBREF4 we introduced the psr , which at the time of publication significantly outperformed all other architectures on the CNN, DM and CBT datasets  This model is built to leverage the fact that the answer is a single word from the context document  Similarly to many other models it uses attention over the document – intuitively a measure of how relevant each word is to answering the question  However while most previous models used this attention as weights to calculate a blended representation of the answer word, we simply sum the attention across all occurrences of each unique words and then simply select the word with the highest sum as the final answer  While simple, this trick seems both to improve accuracy and to speed-up training  It was adopted by many subsequent models BIBREF8 , BIBREF6 , BIBREF7 , BIBREF10 , BIBREF11 , BIBREF24  
Let us now describe the model in more detail  Figure FIGREF21 may help you in understanding the following paragraphs 
The words from the document and the question are first converted into vector embeddings using a look-up matrix INLINEFORM0   The document is then read by a bidirectional GRU network BIBREF29   A concatenation of the hidden states of the forward and backward GRUs at each word is then used as a contextual embedding of this word, intuitively representing the context in which the word is appearing  We can also understand it as representing the set of questions to which this word may be an answer 
Similarly the question is read by a bidirectional GRU but in this case only the final hidden states are concatenated to form the question embedding 
The attention over each word in the context is then calculated as the dot product of its contextual embedding with the question embedding  This attention is then normalized by the softmax function and summed across all occurrences of each answer candidate  The candidate with most accumulated attention is selected as the final answer 
For a more detailed description of the model including equations check BIBREF4   More details about the training setup and model hyperparameters can be found in the Appendix 
During our past experiments on the CNN, DM and CBT datasets BIBREF4 each unique word from the training, validation and test datasets had its row in the look-up matrix INLINEFORM0   However as we radically increased the dataset size, this would result in an extremely large number of model parameters so we decided to limit the vocabulary size to INLINEFORM1 most frequent words  For each example, each unique out-of-vocabulary word is now mapped on one of 1000 anonymous tokens which are randomly initialized and untrained  Fixing the embeddings of these anonymous tags proved to significantly improve the performance 
While mostly using the original AS Reader model, we have also tried introducing a minor tweak in some instances of the model  We tried initializing the context encoder GRU's hidden state by letting the encoder read the question first before proceeding to read the context document  Intuitively this allows the encoder to know in advance what to look for when reading over the context document 
Including models of this kind in the ensemble helped to improve the performance 
Results
Table TABREF25 shows the accuracy of the psr and other architectures on the CBT validation and test data  The last two rows show the performance of the psr trained on the BookTest dataset; all the other models have been trained on the original CBT training data 
If we take the best psr ensemble trained on CBT as a baseline, improving the model architecture as in BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , continuing to use the original CBT training data, lead to improvements of INLINEFORM0 and INLINEFORM1 absolute on named entities and common nouns respectively  By contrast, inflating the training dataset provided a boost of INLINEFORM2 while using the same model  The ensemble of our models even exceeded the human baseline provided by Facebook BIBREF2 on the Common Noun dataset 
Our model takes approximately two weeks to converge when trained on the BookTest dataset on a single Nvidia Tesla K40 GPU 
Discussion
Embracing the abundance of data may mean focusing on other aspects of system design than with smaller data  Here are some of the challenges that we need to face in this situation 
Firstly, since the amount of data is practically unlimited – we could even generate them on the fly resulting in continuous learning similar to the Never-Ending Language Learning by Carnegie Mellon University BIBREF30 – it is now the speed of training that determines how much data the model is able to see  Since more training data significantly help the model performance, focusing on speeding up the algorithm may be more important than ever before  This may for instance influence the decision whether to use regularization such as dropout which does seem to somewhat improve the model performance, however usually at a cost of slowing down training 
Thanks to its simplicity, the psr seems to be training fast - for example around seven times faster than the models proposed by Chen et al  BIBREF5   Hence the psr may be particularly suitable for training on large datasets 
The second challenge is how to generalize the performance gains from large data to a specific target domain  While there are huge amounts of natural language data in general, it may not be the case in the domain where we may want to ultimately apply our model 
Hence we are usually not facing a scenario of simply using a larger amount of the same training data, but rather extending training to a related domain of data, hoping that some of what the model learns on the new data will still help it on the original task 
This is highlighted by our observations from applying a model trained on the BookTest to Children's Book Test test data  If we move model training from joint CBT NE+CN training data to a subset of the BookTest of the same size (230k examples), we see a drop in accuracy of around 10% on the CBT test datasets 
Hence even though the Children's Book Test and BookTest datasets are almost as close as two disjoint datasets can get, the transfer is still very imperfect   Rightly choosing data to augment the in-domain training data is certainly a problem worth exploring in future work 
Our results show that given enough data the AS Reader was able to exceed the human performance on CBT CN reported by Facebook  However we hypothesized that the system is still not achieving its full potential so we decided to examine the room for improvement in our own small human study 
Human Study
After adding more data we have the performance on the CBT validation and test datasets soaring  However is there still potential for much further growth beyond the results we have observed?
We decided to explore the remaining space for improvement on the CBT by testing humans on a random subset of 50 named entity and 50 common noun validation questions that the psr ensemble could not answer correctly  These questions were answered by 10 non-native English speakers from our research laboratory, each on a disjoint subset of questions   Participants had unlimited time to answer the questions and were told that these questions were not correctly answered by a machine, providing additional motivation to prove they are better than computers  The results of the human study are summarized in Table TABREF28   They show that a majority of questions that our system could not answer so far are in fact answerable  This suggests that 1) the original human baselines might have been underestimated, however, it might also be the case that there are some examples that can be answered by machines and not by humans; 2) there is still space for improvement 
A system that would answer correctly every time when either our ensemble or human answered correctly would achieve accuracy over 92% percent on both validation and test NE datasets and over 96% on both CN datasets  Hence it still makes sense to use CBT dataset to study further improvements of text-comprehension systems 
Conclusion
Few ways of improving model performance are as solidly established as using more training data  Yet we believe this principle has been somewhat neglected by recent research in text comprehension  While there is a practically unlimited amount of data available in this field, most research was performed on unnecessarily small datasets 
As a gentle reminder to the community we have shown that simply infusing a model with more data can yield performance improvements of up to INLINEFORM0 where several attempts to improve the model architecture on the same training data have given gains of at most INLINEFORM1 compared to our best ensemble result  Yes, experiments on small datasets certainly can bring useful insights  However we believe that the community should also embrace the real-world scenario of data abundance 
The BookTest dataset we are proposing gives the reading-comprehension community an opportunity to make a step in that direction 
Training Details
The training details are similar to those in BIBREF4 however we are including them here for completeness 
To train the model we used stochastic gradient descent with the ADAM update rule BIBREF32 and learning rates of INLINEFORM0 , INLINEFORM1 and INLINEFORM2   The best learning rate in our experiments was INLINEFORM3   We minimized negative log-likelihood as the training objective 
The initial weights in the word-embedding matrix were drawn randomly uniformly from the interval INLINEFORM0   Weights in the GRU networks were initialized by random orthogonal matrices BIBREF34 and biases were initialized to zero  We also used a gradient clipping BIBREF33 threshold of 10 and batches of sizes between 32 or 256  Increasing the batch from 32 to 128 seems to significantly improve performance on the large dataset - something we did not observe on the original CBT data  Increasing the batch size much above 128 is currently difficult due to memory constraints of the GPU 
During training we randomly shuffled all examples at the beginning of each epoch  To speed up training, we always pre-fetched 10 batches worth of examples and sorted them according to document length  Hence each batch contained documents of roughly the same length 
We also did not use pre-trained word embeddings 
We did not perform any text pre-processing since the datasets were already tokenized 
During training we evaluated the model performance every 12 hours and at the end of each epoch and stopped training when the error on the 20k BookTest validation set started increasing  We explored the hyperparameter space by training 67 different models The region of the parameter space that we explored together with the parameters of the model with best validation accuracy are summarized in Table TABREF29  
Our model was implemented using Theano BIBREF31 and Blocks BIBREF35  
The ensembles were formed by simply averaging the predictions from the constituent single models  These single models were selected using the following algorithm 
We started with the best performing model according to validation performance  Then in each step we tried adding the best performing model that had not been previously tried  We kept it in the ensemble if it did improve its validation performance and discarded it otherwise  This way we gradually tried each model once  We call the resulting model a greedy ensemble  We used the INLINEFORM0 BookTest validation dataset for this procedure 
The algorithm was offered 10 models and selected 5 of them for the final ensemble 

ground_truth: simply averaging the predictions from the constituent single models

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the languages they use in their experiment?
Context 3: Introduction
While most NLP resources are English-specific, there have been several recent efforts to build multilingual benchmarks  One possibility is to collect and annotate data in multiple languages separately BIBREF0, but most existing datasets have been created through translation BIBREF1, BIBREF2  This approach has two desirable properties: it relies on existing professional translation services rather than requiring expertise in multiple languages, and it results in parallel evaluation sets that offer a meaningful measure of the cross-lingual transfer gap of different models  The resulting multilingual datasets are generally used for evaluation only, relying on existing English datasets for training 
Closely related to that, cross-lingual transfer learning aims to leverage large datasets available in one language—typically English—to build multilingual models that can generalize to other languages  Previous work has explored 3 main approaches to that end: machine translating the test set into English and using a monolingual English model (Translate-Test), machine translating the training set into each target language and training the models on their respective languages (Translate-Train), or using English data to fine-tune a multilingual model that is then transferred to the rest of languages (Zero-Shot) 
The dataset creation and transfer procedures described above result in a mixture of original, human translated and machine translated data when dealing with cross-lingual models  In fact, the type of text a system is trained on does not typically match the type of text it is exposed to at test time: Translate-Test systems are trained on original data and evaluated on machine translated test sets, Zero-Shot systems are trained on original data and evaluated on human translated test sets, and Translate-Train systems are trained on machine translated data and evaluated on human translated test sets 
Despite overlooked to date, we show that such mismatch has a notable impact in the performance of existing cross-lingual models  By using back-translation BIBREF3 to paraphrase each training instance, we obtain another English version of the training set that better resembles the test set, obtaining substantial improvements for the Translate-Test and Zero-Shot approaches in cross-lingual Natural Language Inference (NLI)  While improvements brought by machine translation have previously been attributed to data augmentation BIBREF4, we reject this hypothesis and show that the phenomenon is only present in translated test sets, but not in original ones  Instead, our analysis reveals that this behavior is caused by subtle artifacts arising from the translation process itself  In particular, we show that translating different parts of each instance separately (e g  the premise and the hypothesis in NLI) can alter superficial patterns in the data (e g  the degree of lexical overlap between them), which severely affects the generalization ability of current models  Based on the gained insights, we improve the state-of-the-art in XNLI, and show that some previous findings need to be reconsidered in the light of this phenomenon 
Related work ::: Cross-lingual transfer learning 
Current cross-lingual models work by pre-training multilingual representations using some form of language modeling, which are then fine-tuned on the relevant task and transferred to different languages  Some authors leverage parallel data to that end BIBREF5, BIBREF6, but training a model akin to BERT BIBREF7 on the combination of monolingual corpora in multiple languages is also effective BIBREF8  Closely related to our work, BIBREF4 showed that replacing segments of the training data with their translation during fine-tuning is helpful  However, they attribute this behavior to a data augmentation effect, which we believe should be reconsidered given the new evidence we provide 
Related work ::: Multilingual benchmarks 
Most benchmarks covering a wide set of languages have been created through translation, as it is the case of XNLI BIBREF1 for NLI, PAWS-X BIBREF9 for adversarial paraphrase identification, and XQuAD BIBREF2 and MLQA BIBREF10 for Question Answering (QA)  A notable exception is TyDi QA BIBREF0, a contemporaneous QA dataset that was separately annotated in 11 languages  Other cross-lingual datasets leverage existing multilingual resources, as it is the case of MLDoc BIBREF11 for document classification and Wikiann BIBREF12 for named entity recognition  Concurrent to our work, BIBREF13 combine some of these datasets into a single multilingual benchmark, and evaluate some well-known methods on it 
Related work ::: Annotation artifacts 
Several studies have shown that NLI datasets like SNLI BIBREF14 and MultiNLI BIBREF15 contain spurious patterns that can be exploited to obtain strong results without making real inferential decisions  For instance, BIBREF16 and BIBREF17 showed that a hypothesis-only baseline performs better than chance due to cues on their lexical choice and sentence length  Similarly, BIBREF18 showed that NLI models tend to predict entailment for sentence pairs with a high lexical overlap  Several authors have worked on adversarial datasets to diagnose these issues and provide a more challenging benchmark BIBREF19, BIBREF20, BIBREF21  Besides NLI, other tasks like QA have also been found to be susceptible to annotation artifacts BIBREF22, BIBREF23  While previous work has focused on the monolingual scenario, we show that translation can interfere with these artifacts in multilingual settings 
Related work ::: Translationese 
Translated texts are known to have unique features like simplification, explicitation, normalization and interference, which are refer to as translationese BIBREF24  This phenomenon has been reported to have a notable impact in machine translation evaluation BIBREF25, BIBREF26  For instance, back-translation brings large BLEU gains for reversed test sets (i e  when translationese is on the source side and original text is used as reference), but its effect diminishes in the natural direction BIBREF27  While connected, the phenomenon we analyze is different in that it arises from translation inconsistencies due to the lack of context, and affects cross-lingual transfer learning rather than machine translation 
Experimental design
Our goal is to analyze the effect of both human and machine translation in cross-lingual models  For that purpose, the core idea of our work is to (i) use machine translation to either translate the training set into other languages, or generate English paraphrases of it through back-translation, and (ii) evaluate the resulting systems on original, human translated and machine translated test sets in comparison with systems trained on original data  We next describe the models used in our experiments (§SECREF6), the specific training variants explored (§SECREF8), and the evaluation procedure followed (§SECREF10) 
Experimental design ::: Models and transfer methods
We experiment with two models that are representative of the state-of-the-art in monolingual and cross-lingual pre-training: (i) Roberta BIBREF28, which is an improved version of BERT that uses masked language modeling to pre-train an English Transformer model, and (ii) XLM-R BIBREF8, which is a multilingual extension of the former pre-trained on 100 languages  In both cases, we use the large models released by the authors under the fairseq repository  As discussed next, we explore different variants of the training set to fine-tune each model on different tasks  At test time, we try both machine translating the test set into English (Translate-Test) and, in the case of XLM-R, using the actual test set in the target language (Zero-Shot) 
Experimental design ::: Training variants
We try 3 variants of each training set to fine-tune our models: (i) the original one in English (Orig), (ii) an English paraphrase of it generated through back-translation using Spanish or Finnish as pivot (BT-ES and BT-FI), and (iii) a machine translated version in Spanish or Finnish (MT-ES and MT-FI)  For sentences occurring multiple times in the training set (e g  premises repeated for multiple hypotheses), we use the exact same translation for all occurrences, as our goal is to understand the inherent effect of translation rather than its potential application as a data augmentation method 
In order to train the machine translation systems for MT-XX and BT-XX, we use the big Transformer model BIBREF29 with the same settings as BIBREF30 and SentencePiece tokenization BIBREF31 with a joint vocabulary of 32k subwords  For English-Spanish, we train for 10 epochs on all parallel data from WMT 2013 BIBREF32 and ParaCrawl v5 0 BIBREF33  For English-Finnish, we train for 40 epochs on Europarl and Wiki Titles from WMT 2019 BIBREF34, ParaCrawl v5 0, and DGT, EUbookshop and TildeMODEL from OPUS BIBREF35  In both cases, we remove sentences longer than 250 tokens, with a source/target ratio exceeding 1 5, or for which langid py BIBREF36 predicts a different language, resulting in a final corpus size of 48M and 7M sentence pairs, respectively  We use sampling decoding with a temperature of 0 5 for inference, which produces more diverse translations than beam search BIBREF37 and performed better in our preliminary experiments 
Experimental design ::: Tasks and evaluation procedure
We use the following tasks for our experiments:
Experimental design ::: Tasks and evaluation procedure ::: Natural Language Inference (NLI) 
Given a premise and a hypothesis, the task is to determine whether there is an entailment, neutral or contradiction relation between them  We fine-tune our models on MultiNLI BIBREF15 for 10 epochs using the same settings as BIBREF28  In most of our experiments, we evaluate on XNLI BIBREF1, which comprises 2490 development and 5010 test instances in 15 languages  These were originally annotated in English, and the resulting premises and hypotheses were independently translated into the rest of the languages by professional translators  For the Translate-Test approach, we use the machine translated versions from the authors  Following BIBREF8, we select the best epoch checkpoint according to the average accuracy in the development set 
Experimental design ::: Tasks and evaluation procedure ::: Question Answering (QA) 
Given a context paragraph and a question, the task is to identify the span answering the question in the context  We fine-tune our models on SQuAD v1 1 BIBREF38 for 2 epochs using the same settings as BIBREF28, and report test results for the last epoch  We use two datasets for evaluation: XQuAD BIBREF2, a subset of the SQuAD development set translated into 10 other languages, and MLQA BIBREF10 a dataset consisting of parallel context paragraphs plus the corresponding questions annotated in English and translated into 6 other languages  In both cases, the translation was done by professional translators at the document level (i e  when translating a question, the text answering it was also shown)  For our BT-XX and MT-XX variants, we translate the context paragraph and the questions independently, and map the answer spans using the same procedure as BIBREF39  For the Translate-Test approach, we use the official machine translated versions of MLQA, run inference over them, and map the predicted answer spans back to the target language 
Both for NLI and QA, we run each system 5 times with different random seeds and report the average results  Space permitting, we also report the standard deviation across the 5 runs 
NLI experiments
We next discuss our main results in the XNLI development set (§SECREF15, §SECREF16), run additional experiments to better understand the behavior of our different variants (§SECREF17, §SECREF22, §SECREF25), and compare our results to previous work in the XNLI test set (§SECREF30) 
NLI experiments ::: Translate-Test results
We start by analyzing XNLI development results for Translate-Test  Recall that, in this approach, the test set is machine translated into English, but training is typically done on original English data  Our BT-ES and BT-FI variants close this gap by training on a machine translated English version of the training set generated through back-translation  As shown in Table TABREF9, this brings substantial gains for both Roberta and XLM-R, with an average improvement of 4 6 points in the best case  Quite remarkably, MT-ES and MT-FI also outperform Orig by a substantial margin, and are only 0 8 points below their BT-ES and BT-FI counterparts  Recall that, for these two systems, training is done in machine translated Spanish or Finnish, while inference is done in machine translated English  This shows that the loss of performance when generalizing from original data to machine translated data is substantially larger than the loss of performance when generalizing from one language to another 
NLI experiments ::: Zero-Shot results
We next analyze the results for the Zero-Shot approach  In this case, inference is done in the test set in each target language which, in the case of XNLI, was human translated from English  As such, different from the Translate-Test approach, neither training on original data (Orig) nor training on machine translated data (BT-XX and MT-XX) makes use of the exact same type of text that the system is exposed to at test time  However, as shown in Table TABREF9, both BT-XX and MT-XX outperform Orig by approximately 2 points, which suggests that our (back-)translated versions of the training set are more similar to the human translated test sets than the original one  This also provides a new perspective on the Translate-Train approach, which was reported to outperform Orig in previous work BIBREF5: while the original motivation was to train the model on the same language that it is tested on, our results show that machine translating the training set is beneficial even when the target language is different 
NLI experiments ::: Original vs  translated test sets
So as to understand whether the improvements observed so far are limited to translated test sets or apply more generally, we conduct additional experiments comparing translated test sets to original ones  However, to the best of our knowledge, all existing non-English NLI benchmarks were created through translation  For that reason, we build a new test set that mimics XNLI, but is annotated in Spanish rather than English  We first collect the premises from a filtered version of CommonCrawl BIBREF42, taking a subset of 5 websites that represent a diverse set of genres: a newspaper, an economy forum, a celebrity magazine, a literature blog, and a consumer magazine  We then ask native Spanish annotators to generate an entailment, a neutral and a contradiction hypothesis for each premise  We collect a total of 2490 examples using this procedure, which is the same size as the XNLI development set  Finally, we create a human translated and a machine translated English version of the dataset using professional translators from Gengo and our machine translation system described in §SECREF8, respectively  We report results for the best epoch checkpoint on each set 
As shown in Table TABREF18, both BT-XX and MT-XX clearly outperform Orig in all test sets created through translation, which is consistent with our previous results  In contrast, the best results on the original English set are obtained by Orig, and neither BT-XX nor MT-XX obtain any clear improvement on the one in Spanish either  This confirms that the underlying phenomenon is limited to translated test sets  In addition, it is worth mentioning that the results for the machine translated test set in English are slightly better than those for the human translated one, which suggests that the difficulty of the task does not only depend on the translation quality  Finally, it is also interesting that MT-ES is only marginally better than MT-FI in both Spanish test sets, even if it corresponds to the Translate-Train approach, whereas MT-FI needs to Zero-Shot transfer from Finnish into Spanish  This reinforces the idea that it is training on translated data rather than training on the target language that is key in Translate-Train 
NLI experiments ::: Stress tests
In order to better understand how systems trained on original and translated data differ, we run additional experiments on the NLI Stress Tests BIBREF19, which were designed to test the robustness of NLI models to specific linguistic phenomena in English  The benchmark consists of a competence test, which evaluates the ability to understand antonymy relation and perform numerical reasoning, a distraction test, which evaluates the robustness to shallow patterns like lexical overlap and the presence of negation words, and a noise test, which evaluates robustness to spelling errors  Just as with previous experiments, we report results for the best epoch checkpoint in each test set 
As shown in Table TABREF23, Orig outperforms BT-FI and MT-FI on the competence test by a large margin, but the opposite is true on the distraction test  In particular, our results show that BT-FI and MT-FI are less reliant on lexical overlap and the presence of negative words  This feels intuitive, as translating the premise and hypothesis independently—as BT-FI and MT-FI do—is likely to reduce the lexical overlap between them  More generally, the translation process can alter similar superficial patterns in the data, which NLI models are sensitive to (§SECREF2)  This would explain why the resulting models have a different behavior on different stress tests 
NLI experiments ::: Output class distribution
With the aim to understand the effect of the previous phenomenon in cross-lingual settings, we look at the output class distribution of our different models in the XNLI development set  As shown in Table TABREF28, the predictions of all systems are close to the true class distribution in the case of English  Nevertheless, Orig is strongly biased for the rest of languages, and tends to underpredict entailment and overpredict neutral  This can again be attributed to the fact that the English test set is original, whereas the rest are human translated  In particular, it is well-known that NLI models tend to predict entailment when there is a high lexical overlap between the premise and the hypothesis (§SECREF2)  However, the degree of overlap will be smaller in the human translated test sets given that the premise and the hypothesis were translated independently, which explains why entailment is underpredicted  In contrast, BT-FI and MT-FI are exposed to the exact same phenomenon during training, which explains why they are not that heavily affected 
So as to measure the impact of this phenomenon, we explore a simple approach to correct this bias: having fine-tuned each model, we adjust the bias term added to the logit of each class so the model predictions match the true class distribution for each language  As shown in Table TABREF29, this brings large improvements for Orig, but is less effective for BT-FI and MT-FI  This shows that the performance of Orig was considerably hindered by this bias, which BT-FI and MT-FI effectively mitigate 
NLI experiments ::: Comparison with the state-of-the-art
So as to put our results into perspective, we compare our best variant to previous work on the XNLI test set  As shown in Table TABREF31, our method improves the state-of-the-art for both the Translate-Test and the Zero-Shot approaches by 4 3 and 2 8 points, respectively  It also obtains the best overall results published to date, with the additional advantage that the previous state-of-the-art required a machine translation system between English and each of the 14 target languages, whereas our method uses a single machine translation system between English and Finnish (which is not one of the target languages)  While the main goal of our work is not to design better cross-lingual models, but to analyze their behavior in connection to translation, this shows that the phenomenon under study is highly relevant, to the extent that it can be exploited to improve the state-of-the-art 
QA experiments
So as to understand whether our previous findings apply to other tasks besides NLI, we run additional experiments on QA  As shown in Table TABREF32, BT-FI and BT-ES do indeed outperform Orig for the Translate-Test approach on MLQA  The improvement is modest, but very consistent across different languages, models and runs  The results for MT-ES and MT-FI are less conclusive, presumably because mapping the answer spans across languages might introduce some noise  In contrast, we do not observe any clear improvement for the Zero-Shot approach on this dataset  Our XQuAD results in Table TABREF33 are more positive, but still inconclusive 
These results can partly be explained by the translation procedure used to create the different benchmarks: the premises and hypotheses of XNLI were translated independently, whereas the questions and context paragraphs of XQuAD were translated together  Similarly, MLQA made use of parallel contexts, and translators were shown the sentence containing each answer when translating the corresponding question  As a result, one can expect both QA benchmarks to have more consistent translations than XNLI, which would in turn diminish this phenomenon  In contrast, the questions and context paragraphs are independently translated when using machine translation, which explains why BT-ES and BT-FI outperform Orig for the Translate-Test approach  We conclude that the translation artifacts revealed by our analysis are not exclusive to NLI, as they also show up on QA for the Translate-Test approach, but their actual impact can be highly dependent on the translation procedure used and the nature of the task 
Discussion
Our analysis prompts to reconsider previous findings in cross-lingual transfer learning as follows:
Discussion ::: The cross-lingual transfer gap on XNLI was overestimated 
Given the parallel nature of XNLI, accuracy differences across languages are commonly interpreted as the loss of performance when generalizing from English to the rest of languages  However, our work shows that there is another factor that can have a much larger impact: the loss of performance when generalizing from original to translated data  Our results suggest that the real cross-lingual generalization ability of XLM-R is considerably better than what the accuracy numbers in XNLI reflect 
Discussion ::: Overcoming the cross-lingual gap is not what makes Translate-Train work 
The original motivation for Translate-Train was to train the model on the same language it is tested on  However, we show that it is training on translated data, rather than training on the target language, that is key for this approach to outperform Zero-Shot as reported by previous authors 
Discussion ::: Improvements previously attributed to data augmentation should be reconsidered 
The method by BIBREF4 combines machine translated premises and hypotheses in different languages (§SECREF2), resulting in an effect similar to BT-XX and MT-XX  As such, we believe that this method should be analyzed from the point of view of dataset artifacts rather than data augmentation, as the authors do  From this perspective, having the premise and the hypotheses in different languages can reduce the superficial patterns between them, which would explain why this approach is better than using examples in a single language 
Discussion ::: The potential of Translate-Test was underestimated 
The previous best results for Translate-Test on XNLI lagged behind the state-of-the-art by 4 6 points  Our work reduces this gap to only 0 8 points by addressing the underlying translation artifacts  The reason why Translate-Test is more severely affected by this phenomenon is twofold: (i) the effect is doubled by first using human translation to create the test set and then machine translation to translate it back to English, and (ii) Translate-Train was inadvertently mitigating this issue (see above), but equivalent techniques were never applied to Translate-Test 
Discussion ::: Future evaluation should better account for translation artifacts 
The evaluation issues raised by our analysis do not have a simple solution  In fact, while we use the term translation artifacts to highlight that they are an unintended effect of translation that impacts final evaluation, one could also argue that it is the original datasets that contain the artifacts, which translation simply alters or even mitigates  In any case, this is a more general issue that falls beyond the scope of cross-lingual transfer learning, so we argue that it should be carefully controlled when evaluating cross-lingual models  In the absence of more robust datasets, we recommend that future multilingual benchmarks should at least provide consistent test sets for English and the rest of languages  This can be achieved by (i) using original annotations in all languages, (ii) using original annotations in a non-English language and translating them into English and other languages, or (iii) if translating from English, doing so at the document level to minimize translation inconsistencies 
Conclusions
In this paper, we have shown that both human and machine translation can alter superficial patterns in data, which requires reconsidering previous findings in cross-lingual transfer learning  Based on the gained insights, we have improved the state-of-the-art in XNLI for the Translate-Test and Zero-Shot approaches by a substantial margin  Finally, we have shown that the phenomenon is not specific to NLI but also affects QA, although it is less pronounced there thanks to the translation procedure used in the corresponding benchmarks  So as to facilitate similar studies in the future, we release our NLI dataset, which, unlike previous benchmarks, was annotated in a non-English language and human translated into English 
Acknowledgments
We thank Nora Aranberri and Uxoa Iñurrieta for helpful discussion during the development of this work, as well as the rest of our colleagues from the IXA group that worked as annotators for our NLI dataset 
This research was partially funded by a Facebook Fellowship, the Basque Government excellence research group (IT1343-19), the Spanish MINECO (UnsupMT TIN2017‐91692‐EXP MCIU/AEI/FEDER, UE), Project BigKnowledge (Ayudas Fundación BBVA a equipos de investigación científica 2018), and the NVIDIA GPU grant program 

ground_truth: English
French
Spanish
German
Greek
Bulgarian
Russian
Turkish
Arabic
Vietnamese
Thai
Chinese
Hindi
Swahili
Urdu
Finnish

English, Spanish, Finnish