prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Was PolyReponse evaluated against some baseline?
Context 1: Introduction and Background
Task-oriented dialogue systems are primarily designed to search and interact with large databases which contain information pertaining to a certain dialogue domain: the main purpose of such systems is to assist the users in accomplishing a well-defined task such as flight booking BIBREF0, tourist information BIBREF1, restaurant search BIBREF2, or booking a taxi BIBREF3  These systems are typically constructed around rigid task-specific ontologies BIBREF1, BIBREF4 which enumerate the constraints the users can express using a collection of slots (e g , price range for restaurant search) and their slot values (e g , cheap, expensive for the aforementioned slots)  Conversations are then modelled as a sequence of actions that constrain slots to particular values  This explicit semantic space is manually engineered by the system designer  It serves as the output of the natural language understanding component as well as the input to the language generation component both in traditional modular systems BIBREF5, BIBREF6 and in more recent end-to-end task-oriented dialogue systems BIBREF7, BIBREF8, BIBREF9, BIBREF3 
Working with such explicit semantics for task-oriented dialogue systems poses several critical challenges on top of the manual time-consuming domain ontology design  First, it is difficult to collect domain-specific data labelled with explicit semantic representations  As a consequence, despite recent data collection efforts to enable training of task-oriented systems across multiple domains BIBREF0, BIBREF3, annotated datasets are still few and far between, as well as limited in size and the number of domains covered  Second, the current approach constrains the types of dialogue the system can support, resulting in artificial conversations, and breakdowns when the user does not understand what the system can and cannot support  In other words, training a task-based dialogue system for voice-controlled search in a new domain always implies the complex, expensive, and time-consuming process of collecting and annotating sufficient amounts of in-domain dialogue data 
In this paper, we present a demo system based on an alternative approach to task-oriented dialogue  Relying on non-generative response retrieval we describe the PolyResponse conversational search engine and its application in the task of restaurant search and booking  The engine is trained on hundreds of millions of real conversations from a general domain (i e , Reddit), using an implicit representation of semantics that directly optimizes the task at hand  It learns what responses are appropriate in different conversational contexts, and consequently ranks a large pool of responses according to their relevance to the current user utterance and previous dialogue history (i e , dialogue context) 
The technical aspects of the underlying conversational search engine are explained in detail in our recent work BIBREF11, while the details concerning the Reddit training data are also available in another recent publication BIBREF12  In this demo, we put focus on the actual practical usefulness of the search engine by demonstrating its potential in the task of restaurant search, and extending it to deal with multi-modal data  We describe a PolyReponse system that assists the users in finding a relevant restaurant according to their preference, and then additionally helps them to make a booking in the selected restaurant  Due to its retrieval-based design, with the PolyResponse engine there is no need to engineer a structured ontology, or to solve the difficult task of general language generation  This design also bypasses the construction of dedicated decision-making policy modules  The large ranking model already encapsulates a lot of knowledge about natural language and conversational flow 
Since retrieved system responses are presented visually to the user, the PolyResponse restaurant search engine is able to combine text responses with relevant visual information (e g , photos from social media associated with the current restaurant and related to the user utterance), effectively yielding a multi-modal response  This setup of using voice as input, and responding visually is becoming more and more prevalent with the rise of smart screens like Echo Show and even mixed reality  Finally, the PolyResponse restaurant search engine is multilingual: it is currently deployed in 8 languages enabling search over restaurants in 8 cities around the world  System snapshots in four different languages are presented in Figure FIGREF16, while screencast videos that illustrate the dialogue flow with the PolyResponse engine are available at: https://tinyurl com/y3evkcfz 
PolyResponse: Conversational Search
The PolyResponse system is powered by a single large conversational search engine, trained on a large amount of conversational and image data, as shown in Figure FIGREF2  In simple words, it is a ranking model that learns to score conversational replies and images in a given conversational context  The highest-scoring responses are then retrieved as system outputs  The system computes two sets of similarity scores: 1) $S(r,c)$ is the score of a candidate reply $r$ given a conversational context $c$, and 2) $S(p,c)$ is the score of a candidate photo $p$ given a conversational context $c$  These scores are computed as a scaled cosine similarity of a vector that represents the context ($h_c$), and a vector that represents the candidate response: a text reply ($h_r$) or a photo ($h_p$)  For instance, $S(r,c)$ is computed as $S(r,c)=C cos(h_r,h_c)$, where $C$ is a learned constant  The part of the model dealing with text input (i e , obtaining the encodings $h_c$ and $h_r$) follows the architecture introduced recently by Henderson:2019acl  We provide only a brief recap here; see the original paper for further details 
PolyResponse: Conversational Search ::: Text Representation 
The model, implemented as a deep neural network, learns to respond by training on hundreds of millions context-reply $(c,r)$ pairs  First, similar to Henderson:2017arxiv, raw text from both $c$ and $r$ is converted to unigrams and bigrams  All input text is first lower-cased and tokenised, numbers with 5 or more digits get their digits replaced by a wildcard symbol #, while words longer than 16 characters are replaced by a wildcard token LONGWORD  Sentence boundary tokens are added to each sentence  The vocabulary consists of the unigrams that occur at least 10 times in a random 10M subset of the Reddit training set (see Figure FIGREF2) plus the 200K most frequent bigrams in the same random subset 
During training, we obtain $d$-dimensional feature representations ($d=320$) shared between contexts and replies for each unigram and bigram jointly with other neural net parameters  A state-of-the-art architecture based on transformers BIBREF13 is then applied to unigram and bigram vectors separately, which are then averaged to form the final 320-dimensional encoding  That encoding is then passed through three fully-connected non-linear hidden layers of dimensionality $1,024$  The final layer is linear and maps the text into the final $l$-dimensional ($l=512$) representation: $h_c$ and $h_r$  Other standard and more sophisticated encoder models can also be used to provide final encodings $h_c$ and $h_r$, but the current architecture shows a good trade-off between speed and efficacy with strong and robust performance in our empirical evaluations on the response retrieval task using Reddit BIBREF14, OpenSubtitles BIBREF15, and AmazonQA BIBREF16 conversational test data, see BIBREF12 for further details 
In training the constant $C$ is constrained to lie between 0 and $\sqrt{l}$  Following Henderson:2017arxiv, the scoring function in the training objective aims to maximise the similarity score of context-reply pairs that go together, while minimising the score of random pairings: negative examples  Training proceeds via SGD with batches comprising 500 pairs (1 positive and 499 negatives) 
PolyResponse: Conversational Search ::: Photo Representation 
Photos are represented using convolutional neural net (CNN) models pretrained on ImageNet BIBREF17  We use a MobileNet model with a depth multiplier of 1 4, and an input dimension of $224 \times 224$ pixels as in BIBREF18  This provides a $1,280 \times 1 4 = 1,792$-dimensional representation of a photo, which is then passed through a single hidden layer of dimensionality $1,024$ with ReLU activation, before being passed to a hidden layer of dimensionality 512 with no activation to provide the final representation $h_p$ 
PolyResponse: Conversational Search ::: Data Source 1: Reddit 
For training text representations we use a Reddit dataset similar to AlRfou:2016arxiv  Our dataset is large and provides natural conversational structure: all Reddit data from January 2015 to December 2018, available as a public BigQuery dataset, span almost 3 7B comments BIBREF12  We preprocess the dataset to remove uninformative and long comments by retaining only sentences containing more than 8 and less than 128 word tokens  After pairing all comments/contexts $c$ with their replies $r$, we obtain more than 727M context-reply $(c,r)$ pairs for training, see Figure FIGREF2 
PolyResponse: Conversational Search ::: Data Source 2: Yelp 
Once the text encoding sub-networks are trained, a photo encoder is learned on top of a pretrained MobileNet CNN, using data taken from the Yelp Open dataset: it contains around 200K photos and their captions  Training of the multi-modal sub-network then maximises the similarity of captions encoded with the response encoder $h_r$ to the photo representation $h_p$  As a result, we can compute the score of a photo given a context using the cosine similarity of the respective vectors  A photo will be scored highly if it looks like its caption would be a good response to the current context 
PolyResponse: Conversational Search ::: Index of Responses 
The Yelp dataset is used at inference time to provide text and photo candidates to display to the user at each step in the conversation  Our restaurant search is currently deployed separately for each city, and we limit the responses to a given city  For instance, for our English system for Edinburgh we work with 396 restaurants, 4,225 photos (these include additional photos obtained using the Google Places API without captions), 6,725 responses created from the structured information about restaurants that Yelp provides, converted using simple templates to sentences of the form such as “Restaurant X accepts credit cards ”, 125,830 sentences extracted from online reviews 
PolyResponse: Conversational Search ::: PolyResponse in a Nutshell 
The system jointly trains two encoding functions (with shared word embeddings) $f(context)$ and $g(reply)$ which produce encodings $h_c$ and $h_r$, so that the similarity $S(c,r)$ is high for all $(c,r)$ pairs from the Reddit training data and low for random pairs  The encoding function $g()$ is then frozen, and an encoding function $t(photo)$ is learnt which makes the similarity between a photo and its associated caption high for all (photo, caption) pairs from the Yelp dataset, and low for random pairs  $t$ is a CNN pretrained on ImageNet, with a shallow one-layer DNN on top  Given a new context/query, we then provide its encoding $h_c$ by applying $f()$, and find plausible text replies and photo responses according to functions $g()$ and $t()$, respectively  These should be responses that look like answers to the query, and photos that look like they would have captions that would be answers to the provided query 
At inference, finding relevant candidates given a context reduces to computing $h_c$ for the context $c$ , and finding nearby $h_r$ and $h_p$ vectors  The response vectors can all be pre-computed, and the nearest neighbour search can be further optimised using standard libraries such as Faiss BIBREF19 or approximate nearest neighbour retrieval BIBREF20, giving an efficient search that scales to billions of candidate responses 
The system offers both voice and text input and output  Speech-to-text and text-to-speech conversion in the PolyResponse system is currently supported by the off-the-shelf Google Cloud tools 
Dialogue Flow
The ranking model lends itself to the one-shot task of finding the most relevant responses in a given context  However, a restaurant-browsing system needs to support a dialogue flow where the user finds a restaurant, and then asks questions about it  The dialogue state for each search scenario is represented as the set of restaurants that are considered relevant  This starts off as all the restaurants in the given city, and is assumed to monotonically decrease in size as the conversation progresses until the user converges to a single restaurant  A restaurant is only considered valid in the context of a new user input if it has relevant responses corresponding to it  This flow is summarised here:
S1  Initialise $R$ as the set of all restaurants in the city  Given the user's input, rank all the responses in the response pool pertaining to restaurants in $R$ 
S2  Retrieve the top $N$ responses $r_1, r_2, \ldots , r_N$ with corresponding (sorted) cosine similarity scores: $s_1 \ge s_2 \ge \ldots \ge s_N$ 
S3  Compute probability scores $p_i \propto \exp (a \cdot s_i)$ with $\sum _{i=1}^N p_i$, where $a>0$ is a tunable constant 
S4  Compute a score $q_e$ for each restaurant/entity $e \in R$, $q_e = \sum _{i: r_i \in e} p_i$ 
S5  Update $R$ to the smallest set of restaurants with highest $q$ whose $q$-values sum up to more than a predefined threshold $t$ 
S6  Display the most relevant responses associated with the updated $R$, and return to S2 
If there are multiple relevant restaurants, one response is shown from each  When only one restaurant is relevant, the top $N$ responses are all shown, and relevant photos are also displayed  The system does not require dedicated understanding, decision-making, and generation modules, and this dialogue flow does not rely on explicit task-tailored semantics  The set of relevant restaurants is kept internally while the system narrows it down across multiple dialogue turns  A simple set of predefined rules is used to provide a templatic spoken system response: e g , an example rule is “One review of $e$ said $r$”, where $e$ refers to the restaurant, and $r$ to a relevant response associated with $e$  Note that while the demo is currently focused on the restaurant search task, the described “narrowing down” dialogue flow is generic and applicable to a variety of applications dealing with similar entity search 
The system can use a set of intent classifiers to allow resetting the dialogue state, or to activate the separate restaurant booking dialogue flow  These classifiers are briefly discussed in §SECREF4 
Other Functionality ::: Multilinguality 
The PolyResponse restaurant search is currently available in 8 languages and for 8 cities around the world: English (Edinburgh), German (Berlin), Spanish (Madrid), Mandarin (Taipei), Polish (Warsaw), Russian (Moscow), Korean (Seoul), and Serbian (Belgrade)  Selected snapshots are shown in Figure FIGREF16, while we also provide videos demonstrating the use and behaviour of the systems at: https://tinyurl com/y3evkcfz  A simple MT-based translate-to-source approach at inference time is currently used to enable the deployment of the system in other languages: 1) the pool of responses in each language is translated to English by Google Translate beforehand, and pre-computed encodings of their English translations are used as representations of each foreign language response; 2) a provided user utterance (i e , context) is translated to English on-the-fly and its encoding $h_c$ is then learned  We plan to experiment with more sophisticated multilingual models in future work 
Other Functionality ::: Voice-Controlled Menu Search 
An additional functionality enables the user to get parts of the restaurant menu relevant to the current user utterance as responses  This is achieved by performing an additional ranking step of available menu items and retrieving the ones that are semantically relevant to the user utterance using exactly the same methodology as with ranking other responses  An example of this functionality is shown in Figure FIGREF21 
Other Functionality ::: Resetting and Switching to Booking 
The restaurant search system needs to support the discrete actions of restarting the conversation (i e , resetting the set $R$), and should enable transferring to the slot-based table booking flow  This is achieved using two binary intent classifiers, that are run at each step in the dialogue  These classifiers make use of the already-computed $h_c$ vector that represents the user's latest text  A single-layer neural net is learned on top of the 512-dimensional encoding, with a ReLU activation and 100 hidden nodes  To train the classifiers, sets of 20 relevant paraphrases (e g , “Start again”) are provided as positive examples  Finally, when the system successfully switches to the booking scenario, it proceeds to the slot filling task: it aims to extract all the relevant booking information from the user (e g , date, time, number of people to dine)  The entire flow of the system illustrating both the search phase and the booking phase is provided as the supplemental video material 
Conclusion and Future Work
This paper has presented a general approach to search-based dialogue that does not rely on explicit semantic representations such as dialogue acts or slot-value ontologies, and allows for multi-modal responses  In future work, we will extend the current demo system to more tasks and languages, and work with more sophisticated encoders and ranking functions  Besides the initial dialogue flow from this work (§SECREF3), we will also work with more complex flows dealing, e g , with user intent shifts 
Context 2: Introduction
In June 2015, the operators of the online discussion site Reddit banned several communities under new anti-harassment rules  BIBREF0 used this opportunity to combine rich online data with computational methods to study a current question: Does eliminating these “echo chambers” diminish the amount of hate speech overall? Exciting opportunities like these, at the intersection of “thick” cultural and societal questions on the one hand, and the computational analysis of rich textual data on larger-than-human scales on the other, are becoming increasingly common 
Indeed, computational analysis is opening new possibilities for exploring challenging questions at the heart of some of the most pressing contemporary cultural and social issues  While a human reader is better equipped to make logical inferences, resolve ambiguities, and apply cultural knowledge than a computer, human time and attention are limited  Moreover, many patterns are not obvious in any specific context, but only stand out in the aggregate  For example, in a landmark study, BIBREF1 analyzed the authorship of The Federalist Papers using a statistical text analysis by focusing on style, based on the distribution of function words, rather than content  As another example, BIBREF2 studied what defines English haiku and showed how computational analysis and close reading can complement each other  Computational approaches are valuable precisely because they help us identify patterns that would not otherwise be discernible 
Yet these approaches are not a panacea  Examining thick social and cultural questions using computational text analysis carries significant challenges  For one, texts are culturally and socially situated  They reflect the ideas, values and beliefs of both their authors and their target audiences, and such subtleties of meaning and interpretation are difficult to incorporate in computational approaches  For another, many of the social and cultural concepts we seek to examine are highly contested — hate speech is just one such example  Choices regarding how to operationalize and analyze these concepts can raise serious concerns about conceptual validity and may lead to shallow or obvious conclusions, rather than findings that reflect the depth of the questions we seek to address 
These are just a small sample of the many opportunities and challenges faced in computational analyses of textual data  New possibilities and frustrating obstacles emerge at every stage of research, from identification of the research question to interpretation of the results  In this article, we take the reader through a typical research process that involves measuring social or cultural concepts using computational methods, discussing both the opportunities and complications that often arise  In the Reddit case, for example, hate speech is measured, however imperfectly, by the presence of particular words semi-automatically extracted from a machine learning algorithm  Operationalizations are never perfect translations, and are often refined over the course of an investigation, but they are crucial 
We begin our exploration with the identification of research questions, proceed through data selection, conceptualization, and operationalization, and end with analysis and the interpretation of results  The research process sounds more or less linear this way, but each of these phases overlaps, and in some instances turns back upon itself  The analysis phase, for example, often feeds back into the original research questions, which may continue to evolve for much of the project  At each stage, our discussion is critically informed by insights from the humanities and social sciences, fields that have focused on, and worked to tackle, the challenges of textual analysis—albeit at smaller scales—since their inception 
In describing our experiences with computational text analysis, we hope to achieve three primary goals  First, we aim to shed light on thorny issues not always at the forefront of discussions about computational text analysis methods  Second, we hope to provide a set of best practices for working with thick social and cultural concepts  Our guidance is based on our own experiences and is therefore inherently imperfect  Still, given our diversity of disciplinary backgrounds and research practices, we hope to capture a range of ideas and identify commonalities that will resonate for many  And this leads to our final goal: to help promote interdisciplinary collaborations  Interdisciplinary insights and partnerships are essential for realizing the full potential of any computational text analysis that involves social and cultural concepts, and the more we are able to bridge these divides, the more fruitful we believe our work will be 
Research questions
We typically start by identifying the questions we wish to explore  Can text analysis provide a new perspective on a “big question” that has been attracting interest for years? Or can we raise new questions that have only recently emerged, for example about social media? For social scientists working in computational analysis, the questions are often grounded in theory, asking: How can we explain what we observe? These questions are also influenced by the availability and accessibility of data sources  For example, the choice to work with data from a particular social media platform may be partly determined by the fact that it is freely available, and this will in turn shape the kinds of questions that can be asked  A key output of this phase are the concepts to measure, for example: influence; copying and reproduction; the creation of patterns of language use; hate speech  Computational analysis of text motivated by these questions is insight driven: we aim to describe a phenomenon or explain how it came about  For example, what can we learn about how and why hate speech is used or how this changes over time? Is hate speech one thing, or does it comprise multiple forms of expression? Is there a clear boundary between hate speech and other types of speech, and what features make it more or less ambiguous? In these cases, it is critical to communicate high-level patterns in terms that are recognizable 
This contrasts with much of the work in computational text analysis, which tends to focus on automating tasks that humans perform inefficiently  These tasks range from core linguistically motivated tasks that constitute the backbone of natural language processing, such as part-of-speech tagging and parsing, to filtering spam and detecting sentiment  Many tasks are motivated by applications, for example to automatically block online trolls  Success, then, is often measured by performance, and communicating why a certain prediction was made—for example, why a document was labeled as positive sentiment, or why a word was classified as a noun—is less important than the accuracy of the prediction itself  The approaches we use and what we mean by `success' are thus guided by our research questions 
Domain experts and fellow researchers can provide feedback on questions and help with dynamically revising them  For example, they may say “we already think we know that”, “that's too naïve”, “that doesn't reflect social reality” (negative); “two major camps in the field would give different answers to that question” (neutral); “we tried to look at that back in the 1960s, but we didn't have the technology” (positive); and “that sounds like something that people who made that archive would love”, “that's a really fundamental question” (very positive) 
Sometimes we also hope to connect to multiple disciplines  For example, while focusing on the humanistic concerns of an archive, we could also ask social questions such as “is this archive more about collaborative processes, culture-building or norm creation?” or “how well does this archive reflect the society in which it is embedded?" BIBREF3 used quantitative methods to tell a story about Darwin's intellectual development—an essential biographical question for a key figure in the history of science  At the same time, their methods connected Darwin's development to the changing landscape of Victorian scientific culture, allowing them to contrast Darwin's “foraging” in the scientific literature of his time to the ways in which that literature was itself produced  Finally, their methods provided a case study, and validation of technical approaches, for cognitive scientists who are interested in how people explore and exploit sources of knowledge 
Questions about potential “dual use” may also arise  Returning to our introductory example, BIBREF0 started with a deceptively simple question: if an internet platform eliminates forums for hate speech, does this impact hate speech in other forums? The research was motivated by the belief that a rising tide of online hate speech was (and is) making the internet increasingly unfriendly for disempowered groups, including minorities, women, and LBGTQ individuals  Yet the possibility of dual use troubled the researchers from the onset  Could the methodology be adopted to target the speech of groups like Black Lives Matter? Could it be adopted by repressive governments to minimize online dissent? While these concerns remained, they concluded that hypothetical dual use scenarios did not outweigh the tangible contribution this research could offer towards making the online environment more equal and just 
Data
The next step involves deciding on the data sources, collecting and compiling the dataset, and inspecting its metadata 
Data acquisition
Many scholars in the humanities and the social sciences work with sources that are not available in digital form, and indeed may never be digitized  Others work with both analogue and digitized materials, and the increasing digitization of archives has opened opportunities to study these archives in new ways  We can go to the canonical archive or open up something that nobody has studied before  For example, we might focus on major historical moments (French Revolution, post-Milosevic Serbia) or critical epochs (Britain entering the Victorian era, the transition from Latin to proto-Romance)  Or, we could look for records of how people conducted science, wrote and consumed literature, and worked out their philosophies 
A growing number of researchers work with born-digital sources or data  Born-digital data, e g , from social media, generally do not involve direct elicitation from participants and therefore enable unobtrusive measurements BIBREF5 , BIBREF6   In contrast, methods like surveys sometimes elicit altered responses from participants, who might adapt their responses to what they think is expected  Moreover, born-digital data is often massive, enabling large-scale studies of language and behavior in a variety of social contexts 
Still, many scholars in the social sciences and humanities work with multiple data sources  The variety of sources typically used means that more than one data collection method is often required  For example, a project examining coverage of a UK General Election, could draw data from traditional media, web archives, Twitter and Facebook, campaign manifestos, etc  and might combine textual analysis of these materials with surveys, laboratory experiments, or field observations offline  In contrast, many computational studies based on born-digital data have focused on one specific source, such as Twitter 
The use of born-digital data raises ethical concerns  Although early studies often treated privacy as a binary construct, many now acknowledge its complexity BIBREF7   Conversations on private matters can be posted online, visible for all, but social norms regarding what should be considered public information may differ from the data's explicit visibility settings  Often no informed consent has been obtained, raising concerns and challenges regarding publishing content and potentially harmful secondary uses BIBREF8 , BIBREF4  
Recently, concerns about potential harms stemming from secondary uses have led a number of digital service providers to restrict access to born-digital data  Facebook and Twitter, for example, have reduced or eliminated public access to their application programming interfaces (APIs) and expressed hesitation about allowing academic researchers to use data from their platforms to examine certain sensitive or controversial topics  Despite the seeming abundance of born-digital data, we therefore cannot take its availability for granted 
Working with data that someone else has acquired presents additional problems related to provenance and contextualisation  It may not always be possible to determine the criteria applied during the creation process  For example, why were certain newspapers digitized but not others, and what does this say about the collection? Similar questions arise with the use of born-digital data  For instance, when using the Internet Archive’s Wayback Machine to gather data from archived web pages, we need to consider what pages were captured, which are likely missing, and why 
We must often repurpose born-digital data (e g , Twitter was not designed to measure public opinion), but data biases may lead to spurious results and limit justification for generalization  In particular, data collected via black box APIs designed for commercial, not research, purposes are likely to introduce biases into the inferences we draw, and the closed nature of these APIs means we rarely know what biases are introduced, let alone how severely they might impact our research BIBREF10   These, however, are not new problems  Historians, for example, have always understood that their sources were produced within particular contexts and for particular purposes, which are not always apparent to us 
Non-representative data can still be useful for making comparisons within a sample  In the introductory example on hate speech BIBREF0 , the Reddit forums do not present a comprehensive or balanced picture of hate speech: the writing is almost exclusively in English, the targets of hate speech are mainly restricted (e g , to black people, or women), and the population of writers is shaped by Reddit's demographics, which skew towards young white men  These biases limit the generalizability of the findings, which cannot be extrapolated to other languages, other types of hate speech, and other demographic groups  However, because the findings are based on measurements on the same sort of hate speech and the same population of writers, as long as the collected data are representative of this specific population, these biases do not pose an intractable validity problem if claims are properly restricted 
The size of many newly available datasets is one of their most appealing characteristics  Bigger datasets often make statistics more robust  The size needed for a computational text analysis depends on the research goal: When it involves studying rare events, bigger datasets are needed  However, larger is not always better  Some very large archives are “secretly” collections of multiple and distinct processes that no in-field scholar would consider related  For example, Google Books is frequently used to study cultural patterns, but the over-representation of scientific articles in Google books can be problematic BIBREF11   Even very large born-digital datasets usually cover limited timespans compared to, e g , the Gutenberg archive of British novels 
This stage of the research also raises important questions about fairness  Are marginalized groups, for example, represented in the tweets we have collected? If not, what types of biases might result from analyses relying on those tweets?
Local experts and “informants” can help navigate the data  They can help understand the role an archive plays in the time and place  They might tell us: Is this the central archive, or a peripheral one? What makes it unusual? Or they might tell us how certain underrepresented communities use a social media platform and advise us on strategies for ensuring our data collection includes their perspectives 
However, when it is practically infeasible to navigate the data in this way—for instance, when we cannot determine what is missing from Twitter's Streaming API or what webpages are left out of the Internet Archive—we should be open about the limitations of our analyses, acknowledging the flaws in our data and drawing cautious and reasonable conclusions from them  In all cases, we should report the choices we have made when creating or re-using any dataset 
Compiling data
After identifying the data source(s), the next step is compiling the data  This step is fundamental: if the sources cannot support a convincing result, no result will be convincing  In many cases, this involves defining a “core" set of documents and a “comparison" set  We often have a specific set of documents in mind: an author's work, a particular journal, a time period  But if we want to say that this “core" set has some distinctive property, we need a “comparison" set  Expanding the collection beyond the documents that we would immediately think of has the beneficial effect of increasing our sample size  Having more sources increases the chance that we will notice something consistent across many individually varying contexts 
Comparing sets of documents can sometimes support causal inference, presented as a contrast between a treatment group and a control  In BIBREF0 , the treatment consisted of the text written in the two forums that were eventually closed by Reddit  However, identifying a control group required a considerable amount of time and effort  Reddit is a diverse platform, with a wide variety of interactional and linguistic styles; it would be pointless to compare hate speech forums against forums dedicated to, say, pictures of wrecked bicycles  Chandrasekharan et al  used a matching design, populating the control group with forums that were as similar as possible to the treatment group, but were not banned from Reddit  The goal is to estimate the counterfactual scenario: in this case, what would have happened had the site not taken action against these specific forums? An ideal control would make it possible to distinguish the effect of the treatment — closing the forums — from other idiosyncratic properties of texts that were treated 
We also look for categories of documents that might not be useful  We might remove documents that are meta-discourse, like introductions and notes, or documents that are in a language that is not the primary language of the collection, or duplicates when we are working with archived web pages  However, we need to carefully consider the potential consequences of information we remove  Does its removal alter the data, or the interpretation of the data, we are analyzing? Are we losing anything that might be valuable at a later stage?
Labels and metadata
Sometimes all we have is documents, but often we want to look at documents in the context of some additional information, or metadata  This additional information could tell us about the creation of documents (date, author, forum), or about the reception of documents (flagged as hate speech, helpful review)  Information about text segments can be extremely valuable, but it is also prone to errors, inconsistencies, bias, and missing information  Examining metadata is a good way to check a collection's balance and representativeness  Are sources disproportionately of one form? Is the collection missing a specific time window? This type of curation can be extremely time consuming as it may require expert labeling, but it often leads to the most compelling results  Sometimes metadata are also used as target labels to develop machine learning models  But using them as a “ground truth” requires caution  Labels sometimes mean something different than we expect  For example, a down vote for a social media post could indicate that the content is offensive, or that the voter simply disagreed with the expressed view 
Conceptualization
A core step in many analyses is translating social and cultural concepts (such as hate speech, rumor, or conversion) into measurable quantities  Before we can develop measurements for these concepts (the operationalization step, or the “implementation” step as denoted by BIBREF12 ), we need to define them  In the conceptualization phase we often start with questions such as: who are the domain experts, and how have they approached the topic? We are looking for a definition of the concept that is flexible enough to apply on our dataset, yet formal enough for computational research  For example, our introductory study on hate speech BIBREF0 used a statement on hate speech produced by the European Union Court of Human Rights  The goal was not to implement this definition directly in software but to use it as a reference point to anchor subsequent analyses 
If we want to move beyond the use of ad hoc definitions, it can be useful to distinguish between what political scientists Adcock and Collier call the “background concept” and the “systematized concept” BIBREF13   The background concept comprises the full and diverse set of meanings that might be associated with a particular term  This involves delving into theoretical, conceptual, and empirical studies to assess how a concept has been defined by other scholars and, most importantly, to determine which definition is most appropriate for the particular research question and the theoretical framework in which it is situated  That definition, in turn, represents the systematized concept: the formulation that is adopted for the study 
It is important to consider that for social and cultural concepts there is no absolute ground truth  There are often multiple valid definitions for a concept (the “background” concept in the terms of Adcock and Collier), and definitions might be contested over time  This may be uncomfortable for computer scientists, whose primary measure of success is often based on comparing a model's output against “ground truth” or a “gold standard”, e g , by comparing a sentiment classifier's output against manual annotations  However, the notion of ground truth is uncommon in the humanities and the social sciences and it is often taken too far in machine learning  BIBREF14 notes that in literary criticism and the digital humanities more broadly “interpretation, ambiguity, and argumentation are prized far above ground truth and definitive conclusions"  BIBREF15 draw attention to the different attitudes of literary scholars and computational linguists towards ambiguity, stating that “In Computational Linguistics [  ] ambiguity is almost uniformly treated as a problem to be solved; the focus is on disambiguation, with the assumption that one true, correct interpretation exists " The latter is probably true for tasks such as spam filtering, but in the social sciences and the humanities many relevant concepts are fundamentally unobservable, such as latent traits of political actors BIBREF16 or cultural fit in organizations BIBREF17 , leading to validation challenges  Moreover, when the ground truth comes from people, it may be influenced by ideological priors, priming, simple differences of opinion or perspective, and many other factors BIBREF18   We return to this issue in our discussions on validation and analysis 
Operationalization
In this phase we develop measures (or, “operationalizations”, or “indicators”) for the concepts of interest, a process called “operationalization”  Regardless of whether we are working with computers, the output produced coincides with Adcock and Collier's “scores”—the concrete translation and output of the systematized concept into numbers or labels BIBREF13   Choices made during this phase are always tied to the question “Are we measuring what we intend to measure?” Does our operationalization match our conceptual definition? To ensure validity we must recognize gaps between what is important and what is easy to measure  We first discuss modeling considerations  Next, we describe several frequently used computational approaches and their limitations and strengths 
Modeling considerations
The variables (both predictors and outcomes) are rarely simply binary or categorical  For example, a study on language use and age could focus on chronological age (instead of, e g , social age BIBREF19 )  However, even then, age can be modeled in different ways  Discretization can make the modeling easier and various NLP studies have modeled age as a categorical variable BIBREF20   But any discretization raises questions: How many categories? Where to place the boundaries? Fine distinctions might not always be meaningful for the analysis we are interested in, but categories that are too broad can threaten validity  Other interesting variables include time, space, and even the social network position of the author  It is often preferable to keep the variable in its most precise form  For example, BIBREF21 perform exploration in the context of hypothesis testing by using latitude and longitude coordinates — the original metadata attached to geotagged social media such as tweets — rather than aggregating into administrative units such as counties or cities  This is necessary when such administrative units are unlikely to be related to the target concept, as is the case in their analysis of dialect differences  Focusing on precise geographical coordinates also makes it possible to recognize fine-grained effects, such as language variation across the geography of a city 
Using a particular classification scheme means deciding which variations are visible, and which ones are hidden BIBREF22   We are looking for a categorization scheme for which it is feasible to collect a large enough labeled document collection (e g , to train supervised models), but which is also fine-grained enough for our purposes  Classification schemes rarely exhibit the ideal properties, i e , that they are consistent, their categories are mutually exclusive, and that the system is complete BIBREF22   Borderline cases are challenging, especially with social and cultural concepts, where the boundaries are often not clear-cut  The choice of scheme can also have ethical implications BIBREF22   For example, gender is usually represented as a binary variable in NLP and computational models tend to learn gender-stereotypical patterns  The operationalization of gender in NLP has been challenged only recently BIBREF23 , BIBREF24 , BIBREF25  
Supervised and unsupervised learning are the most common approaches to learning from data  With supervised learning, a model learns from labeled data (e g , social media messages labeled by sentiment) to infer (or predict) these labels from unlabeled texts  In contrast, unsupervised learning uses unlabeled data  Supervised approaches are especially suitable when we have a clear definition of the concept of interest and when labels are available (either annotated or native to the data)  Unsupervised approaches, such as topic models, are especially useful for exploration  In this setting, conceptualization and operationalization may occur simultaneously, with theory emerging from the data BIBREF26   Unsupervised approaches are also used when there is a clear way of measuring a concept, often based on strong assumptions  For example, BIBREF3 measure “surprise” in an analysis of Darwin's reading decisions based on the divergence between two probability distributions 
From an analysis perspective, the unit of text that we are labeling (or annotating, or coding), either automatic or manual, can sometimes be different than one's final unit of analysis  For example, if in a study on media frames in news stories, the theoretical framework and research question point toward frames at the story level (e g , what is the overall causal analysis of the news article?), the story must be the unit of analysis  Yet it is often difficult to validly and reliably code a single frame at the story level  Multiple perspectives are likely to sit side-by-side in a story  Thus, an article on income inequality might point to multiple causes, such as globalization, education, and tax policies  Coding at the sentence level would detect each of these causal explanations individually, but this information would need to be somehow aggregated to determine the overall story-level frame  Sometimes scholars solve this problem by only examining headlines and lead paragraphs, arguing that based on journalistic convention, the most important information can be found at the beginning of a story  However, this leads to a return to a shorter, less nuanced analysis 
From a computational perspective, the unit of text can also make a huge difference, especially when we are using bag-of-words models, where word order within a unit does not matter  Small segments, like tweets, sometimes do not have enough information to make their semantic context clear  In contrast, larger segments, like novels, have too much variation, making it difficult to train focused models  Finding a good segmentation sometimes means combining short documents and subdividing long documents  The word “document" can therefore be misleading  But it is so ingrained in the common NLP lexicon that we use it anyway in this article 
For insight-driven text analysis, it is often critical that high-level patterns can be communicated  Furthermore, interpretable models make it easier to find spurious features, to do error analysis, and to support interpretation of results  Some approaches are effective for prediction, but harder to interpret  The value we place on interpretability can therefore influence the approach we choose  There is an increasing interest in developing interpretable or transparent models in the NLP and machine learning communities 
Annotation
Many studies involve human coders  Sometimes the goal is to fully code the data, but in a computational analysis we often use the labels (or annotations) to train machine learning models to automatically recognize them, and to identify language patterns that are associated with these labels  For example, for a project analyzing rumors online BIBREF27 , conversation threads were annotated along different dimensions, including rumor versus non-rumor and stance towards a rumor 
The collection of annotation choices make up an annotation scheme (or “codebook”)  Existing schemes and annotations can be useful as starting points
Context 3: Introduction
In recent years, gender has become a hot topic within the political, societal and research spheres  Numerous studies have been conducted in order to evaluate the presence of women in media, often revealing their under-representation, such as the Global Media Monitoring Project BIBREF0  In the French context, the CSA BIBREF1 produces a report on gender representation in media on a yearly basis  The 2017 report shows that women represent 40% of French media speakers, with a significant drop during high-audience hours (6:00-8:00pm) reaching a value of only 29%  Another large scale study confirmed this trend with an automatic analysis of gender in French audiovisuals streams, highlighting a huge variation across type of shows BIBREF2 
Besides the social impact of gender representation, broadcast recordings are also a valuable source of data for the speech processing community  Indeed, automatic speech recognition (ASR) systems require large amount of annotated speech data to be efficiently trained, which leaves us facing the emerging concern about the fact that "AI artifacts tend to reflect the goals, knowledge and experience of their creators" BIBREF3  Since we know that women are under-represented in media and that the AI discipline has retained a male-oriented focus BIBREF4, we can legitimately wonder about the impact of using such data as a training set for ASR technologies  This concern is strengthened by the recent works uncovering gender bias in several natural language processing (NLP) tools such as BIBREF5, BIBREF6, BIBREF7, BIBREF8 
In this paper, we first highlight the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have  We then perform a statistical analysis of gender representation in a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community  Finally we question the impact of such a representation on the systems developed on this data, through the perspective of an ASR system 
From gender representation in data to gender bias in AI ::: On the importance of data
The ever growing use of machine learning in science has been enabled by several progresses among which the exponential growth of data available  The quality of a system now depends mostly on the quality and quantity of the data it has been trained on  If it does not discard the importance of an appropriate architecture, it reaffirms the fact that rich and large corpora are a valuable resource  Corpora are research contributions which do not only allow to save and observe certain phenomena or validate a hypothesis or model, but are also a mandatory part of the technology development  This trend is notably observable within the NLP field, where industrial technologies, such as Apple, Amazon or Google vocal assistants now reach high performance level partly due to the amount of data possessed by these companies BIBREF9 
Surprisingly, as data is said to be “the new oil", few data sets are available for ASR systems  The best known are corpora like TIMIT BIBREF10, Switchboard BIBREF11 or Fisher BIBREF12 which date back to the early 1990s  The scarceness of available corpora is justified by the fact that gathering and annotating audio data is costly both in terms of money and time  Telephone conversations and broadcast recordings have been the primary source of spontaneous speech used  Out of all the 130 audio resources proposed by LDC to train automatic speech recognition systems in English, approximately 14% of them are based on broadcast news and conversation  For French speech technologies, four corpora containing radio and TV broadcast are the most widely used: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16  These four corpora have been built alongside evaluation campaigns and are still, to our knowledge, the largest French ones of their type available to date 
From gender representation in data to gender bias in AI ::: From data to bias
The gender issue has returned to the forefront of the media scene in recent years and with the emergence of AI technologies in our daily lives, gender bias has become a scientific topic that researchers are just beginning to address  Several studies revealed the existence of gender bias in AI technologies such as face recognition (GenderShades BIBREF17), NLP (word embeddings BIBREF5 and semantics BIBREF6) and machine translation (BIBREF18, BIBREF7)  The impact of the training data used within these deep-learning algorithms is therefore questioned 
Bias can be found at different levels as pointed out by BIBREF19  BIBREF20 defines bias as a skew that produces a type of harm  She distinguishes two types of harms that are allocation harm and representation harm  The allocation harm occurs when a system is performing better or worse for a certain group while representational harm contributes to the perpetuation of stereotypes  Both types of harm are the results of bias in machine learning that often comes from the data systems are trained on  Disparities in representation in our social structures is captured and reflected by the training data, through statistical patterns  The GenderShades study is a striking example of what data disparity and lack of representation can produce: the authors tested several gender recognition modules used by facial recognition tools and found difference in error-rate as high as 34 percentage points between recognition of white male and black female faces  The scarce presence of women and colored people in training set resulted in bias in performance towards these two categories, with a strong intersectional bias  As written by BIBREF21 "A data set may have many millions of pieces of data, but this does not mean it is random or representative  To make statistical claims about a data set, we need to know where data is coming from; it is similarly important to know and account for the weaknesses in that data " (p 668) 
Regarding ASR technology, little work has explored the presence of gender bias within the systems and no consensus has been reached  BIBREF22 found that speech recognizers perform better on female voice on a broadcast news and telephone corpus  They proposed several explanations to this observation, such as the larger presence of non-professional male speech in the broadcast data, implying a less prepared speech for these speakers or a more normative language and standard pronunciation for women linked to the traditional role of women in language acquisition and education  The same trend was observed by BIBREF23  More recently, BIBREF24 discovered a gender bias within YouTube's automatic captioning system but this bias was not observed in a second study evaluating Bing Speech system and YouTube Automatic Captions on a larger data set BIBREF8  However race and dialect bias were found  General American speakers and white speakers had the lowest error rate for both systems  If the better performance on General American speakers could be explained by the fact that they are all voice professionals, producing clear and articulated speech, but no explanation is provided for biases towards non-white speakers 
Gender bias in ASR technology is still an open research question as no clear answer has been reached so far  It seems that many parameters are to take into account to achieve a general agreement  As we established the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have, the following content of this paper is structured as this: we first describe statistically the gender representation of a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community, introducing the notion of speaker's role to refine our analysis in terms of voice professionalism  We then question the impact of such a representation on a ASR system trained on these data  BIBREF25
Methodology
This section is organized as follows: we first present the data we are working on  In a second time we explain how we proceed to describe the gender representation in our corpus and introduce the notion of speaker's role  The third subsection introduces the ASR system and metrics used to evaluate gender bias in performance 
Methodology ::: Data presentation
Our data consists of two sets used to train and evaluate our automatic speech recognition system  Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16  These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR  Show duration varies between 10min and an hour  As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed  ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition 
Our training set contains 27,085 speech utterances produced by 2,506 speakers, accounting for approximately 100 hours of speech  Our evaluation set contains 74,064 speech utterances produced by 1,268 speakers for a total of 70 hours of speech

ground_truth: No

No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What background do they have?
Context 2:  Usually settling on an annotation scheme requires several iterations, in which the guidelines are updated and annotation examples are added  For example, a political scientist could use a mixed deductive-inductive strategy for developing a codebook  She starts by laying out a set of theory-driven deductive coding rules, which means that the broad principles of the coding rules are laid out without examining examples first  These are then tested (and possibly adjusted) based on a sample of the data  In line with Adcock and Collier's notion of “content validity” BIBREF13 , the goal is to assess whether the codebook adequately captures the systematized concept  By looking at the data themselves, she gains a better sense of whether some things have been left out of the coding rules and whether anything is superfluous, misleading, or confusing  Adjustments are made and the process is repeated, often with another researcher involved 
The final annotations can be collected using a crowdsourcing platform, a smaller number of highly-trained annotators, or a group of experts  Which type of annotator to use should be informed by the complexity and specificity of the concept  For more complex concepts, highly-trained or expert annotators tend to produce more reliable results  However, complex concepts can sometimes be broken down into micro-tasks that can be performed independently in parallel by crowdsourced annotators  Concepts from highly specialized domains may require expert annotators  In all cases, however, some training will be required, and the training phase should involve continual checks of inter-annotator agreement (i e  intercoder reliability) or checks against a gold standard (e g  quizzes in crowdsourcing platforms) 
We also need to decide how inter-annotator agreement will be measured and what an acceptable level of agreement would be  Krippendorff's alpha is frequently used in the social sciences, but the right measure depends on the type of data and task  For manual coding, we can continually check inter-annotator agreement and begin introducing checks of intra-annotator agreement, too  For most communication scholars using only manual content analysis, an acceptable rate of agreement is achieved when Krippendorf's alpha reaches 0 80 or above  When human-coded data are used to validate machine learning algorithms, the reliability of the human-coded data is even more important  Disagreement between annotators can signal weaknesses of the annotation scheme, or highlight the inherent ambiguity in what we are trying to measure  Disagreement itself can be meaningful and can be integrated in subsequent analyses BIBREF28 , BIBREF29  
Data pre-processing
Preparing the data can be a complex and time-consuming process, often involving working with partially or wholly unstructured data  The pre-processing steps have a big impact on the operationalizations, subsequent analyses and reproducibility efforts BIBREF30 , and they are usually tightly linked to what we intend to measure  Unfortunately, these steps tend to be underreported, but documenting the pre-processing choices made is essential and is analogous to recording the decisions taken during the production of a scholarly edition or protocols in biomedical research  Data may also vary enormously in quality, depending on how it has been generated  Many historians, for example, work with text produced from an analogue original using Optical Character Recognition (OCR)  Often, there will be limited information available regarding the accuracy of the OCR, and the degree of accuracy may even vary within a single corpus (e g  where digitized text has been produced over a period of years, and the software has gradually improved)  The first step, then, is to try to correct for common OCR errors  These will vary depending on the type of text, the date at which the `original' was produced, and the nature of the font and typesetting 
One step that almost everyone takes is to tokenize the original character sequence into the words and word-like units  Tokenization is a more subtle and more powerful process than people expect  It is often done using regular expressions or scripts that have been circulating within the NLP community  Tokenization heuristics, however, can be badly confused by emoticons, creative orthography (e g , U$A, sh!t), and missing whitespace  Multi-word terms are also challenging  Treating them as a single unit can dramatically alter the patterns in text  Many words that are individually ambiguous have clear, unmistakable meanings as terms, like “black hole" or “European Union"  However, deciding what constitutes a multi-word term is a difficult problem  In writing systems like Chinese, tokenization is a research problem in its own right 
Beyond tokenization, common steps include lowercasing, removing punctuation, stemming (removing suffixes), lemmatization (converting inflections to a base lemma), and normalization, which has never been clearly defined, but often includes grouping abbreviations like “U S A " and “USA", ordinals like “1st" and “first", and variant spellings like “noooooo"  The main goal of these steps is to improve the ratio of tokens (individual occurrences) to types (the distinct things in a corpus)  Each step requires making additional assumptions about which distinctions are relevant: is “apple” different from “Apple”? Is “burnt” different from “burned”? Is “cool" different from “coooool"? Sometimes these steps can actively hide useful patterns, like social meaning BIBREF32   Some of us therefore try do as little modification as possible 
From a multilingual perspective, English and Chinese have an unusually simple inflectional system, and so it is statistically reasonable to treat each inflection as a unique word type  Romance languages have considerably more inflections than English; many indigenous North American languages have still more  For these languages, unseen data is far more likely to include previously-unseen inflections, and therefore, dealing with inflections is more important  On the other hand, the resources for handling inflections vary greatly by language, with European languages dominating the attention of the computational linguistics community thus far 
We sometimes also remove words that are not relevant to our goals, for example by calculating vocabulary frequencies  We construct a “stoplist” of words that we are not interested in  If we are looking for semantic themes we might remove function words like determiners and prepositions  If we are looking for author-specific styles, we might remove all words except function words  Some words are generally meaningful but too frequent to be useful within a specific collection  We sometimes also remove very infrequent words  Their occurrences are too low for robust patterns and removing them helps reducing the vocabulary size 
The choice of processing steps can be guided by theory or knowledge about the domain as well as experimental investigation  When we have labels, predictive accuracy of a model is a way to assess the effect of the processing steps  In unsupervised settings, it is more challenging to understand the effects of different steps  Inferences drawn from unsupervised settings can be sensitive to pre-processing choices BIBREF33   Stemming has been found to provide little measurable benefits for topic modeling and can sometimes even be harmful BIBREF34   All in all, this again highlights the need to document these steps 
Finally, we can also mark up the data, e g , by identifying entities (people, places, organizations, etc ) or parts of speech  Although many NLP tools are available for such tasks, they are often challenged by linguistic variation, such as orthographic variation in historical texts BIBREF35 and social media BIBREF32   Moreover, the performance of NLP tools often drops when applying them outside the training domain, such as applying tools developed on newswire texts to texts written by younger authors BIBREF36   Problems (e g , disambiguation in named entity recognition) are sometimes resolved using considerable manual intervention  This combination of the automated and the manual, however, becomes more difficult as the scale of the data increases, and the `certainty' brought by the latter may have to be abandoned 
Dictionary-based approaches
Dictionaries are frequently used to code texts in content analyses BIBREF37   Dictionaries consist of one or more categories (i e  word lists)  Sometimes the output is simply the number of category occurrences (e g , positive sentiment), thus weighting words within a category equally  In some other cases, words are assigned continuous scores  The high transparency of dictionaries makes them sometimes more suitable than supervised machine learning models  However, dictionaries should only be used if the scores assigned to words match how the words are used in the data (see BIBREF38 for a detailed discussion on limitations)  There are many off-the-shelf dictionaries available (e g , LIWC BIBREF39 )  These are often well-validated, but applying them on a new domain may not be appropriate without additional validation  Corpus- or domain-specific dictionaries can overcome limitations of general-purpose dictionaries 
The dictionaries are often manually compiled, but increasingly they are constructed semi-automatically (e g , BIBREF40 )  When we semi-automatically create a word list, we use automation to identify an initial word list, and human insight to filter it  By automatically generating the initial words lists, words can be identified that human annotators might have difficulty intuiting  By manually filtering the lists, we use our theoretical understanding of the target concept to remove spurious features 
In the introduction study, SAGE BIBREF41 was used to obtain a list of words that distinguished the text in the treatment group (subreddits that were closed by Reddit) from text in the control group (similar subreddits that were not closed)  The researchers then returned to the hate speech definition provided by the European Court of Human Rights, and manually filtered the top SAGE words based on this definition  Not all identified words fitted the definition  The others included: the names of the subreddits themselves, names of related subreddits, community-specific jargon that was not directly related to hate speech, and terms such as IQ and welfare, which were frequently used in discourses of hate speech, but had significant other uses  The word lists provided the measurement instrument for their main result, which is that the use of hate speech throughout Reddit declined after the two treatment subreddits were closed 
Supervised models
Supervised learning is frequently used to scale up analyses  For example, BIBREF42 wanted to analyze the motivations of Movember campaign participants  By developing a classifier based on a small set of annotations, they were able to expand the analysis to over 90k participants 
The choice of supervised learning model is often guided by the task definition and the label types  For example, to identify stance towards rumors based on sequential annotations, an algorithm for learning from sequential BIBREF43 or time series data BIBREF44 could be used  The features (sometimes called variables or predictors) are used by the model to make the predictions  They may vary from content-based features such as single words, sequences of words, or information about their syntactic structure, to meta-information such as user or network information  Deciding on the features requires experimentation and expert insight and is often called feature engineering  For insight-driven analysis, we are often interested in why a prediction has been made and features that can be interpreted by humans may be preferred  Recent neural network approaches often use simple features as input (such as word embeddings or character sequences), which requires less feature engineering but make interpretation more difficult 
Supervised models are powerful, but they can latch on to spurious features of the dataset  This is particularly true for datasets that are not well-balanced, and for annotations that are noisy  In our introductory example on hate speech in Reddit BIBREF0 , the annotations are automatically derived from the forum in which each post appears, and indeed, many of the posts in the forums (subreddits) that were banned by Reddit would be perceived by many as hate speech  But even in banned subreddits, not all of the content is hate speech (e g , some of the top features were self-referential like the name of the subreddit) but a classifier would learn a high weight for these features 
Even when expert annotations are available on the level of individual posts, spurious features may remain  BIBREF45 produced expert annotations of hate speech on Twitter  They found that one of the strongest features for sexism is the name of an Australian TV show, because people like to post sexist comments about the contestants  If we are trying to make claims about what inhibits or encourages hate speech, we would not want those claims to be tied to the TV show's popularity  Such problems are inevitable when datasets are not well-balanced over time, across genres, topics, etc  Especially with social media data, we lack a clear and objective definition of `balance' at this time 
The risk of supervised models latching on to spurious features reinforces the need for interpretability  Although the development of supervised models is usually performance driven, placing more emphasis on interpretability could increase the adoption of these models in insight-driven analyses  One way would be to only use models that are already somewhat interpretable, for example models that use a small number of human-interpretable features  Rather than imposing such restrictions, there is also work on generating post-hoc explanations for individual predictions (e g , BIBREF46 ), even when the underlying model itself is very complex 
Topic modeling
Topic models (e g , LDA BIBREF47 ) are usually unsupervised and therefore less biased towards human-defined categories  They are especially suited for insight-driven analysis, because they are constrained in ways that make their output interpretable  Although there is no guarantee that a “topic” will correspond to a recognizable theme or event or discourse, they often do so in ways that other methods do not  Their easy applicability without supervision and ready interpretability make topic models good for exploration  Topic models are less successful for many performance-driven applications  Raw word features are almost always better than topics for search and document classification  LSTMs and other neural network models are better as language models  Continuous word embeddings have more expressive power to represent fine-grained semantic similarities between words 
A topic model provides a different perspective on a collection  It creates a set of probability distributions over the vocabulary of the collection, which, when combined together in different proportions, best match the content of the collection  We can sort the words in each of these distributions in descending order by probability, take some arbitrary number of most-probable words, and get a sense of what (if anything) the topic is “about”  Each of the text segments also has its own distribution over the topics, and we can sort these segments by their probability within a given topic to get a sense of how that topic is used 
One of the most common questions about topic models is how many topics to use, usually with the implicit assumption that there is a “right” number that is inherent in the collection  We prefer to think of this parameter as more like the scale of a map or the magnification of a microscope  The “right” number is determined by the needs of the user, not by the collection  If the analyst is looking for a broad overview, a relatively small number of topics may be best  If the analyst is looking for fine-grained phenomena, a larger number is better 
After fitting the model, it may be necessary to circle back to an earlier phase  Topic models find consistent patterns  When authors repeatedly use a particular theme or discourse, that repetition creates a consistent pattern  But other factors can also create similar patterns, which look as good to the algorithm  We might notice a topic that has highest probability on French stopwords, indicating that we need to do a better job of filtering by language  We might notice a topic of word fragments, such as “ing”, “tion”, “inter”, indicating that we are not handling end-of-line hyphenation correctly  We may need to add to our stoplist or change how we curate multi-word terms 
Validation
The output of our measurement procedures (in the social sciences often called the “scores”) must now be assessed in terms of their reliability and validity with regard to the (systemized) concept  Reliability aims to capture repeatability, i e  the extent to which a given tool provides consistent results 
Validity assesses the extent to which a given measurement tool measures what it is supposed to measure  In NLP and machine learning, most models are primarily evaluated by comparing the machine-generated labels against an annotated sample  This approach presumes that the human output is the “gold standard" against which performance should be tested  In contrast, when the reliability is measured based on the output of different annotators, no coder is taken as the standard and the likelihood of coders reaching agreement by chance (rather than because they are “correct") is factored into the resulting statistic  Comparing against a “gold standard” suggests that the threshold for human inter- and intra-coder reliability should be particularly high 
Accuracy, as well as other measures such as precision, recall and F-score, are sometimes presented as a measure of validity, but if we do not have a genuinely objective determination of what something is supposed measure—as is often the case in text analysis—then accuracy is perhaps a better indication of reliability than of validity  In that case, validity needs to be assessed based on other techniques like those we discuss later in this section  It is also worth asking what level of accuracy is sufficient for our analysis and to what extent there may be an upper bound, especially when the labels are native to the data or when the notion of a “gold standard” is not appropriate 
For some in the humanities, validation takes the form of close reading, not designed to confirm whether the model output is correct, but to present what BIBREF48 refers to as a form of “further discovery in two directions”  Model outputs tell us something about the texts, while a close reading of the texts alongside those outputs tells us something about the models that can be used for more effective model building  Applying this circular, iterative process to 450 18th-century novels written in three languages, Piper was able to uncover a new form of “conversional novel” that was not previously captured in “literary history's received critical categories” BIBREF48  
Along similar lines, we can subject both the machine-generated output and the human annotations to another round of content validation  That is, take a stratified random sample, selecting observations from the full range of scores, and ask: Do these make sense in light of the systematized concept? If not, what seems to be missing? Or is something extraneous being captured? This is primarily a qualitative process that requires returning to theory and interrogating the systematized concept, indicators, and scores together  This type of validation is rarely done in NLP, but it is especially important when it is difficult to assess what drives a given machine learning model  If there is a mismatch between the scores and systematized concept at this stage, the codebook may need to be adjusted, human coders retrained, more training data prepared, algorithms adjusted, or in some instances, even a new analytical method adopted 
Other types of validation are also possible, such as comparing with other approaches that aim to capture the same concept, or comparing the output with external measures (e g , public opinion polls, the occurrence of future events)  We can also go beyond only evaluating the labels (or point estimates)  BIBREF16 used human judgments to not only assess the positional estimates from a scaling method of latent political traits but also to assess uncertainty intervals  Using different types of validation can increase our confidence in the approach, especially when there is no clear notion of ground truth 
Besides focusing on rather abstract evaluation measures, we could also assess the models in task-based settings using human experts  Furthermore, for insight-driven analyses, it can be more useful to focus on improving explanatory power than making small improvements in predictive performance 
Analysis
In this phase, we use our models to explore or answer our research questions  For example, given a topic model we can look at the connection between topics and metadata elements  Tags such as “hate speech" or metadata information imply a certain way of organizing the collection  Computational models provide another organization, which may differ in ways that provide more insight into how these categories manifest themselves, or fail to do so 
Moreover, when using a supervised approach, the “errors”, i e  disagreement between the system output and human-provided labels, can point towards interesting cases for closer analysis and help us reflect on our conceptualizations  In the words of BIBREF2 , they can be “opportunities for interpretation”  Other types of “failures” can be insightful as well  Sometimes there is a “dog that didn't bark” BIBREF49 –i e , something that everyone thinks we should have found, but we did not  Or, sometimes the failures are telling us about the existence of something in the data that nobody noticed, or thought important, until then (e g , the large number of travel journals in Darwin's reading lists) 
Computational text analysis is not a replacement for but rather an addition to the approaches one can take to analyze social and cultural phenomena using textual data  By moving back and forth between large-scale computational analyses and small-scale qualitative analyses, we can combine their strengths so that we can identify large-scale and long-term trends, but also tell individual stories  For example, the Reddit study on hate speech BIBREF0 raised various follow-up questions: Can we distinguish hate speech from people talking about hate speech? Did people find new ways to express hate speech? If so, did the total amount of online hate speech decrease after all? As possible next steps, a qualitative discourse analyst might examine a smaller corpus to investigate whether commenters were indeed expressing hate speech in new ways; a specialist in interview methodologies might reach out to commenters to better understand the role of online hate speech in their lives  Computational text analysis represents a step towards better understanding social and cultural phenomena, and it is in many cases better suited towards opening questions rather than closing them 
Conclusion
Insight-driven computational analysis of text is becoming increasingly common  It not only helps us see more broadly, it helps us see subtle patterns more clearly and allows us to explore radical new questions about culture and society  In this article we have consolidated our experiences, as scholars from very different disciplines, in analyzing text as social and cultural data and described how the research process often unfolds  Each of the steps in the process is time-consuming and labor-intensive  Each presents challenges  And especially when working across disciplines, the research often involves a fair amount of discussion—even negotiation—about what means of operationalization and approaches to analysis are appropriate and feasible  And yet, with a bit of perseverance and mutual understanding, conceptually sound and meaningful work results so that we can truly make use of the exciting opportunities rich textual data offers 
Acknowledgements
This work was supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1  Dong Nguyen is supported with an Alan Turing Institute Fellowship (TU/A/000006)  Maria Liakata is a Turing fellow at 40%  We would also like to thank the participants of the “Bridging disciplines in analysing text as social and cultural data” workshop held at the Turing Institute (2017) for insightful discussions  The workshop was funded by a Turing Institute seed funding award to Nguyen and Liakata 
Context 3:  Training data by show, medium and speech type is summarized in Table and evaluation data in Table   Evaluation data has a higher variety of shows with both prepared (P) and spontaneous (S) speech type (accented speech from African radio broadcast is also included in the evaluation set) 
Methodology ::: Methodology for descriptive analysis of gender representation in training data
We first describe the gender representation in training data  Gender representation is measured in terms of number of speakers, number of utterances (or speech turns), and turn lengths (descriptive statistics are given in Section SECREF16)  Each speech turn was mapped to its speaker in order to associate it with a gender 
As pointed out by the CSA report BIBREF1, women presence tends to be marginal within the high-audience hours, showing that women are represented but less than men and within certain given conditions  It is clear that a small number of speakers is responsible for a large number of speech turns  Most of these speakers are journalists, politicians, presenters and such, who are representative of a show  Therefore, we introduce the notion of speaker's role to refine our exploration of gender disparity, following studies which quantified women's presence in terms of role  Within our work, we define the notion of speaker role by two criteria specifying the speaker's on-air presence, namely the number of speech turns and the cumulative duration of his or her speaking time in a show  Based on the available speech transcriptions and meta-data, we compute for each speaker the number of speech turns uttered as well as their total length  We then use the following criteria to define speaker's role: a speaker is considered as speaking often (respectively seldom) if he/she accumulates a total of turns higher (respectively lower) than 1% of the total number of speech turns in a given show  The same process is applied to identify speakers talking for a long period from those who do not  We end up with two salient roles called Anchors and Punctual speakers:
the Anchor speakers (A) are above the threshold of 1% for both criteria, meaning they are intervening often and for a long time thus holding an important place in interaction;
the Punctual speakers (PS) on the contrary are below the threshold of 1% for both the total number of turns and the total speech time 
These roles are defined at the show level  They could be roughly assimilated to the categorization “host/guest” in radio and TV shows  Anchors could be described as professional speakers, producing mostly prepared speech, whereas Punctual speakers are more likely to be “everyday people"  The concept of speaker's role makes sense at both sociological and technical levels  An Anchor speaker is more likely to be known from the audience (society), but he or she will also likely have a professional (clear) way of speaking (as mentioned by BIBREF22 and BIBREF8), as well as a high number of utterances, augmenting the amount of data available for a given gender category 
Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: ASR system
The ASR system used in this work is described in BIBREF25  It uses the KALDI toolkit BIBREF26, following a standard Kaldi recipe  The acoustic model is based on a hybrid HMM-DNN architecture and trained on the data summarized in Table   Acoustic training data correspond to 100h of non-spontaneous speech type (mostly broadcast news) coming from both radio and TV shows  A 5-gram language model is trained from several French corpora (3,323M words in total) using SRILM toolkit BIBREF27  The pronunciation model is developed using the lexical resource BDLEX BIBREF28 as well as automatic grapheme-to-phoneme (G2P) transcription to find pronunciation variants of our vocabulary (limited to 80K)  It is important to re-specify here, for further analysis, that our Kaldi pipeline follows speaker adaptive training (SAT) where we train and decode using speaker adapted features (fMLLR-adapted features) in per-speaker mode  It is well known that speaker adaptation acts as an effective procedure to reduce mismatch between training and evaluation conditions BIBREF29, BIBREF26 
Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: Evaluation
Word Error Rate (WER) is a common metric to evaluate ASR performance  It is measured as the sum of errors (insertions, deletions and substitutions) divided by the total number of words in the reference transcription  As we are investigating the impact on performance of speaker's gender and role, we computed the WER for each speaker at the episode (show occurrence) level  Analyzing at such granularity allows us to avoid large WER variation that could be observed at utterance level (especially for short speech turns) but also makes possible to get several WER values for a given speaker, one for each occurrence of a show in which he/she appears on  Speaker's gender was provided by the meta-data and role was obtained using the criteria from Section SECREF6 computed for each show  This enables us to analyze our results across gender and role categories which was done using Wilcoxon rank sum tests also called Mann-Whitney U test (with $\alpha $= 0 001) BIBREF30  The choice of a Wilcoxon rank sum test and not the commonly used t-test is motivated by the non-normality of our data 
Results ::: Descriptive analysis of gender representation in training data ::: Gender representation
As expected, we observe a disparity in terms of gender representation in our data (see Table )  Women represent 33 16% of the speakers, confirming the figures given by the GMMP report BIBREF0  However, it is worth noticing that women account for only 22 57% of the total speech time, which leads us to conclude that women also speak less than men 
Results ::: Descriptive analysis of gender representation in training data ::: Speaker's role representation
Table presents roles' representation in training data and shows that despite the small number of Anchor speakers in our data (3 79%), they nevertheless concentrate 35 71 % of the total speech time 
Results ::: Descriptive analysis of gender representation in training data ::: Role and gender interaction
When crossing both parameters, we can observe that the gender distribution is not constant throughout roles  Women represent 29 47% of the speakers within the Anchor category, even less than among the Punctual speakers  Their percentage of speech is also smaller  When calculating the average speech time uttered by a female Anchor, we obtain a value of 15 9 min against 25 2 min for a male Anchor, which suggests that even within the Anchor category men tend to speak more  This confirms the existence of gender disparities within French media  It corroborates with the analysis of the CSA BIBREF1, which shows that women were less present during high-audience hours  Our study shows that they are also less present in important roles  These results legitimate our initial questioning on the impact of gender balance on ASR performance trained on broadcast recordings 
Results ::: Performance (WER) analysis on evaluation data ::: Impact of gender on WER
As explained in Section SECREF13, WER is the sum of errors divided by the number of words in the transcription reference  The higher the WER, the poorer the system performance  Our 70h evaluation data contains a large amount of spontaneous speech and is very challenging for the ASR system trained on prepared speech: we observe an overall average WER of 42 9% for women and 34 3% for men  This difference of WER between men and women is statistically significant (med(M) = 25%; med(F) = 29%; U = 709040; p-value < 0 001) 
However, when observing gender differences across shows, no clear trend can be identified, as shown in Figure FIGREF21

ground_truth: Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How big is imbalance in analyzed corpora?
Context 3:  For shows like Africa1 Infos or La Place du Village, we find an average WER lower for women than for men, while the trend is reversed for shows such as Un Temps de Pauchon or Le Masque et la Plume  The disparity of the results depending on the show leads us to believe that other factors may be entangled within the observed phenomenon 
Results ::: Performance (WER) analysis on evaluation data ::: Impact of role on WER
Speaker's role seems to have an impact on WER: we obtain an average WER of 30 8% for the Anchor speakers and 42 23% for the Punctual speakers  This difference is statistically significant with a p-value smaller than $10^{-14}$ (med(A) = 21%; med(P) = 31%; U = 540,430; p-value < 0 001)  
Results ::: Performance (WER) analysis on evaluation data ::: Role and gender interaction
Figure FIGREF25 presents the WER distribution (WER being obtained for each speaker in a show occurrence) according to the speaker's role and gender  It is worth noticing that the gender difference is only significant within the Punctual speakers group  The average WER is of 49 04% for the women and 38 56% for the men with a p-value smaller than $10^{-6}$ (med(F) = 39%; med(M) = 29%; U = 251,450; p-value < 0 001), whereas it is just a trend between male and female Anchors (med(F) = 21%; med(M) = 21%; U = 116,230; p-value = 0 173)  This could be explained by the quantity of data available per speaker 
Results ::: Performance (WER) analysis on evaluation data ::: Speech type as a third entangled factor?
In order to try to explain the observed variation in our results depending on shows and gender (Figure FIGREF21), we add the notion of speech type to shed some light on our results  BIBREF22 and BIBREF24 suggested that the speaker professionalism, associated with clear and hyper-articulated speech could be an explaining factor for better performance 
Based on our categorization in prepared speech (mostly news reports) and spontaneous speech (mostly debates and entertainment shows), we cross this parameter in our performance analysis  As shown on Figure FIGREF26, these results confirm the inherent challenge of spontaneous speech compared to prepared speech  WER scores are similar between men and women when considering prepared speech (med(F) = 18%; med(M) = 21%; U = 217,160; p-value = 0 005) whereas they are worse for women (61 29%) than for men (46 51%) with p-value smaller than $10^{-14}$ for the spontaneous speech type (med(F) = 61%; med(M) = 37%; U = 153,580; p-value < 0 001) 
Discussion
We find a clear disparity in terms of women presence and speech quantity in French media  Our data being recorded between 1998 and 2013, we can expect this disparity to be smaller on more recent broadcast recordings, especially since the French government displays efforts toward parity in media representation  One can also argue that even if our analysis was conducted on a large amount of data it does not reach the exhaustiveness of large-scale studies such as the one of BIBREF2  Nonetheless it does not affect the relevance of our findings, because if real-world gender representation might be more balanced today, these corpora are still used as training data for AI systems 
The performance difference across gender we observed corroborates (on a larger quantity and variety of language data produced by more than 2400 speakers) the results obtained by BIBREF24 on isolated words recognition  However the following study on read speech does not replicate these results  Yet a performance degradation is observed across dialect and race BIBREF8  BIBREF22 found lower WER for women than men on broadcast news and conversational telephone speech for both English and French  The authors suggest that gender stereotypes associated with women role in education and language acquisition induce a more normative elocution  We observed that the higher the degree of normativity of speech the smaller the gender difference  No significant gender bias is observed for prepared speech nor within the Anchor category  Even if we do not find similar results with lower WER for women than men, we obtained a median WER smaller for women on prepared speech and equal to the male median WER for the Anchor speakers 
Another explanation could be the use of adaptation within the pipeline  Most broadcast programs transcription systems have a speaker adaptation step within their decoding pipeline, which is the case for our system  An Anchor speaker intervening more often would have a larger quantity of data to realize such adaptation of the acoustic model  On the contrary, Punctual speakers who appear scarcely in the data are not provided with the same amount of adaptation data  Hence we can hypothesize that gender performance difference observed for Punctual speakers is due to the fact that female speech is further from the (initial non-adapted) acoustic model as it was trained on unbalanced data (as shown in Table )  Considering that Punctual speakers represent 92 78% of the speakers, this explains why gender difference is significant over our entire data set  A way to confirm our hypothesis would be to reproduce our analysis on WER values obtained without using speaker adapted features at the decoding step 
When decoding prepared speech (hence similar to the training data), no significant difference is found in WER between men and women, revealing that the speaker adaptation step could be sufficient to reach same performance for both genders  But when decoding more spontaneous speech, there is a mismatch with the initial acoustic model (trained on prepared speech)  Consequently, the speaker adaptation step might not be enough to recover good ASR performance, especially for women for whom less adaptation data is available (see Section 4 2 3) 
Conclusion
This paper has investigated gender bias in ASR performance through the following research questions: i) what is the proportion of men and women in French radio and TV media data ? ii) what is the impact of the observed disparity on ASR performance ? iii) is this as simple as a problem of gender proportion in the training data or are other factors entangled ? Our contributions are the following:
Descriptive analysis of the broadcast data used to train our ASR system confirms the already known disparity, where 65% of the speakers are men, speaking more than 75% of the time 
When investigating WER scores according to gender, speaker's role and speech type, huge variations are observed  We conclude that gender is clearly a factor of variation in ASR performance, with a WER increase of 24% for women compared to men, exhibiting a clear gender bias 
Gender bias varies across speaker's role and speech spontaneity level  Performance for Punctual speakers respectively spontaneous speech seems to reinforce this gender bias with a WER increase of 27 2% respectively 31 8% between male and female speakers 
We found that an ASR system trained on unbalanced data regarding gender produces gender bias performance  Therefore, in order to create fair systems it is necessary to take into account the representation problems in society that are going to be encapsulated in the data  Understanding how women under-representation in broadcast data can lead to bias in ASR performances is the key to prevent re-implementing and reinforcing discrimination already existing in our societies  This is in line with the concept of “Fairness by Design" proposed by BIBREF31 
Gender, race, religion, nationality are all characteristics that we deem unfair to classify on, and these ethical standpoints needs to be taken into account in systems' design  Characteristics that are not considered as relevant in a given task can be encapsulated in data nonetheless, and lead to bias performance  Being aware of the demographic skews our data set might contain is a first step to track the life cycle of a training data set and a necessary step to control the tools we develop 

ground_truth: Women represent 33.16% of the speakers