prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What architecture does the decoder have?
Context 1: Introduction
This paper describes our approach and results for Task 2 of the CoNLL–SIGMORPHON 2018 shared task on universal morphological reinflection BIBREF0   The task is to generate an inflected word form given its lemma and the context in which it occurs 
Morphological (re)inflection from context is of particular relevance to the field of computational linguistics: it is compelling to estimate how well a machine-learned system can capture the morphosyntactic properties of a word given its context, and map those properties to the correct surface form for a given lemma 
There are two tracks of Task 2 of CoNLL–SIGMORPHON 2018: in Track 1 the context is given in terms of word forms, lemmas and morphosyntactic descriptions (MSD); in Track 2 only word forms are available  See Table TABREF1 for an example  Task 2 is additionally split in three settings based on data size: high, medium and low, with high-resource datasets consisting of up to 70K instances per language, and low-resource datasets consisting of only about 1K instances 
The baseline provided by the shared task organisers is a seq2seq model with attention (similar to the winning system for reinflection in CoNLL–SIGMORPHON 2016, BIBREF1 ), which receives information about context through an embedding of the two words immediately adjacent to the target form  We use this baseline implementation as a starting point and achieve the best overall accuracy of 49 87 on Task 2 by introducing three augmentations to the provided baseline system: (1) We use an LSTM to encode the entire available context; (2) We employ a multi-task learning approach with the auxiliary objective of MSD prediction; and (3) We train the auxiliary component in a multilingual fashion, over sets of two to three languages 
In analysing the performance of our system, we found that encoding the full context improves performance considerably for all languages: 11 15 percentage points on average, although it also highly increases the variance in results  Multi-task learning, paired with multilingual training and subsequent monolingual finetuning, scored highest for five out of seven languages, improving accuracy by another 9 86% on average 
System Description
Our system is a modification of the provided CoNLL–SIGMORPHON 2018 baseline system, so we begin this section with a reiteration of the baseline system architecture, followed by a description of the three augmentations we introduce 
Baseline
The CoNLL–SIGMORPHON 2018 baseline is described as follows:
The system is an encoder-decoder on character sequences  It takes a lemma as input and generates a word form  The process is conditioned on the context of the lemma [   ] The baseline treats the lemma, word form and MSD of the previous and following word as context in track 1  In track 2, the baseline only considers the word forms of the previous and next word  [   ] The baseline system concatenates embeddings for context word forms, lemmas and MSDs into a context vector  The baseline then computes character embeddings for each character in the input lemma  Each of these is concatenated with a copy of the context vector  The resulting sequence of vectors is encoded using an LSTM encoder  Subsequently, an LSTM decoder generates the characters in the output word form using encoder states and an attention mechanism 
To that we add a few details regarding model size and training schedule:
the number of LSTM layers is one;
embedding size, LSTM layer size and attention layer size is 100;
models are trained for 20 epochs;
on every epoch, training data is subsampled at a rate of 0 3;
LSTM dropout is applied at a rate 0 3;
context word forms are randomly dropped at a rate of 0 1;
the Adam optimiser is used, with a default learning rate of 0 001; and
trained models are evaluated on the development data (the data for the shared task comes already split in train and dev sets) 
Our system
Here we compare and contrast our system to the baseline system  A diagram of our system is shown in Figure FIGREF4  
The idea behind this modification is to provide the encoder with access to all morpho-syntactic cues present in the sentence  In contrast to the baseline, which only encodes the immediately adjacent context of a target word, we encode the entire context  All context word forms, lemmas, and MSD tags (in Track 1) are embedded in their respective high-dimensional spaces as before, and their embeddings are concatenated  However, we now reduce the entire past context to a fixed-size vector by encoding it with a forward LSTM, and we similarly represent the future context by encoding it with a backwards LSTM 
We introduce an auxiliary objective that is meant to increase the morpho-syntactic awareness of the encoder and to regularise the learning process—the task is to predict the MSD tag of the target form  MSD tag predictions are conditioned on the context encoding, as described in UID15   Tags are generated with an LSTM one component at a time, e g  the tag PRO;NOM;SG;1 is predicted as a sequence of four components, INLINEFORM0 PRO, NOM, SG, 1 INLINEFORM1  
For every training instance, we backpropagate the sum of the main loss and the auxiliary loss without any weighting 
As MSD tags are only available in Track 1, this augmentation only applies to this track 
The parameters of the entire MSD (auxiliary-task) decoder are shared across languages 
Since a grouping of the languages based on language family would have left several languages in single-member groups (e g  Russian is the sole representative of the Slavic family), we experiment with random groupings of two to three languages  Multilingual training is performed by randomly alternating between languages for every new minibatch  We do not pass any information to the auxiliary decoder as to the source language of the signal it is receiving, as we assume abstract morpho-syntactic features are shared across languages 
After 20 epochs of multilingual training, we perform 5 epochs of monolingual finetuning for each language  For this phase, we reduce the learning rate to a tenth of the original learning rate, i e  0 0001, to ensure that the models are indeed being finetuned rather than retrained 
We keep all hyperparameters the same as in the baseline  Training data is split 90:10 for training and validation  We train our models for 50 epochs, adding early stopping with a tolerance of five epochs of no improvement in the validation loss  We do not subsample from the training data 
We train models for 50 different random combinations of two to three languages in Track 1, and 50 monolingual models for each language in Track 2  Instead of picking the single model that performs best on the development set and thus risking to select a model that highly overfits that data, we use an ensemble of the five best models, and make the final prediction for a given target form with a majority vote over the five predictions 
Results and Discussion
Test results are listed in Table TABREF17   Our system outperforms the baseline for all settings and languages in Track 1 and for almost all in Track 2—only in the high resource setting is our system not definitively superior to the baseline 
Interestingly, our results in the low resource setting are often higher for Track 2 than for Track 1, even though contextual information is less explicit in the Track 2 data and the multilingual multi-tasking approach does not apply to this track  We interpret this finding as an indicator that a simpler model with fewer parameters works better in a setting of limited training data  Nevertheless, we focus on the low resource setting in the analysis below due to time limitations  As our Track 1 results are still substantially higher than the baseline results, we consider this analysis valid and insightful 
Ablation Study
We analyse the incremental effect of the different features in our system, focusing on the low-resource setting in Track 1 and using development data 
Encoding the entire context with an LSTM highly increases the variance of the observed results  So we trained fifty models for each language and each architecture  Figure FIGREF23 visualises the means and standard deviations over the trained models  In addition, we visualise the average accuracy for the five best models for each language and architecture, as these are the models we use in the final ensemble prediction  Below we refer to these numbers only 
The results indicate that encoding the full context with an LSTM highly enhances the performance of the model, by 11 15% on average  This observation explains the high results we obtain also for Track 2 
Adding the auxiliary objective of MSD prediction has a variable effect: for four languages (de, en, es, and sv) the effect is positive, while for the rest it is negative  We consider this to be an issue of insufficient data for the training of the auxiliary component in the low resource setting we are working with 
We indeed see results improving drastically with the introduction of multilingual training, with multilingual results being 7 96% higher than monolingual ones on average 
We studied the five best models for each language as emerging from the multilingual training (listed in Table TABREF27 ) and found no strong linguistic patterns  The en–sv pairing seems to yield good models for these languages, which could be explained in terms of their common language family and similar morphology  The other natural pairings, however, fr–es, and de–sv, are not so frequent among the best models for these pairs of languages 
Finally, monolingual finetuning improves accuracy across the board, as one would expect, by 2 72% on average 
The final observation to be made based on this breakdown of results is that the multi-tasking approach paired with multilingual training and subsequent monolingual finetuning outperforms the other architectures for five out of seven languages: de, en, fr, ru and sv  For the other two languages in the dataset, es and fi, the difference between this approach and the approach that emerged as best for them is less than 1%  The overall improvement of the multilingual multi-tasking approach over the baseline is 18 30% 
Error analysis
Here we study the errors produced by our system on the English test set to better understand the remaining shortcomings of the approach  A small portion of the wrong predictions point to an incorrect interpretation of the morpho-syntactic conditioning of the context, e g  the system predicted plan instead of plans in the context Our _ include raising private capital  The majority of wrong predictions, however, are nonsensical, like bomb for job, fify for fixing, and gnderrate for understand  This observation suggests that generally the system did not learn to copy the characters of lemma into inflected form, which is all it needs to do in a large number of cases  This issue could be alleviated with simple data augmentation techniques that encourage autoencoding BIBREF2  
MSD prediction
Figure FIGREF32 summarises the average MSD-prediction accuracy for the multi-tasking experiments discussed above  Accuracy here is generally higher than on the main task, with the multilingual finetuned setup for Spanish and the monolingual setup for French scoring best: 66 59% and 65 35%, respectively  This observation illustrates the added difficulty of generating the correct surface form even when the morphosyntactic description has been identified correctly 
We observe some correlation between these numbers and accuracy on the main task: for de, en, ru and sv, the brown, pink and blue bars here pattern in the same way as the corresponding INLINEFORM0 's in Figure FIGREF23   One notable exception to this pattern is fr where inflection gains a lot from multilingual training, while MSD prediction suffers greatly  Notice that the magnitude of change is not always the same, however, even when the general direction matches: for ru, for example, multilingual training benefits inflection much more than in benefits MSD prediction, even though the MSD decoder is the only component that is actually shared between languages  This observation illustrates the two-fold effect of multi-task training: an auxiliary task can either inform the main task through the parameters the two tasks share, or it can help the main task learning through its regularising effect 
Related Work
Our system is inspired by previous work on multi-task learning and multi-lingual learning, mainly building on two intuitions: (1) jointly learning related tasks tends to be beneficial BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 ; and (2) jointly learning related languages in an MTL-inspired framework tends to be beneficial BIBREF8 , BIBREF9 , BIBREF10   In the context of computational morphology, multi-lingual approaches have previously been employed for morphological reinflection BIBREF2 and for paradigm completion BIBREF11   In both of these cases, however, the available datasets covered more languages, 40 and 21, respectively, which allowed for linguistically-motivated language groupings and for parameter sharing directly on the level of characters  BIBREF10 explore parameter sharing between related languages for dependency parsing, and find that sharing is more beneficial in the case of closely related languages 
Conclusions
In this paper we described our system for the CoNLL–SIGMORPHON 2018 shared task on Universal Morphological Reinflection, Task 2, which achieved the best performance out of all systems submitted, an overall accuracy of 49 87  We showed in an ablation study that this is due to three core innovations, which extend a character-based encoder-decoder model: (1) a wide context window, encoding the entire available context; (2) multi-task learning with the auxiliary task of MSD prediction, which acts as a regulariser; (3) a multilingual approach, exploiting information across languages  In future work we aim to gain better understanding of the increase in variance of the results introduced by each of our modifications and the reasons for the varying effect of multi-task learning for different languages 
Acknowledgements
We gratefully acknowledge the support of the NVIDIA Corporation with the donation of the Titan Xp GPU used for this research 
Context 3: Introduction
Microblogging such as Twitter and Weibo is a popular social networking service, which allows users to post messages up to 140 characters  There are millions of active users on the platform who stay connected with friends  Unfortunately, spammers also use it as a tool to post malicious links, send unsolicited messages to legitimate users, etc  A certain amount of spammers could sway the public opinion and cause distrust of the social platform  Despite the use of rigid anti-spam rules, human-like spammers whose homepages having photos, detailed profiles etc  have emerged  Unlike previous "simple" spammers, whose tweets contain only malicious links, those "smart" spammers are more difficult to distinguish from legitimate users via content-based features alone BIBREF0  
There is a considerable amount of previous work on spammer detection on social platforms  Researcher from Twitter Inc  BIBREF1 collect bot accounts and perform analysis on the user behavior and user profile features  Lee et al  lee2011seven use the so-called social honeypot by alluring social spammers' retweet to build a benchmark dataset, which has been extensively explored in our paper  Some researchers focus on the clustering of urls in tweets and network graph of social spammers BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , showing the power of social relationship features As for content information modeling, BIBREF6 apply improved sparse learning methods  However, few studies have adopted topic-based features  Some researchers BIBREF7 discuss topic characteristics of spamming posts, indicating that spammers are highly likely to dwell on some certain topics such as promotion  But this may not be applicable to the current scenario of smart spammers 
In this paper, we propose an efficient feature extraction method  In this method, two new topic-based features are extracted and used to discriminate human-like spammers from legitimate users  We consider the historical tweets of each user as a document and use the Latent Dirichlet Allocation (LDA) model to compute the topic distribution for each user  Based on the calculated topic probability, two topic-based features, the Local Outlier Standard Score (LOSS) which captures the user's interests on different topics and the Global Outlier Standard Score (GOSS) which reveals the user's interests on specific topic in comparison with other users', are extracted  The two features contain both local and global information, and the combination of them can distinguish human-like spammers effectively 
To the best of our knowledge, it is the first time that features based on topic distributions are used in spammer classification  Experimental results on one public dataset and one self-collected dataset further validate that the two sets of extracted topic-based features get excellent performance on human-like spammer classification problem compared with other state-of-the-art methods  In addition, we build a Weibo dataset, which contains both legitimate users and spammers 
To summarize, our major contributions are two-fold:
In the following sections, we first propose the topic-based features extraction method in Section 2, and then introduce the two datasets in Section 3  Experimental results are discussed in Section 4, and we conclude the paper in Section 5  Future work is presented in Section 6 
Methodology
In this section, we first provide some observations we obtained after carefully exploring the social network, then the LDA model is introduced  Based on the LDA model, the ways to obtain the topic probability vector for each user and the two topic-based features are provided 
Observation
After exploring the homepages of a substantial number of spammers, we have two observations  1) social spammers can be divided into two categories  One is content polluters, and their tweets are all about certain kinds of advertisement and campaign  The other is fake accounts, and their tweets resemble legitimate users' but it seems they are simply random copies of others to avoid being detected by anti-spam rules  2) For legitimate users, content polluters and fake accounts, they show different patterns on topics which interest them 
Legitimate users mainly focus on limited topics which interest him  They seldom post contents unrelated to their concern 
Content polluters concentrate on certain topics 
Fake accounts focus on a wide range of topics due to random copying and retweeting of other users' tweets 
Spammers and legitimate users show different interests on some topics e g  commercial, weather, etc 
To better illustrate our observation, Figure  1 shows the topic distribution of spammers and legitimate users in two employed datasets(the Honeypot dataset and Weibo dataset)  We can see that on both topics (topic-3 and topic-11) there exists obvious difference between the red bars and green bars, representing spammers and legitimate users  On the Honeypot dataset, spammers have a narrower shape of distribution (the outliers on the red bar tail are not counted) than that of legitimate users  This is because there are more content polluters than fake accounts  In other word, spammers in this dataset tend to concentrate on limited topics  While on the Weibo dataset, fake accounts who are interested in different topics take large proportion of spammers  Their distribution is more flat (i e  red bars) than that of the legitimate users  Therefore we can detect spammers by means of the difference of their topic distribution patterns 
LDA model
Blei et al blei2003latent first presented Latent Dirichlet Allocation(LDA) as an example of topic model 
Each document $i$ is deemed as a bag of words $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $ and $M$ is the number of words

ground_truth: LSTM

LSTM

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How big is imbalance in analyzed corpora?
Context 2: Introduction
In recent years, gender has become a hot topic within the political, societal and research spheres  Numerous studies have been conducted in order to evaluate the presence of women in media, often revealing their under-representation, such as the Global Media Monitoring Project BIBREF0  In the French context, the CSA BIBREF1 produces a report on gender representation in media on a yearly basis  The 2017 report shows that women represent 40% of French media speakers, with a significant drop during high-audience hours (6:00-8:00pm) reaching a value of only 29%  Another large scale study confirmed this trend with an automatic analysis of gender in French audiovisuals streams, highlighting a huge variation across type of shows BIBREF2 
Besides the social impact of gender representation, broadcast recordings are also a valuable source of data for the speech processing community  Indeed, automatic speech recognition (ASR) systems require large amount of annotated speech data to be efficiently trained, which leaves us facing the emerging concern about the fact that "AI artifacts tend to reflect the goals, knowledge and experience of their creators" BIBREF3  Since we know that women are under-represented in media and that the AI discipline has retained a male-oriented focus BIBREF4, we can legitimately wonder about the impact of using such data as a training set for ASR technologies  This concern is strengthened by the recent works uncovering gender bias in several natural language processing (NLP) tools such as BIBREF5, BIBREF6, BIBREF7, BIBREF8 
In this paper, we first highlight the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have  We then perform a statistical analysis of gender representation in a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community  Finally we question the impact of such a representation on the systems developed on this data, through the perspective of an ASR system 
From gender representation in data to gender bias in AI ::: On the importance of data
The ever growing use of machine learning in science has been enabled by several progresses among which the exponential growth of data available  The quality of a system now depends mostly on the quality and quantity of the data it has been trained on  If it does not discard the importance of an appropriate architecture, it reaffirms the fact that rich and large corpora are a valuable resource  Corpora are research contributions which do not only allow to save and observe certain phenomena or validate a hypothesis or model, but are also a mandatory part of the technology development  This trend is notably observable within the NLP field, where industrial technologies, such as Apple, Amazon or Google vocal assistants now reach high performance level partly due to the amount of data possessed by these companies BIBREF9 
Surprisingly, as data is said to be “the new oil", few data sets are available for ASR systems  The best known are corpora like TIMIT BIBREF10, Switchboard BIBREF11 or Fisher BIBREF12 which date back to the early 1990s  The scarceness of available corpora is justified by the fact that gathering and annotating audio data is costly both in terms of money and time  Telephone conversations and broadcast recordings have been the primary source of spontaneous speech used  Out of all the 130 audio resources proposed by LDC to train automatic speech recognition systems in English, approximately 14% of them are based on broadcast news and conversation  For French speech technologies, four corpora containing radio and TV broadcast are the most widely used: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16  These four corpora have been built alongside evaluation campaigns and are still, to our knowledge, the largest French ones of their type available to date 
From gender representation in data to gender bias in AI ::: From data to bias
The gender issue has returned to the forefront of the media scene in recent years and with the emergence of AI technologies in our daily lives, gender bias has become a scientific topic that researchers are just beginning to address  Several studies revealed the existence of gender bias in AI technologies such as face recognition (GenderShades BIBREF17), NLP (word embeddings BIBREF5 and semantics BIBREF6) and machine translation (BIBREF18, BIBREF7)  The impact of the training data used within these deep-learning algorithms is therefore questioned 
Bias can be found at different levels as pointed out by BIBREF19  BIBREF20 defines bias as a skew that produces a type of harm  She distinguishes two types of harms that are allocation harm and representation harm  The allocation harm occurs when a system is performing better or worse for a certain group while representational harm contributes to the perpetuation of stereotypes  Both types of harm are the results of bias in machine learning that often comes from the data systems are trained on  Disparities in representation in our social structures is captured and reflected by the training data, through statistical patterns  The GenderShades study is a striking example of what data disparity and lack of representation can produce: the authors tested several gender recognition modules used by facial recognition tools and found difference in error-rate as high as 34 percentage points between recognition of white male and black female faces  The scarce presence of women and colored people in training set resulted in bias in performance towards these two categories, with a strong intersectional bias  As written by BIBREF21 "A data set may have many millions of pieces of data, but this does not mean it is random or representative  To make statistical claims about a data set, we need to know where data is coming from; it is similarly important to know and account for the weaknesses in that data " (p 668) 
Regarding ASR technology, little work has explored the presence of gender bias within the systems and no consensus has been reached  BIBREF22 found that speech recognizers perform better on female voice on a broadcast news and telephone corpus  They proposed several explanations to this observation, such as the larger presence of non-professional male speech in the broadcast data, implying a less prepared speech for these speakers or a more normative language and standard pronunciation for women linked to the traditional role of women in language acquisition and education  The same trend was observed by BIBREF23  More recently, BIBREF24 discovered a gender bias within YouTube's automatic captioning system but this bias was not observed in a second study evaluating Bing Speech system and YouTube Automatic Captions on a larger data set BIBREF8  However race and dialect bias were found  General American speakers and white speakers had the lowest error rate for both systems  If the better performance on General American speakers could be explained by the fact that they are all voice professionals, producing clear and articulated speech, but no explanation is provided for biases towards non-white speakers 
Gender bias in ASR technology is still an open research question as no clear answer has been reached so far  It seems that many parameters are to take into account to achieve a general agreement  As we established the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have, the following content of this paper is structured as this: we first describe statistically the gender representation of a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community, introducing the notion of speaker's role to refine our analysis in terms of voice professionalism  We then question the impact of such a representation on a ASR system trained on these data  BIBREF25
Methodology
This section is organized as follows: we first present the data we are working on  In a second time we explain how we proceed to describe the gender representation in our corpus and introduce the notion of speaker's role  The third subsection introduces the ASR system and metrics used to evaluate gender bias in performance 
Methodology ::: Data presentation
Our data consists of two sets used to train and evaluate our automatic speech recognition system  Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16  These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR  Show duration varies between 10min and an hour  As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed  ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition 
Our training set contains 27,085 speech utterances produced by 2,506 speakers, accounting for approximately 100 hours of speech  Our evaluation set contains 74,064 speech utterances produced by 1,268 speakers for a total of 70 hours of speech  Training data by show, medium and speech type is summarized in Table and evaluation data in Table   Evaluation data has a higher variety of shows with both prepared (P) and spontaneous (S) speech type (accented speech from African radio broadcast is also included in the evaluation set) 
Methodology ::: Methodology for descriptive analysis of gender representation in training data
We first describe the gender representation in training data  Gender representation is measured in terms of number of speakers, number of utterances (or speech turns), and turn lengths (descriptive statistics are given in Section SECREF16)  Each speech turn was mapped to its speaker in order to associate it with a gender 
As pointed out by the CSA report BIBREF1, women presence tends to be marginal within the high-audience hours, showing that women are represented but less than men and within certain given conditions  It is clear that a small number of speakers is responsible for a large number of speech turns  Most of these speakers are journalists, politicians, presenters and such, who are representative of a show  Therefore, we introduce the notion of speaker's role to refine our exploration of gender disparity, following studies which quantified women's presence in terms of role  Within our work, we define the notion of speaker role by two criteria specifying the speaker's on-air presence, namely the number of speech turns and the cumulative duration of his or her speaking time in a show  Based on the available speech transcriptions and meta-data, we compute for each speaker the number of speech turns uttered as well as their total length  We then use the following criteria to define speaker's role: a speaker is considered as speaking often (respectively seldom) if he/she accumulates a total of turns higher (respectively lower) than 1% of the total number of speech turns in a given show  The same process is applied to identify speakers talking for a long period from those who do not  We end up with two salient roles called Anchors and Punctual speakers:
the Anchor speakers (A) are above the threshold of 1% for both criteria, meaning they are intervening often and for a long time thus holding an important place in interaction;
the Punctual speakers (PS) on the contrary are below the threshold of 1% for both the total number of turns and the total speech time 
These roles are defined at the show level  They could be roughly assimilated to the categorization “host/guest” in radio and TV shows  Anchors could be described as professional speakers, producing mostly prepared speech, whereas Punctual speakers are more likely to be “everyday people"  The concept of speaker's role makes sense at both sociological and technical levels  An Anchor speaker is more likely to be known from the audience (society), but he or she will also likely have a professional (clear) way of speaking (as mentioned by BIBREF22 and BIBREF8), as well as a high number of utterances, augmenting the amount of data available for a given gender category 
Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: ASR system
The ASR system used in this work is described in BIBREF25  It uses the KALDI toolkit BIBREF26, following a standard Kaldi recipe  The acoustic model is based on a hybrid HMM-DNN architecture and trained on the data summarized in Table   Acoustic training data correspond to 100h of non-spontaneous speech type (mostly broadcast news) coming from both radio and TV shows  A 5-gram language model is trained from several French corpora (3,323M words in total) using SRILM toolkit BIBREF27  The pronunciation model is developed using the lexical resource BDLEX BIBREF28 as well as automatic grapheme-to-phoneme (G2P) transcription to find pronunciation variants of our vocabulary (limited to 80K)  It is important to re-specify here, for further analysis, that our Kaldi pipeline follows speaker adaptive training (SAT) where we train and decode using speaker adapted features (fMLLR-adapted features) in per-speaker mode  It is well known that speaker adaptation acts as an effective procedure to reduce mismatch between training and evaluation conditions BIBREF29, BIBREF26 
Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: Evaluation
Word Error Rate (WER) is a common metric to evaluate ASR performance  It is measured as the sum of errors (insertions, deletions and substitutions) divided by the total number of words in the reference transcription  As we are investigating the impact on performance of speaker's gender and role, we computed the WER for each speaker at the episode (show occurrence) level  Analyzing at such granularity allows us to avoid large WER variation that could be observed at utterance level (especially for short speech turns) but also makes possible to get several WER values for a given speaker, one for each occurrence of a show in which he/she appears on  Speaker's gender was provided by the meta-data and role was obtained using the criteria from Section SECREF6 computed for each show  This enables us to analyze our results across gender and role categories which was done using Wilcoxon rank sum tests also called Mann-Whitney U test (with $\alpha $= 0 001) BIBREF30  The choice of a Wilcoxon rank sum test and not the commonly used t-test is motivated by the non-normality of our data 
Results ::: Descriptive analysis of gender representation in training data ::: Gender representation
As expected, we observe a disparity in terms of gender representation in our data (see Table )  Women represent 33 16% of the speakers, confirming the figures given by the GMMP report BIBREF0  However, it is worth noticing that women account for only 22 57% of the total speech time, which leads us to conclude that women also speak less than men 
Results ::: Descriptive analysis of gender representation in training data ::: Speaker's role representation
Table presents roles' representation in training data and shows that despite the small number of Anchor speakers in our data (3 79%), they nevertheless concentrate 35 71 % of the total speech time 
Results ::: Descriptive analysis of gender representation in training data ::: Role and gender interaction
When crossing both parameters, we can observe that the gender distribution is not constant throughout roles  Women represent 29 47% of the speakers within the Anchor category, even less than among the Punctual speakers  Their percentage of speech is also smaller  When calculating the average speech time uttered by a female Anchor, we obtain a value of 15 9 min against 25 2 min for a male Anchor, which suggests that even within the Anchor category men tend to speak more  This confirms the existence of gender disparities within French media  It corroborates with the analysis of the CSA BIBREF1, which shows that women were less present during high-audience hours  Our study shows that they are also less present in important roles  These results legitimate our initial questioning on the impact of gender balance on ASR performance trained on broadcast recordings 
Results ::: Performance (WER) analysis on evaluation data ::: Impact of gender on WER
As explained in Section SECREF13, WER is the sum of errors divided by the number of words in the transcription reference  The higher the WER, the poorer the system performance  Our 70h evaluation data contains a large amount of spontaneous speech and is very challenging for the ASR system trained on prepared speech: we observe an overall average WER of 42 9% for women and 34 3% for men  This difference of WER between men and women is statistically significant (med(M) = 25%; med(F) = 29%; U = 709040; p-value < 0 001) 
However, when observing gender differences across shows, no clear trend can be identified, as shown in Figure FIGREF21  For shows like Africa1 Infos or La Place du Village, we find an average WER lower for women than for men, while the trend is reversed for shows such as Un Temps de Pauchon or Le Masque et la Plume  The disparity of the results depending on the show leads us to believe that other factors may be entangled within the observed phenomenon 
Results ::: Performance (WER) analysis on evaluation data ::: Impact of role on WER
Speaker's role seems to have an impact on WER: we obtain an average WER of 30 8% for the Anchor speakers and 42 23% for the Punctual speakers  This difference is statistically significant with a p-value smaller than $10^{-14}$ (med(A) = 21%; med(P) = 31%; U = 540,430; p-value < 0 001)  
Results ::: Performance (WER) analysis on evaluation data ::: Role and gender interaction
Figure FIGREF25 presents the WER distribution (WER being obtained for each speaker in a show occurrence) according to the speaker's role and gender  It is worth noticing that the gender difference is only significant within the Punctual speakers group  The average WER is of 49 04% for the women and 38 56% for the men with a p-value smaller than $10^{-6}$ (med(F) = 39%; med(M) = 29%; U = 251,450; p-value < 0 001), whereas it is just a trend between male and female Anchors (med(F) = 21%; med(M) = 21%; U = 116,230; p-value = 0 173)  This could be explained by the quantity of data available per speaker 
Results ::: Performance (WER) analysis on evaluation data ::: Speech type as a third entangled factor?
In order to try to explain the observed variation in our results depending on shows and gender (Figure FIGREF21), we add the notion of speech type to shed some light on our results  BIBREF22 and BIBREF24 suggested that the speaker professionalism, associated with clear and hyper-articulated speech could be an explaining factor for better performance 
Based on our categorization in prepared speech (mostly news reports) and spontaneous speech (mostly debates and entertainment shows), we cross this parameter in our performance analysis  As shown on Figure FIGREF26, these results confirm the inherent challenge of spontaneous speech compared to prepared speech  WER scores are similar between men and women when considering prepared speech (med(F) = 18%; med(M) = 21%; U = 217,160; p-value = 0 005) whereas they are worse for women (61 29%) than for men (46 51%) with p-value smaller than $10^{-14}$ for the spontaneous speech type (med(F) = 61%; med(M) = 37%; U = 153,580; p-value < 0 001) 
Discussion
We find a clear disparity in terms of women presence and speech quantity in French media  Our data being recorded between 1998 and 2013, we can expect this disparity to be smaller on more recent broadcast recordings, especially since the French government displays efforts toward parity in media representation  One can also argue that even if our analysis was conducted on a large amount of data it does not reach the exhaustiveness of large-scale studies such as the one of BIBREF2  Nonetheless it does not affect the relevance of our findings, because if real-world gender representation might be more balanced today, these corpora are still used as training data for AI systems 
The performance difference across gender we observed corroborates (on a larger quantity and variety of language data produced by more than 2400 speakers) the results obtained by BIBREF24 on isolated words recognition  However the following study on read speech does not replicate these results  Yet a performance degradation is observed across dialect and race BIBREF8  BIBREF22 found lower WER for women than men on broadcast news and conversational telephone speech for both English and French  The authors suggest that gender stereotypes associated with women role in education and language acquisition induce a more normative elocution  We observed that the higher the degree of normativity of speech the smaller the gender difference  No significant gender bias is observed for prepared speech nor within the Anchor category  Even if we do not find similar results with lower WER for women than men, we obtained a median WER smaller for women on prepared speech and equal to the male median WER for the Anchor speakers 
Another explanation could be the use of adaptation within the pipeline  Most broadcast programs transcription systems have a speaker adaptation step within their decoding pipeline, which is the case for our system  An Anchor speaker intervening more often would have a larger quantity of data to realize such adaptation of the acoustic model  On the contrary, Punctual speakers who appear scarcely in the data are not provided with the same amount of adaptation data  Hence we can hypothesize that gender performance difference observed for Punctual speakers is due to the fact that female speech is further from the (initial non-adapted) acoustic model as it was trained on unbalanced data (as shown in Table )  Considering that Punctual speakers represent 92 78% of the speakers, this explains why gender difference is significant over our entire data set  A way to confirm our hypothesis would be to reproduce our analysis on WER values obtained without using speaker adapted features at the decoding step 
When decoding prepared speech (hence similar to the training data), no significant difference is found in WER between men and women, revealing that the speaker adaptation step could be sufficient to reach same performance for both genders  But when decoding more spontaneous speech, there is a mismatch with the initial acoustic model (trained on prepared speech)  Consequently, the speaker adaptation step might not be enough to recover good ASR performance, especially for women for whom less adaptation data is available (see Section 4 2 3) 
Conclusion
This paper has investigated gender bias in ASR performance through the following research questions: i) what is the proportion of men and women in French radio and TV media data ? ii) what is the impact of the observed disparity on ASR performance ? iii) is this as simple as a problem of gender proportion in the training data or are other factors entangled ? Our contributions are the following:
Descriptive analysis of the broadcast data used to train our ASR system confirms the already known disparity, where 65% of the speakers are men, speaking more than 75% of the time 
When investigating WER scores according to gender, speaker's role and speech type, huge variations are observed  We conclude that gender is clearly a factor of variation in ASR performance, with a WER increase of 24% for women compared to men, exhibiting a clear gender bias 
Gender bias varies across speaker's role and speech spontaneity level  Performance for Punctual speakers respectively spontaneous speech seems to reinforce this gender bias with a WER increase of 27 2% respectively 31 8% between male and female speakers 
We found that an ASR system trained on unbalanced data regarding gender produces gender bias performance  Therefore, in order to create fair systems it is necessary to take into account the representation problems in society that are going to be encapsulated in the data  Understanding how women under-representation in broadcast data can lead to bias in ASR performances is the key to prevent re-implementing and reinforcing discrimination already existing in our societies  This is in line with the concept of “Fairness by Design" proposed by BIBREF31 
Gender, race, religion, nationality are all characteristics that we deem unfair to classify on, and these ethical standpoints needs to be taken into account in systems' design  Characteristics that are not considered as relevant in a given task can be encapsulated in data nonetheless, and lead to bias performance  Being aware of the demographic skews our data set might contain is a first step to track the life cycle of a training data set and a necessary step to control the tools we develop 
Context 3:  Each word is attributable to one of the document's topics $Z=\left\lbrace  z_{i1},z_{i2},   ,z_{iK}\right\rbrace $ and $K$ is the number of topics  $\psi _{k}$ is a multinomial distribution over words for topic $k$   $\theta _i$ is another multinomial distribution over topics for document $i$   The smoothed generative model is illustrated in Figure  2   $\alpha $ and $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $0 are hyper parameter that affect scarcity of the document-topic and topic-word distributions  In this paper, $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $1 , $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $2 and $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $3 are empirically set to 0 3, 0 01 and 15  The entire content of each Twitter user is regarded as one document  We adopt Gibbs Sampling BIBREF8 to speed up the inference of LDA  Based on LDA, we can get the topic probabilities for all users in the employed dataset as: $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $4 , where $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $5 is the number of users  Each element $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $6 is a topic probability vector for the $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $7 document  $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $8 is the raw topic probability vector and our features are developed on top of it 
Topic-based Features
Using the LDA model, each person in the dataset is with a topic probability vector $X_i$   Assume $x_{ik}\in X_{i}$ denotes the likelihood that the $\emph {i}^{th}$ tweet account favors $\emph {k}^{th}$ topic in the dataset  Our topic based features can be calculated as below 
Global Outlier Standard Score measures the degree that a user's tweet content is related to a certain topic compared to the other users  Specifically, the "GOSS" score of user $i$ on topic $k$ can be calculated as Eq ( 12 ):
$$\centering \begin{array}{ll} \mu \left(x_{k}\right)=\frac{\sum _{i=1}^{n} x_{ik}}{n},\\ GOSS\left(x_{ik}\right)=\frac{x_{ik}-\mu \left(x_k\right)}{\sqrt{\underset{i}{\sum }\left(x_{ik}-\mu \left(x_{k}\right)\right)^{2}}}  \end{array}$$   (Eq  12)
The value of $GOSS\left(x_{ik}\right)$ indicates the interesting degree of this person to the $\emph {k}^{th}$ topic  Specifically, if $GOSS\left(x_{ik}\right)$ > $GOSS\left(x_{jk}\right)$ , it means that the $\emph {i}^{th}$ person has more interest in topic $k$ than the $\emph {j}^{th}$ person  If the value $GOSS\left(x_{ik}\right)$ is extremely high or low, the $\emph {i}^{th}$ person showing extreme interest or no interest on topic $k$ which will probably be a distinctive pattern in the fowllowing classfication  Therefore, the topics interested or disliked by the $\emph {k}^{th}$0 person can be manifested by $\emph {k}^{th}$1 , from which the pattern of the interested topics with regarding to this person is found  Denote $\emph {k}^{th}$2 our first topic-based feature, and it hopefully can get good performance on spammer detection 
Local Outlier Standard Score measures the degree of interest someone shows to a certain topic by considering his own homepage content only  For instance, the "LOSS" score of account $i$ on topic $k$ can be calculated as Eq ( 13 ):
$$\centering \begin{array}{ll} \mu \left(x_{i}\right)=\frac{\sum _{k=1}^{K} x_{ik}}{K},\\ LOSS\left(x_{ik}\right)=\frac{x_{ik}-\mu \left(x_i\right)}{\sqrt{\underset{k}{\sum }\left(x_{ik}-\mu \left(x_{i}\right)\right)^{2}}}

ground_truth: Women represent 33.16% of the speakers

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the benchmark dataset and is its quality high?
Context 3:  \end{array}$$   (Eq  13)
$\mu (x_i)$ represents the averaged interesting degree for all topics with regarding to $\emph {i}^{th}$ user and his tweet content  Similarly to $GOSS$ , the topics interested or disliked by the $\emph {i}^{th}$ person via considering his single post information can be manifested by $f_{LOSS}^{i}=[LOSS(x_{i1})\cdots LOSS(x_{iK})]$ , and $LOSS$ becomes our second topic-based features for the $\emph {i}^{th}$ person 
Dataset
We use one public dataset Social Honeypot dataset and one self-collected dataset Weibo dataset to validate the effectiveness of our proposed features 
Social Honeypot Dataset: Lee et al  lee2010devils created and deployed 60 seed social accounts on Twitter to attract spammers by reporting back what accounts interact with them  They collected 19,276 legitimate users and 22,223 spammers in their datasets along with their tweet content in 7 months  This is our first test dataset 
Our Weibo Dataset: Sina Weibo is one of the most famous social platforms in China  It has implemented many features from Twitter  The 2197 legitimate user accounts in this dataset are provided by the Tianchi Competition held by Sina Weibo  The spammers are all purchased commercially from multiple vendors on the Internet  We checked them manually and collected 802 suitable "smart" spammers accounts 
Preprocessing: Before directly performing the experiments on the employed datasets, we first delete some accounts with few posts in the two employed since the number of tweets is highly indicative of spammers  For the English Honeypot dataset, we remove stopwords, punctuations, non-ASCII words and apply stemming  For the Chinese Weibo dataset, we perform segmentation with "Jieba", a Chinese text segmentation tool  After preprocessing steps, the Weibo dataset contains 2197 legitimate users and 802 spammers, and the honeypot dataset contains 2218 legitimate users and 2947 spammers  It is worth mentioning that the Honeypot dataset has been slashed because most of the Twitter accounts only have limited number of posts, which are not enough to show their interest inclination 
Evaluation Metrics
The evaluating indicators in our model are show in 2   We calculate precision, recall and F1-score (i e  F1 score) as in Eq  ( 19 )  Precision is the ratio of selected accounts that are spammers  Recall is the ratio of spammers that are detected so  F1-score is the harmonic mean of precision and recall 
$$precision =\frac{TP}{TP+FP}, recall =\frac{TP}{TP+FN}\nonumber \\ F1-score = \frac{2\times precision \times recall}{precision + recall}$$   (Eq  19)
Performance Comparisons with Baseline
Three baseline classification methods: Support Vector Machines (SVM), Adaboost, and Random Forests are adopted to evaluate our extracted features  We test each classification algorithm with scikit-learn BIBREF9 and run a 10-fold cross validation  On each dataset, the employed classifiers are trained with individual feature first, and then with the combination of the two features  From 1 , we can see that GOSS+LOSS achieves the best performance on F1-score among all others  Besides, the classification by combination of LOSS and GOSS can increase accuracy by more than 3% compared with raw topic distribution probability 
Comparison with Other Features
To compare our extracted features with previously used features for spammer detection, we use three most discriminative feature sets according to Lee et al  lee2011seven( 4 )  Two classifiers (Adaboost and SVM) are selected to conduct feature performance comparisons  Using Adaboost, our LOSS+GOSS features outperform all other features except for UFN which is 2% higher than ours with regard to precision on the Honeypot dataset  It is caused by the incorrectly classified spammers who are mostly news source after our manual check  They keep posting all kinds of news pieces covering diverse topics, which is similar to the behavior of fake accounts  However, UFN based on friendship networks is more useful for public accounts who possess large number of followers  The best recall value of our LOSS+GOSS features using SVM is up to 6% higher than the results by other feature groups  Regarding F1-score, our features outperform all other features  To further show the advantages of our proposed features, we compare our combined LOSS+GOSS with the combination of all the features from Lee et al  lee2011seven (UFN+UC+UH)  It's obvious that LOSS+GOSS have a great advantage over UFN+UC+UH in terms of recall and F1-score  Moreover, by combining our LOSS+GOSS features and UFN+UC+UH features together, we obtained another 7 1% and 2 3% performance gain with regard to precision and F1-score by Adaboost  Though there is a slight decline in terms of recall  By SVM, we get comparative results on recall and F1-score but about 3 5% improvement on precision 
Conclusion
In this paper, we propose a novel feature extraction method to effectively detect "smart" spammers who post seemingly legitimate tweets and are thus difficult to identify by existing spammer classification methods  Using the LDA model, we obtain the topic probability for each Twitter user  By utilizing the topic probability result, we extract our two topic-based features: GOSS and LOSS which represent the account with global and local information  Experimental results on a public dataset and a self-built Chinese microblog dataset validate the effectiveness of the proposed features 
Future Work
In future work, the combination method of local and global information can be further improved to maximize their individual strengths  We will also apply decision theory to enhancing the performance of our proposed features  Moreover, we are also building larger datasets on both Twitter and Weibo to validate our method  Moreover, larger datasets on both Twitter and Weibo will be built to further validate our method 

ground_truth: Social Honeypot dataset (public) and Weibo dataset (self-collected); yes

Social Honeypot, which is not of high quality