prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What metrics are used for evaluation?
Context 1: Introduction
Pre-trained models BIBREF0, BIBREF1 have received much of attention recently thanks to their impressive results in many down stream NLP tasks  Additionally, multilingual pre-trained models enable many NLP applications for other languages via zero-short cross-lingual transfer  Zero-shot cross-lingual transfer has shown promising results for rapidly building applications for low resource languages  BIBREF2 show the potential of multilingual-BERT BIBREF0 in zero-shot transfer for a large number of languages from different language families on five NLP tasks, namely, natural language inference, document classification, named entity recognition, part-of-speech tagging, and dependency parsing 
Although multilingual models are an important ingredient for enhancing language technology in many languages, recent research on improving pre-trained models puts much emphasis on English BIBREF3, BIBREF4, BIBREF5  The current state of affairs makes it difficult to translate advancements in pre-training from English to non-English languages  To our best knowledge, there are only three available multilingual pre-trained models to date: (1) the multilingual-BERT (mBERT) that supports 104 languages, (2) cross-lingual language model BIBREF6 that supports 100 languages, and (3) Language Agnostic SEntence Representations BIBREF7 that supports 93 languages  Among the three models, LASER is based on neural machine translation approach and strictly requires parallel data to train 
Do multilingual models always need to be trained from scratch? Can we transfer linguistic knowledge learned by English pre-trained models to other languages? In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way BIBREF8  As the first step, we focus on building a bilingual language model (LM) of English and a target language  Starting from a pre-trained English LM, we learn the target language specific parameters (i e , word embeddings), while keeping the encoder layers of the pre-trained English LM fixed  We then fine-tune both English and target model to obtain the bilingual LM  We apply our approach to autoencoding language models with masked language model objective and show the advantage of the proposed approach in zero-shot transfer  Our main contributions in this work are:
We propose a fast adaptation method for obtaining a bilingual BERT$_{\textsc {base}}$ of English and a target language within a day using one Tesla V100 16GB GPU 
We evaluate our bilingual LMs for six languages on two zero-shot cross-lingual transfer tasks, namely natural language inference BIBREF9 and universal dependency parsing  We show that our models offer competitive performance or even better that mBERT 
We illustrate that our bilingual LMs can serve as an excellent feature extractor in supervised dependency parsing task 
Bilingual Pre-trained LMs
We first provide some background of pre-trained language models  Let $_e$ be English word-embeddings and $\Psi ()$ be the Transformer BIBREF10 encoder with parameters $$  Let $_{w_i}$ denote the embedding of word $w_i$ (i e , $_{w_i} = _e[w_1]$)  We omit positional embeddings and bias for clarity  A pre-trained LM typically performs the following computations: (i) transform a sequence of input tokens to contextualized representations $[_{w_1},\dots ,_{w_n}] = \Psi (_{w_1}, \dots , _{w_n}; )$, and (ii) predict an output word $y_i$ at $i^{\text{th}}$ position $p(y_i | _{w_i}) \propto \exp (_{w_i}^\top _{y_i})$ 
Autoencoding LM BIBREF0 corrupts some input tokens $w_i$ by replacing them with a special token [MASK]  It then predicts the original tokens $y_i = w_i$ from the corrupted tokens  Autoregressive LM BIBREF3 predicts the next token ($y_i = w_{i+1}$) given all the previous tokens  The recently proposed XLNet model BIBREF5 is an autoregressive LM that factorizes output with all possible permutations, which shows empirical performance improvement over GPT-2 due to the ability to capture bidirectional context  Here we assume that the encoder performs necessary masking with respect to each training objective 
Given an English pre-trained LM, we wish to learn a bilingual LM for English and a given target language $f$ under a limited computational resource budget  To quickly build a bilingual LM, we directly adapt the English pre-traind model to the target model  Our approach consists of three steps  First, we initialize target language word-embeddings $_f$ in the English embedding space such that embeddings of a target word and its English equivalents are close together (§SECREF8)  Next, we create a target LM from the target embeddings and the English encoder $\Psi ()$  We then fine-tune target embeddings while keeping $\Psi ()$ fixed (§SECREF14)  Finally, we construct a bilingual LM of $_e$, $_f$, and $\Psi ()$ and fine-tune all the parameters (§SECREF15)  Figure FIGREF7 illustrates the last two steps in our approach 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings
Our approach to learn the initial foreign word embeddings $_f \in ^{|V_f| \times d}$ is based on the idea of mapping the trained English word embeddings $_e \in ^{|V_e| \times d}$ to $_f$ such that if a foreign word and an English word are similar in meaning then their embeddings are similar  Borrowing the idea of universal lexical sharing from BIBREF11, we represent each foreign word embedding $_f[i] \in ^d$ as a linear combination of English word embeddings $_e[j] \in ^d$
where $_i\in ^{|V_e|}$ is a sparse vector and $\sum _j^{|V_e|} \alpha _{ij} = 1$ 
In this step of initializing foreign embeddings, having a good estimation of $$ could speed of the convergence when tuning the foreign model and enable zero-shot transfer (§SECREF5)  In the following, we discuss how to estimate $_i\;\forall i\in \lbrace 1,2, \dots , |V_f|\rbrace $ under two scenarios: (i) we have parallel data of English-foreign, and (ii) we only rely on English and foreign monolingual data 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Parallel Corpus
Given an English-foreign parallel corpus, we can estimate word translation probability $p(e\,|\,f)$ for any (English-foreign) pair $(e, f)$ using popular word-alignment BIBREF12 toolkits such as fast-align BIBREF13  We then assign:
Since $_i$ is estimated from word alignment, it is a sparse vector 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Monolingual Corpus
For low resource languages, parallel data may not be available  In this case, we rely only on monolingual data (e g , Wikipedias)  We estimate word translation probabilities from word embeddings of the two languages  Word vectors of these languages can be learned using fastText BIBREF14 and then are aligned into a shared space with English BIBREF15, BIBREF16  Unlike learning contextualized representations, learning word vectors is fast and computationally cheap  Given the aligned vectors $\bar{}_f$ of foreign and $\bar{}_e$ of English, we calculate the word translation matrix $\in ^{|V_f|\times |V_e|}$ as
Here, we use $\mathrm {sparsemax}$ BIBREF17 instead of softmax  Sparsemax is a sparse version of softmax and it puts zero probabilities on most of the word in the English vocabulary except few English words that are similar to a given foreign word  This property is desirable in our approach since it leads to a better initialization of the foreign embeddings 
Bilingual Pre-trained LMs ::: Fine-tuning Target Embeddings
After initializing foreign word-embeddings, we replace English word-embeddings in the English pre-trained LM with foreign word-embeddings to obtain the foreign LM  We then fine-tune only foreign word-embeddings on monolingual data  The training objective is the same as the training objective of the English pre-trained LM (i e , masked LM for BERT)  Since the trained encoder $\Psi ()$ is good at capturing association, the purpose of this step is to further optimize target embeddings such that the target LM can utilized the trained encoder for association task  For example, if the words Albert Camus presented in a French input sequence, the self-attention in the encoder more likely attends to words absurde and existentialisme once their embeddings are tuned 
Bilingual Pre-trained LMs ::: Fine-tuning Bilingual LM
We create a bilingual LM by plugging foreign language specific parameters to the pre-trained English LM (Figure FIGREF7)  The new model has two separate embedding layers and output layers, one for English and one for foreign language  The encoder layer in between is shared  We then fine-tune this model using English and foreign monolingual data  Here, we keep tuning the model on English to ensure that it does not forget what it has learned in English and that we can use the resulting model for zero-shot transfer (§SECREF3)  In this step, the encoder parameters are also updated so that in can learn syntactic aspects (i e , word order, morphological agreement) of the target languages 
Zero-shot Experiments
We build our bilingual LMs, named RAMEN, starting from BERT$_{\textsc {base}}$, BERT$_{\textsc {large}}$, RoBERTa$_{\textsc {base}}$, and RoBERTa$_{\textsc {large}}$ pre-trained models  Using BERT$_{\textsc {base}}$ allows us to compare the results with mBERT model  Using BERT$_{\textsc {large}}$ and RoBERTa allows us to investigate whether the performance of the target LM correlates with the performance of the source LM  We evaluate our models on two cross-lingual zero-shot tasks: (1) Cross-lingual Natural Language Inference (XNLI) and (2) dependency parsing 
Zero-shot Experiments ::: Data
We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi)  These languages belong to four different language families  French, Russian, and Hindi are Indo-European languages, similar to English  Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan, and Austro-Asiatic family respectively  The choice of the six languages also reflects different training conditions depending on the amount of monolingual data  French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource 
For experiments that use parallel data to initialize foreign specific parameters, we use the same datasets in the work of BIBREF6  Specifically, we use United Nations Parallel Corpus BIBREF18 for en-ru, en-ar, en-zh, and en-fr  We collect en-hi parallel data from IIT Bombay corpus BIBREF19 and en-vi data from OpenSubtitles 2018  For experiments that use only monolingual data to initialize foreign parameters, instead of training word-vectors from the scratch, we use the pre-trained word vectors from fastText BIBREF14 to estimate word translation probabilities (Eq  DISPLAY_FORM13)  We align these vectors into a common space using orthogonal Procrustes BIBREF20, BIBREF15, BIBREF16  We only use identical words between the two languages as the supervised signal  We use WikiExtractor to extract extract raw sentences from Wikipedias as monolingual data for fine-tuning target embeddings and bilingual LMs (§SECREF15)  We do not lowercase or remove accents in our data preprocessing pipeline 
We tokenize English using the provided tokenizer from pre-trained models  For target languages, we use fastBPE to learn 30,000 BPE codes and 50,000 codes when transferring from BERT and RoBERTa respectively  We truncate the BPE vocabulary of foreign languages to match the size of the English vocabulary in the source models  Precisely, the size of foreign vocabulary is set to 32,000 when transferring from BERT and 50,000 when transferring from RoBERTa 
We use XNLI dataset BIBREF9 for classification task and Universal Dependencies v2 4 BIBREF21 for parsing task  Since a language might have more than one treebank in Universal Dependencies, we use the following treebanks: en_ewt (English), fr_gsd (French), ru_syntagrus (Russian) ar_padt (Arabic), vi_vtb (Vietnamese), hi_hdtb (Hindi), and zh_gsd (Chinese) 
Zero-shot Experiments ::: Data ::: Remark on BPE
BIBREF22 show that sharing subwords between languages improves alignments between embedding spaces  BIBREF2 observe a strong correlation between the percentage of overlapping subwords and mBERT's performances for cross-lingual zero-shot transfer  However, in our current approach, subwords between source and target are not shared  A subword that is in both English and foreign vocabulary has two different embeddings 
Zero-shot Experiments ::: Estimating translation probabilities
Since pre-trained models operate on subword level, we need to estimate subword translation probabilities  Therefore, we subsample 2M sentence pairs from each parallel corpus and tokenize the data into subwords before running fast-align BIBREF13 
Estimating subword translation probabilities from aligned word vectors requires an additional processing step since the provided vectors from fastText are not at subword level  We use the following approximation to obtain subword vectors: the vector $_s$ of subword $s$ is the weighted average of all the aligned word vectors $_{w_i}$ that have $s$ as an subword
where $p(w_j)$ is the unigram probability of word $w_j$ and $n_s = \sum _{w_j:\, s\in w_j} p(w_j)$  We take the top 50,000 words in each aligned word-vectors to compute subword vectors 
In both cases, not all the words in the foreign vocabulary can be initialized from the English word-embeddings  Those words are initialized randomly from a Gaussian $\mathcal {N}(0, {1}{d^2})$ 
Zero-shot Experiments ::: Hyper-parameters
In all the experiments, we tune RAMEN$_{\textsc {base}}$ for 175,000 updates and RAMEN$_{\textsc {large}}$ for 275,000 updates where the first 25,000 updates are for language specific parameters  The sequence length is set to 256  The mini-batch size are 64 and 24 when tuning language specific parameters using RAMEN$_{\textsc {base}}$ and RAMEN$_{\textsc {large}}$ respectively  For tuning bilingual LMs, we use a mini-batch size of 64 for RAMEN$_{\textsc {base}}$ and 24 for RAMEN$_{\textsc {large}}$ where half of the batch are English sequences and the other half are foreign sequences  This strategy of balancing mini-batch has been used in multilingual neural machine translation BIBREF23, BIBREF24 
We optimize RAMEN$_{\textsc {base}}$ using Lookahead optimizer BIBREF25 wrapped around Adam with the learning rate of $10^{-4}$, the number of fast weight updates $k=5$, and interpolation parameter $\alpha =0 5$  We choose Lookahead optimizer because it has been shown to be robust to the initial parameters of the based optimizer (Adam)  For Adam optimizer, we linearly increase the learning rate from $10^{-7}$ to $10^{-4}$ in the first 4000 updates and then follow an inverse square root decay  All RAMEN$_{\textsc {large}}$ models are optimized with Adam due to memory limit 
When fine-tuning RAMEN on XNLI and UD, we use a mini-batch size of 32, Adam's learning rate of $10^{-5}$  The number of epochs are set to 4 and 50 for XNLI and UD tasks respectively  All experiments are carried out on a single Tesla V100 16GB GPU  Each RAMEN$_{\textsc {base}}$ model is trained within a day and each RAMEN$_{\textsc {large}}$ is trained within two days 
Results
In this section, we present the results of out models for two zero-shot cross lingual transfer tasks: XNLI and dependency parsing 
Results ::: Cross-lingual Natural Language Inference
Table TABREF32 shows the XNLI test accuracy  For reference, we also include the scores from the previous work, notably the state-of-the-art system XLM BIBREF6  Before discussing the results, we spell out that the fairest comparison in this experiment is the comparison between mBERT and RAMEN$_{\textsc {base}}$+BERT trained with monolingual only 
We first discuss the transfer results from BERT  Initialized from fastText vectors, RAMEN$_{\textsc {base}}$ slightly outperforms mBERT by 1 9 points on average and widen the gap of 3 3 points on Arabic  RAMEN$_{\textsc {base}}$ gains extra 0 8 points on average when initialized from parallel data  With triple number of parameters, RAMEN$_{\textsc {large}}$ offers an additional boost in term of accuracy and initialization with parallel data consistently improves the performance  It has been shown that BERT$_{\textsc {large}}$ significantly outperforms BERT$_{\textsc {base}}$ on 11 English NLP tasks BIBREF0, the strength of BERT$_{\textsc {large}}$ also shows up when adapted to foreign languages 
Transferring from RoBERTa leads to better zero-shot accuracies  With the same initializing condition, RAMEN$_{\textsc {base}}$+RoBERTa outperforms RAMEN$_{\textsc {base}}$+BERT on average by 2 9 and 2 3 points when initializing from monolingual and parallel data respectively  This result show that with similar number of parameters, our approach benefits from a better English pre-trained model  When transferring from RoBERTa$_{\textsc {large}}$, we obtain state-of-the-art results for five languages 
Currently, RAMEN only uses parallel data to initialize foreign embeddings  RAMEN can also exploit parallel data through translation objective proposed in XLM  We believe that by utilizing parallel data during the fine-tuning of RAMEN would bring additional benefits for zero-shot tasks  We leave this exploration to future work  In summary, starting from BERT$_{\textsc {base}}$, our approach obtains competitive bilingual LMs with mBERT for zero-shot XNLI  Our approach shows the accuracy gains when adapting from a better pre-trained model 
Results ::: Universal Dependency Parsing
We build on top of RAMEN a graph-based dependency parser BIBREF27  For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags  Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores  Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing 
We first look at the fairest comparison between mBERT and monolingually initialized RAMEN$_{\textsc {base}}$+BERT  The latter outperforms the former on five languages except Arabic  We observe the largest gain of +5 2 LAS for French  Chinese enjoys +3 1 LAS from our approach  With similar architecture (12 or 24 layers) and initialization (using monolingual or parallel data), RAMEN+RoBERTa performs better than RAMEN+BERT for most of the languages  Arabic and Hindi benefit the most from bigger models  For the other four languages, RAMEN$_{\textsc {large}}$ renders a modest improvement over RAMEN$_{\textsc {base}}$ 
Analysis ::: Impact of initialization
Initializing foreign embeddings is the backbone of our approach  A good initialization leads to better zero-shot transfer results and enables fast adaptation  To verify the importance of a good initialization, we train a RAMEN$_{\textsc {base}}$+RoBERTa with foreign word-embeddings are initialized randomly from $\mathcal {N}(0, {1}{d^2})$  For a fair comparison, we use the same hyper-parameters in §SECREF27  Table TABREF36 shows the results of XNLI and UD parsing of random initialization  In comparison to the initialization using aligned fastText vectors, random initialization decreases the zero-shot performance of RAMEN$_{\textsc {base}}$ by 15 9% for XNLI and 27 8 points for UD parsing on average  We also see that zero-shot parsing of SOV languages (Arabic and Hindi) suffers random initialization 
Analysis ::: Are contextual representations from RAMEN also good for supervised parsing?
All the RAMEN models are built from English and tuned on English for zero-shot cross-lingual tasks  It is reasonable to expect RAMENs do well in those tasks as we have shown in our experiments  But are they also a good feature extractor for supervised tasks? We offer a partial answer to this question by evaluating our model for supervised dependency parsing on UD datasets 
We used train/dev/test splits provided in UD to train and evaluate our RAMEN-based parser  Table TABREF38 summarizes the results (LAS) of our supervised parser  For a fair comparison, we choose mBERT as the baseline and all the RAMEN models are initialized from aligned fastText vectors  With the same architecture of 12 Transformer layers, RAMEN$_{\textsc {base}}$+BERT performs competitive to mBERT and outshines mBERT by +1 2 points for Vietnamese  The best LAS results are obtained by RAMEN$_{\textsc {large}}$+RoBERTa with 24 Transformer layers  Overall, our results indicate the potential of using contextual representations from RAMEN for supervised tasks 
Analysis ::: How does linguistic knowledge transfer happen through each training stages?
We evaluate the performance of RAMEN+RoBERTa$_{\textsc {base}}$ (initialized from monolingual data) at each training steps: initialization of word embeddings (0K update), fine-tuning target embeddings (25K), and fine-tuning the model on both English and target language (at each 25K updates)  The results are presented in Figure FIGREF40 
Without fine-tuning, the average accuracy of XLNI is 39 7% for a three-ways classification task, and the average LAS score is 3 6 for dependency parsing  We see the biggest leap in the performance after 50K updates  While semantic similarity task profits significantly at 25K updates of the target embeddings, syntactic task benefits with further fine-tuning the encoder  This is expected since the target languages might exhibit different syntactic structures than English and fine-tuning encoder helps to capture language specific structures  We observe a substantial gain of 19-30 LAS for all languages except French after 50K updates 
Language similarities have more impact on transferring syntax than semantics  Without tuning the English encoder, French enjoys 50 3 LAS for being closely related to English, whereas Arabic and Hindi, SOV languages, modestly reach 4 2 and 6 4 points using the SVO encoder  Although Chinese has SVO order, it is often seen as head-final while English is strong head-initial  Perhaps, this explains the poor performance for Chinese 
Limitations
While we have successfully adapted autoencoding pre-trained LMs from English to other languages, the question whether our approach can also be applied for autoregressive LM such as XLNet still remains  We leave the investigation to future work 
Conclusions
In this work, we have presented a simple and effective approach for rapidly building a bilingual LM under a limited computational budget  Using BERT as the starting point, we demonstrate that our approach produces better than mBERT on two cross-lingual zero-shot sentence classification and dependency parsing  We find that the performance of our bilingual LM, RAMEN, correlates with the performance of the original pre-trained English models  We also find that RAMEN is also a powerful feature extractor in supervised dependency parsing  Finally, we hope that our work sparks of interest in developing fast and effective methods for transferring pre-trained English models to other languages 
Context 2: Introduction
This work is licenced under a Creative Commons Attribution 4 0 International Licence  Licence details: http://creativecommons org/licenses/by/4 0/
In the spirit of the brevity of social media's messages and reactions, people have got used to express feelings minimally and symbolically, as with hashtags on Twitter and Instagram  On Facebook, people tend to be more wordy, but posts normally receive more simple “likes” than longer comments  Since February 2016, Facebook users can express specific emotions in response to a post thanks to the newly introduced reaction feature (see Section SECREF2 ), so that now a post can be wordlessly marked with an expression of say “joy" or “surprise" rather than a generic “like” 
It has been observed that this new feature helps Facebook to know much more about their users and exploit this information for targeted advertising BIBREF0 , but interest in people's opinions and how they feel isn't limited to commercial reasons, as it invests social monitoring, too, including health care and education BIBREF1   However, emotions and opinions are not always expressed this explicitly, so that there is high interest in developing systems towards their automatic detection  Creating manually annotated datasets large enough to train supervised models is not only costly, but also—especially in the case of opinions and emotions—difficult, due to the intrinsic subjectivity of the task BIBREF2 , BIBREF3   Therefore, research has focused on unsupervised methods enriched with information derived from lexica, which are manually created BIBREF3 , BIBREF4   Since go2009twitter have shown that happy and sad emoticons can be successfully used as signals for sentiment labels, distant supervision, i e  using some reasonably safe signals as proxies for automatically labelling training data BIBREF5 , has been used also for emotion recognition, for example exploiting both emoticons and Twitter hashtags BIBREF6 , but mainly towards creating emotion lexica  mohammad2015using use hashtags, experimenting also with highly fine-grained emotion sets (up to almost 600 emotion labels), to create the large Hashtag Emotion Lexicon  Emoticons are used as proxies also by hallsmarmulti, who use distributed vector representations to find which words are interchangeable with emoticons but also which emoticons are used in a similar context 
We take advantage of distant supervision by using Facebook reactions as proxies for emotion labels, which to the best of our knowledge hasn't been done yet, and we train a set of Support Vector Machine models for emotion recognition  Our models, differently from existing ones, exploit information which is acquired entirely automatically, and achieve competitive or even state-of-the-art results for some of the emotion labels on existing, standard evaluation datasets  For explanatory purposes, related work is discussed further and more in detail when we describe the benchmarks for evaluation (Section SECREF3 ) and when we compare our models to existing ones (Section SECREF5 )  We also explore and discuss how choosing different sets of Facebook pages as training data provides an intrinsic domain-adaptation method 
Facebook reactions as labels
For years, on Facebook people could leave comments to posts, and also “like” them, by using a thumbs-up feature to explicitly express a generic, rather underspecified, approval  A “like” could thus mean “I like what you said", but also “I like that you bring up such topic (though I find the content of the article you linked annoying)" 
In February 2016, after a short trial, Facebook made a more explicit reaction feature available world-wide  Rather than allowing for the underspecified “like” as the only wordless response to a post, a set of six more specific reactions was introduced, as shown in Figure FIGREF1 : Like, Love, Haha, Wow, Sad and Angry  We use such reactions as proxies for emotion labels associated to posts 
We collected Facebook posts and their corresponding reactions from public pages using the Facebook API, which we accessed via the Facebook-sdk python library  We chose different pages (and therefore domains and stances), aiming at a balanced and varied dataset, but we did so mainly based on intuition (see Section SECREF4 ) and with an eye to the nature of the datasets available for evaluation (see Section SECREF5 )  The choice of which pages to select posts from is far from trivial, and we believe this is actually an interesting aspect of our approach, as by using different Facebook pages one can intrinsically tackle the domain-adaptation problem (See Section SECREF6 for further discussion on this)  The final collection of Facebook pages for the experiments described in this paper is as follows: FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney 
Note that thankful was only available during specific time spans related to certain events, as Mother's Day in May 2016 
For each page, we downloaded the latest 1000 posts, or the maximum available if there are fewer, from February 2016, retrieving the counts of reactions for each post  The output is a JSON file containing a list of dictionaries with a timestamp, the post and a reaction vector with frequency values, which indicate how many users used that reaction in response to the post (Figure FIGREF3 )  The resulting emotion vectors must then be turned into an emotion label 
In the context of this experiment, we made the simple decision of associating to each post the emotion with the highest count, ignoring like as it is the default and most generic reaction people tend to use  Therefore, for example, to the first post in Figure FIGREF3 , we would associate the label sad, as it has the highest score (284) among the meaningful emotions we consider, though it also has non-zero scores for other emotions  At this stage, we didn't perform any other entropy-based selection of posts, to be investigated in future work 
Emotion datasets
Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset  In order to compare our performance to state-of-the-art results, we have used them as well  In this Section, in addition to a description of each dataset, we provide an overview of the emotions used, their distribution, and how we mapped them to those we obtained from Facebook posts in Section SECREF7   A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation 
Affective Text dataset
Task 14 at SemEval 2007 BIBREF7 was concerned with the classification of emotions and valence in news headlines  The headlines where collected from several news websites including Google news, The New York Times, BBC News and CNN  The used emotion labels were Anger, Disgust, Fear, Joy, Sadness, Surprise, in line with the six basic emotions of Ekman's standard model BIBREF8   Valence was to be determined as positive or negative  Classification of emotion and valence were treated as separate tasks  Emotion labels were not considered as mututally exclusive, and each emotion was assigned a score from 0 to 100  Training/developing data amounted to 250 annotated headlines (Affective development), while systems were evaluated on another 1000 (Affective test)  Evaluation was done using two different methods: a fine-grained evaluation using Pearson's r to measure the correlation between the system scores and the gold standard; and a coarse-grained method where each emotion score was converted to a binary label, and precision, recall, and f-score were computed to assess performance  As it is done in most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , we also treat this as a classification problem (coarse-grained)  This dataset has been extensively used for the evaluation of various unsupervised methods BIBREF2 , but also for testing different supervised learning techniques and feature portability BIBREF10  
Fairy Tales dataset
This is a dataset collected by alm2008affect, where about 1,000 sentences from fairy tales (by B  Potter, H C  Andersen and Grimm) were annotated with the same six emotions of the Affective Text dataset, though with different names: Angry, Disgusted, Fearful, Happy, Sad, and Surprised  In most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , only sentences where all annotators agreed are used, and the labels angry and disgusted are merged  We adopt the same choices 
ISEAR
The ISEAR (International Survey on Emotion Antecedents and Reactions BIBREF11 , BIBREF12 ) is a dataset created in the context of a psychology project of the 1990s, by collecting questionnaires answered by people with different cultural backgrounds  The main aim of this project was to gather insights in cross-cultural aspects of emotional reactions  Student respondents, both psychologists and non-psychologists, were asked to report situations in which they had experienced all of seven major emotions (joy, fear, anger, sadness, disgust, shame and guilt)  In each case, the questions covered the way they had appraised a given situation and how they reacted  The final dataset contains reports by approximately 3000 respondents from all over the world, for a total of 7665 sentences labelled with an emotion, making this the largest dataset out of the three we use 
Overview of datasets and emotions
We summarise datasets and emotion distribution from two viewpoints  First, because there are different sets of emotions labels in the datasets and Facebook data, we need to provide a mapping and derive a subset of emotions that we are going to use for the experiments  This is shown in Table TABREF8 , where in the “Mapped” column we report the final emotions we use in this paper: anger, joy, sadness, surprise  All labels in each dataset are mapped to these final emotions, which are therefore the labels we use for training and testing our models 
Second, the distribution of the emotions for each dataset is different, as can be seen in Figure FIGREF9  
In Figure FIGREF9 we also provide the distribution of the emotions anger, joy, sadness, surprise per Facebook page, in terms of number of posts (recall that we assign to a post the label corresponding to the majority emotion associated to it, see Section SECREF2 )  We can observe that for example pages about news tend to have more sadness and anger posts, while pages about cooking and tv-shows have a high percentage of joy posts  We will use this information to find the best set of pages for a given target domain (see Section SECREF5 ) 
Model
There are two main decisions to be taken in developing our model: (i) which Facebook pages to select as training data, and (ii) which features to use to train the model, which we discuss below  Specifically, we first set on a subset of pages and then experiment with features  Further exploration of the interaction between choice of pages and choice of features is left to future work, and partly discussed in Section SECREF6   For development, we use a small portion of the Affective data set described in Section SECREF4 , that is the portion that had been released as development set for SemEval's 2007 Task 14 BIBREF7 , which contains 250 annotated sentences (Affective development, Section SECREF4 )  All results reported in this section are on this dataset  The test set of Task 14 as well as the other two datasets described in Section SECREF3 will be used to evaluate the final models (Section SECREF4 ) 
Selecting Facebook pages
Although page selection is a crucial ingredient of this approach, which we believe calls for further and deeper, dedicated investigation, for the experiments described here we took a rather simple approach  First, we selected the pages that would provide training data based on intuition and availability, then chose different combinations according to results of a basic model run on development data, and eventually tested feature combinations, still on the development set 
For the sake of simplicity and transparency, we first trained an SVM with a simple bag-of-words model and default parameters as per the Scikit-learn implementation BIBREF13 on different combinations of pages  Based on results of the attempted combinations as well as on the distribution of emotions in the development dataset (Figure FIGREF9 ), we selected a best model (B-M), namely the combined set of Time, The Guardian and Disney, which yields the highest results on development data  Time and The Guardian perform well on most emotions but Disney helps to boost the performance for the Joy class 
Features
In selecting appropriate features, we mainly relied on previous work and intuition  We experimented with different combinations, and all tests were still done on Affective development, using the pages for the best model (B-M) described above as training data  Results are in Table TABREF20   Future work will further explore the simultaneous selection of features and page combinations 
We use a set of basic text-based features to capture the emotion class  These include a tf-idf bag-of-words feature, word (2-3) and character (2-5) ngrams, and features related to the presence of negation words, and to the usage of punctuation 
This feature is used in all unsupervised models as a source of information, and we mainly include it to assess its contribution, but eventually do not use it in our final model 
We used the NRC10 Lexicon because it performed best in the experiments by BIBREF10 , which is built around the emotions anger, anticipation, disgust, fear, joy, sadness, and surprise, and the valence values positive and negative  For each word in the lexicon, a boolean value indicating presence or absence is associated to each emotion  For a whole sentence, a global score per emotion can be obtained by summing the vectors for all content words of that sentence included in the lexicon, and used as feature 
As additional feature, we also included Word Embeddings, namely distributed representations of words in a vector space, which have been exceptionally successful in boosting performance in a plethora of NLP tasks  We use three different embeddings:
Google embeddings: pre-trained embeddings trained on Google News and obtained with the skip-gram architecture described in BIBREF14   This model contains 300-dimensional vectors for 3 million words and phrases 
Facebook embeddings: embeddings that we trained on our scraped Facebook pages for a total of 20,000 sentences  Using the gensim library BIBREF15 , we trained the embeddings with the following parameters: window size of 5, learning rate of 0 01 and dimensionality of 100  We filtered out words with frequency lower than 2 occurrences 
Retrofitted embeddings: Retrofitting BIBREF16 has been shown as a simple but efficient way of informing trained embeddings with additional information derived from some lexical resource, rather than including it directly at the training stage, as it's done for example to create sense-aware BIBREF17 or sentiment-aware BIBREF18 embeddings  In this work, we retrofit general embeddings to include information about emotions, so that emotion-similar words can get closer in space  Both the Google as well as our Facebook embeddings were retrofitted with lexical information obtained from the NRC10 Lexicon mentioned above, which provides emotion-similarity for each token  Note that differently from the previous two types of embeddings, the retrofitted ones do rely on handcrafted information in the form of a lexical resource 
Results on development set
We report precision, recall, and f-score on the development set  The average f-score is reported as micro-average, to better account for the skewed distribution of the classes as well as in accordance to what is usually reported for this task BIBREF19  
From Table TABREF20 we draw three main observations  First, a simple tf-idf bag-of-word mode works already very well, to the point that the other textual and lexicon-based features don't seem to contribute to the overall f-score (0 368), although there is a rather substantial variation of scores per class  Second, Google embeddings perform a lot better than Facebook embeddings, and this is likely due to the size of the corpus used for training  Retrofitting doesn't seem to help at all for the Google embeddings, but it does boost the Facebook embeddings, leading to think that with little data, more accurate task-related information is helping, but corpus size matters most  Third, in combination with embeddings, all features work better than just using tf-idf, but removing the Lexicon feature, which is the only one based on hand-crafted resources, yields even better results  Then our best model (B-M) on development data relies entirely on automatically obtained information, both in terms of training data as well as features 
Results
In Table TABREF26 we report the results of our model on the three datasets standardly used for the evaluation of emotion classification, which we have described in Section SECREF3  
Our B-M model relies on subsets of Facebook pages for training, which were chosen according to their performance on the development set as well as on the observation of emotions distribution on different pages and in the different datasets, as described in Section SECREF4   The feature set we use is our best on the development set, namely all the features plus Google-based embeddings, but excluding the lexicon  This makes our approach completely independent of any manual annotation or handcrafted resource  Our model's performance is compared to the following systems, for which results are reported in the referred literature  Please note that no other existing model was re-implemented, and results are those reported in the respective papers 
Discussion, conclusions and future work
We have explored the potential of using Facebook reactions in a distant supervised setting to perform emotion classification  The evaluation on standard benchmarks shows that models trained as such, especially when enhanced with continuous vector representations, can achieve competitive results without relying on any handcrafted resource  An interesting aspect of our approach is the view to domain adaptation via the selection of Facebook pages to be used as training data 
We believe that this approach has a lot of potential, and we see the following directions for improvement  Feature-wise, we want to train emotion-aware embeddings, in the vein of work by tang:14, and iacobacci2015sensembed  Retrofitting FB-embeddings trained on a larger corpus might also be successful, but would rely on an external lexicon 
The largest room for yielding not only better results but also interesting insights on extensions of this approach lies in the choice of training instances, both in terms of Facebook pages to get posts from, as well as in which posts to select from the given pages  For the latter, one could for example only select posts that have a certain length, ignore posts that are only quotes or captions to images, or expand posts by including content from linked html pages, which might provide larger and better contexts BIBREF23   Additionally, and most importantly, one could use an entropy-based measure to select only posts that have a strong emotion rather than just considering the majority emotion as training label  For the former, namely the choice of Facebook pages, which we believe deserves the most investigation, one could explore several avenues, especially in relation to stance-based issues BIBREF24   In our dataset, for example, a post about Chile beating Colombia in a football match during the Copa America had very contradictory reactions, depending on which side readers would cheer for  Similarly, the very same political event, for example, would get very different reactions from readers if it was posted on Fox News or The Late Night Show, as the target audience is likely to feel very differently about the same issue  This also brings up theoretical issues related more generally to the definition of the emotion detection task, as it's strongly dependent on personal traits of the audience  Also, in this work, pages initially selected on availability and intuition were further grouped into sets to make training data according to performance on development data, and label distribution  Another criterion to be exploited would be vocabulary overlap between the pages and the datasets 
Lastly, we could develop single models for each emotion, treating the problem as a multi-label task  This would even better reflect the ambiguity and subjectivity intrinsic to assigning emotions to text, where content could be at same time joyful or sad, depending on the reader 
Acknowledgements
In addition to the anonymous reviewers, we want to thank Lucia Passaro and Barbara Plank for insightful discussions, and for providing comments on draft versions of this paper 
Context 3: Introduction
End-to-end speech-to-text translation (ST) has attracted much attention recently BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 given its simplicity against cascading automatic speech recognition (ASR) and machine translation (MT) systems  The lack of labeled data, however, has become a major blocker for bridging the performance gaps between end-to-end models and cascading systems  Several corpora have been developed in recent years  post2013improved introduced a 38-hour Spanish-English ST corpus by augmenting the transcripts of the Fisher and Callhome corpora with English translations  di-gangi-etal-2019-must created the largest ST corpus to date from TED talks but the language pairs involved are out of English only  beilharz2019librivoxdeen created a 110-hour German-English ST corpus from LibriVox audiobooks  godard-etal-2018-low created a Moboshi-French ST corpus as part of a rare language documentation effort  woldeyohannis provided an Amharic-English ST corpus in the tourism domain  boito2019mass created a multilingual ST corpus involving 8 languages from a multilingual speech corpus based on Bible readings BIBREF7  Previous work either involves language pairs out of English, very specific domains, very low resource languages or a limited set of language pairs  This limits the scope of study, including the latest explorations on end-to-end multilingual ST BIBREF8, BIBREF9  Our work is mostly similar and concurrent to iranzosnchez2019europarlst who created a multilingual ST corpus from the European Parliament proceedings  The corpus we introduce has larger speech durations and more translation tokens  It is diversified with multiple speakers per transcript/translation  Finally, we provide additional out-of-domain test sets 
In this paper, we introduce CoVoST, a multilingual ST corpus based on Common Voice BIBREF10 for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora  We also collect an additional evaluation corpus from Tatoeba for French, German, Dutch, Russian and Spanish, resulting in a total of 9 3 hours of speech  Both corpora are created at the sentence level and do not require additional alignments or segmentation  Using the official Common Voice train-development-test split, we also provide baseline models, including, to our knowledge, the first end-to-end many-to-one multilingual ST models  CoVoST is released under CC0 license and free to use  The Tatoeba evaluation samples are also available under friendly CC licenses  All the data can be acquired at https://github com/facebookresearch/covost 
Data Collection and Processing ::: Common Voice (CoVo)
Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license  Contributors record voice clips by reading from a bank of donated sentences  Each voice clip was validated by at least two other users  Most of the sentences are covered by multiple speakers, with potentially different genders, age groups or accents 
Raw CoVo data contains samples that passed validation as well as those that did not  To build CoVoST, we only use the former one and reuse the official train-development-test partition of the validated data  As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages  CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese 
Validated transcripts were sent to professional translators  Note that the translators had access to the transcripts but not the corresponding voice clips since clips would not carry additional information  Since transcripts were duplicated due to multiple speakers, we deduplicated the transcripts before sending them to translators  As a result, different voice clips of the same content (transcript) will have identical translations in CoVoST for train, development and test splits 
In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11  1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU BIBREF12 with the NLTK BIBREF13 implementation between the human translations and the automatic translations produced by a state-of-the-art system BIBREF14 (the French-English system was a Transformer big BIBREF15 separately trained on WMT14)  We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems  Translations with a score that was too low were manually inspected and sent back to the translators when needed  2) We manually inspected examples where the source transcript was identical to the translation  3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data BIBREF14  We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly  4) We computed the ratio of English characters in the translations

ground_truth: translation probabilities, Labeled Attachment Scores (LAS)

accuracy, Labeled Attachment Scores (LAS)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which Facebook pages did they look at?
Context 3:  We manually inspected examples with a low ratio and sent them back to translators accordingly  5) Finally, we used VizSeq BIBREF16 to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings BIBREF17  Samples with low scores were manually inspected and sent back for translation when needed 
We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint 
Data Collection and Processing ::: Tatoeba (TT)
Tatoeba (TT) is a community built language learning corpus having sentences aligned across multiple languages with the corresponding speech partially available  Its sentences are on average shorter than those in CoVoST (see also Table TABREF2) given the original purpose of language learning  Sentences in TT are licensed under CC BY 2 0 FR and part of the speeches are available under various CC licenses 
We construct an evaluation set from TT (for French, German, Dutch, Russian and Spanish) as a complement to CoVoST development and test sets  We collect (speech, transcript, English translation) triplets for the 5 languages and do not include those whose speech has a broken URL or is not CC licensed  We further filter these samples by sentence lengths (minimum 4 words including punctuations) to reduce the portion of short sentences  This makes the resulting evaluation set closer to real-world scenarios and more challenging 
We run the same quality checks for TT as for CoVoST but we do not find poor quality translations according to our criteria  Finally, we report the overlap between CoVo transcripts and TT sentences in Table TABREF5  We found a minimal overlap, which makes the TT evaluation set a suitable additional test set when training on CoVoST 
Data Analysis ::: Basic Statistics
Basic statistics for CoVoST and TT are listed in Table TABREF2 including (unique) sentence counts, speech durations, speaker demographics (partially available) as well as vocabulary and token statistics (based on Moses-tokenized sentences by sacreMoses) on both transcripts and translations  We see that CoVoST has over 327 hours of German speeches and over 171 hours of French speeches, which, to our knowledge, corresponds to the largest corpus among existing public ST corpora (the second largest is 110 hours BIBREF18 for German and 38 hours BIBREF19 for French)  Moreover, CoVoST has a total of 18 hours of Dutch speeches, to our knowledge, contributing the first public Dutch ST resource  CoVoST also has around 27-hour Russian speeches, 37-hour Italian speeches and 67-hour Persian speeches, which is 1 8 times, 2 5 times and 13 3 times of the previous largest public one BIBREF7  Most of the sentences (transcripts) in CoVoST are covered by multiple speakers with potentially different accents, resulting in a rich diversity in the speeches  For example, there are over 1,000 speakers and over 10 accents in the French and German development / test sets  This enables good coverage of speech variations in both model training and evaluation 
Data Analysis ::: Speaker Diversity
As we can see from Table TABREF2, CoVoST is diversified with a rich set of speakers and accents  We further inspect the speaker demographics in terms of sample distributions with respect to speaker counts, accent counts and age groups, which is shown in Figure FIGREF6, FIGREF7 and FIGREF8  We observe that for 8 of the 11 languages, at least 60% of the sentences (transcripts) are covered by multiple speakers  Over 80% of the French sentences have at least 3 speakers  And for German sentences, even over 90% of them have at least 5 speakers  Similarly, we see that a large portion of sentences are spoken in multiple accents for French, German, Dutch and Spanish  Speakers of each language also spread widely across different age groups (below 20, 20s, 30s, 40s, 50s, 60s and 70s) 
Baseline Results
We provide baselines using the official train-development-test split on the following tasks: automatic speech recognition (ASR), machine translation (MT) and speech translation (ST) 
Baseline Results ::: Experimental Settings ::: Data Preprocessing
We convert raw MP3 audio files from CoVo and TT into mono-channel waveforms, and downsample them to 16,000 Hz  For transcripts and translations, we normalize the punctuation, we tokenize the text with sacreMoses and lowercase it  For transcripts, we further remove all punctuation markers except for apostrophes  We use character vocabularies on all the tasks, with 100% coverage of all the characters  Preliminary experimentation showed that character vocabularies provided more stable training than BPE  For MT, the vocabulary is created jointly on both transcripts and translations  We extract 80-channel log-mel filterbank features, computed with a 25ms window size and 10ms window shift using torchaudio  The features are normalized to 0 mean and 1 0 standard deviation  We remove samples having more than 3,000 frames or more than 256 characters for GPU memory efficiency (less than 25 samples are removed for all languages) 
Baseline Results ::: Experimental Settings ::: Model Training
Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing  For MT, we use a Transformer base architecture BIBREF15, but with 3 encoder layers, 3 decoder layers and 0

ground_truth: FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney

FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is Arabic one of the 11 languages in CoVost?
Context 3: 3 dropout  We use a batch size of 10,000 frames for ASR and ST, and a batch size of 4,000 tokens for MT  We train all models using Fairseq BIBREF20 for up to 200,000 updates  We use SpecAugment BIBREF21 for ASR and ST to alleviate overfitting 
Baseline Results ::: Experimental Settings ::: Inference and Evaluation
We use a beam size of 5 for all models  We use the best checkpoint by validation loss for MT, and average the last 5 checkpoints for ASR and ST  For MT and ST, we report case-insensitive tokenized BLEU BIBREF22 using sacreBLEU BIBREF23  For ASR, we report word error rate (WER) and character error rate (CER) using VizSeq 
Baseline Results ::: Automatic Speech Recognition (ASR)
For simplicity, we use the same model architecture for ASR and ST, although we do not leverage ASR models to pretrain ST model encoders later  Table TABREF18 shows the word error rate (WER) and character error rate (CER) for ASR models  We see that French and German perform the best given they are the two highest resource languages in CoVoST  The other languages are relatively low resource (especially Turkish and Swedish) and the ASR models are having difficulties to learn from this data 
Baseline Results ::: Machine Translation (MT)
MT models take transcripts (without punctuation) as inputs and outputs translations (with punctuation)  For simplicity, we do not change the text preprocessing methods for MT to correct this mismatch  Moreover, this mismatch also exists in cascading ST systems, where MT model inputs are the outputs of an ASR model  Table TABREF20 shows the BLEU scores of MT models  We notice that the results are consistent with what we see from ASR models  For example thanks to abundant training data, French has a decent BLEU score of 29 8/25 4  German doesn't perform well, because of less richness of content (transcripts)  The other languages are low resource in CoVoST and it is difficult to train decent models without additional data or pre-training techniques 
Baseline Results ::: Speech Translation (ST)
CoVoST is a many-to-one multilingual ST corpus  While end-to-end one-to-many and many-to-many multilingual ST models have been explored very recently BIBREF8, BIBREF9, many-to-one multilingual models, to our knowledge, have not  We hence use CoVoST to examine this setting  Table TABREF22 and TABREF23 show the BLEU scores for both bilingual and multilingual end-to-end ST models trained on CoVoST  We observe that combining speeches from multiple languages is consistently bringing gains to low-resource languages (all besides French and German)  This includes combinations of distant languages, such as Ru+Fr, Tr+Fr and Zh+Fr  Moreover, some combinations do bring gains to high-resource language (French) as well: Es+Fr, Tr+Fr and Mn+Fr  We simply provide the most basic many-to-one multilingual baselines here, and leave the full exploration of the best configurations to future work  Finally, we note that for some language pairs, absolute BLEU numbers are relatively low as we restrict model training to the supervised data  We encourage the community to improve upon those baselines, for example by leveraging semi-supervised training 
Baseline Results ::: Multi-Speaker Evaluation
In CoVoST, large portion of transcripts are covered by multiple speakers with different genders, accents and age groups  Besides the standard corpus-level BLEU scores, we also want to evaluate model output variance on the same content (transcript) but different speakers  We hence propose to group samples (and their sentence BLEU scores) by transcript, and then calculate average per-group mean and average coefficient of variation defined as follows:
and
where $G$ is the set of sentence BLEU scores grouped by transcript and $G^{\prime } = \lbrace g | g\in G, |g|>1, \textrm {Mean}(g) > 0 \rbrace $ 
$\textrm {BLEU}_{MS}$ provides a normalized quality score as oppose to corpus-level BLEU or unnormalized average of sentence BLEU  And $\textrm {CoefVar}_{MS}$ is a standardized measure of model stability against different speakers (the lower the better)  Table TABREF24 shows the $\textrm {BLEU}_{MS}$ and $\textrm {CoefVar}_{MS}$ of our ST models on CoVoST test set  We see that German and Persian have the worst $\textrm {CoefVar}_{MS}$ (least stable) given their rich speaker diversity in the test set and relatively small train set (see also Figure FIGREF6 and Table TABREF2)  Dutch also has poor $\textrm {CoefVar}_{MS}$ because of the lack of training data  Multilingual models are consistantly more stable on low-resource languages  Ru+Fr, Tr+Fr, Fa+Fr and Zh+Fr even have better $\textrm {CoefVar}_{MS}$ than all individual languages 
Conclusion
We introduce a multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  We also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation  CoVoST is free to use with a CC0 license, and the additional Tatoeba evaluation samples are also CC-licensed 

ground_truth: No

No