prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what was the baseline?
Context 1: Introduction
Neural machine translation (NMT) BIBREF0 , BIBREF1 , BIBREF2 has enabled end-to-end training of a translation system without needing to deal with word alignments, translation rules, and complicated decoding algorithms, which are the characteristics of phrase-based statistical machine translation (PBSMT) BIBREF3   Although NMT can be significantly better than PBSMT in resource-rich scenarios, PBSMT performs better in low-resource scenarios BIBREF4   Only by exploiting cross-lingual transfer learning techniques BIBREF5 , BIBREF6 , BIBREF7 , can the NMT performance approach PBSMT performance in low-resource scenarios 
However, such methods usually require an NMT model trained on a resource-rich language pair like French INLINEFORM0 English (parent), which is to be fine-tuned for a low-resource language pair like Uzbek INLINEFORM1 English (child)  On the other hand, multilingual approaches BIBREF8 propose to train a single model to translate multiple language pairs  However, these approaches are effective only when the parent target or source language is relatively resource-rich like English (En)  Furthermore, the parents and children models should be trained on similar domains; otherwise, one has to take into account an additional problem of domain adaptation BIBREF9  
In this paper, we work on a linguistically distant and thus challenging language pair Japanese INLINEFORM0 Russian (Ja INLINEFORM1 Ru) which has only 12k lines of news domain parallel corpus and hence is extremely resource-poor  Furthermore, the amount of indirect in-domain parallel corpora, i e , Ja INLINEFORM2 En and Ru INLINEFORM3 En, are also small  As we demonstrate in Section SECREF4 , this severely limits the performance of prominent low-resource techniques, such as multilingual modeling, back-translation, and pivot-based PBSMT  To remedy this, we propose a novel multistage fine-tuning method for NMT that combines multilingual modeling BIBREF8 and domain adaptation BIBREF9  
We have addressed two important research questions (RQs) in the context of extremely low-resource machine translation (MT) and our explorations have derived rational contributions (CTs) as follows:
To the best of our knowledge, we are the first to perform such an extensive evaluation of extremely low-resource MT problem and propose a novel multilingual multistage fine-tuning approach involving multilingual modeling and domain adaptation to address it 
Our Japanese–Russian Setting
In this paper, we deal with Ja INLINEFORM0 Ru news translation  This language pair is very challenging because the languages involved have completely different writing system, phonology, morphology, grammar, and syntax  Among various domains, we experimented with translations in the news domain, considering the importance of sharing news between different language speakers  Moreover, news domain is one of the most challenging tasks, due to large presence of out-of-vocabulary (OOV) tokens and long sentences  To establish and evaluate existing methods, we also involved English as the third language  As direct parallel corpora are scarce, involving a language such as English for pivoting is quite common BIBREF10  
There has been no clean held-out parallel data for Ja INLINEFORM0 Ru and Ja INLINEFORM1 En news translation  Therefore, we manually compiled development and test sets using News Commentary data as a source  Since the given Ja INLINEFORM2 Ru and Ja INLINEFORM3 En data share many lines in the Japanese side, we first compiled tri-text data  Then, from each line, corresponding parts across languages were manually identified, and unaligned parts were split off into a new line  Note that we have never merged two or more lines  As a result, we obtained 1,654 lines of data comprising trilingual, bilingual, and monolingual segments (mainly sentences) as summarized in Table TABREF8   Finally, for the sake of comparability, we randomly chose 600 trilingual sentences to create a test set, and concatenated the rest of them and bilingual sentences to form development sets 
Our manually aligned development and test sets are publicly available 
Related Work
koehn-knowles:2017:NMT showed that NMT is unable to handle low-resource language pairs as opposed to PBSMT  Transfer learning approaches BIBREF5 , BIBREF6 , BIBREF7 work well when a large helping parallel corpus is available  This restricts one of the source or the target languages to be English which, in our case, is not possible  Approaches involving bi-directional NMT modeling is shown to drastically improve low-resource translation BIBREF11   However, like most other, this work focuses on translation from and into English 
Remaining options include (a) unsupervised MT BIBREF12 , BIBREF13 , BIBREF14 , (b) parallel sentence mining from non-parallel or comparable corpora BIBREF15 , BIBREF16 , (c) generating pseudo-parallel data BIBREF17 , and (d) MT based on pivot languages BIBREF10   The linguistic distance between Japanese and Russian makes it extremely difficult to learn bilingual knowledge, such as bilingual lexicons and bilingual word embeddings  Unsupervised MT is thus not promising yet, due to its heavy reliance on accurate bilingual word embeddings  Neither does parallel sentence mining, due to the difficulty of obtaining accurate bilingual lexicons  Pseudo-parallel data can be used to augment existing parallel corpora for training, and previous work has reported that such data generated by so-called back-translation can substantially improve the quality of NMT  However, this approach requires base MT systems that can generate somewhat accurate translations  It is thus infeasible in our scenario, because we can obtain only a weak system which is the consequence of an extremely low-resource situation  MT based on pivot languages requires large in-domain parallel corpora involving the pivot languages  This technique is thus infeasible, because the in-domain parallel corpora for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs are also extremely limited, whereas there are large parallel corpora in other domains  Section SECREF4 empirically confirms the limit of these existing approaches 
Fortunately, there are two useful transfer learning solutions using NMT: (e) multilingual modeling to incorporate multiple language pairs into a single model BIBREF8 and (f) domain adaptation to incorporate out-of-domain data BIBREF9   In this paper, we explore a novel method involving step-wise fine-tuning to combine these two methods  By improving the translation quality in this way, we can also increase the likelihood of pseudo-parallel data being useful to further improve translation quality 
Limit of Using only In-domain Data
This section answers our first research question, [RQ1], about the translation quality that we can achieve using existing methods and in-domain parallel and monolingual data  We then use the strongest model to conduct experiments on generating and utilizing back-translated pseudo-parallel data for augmenting NMT  Our intention is to empirically identify the most effective practices as well as recognize the limitations of relying only on in-domain parallel corpora 
Data
To train MT systems among the three languages, i e , Japanese, Russian, and English, we used all the parallel data provided by Global Voices, more specifically those available at OPUS  Table TABREF9 summarizes the size of train/development/test splits used in our experiments  The number of parallel sentences for Ja INLINEFORM0 Ru is 12k, for Ja INLINEFORM1 En is 47k, and for Ru INLINEFORM2 En is 82k  Note that the three corpora are not mutually exclusive: 9k out of 12k sentences in the Ja INLINEFORM3 Ru corpus were also included in the other two parallel corpora, associated with identical English translations  This puts a limit on the positive impact that the helping corpora can have on the translation quality 
Even when one focuses on low-resource language pairs, we often have access to larger quantities of in-domain monolingual data of each language  Such monolingual data are useful to improve quality of MT, for example, as the source of pseudo-parallel data for augmenting training data for NMT BIBREF17 and as the training data for large and smoothed language models for PBSMT BIBREF4   Table TABREF13 summarizes the statistics on our monolingual corpora for several domains including the news domain  Note that we removed from the Global Voices monolingual corpora those sentences that are already present in the parallel corpus 
https://dumps wikimedia org/backup-index html (20180501) http://www statmt org/wmt18/translation-task html https://www yomiuri co jp/database/glossary/ http://www cs jhu edu/~kevinduh/a/multitarget-tedtalks/ http://opus nlpl eu/Tatoeba-v2 php
We tokenized English and Russian sentences using tokenizer perl of Moses BIBREF3   To tokenize Japanese sentences, we used MeCab with the IPA dictionary  After tokenization, we eliminated duplicated sentence pairs and sentences with more than 100 tokens for all the languages 
MT Methods Examined
We began with evaluating standard MT paradigms, i e , PBSMT BIBREF3 and NMT BIBREF1   As for PBSMT, we also examined two advanced methods: pivot-based translation relying on a helping language BIBREF10 and induction of phrase tables from monolingual data BIBREF14  
As for NMT, we compared two types of encoder-decoder architectures: attentional RNN-based model (RNMT) BIBREF2 and the Transformer model BIBREF18   In addition to standard uni-directional modeling, to cope with the low-resource problem, we examined two multi-directional models: bi-directional model BIBREF11 and multi-to-multi (M2M) model BIBREF8  
After identifying the best model, we also examined the usefulness of a data augmentation method based on back-translation BIBREF17  
First, we built a PBSMT system for each of the six translation directions  We obtained phrase tables from parallel corpus using SyMGIZA++ with the grow-diag-final heuristics for word alignment, and Moses for phrase pair extraction  Then, we trained a bi-directional MSD (monotone, swap, and discontinuous) lexicalized reordering model  We also trained three 5-gram language models, using KenLM on the following monolingual data: (1) the target side of the parallel data, (2) the concatenation of (1) and the monolingual data from Global Voices, and (3) the concatenation of (1) and all monolingual data in the news domain in Table TABREF13  
Subsequently, using English as the pivot language, we examined the following three types of pivot-based PBSMT systems BIBREF10 , BIBREF19 for each of Ja INLINEFORM0 Ru and Ru INLINEFORM1 Ja 
2-step decoding using the source-to-English and English-to-target systems 
Obtain a new phrase table from synthetic parallel data generated by translating English side of the target–English training parallel data to the source language with the English-to-source system 
Compile a new phrase table combining those for the source-to-English and English-to-target systems 
Among these three, triangulation is the most computationally expensive method  Although we had filtered the component phrase tables using the statistical significance pruning method BIBREF20 , triangulation can generate an enormous number of phrase pairs  To reduce the computational cost during decoding and the negative effects of potentially noisy phrase pairs, we retained for each source phrase INLINEFORM0 only the INLINEFORM1 -best translations INLINEFORM2 according to the forward translation probability INLINEFORM3 calculated from the conditional probabilities in the component models as defined in utiyama:07  For each of the retained phrase pairs, we also calculated the backward translation probability, INLINEFORM4 , and lexical translation probabilities, INLINEFORM5 and INLINEFORM6 , in the same manner as INLINEFORM7  
We also investigated the utility of recent advances in unsupervised MT  Even though we began with a publicly available implementation of unsupervised PBSMT BIBREF13 , it crashed due to unknown reasons  We therefore followed another method described in marie:usmt-unmt  Instead of short INLINEFORM0 -grams BIBREF12 , BIBREF13 , we collected a set of phrases in Japanese and Russian from respective monolingual data using the word2phrase algorithm BIBREF21 , as in marie:usmt-unmt  To reduce the complexity, we used randomly selected 10M monolingual sentences, and 300k most frequent phrases made of words among the 300k most frequent words  For each source phrase INLINEFORM1 , we selected 300-best target phrases INLINEFORM2 according to the translation probability as in D18-1549: INLINEFORM3 where INLINEFORM4 stands for a bilingual embedding of a given phrase, obtained through averaging bilingual embeddings of constituent words learned from the two monolingual data using fastText and vecmap  For each of the retained phrase pair, INLINEFORM5 was computed analogously  We also computed lexical translation probabilities relying on those learned from the given small parallel corpus 
Up to four phrase tables were jointly exploited by the multiple decoding path ability of Moses  Weights for the features were tuned using KB-MIRA BIBREF22 on the development set; we took the best weights after 15 iterations  Two hyper-parameters, namely, INLINEFORM0 for the number of pivot-based phrase pairs per source phrase and INLINEFORM1 for distortion limit, were determined by a grid search on INLINEFORM2 and INLINEFORM3   In contrast, we used predetermined hyper-parameters for phrase table induction from monolingual data, following the convention: 200 for the dimension of word and phrase embeddings and INLINEFORM4  
We used the open-source implementation of the RNMT and the Transformer models in tensor2tensor  A uni-directional model for each of the six translation directions was trained on the corresponding parallel corpus  Bi-directional and M2M models were realized by adding an artificial token that specifies the target language to the beginning of each source sentence and shuffling the entire training data BIBREF8  
Table TABREF22 contains some specific hyper-parameters for our baseline NMT models  The hyper-parameters not mentioned in this table used the default values in tensor2tensor  For M2M systems, we over-sampled Ja INLINEFORM0 Ru and Ja INLINEFORM1 En training data so that their sizes match the largest Ru INLINEFORM2 En data  To reduce the number of unknown words, we used tensor2tensor's internal sub-word segmentation mechanism  Since we work in a low-resource setting, we used shared sub-word vocabularies of size 16k for the uni- and bi-directional models and 32k for the M2M models  The number of training iterations was determined by early-stopping: we evaluated our models on the development set every 1,000 updates, and stopped training if BLEU score for the development set was not improved for 10,000 updates (10 check-points)  Note that the development set was created by concatenating those for the individual translation directions without any over-sampling 
Having trained the models, we averaged the last 10 check-points and decoded the test sets with a beam size of 4 and a length penalty which was tuned by a linear search on the BLEU score for the development set 
Similarly to PBSMT, we also evaluated “Cascade” and “Synthesize” methods with uni-directional NMT models 
Results
We evaluated MT models using case-sensitive and tokenized BLEU BIBREF23 on test sets, using Moses's multi-bleu perl  Statistical significance ( INLINEFORM0 ) on the difference of BLEU scores was tested by Moses's bootstrap-hypothesis-difference-significance pl 
Tables TABREF27 and TABREF31 show BLEU scores of all the models, except the NMT systems augmented with back-translations  Whereas some models achieved reasonable BLEU scores for Ja INLINEFORM0 En and Ru INLINEFORM1 En translation, all the results for Ja INLINEFORM2 Ru, which is our main concern, were abysmal 
Among the NMT models, Transformer models (b INLINEFORM0 ) were proven to be better than RNMT models (a INLINEFORM1 )  RNMT models could not even outperform the uni-directional PBSMT models (c1)  M2M models (a3) and (b3) outperformed their corresponding uni- and bi-directional models in most cases  It is worth noting that in this extremely low-resource scenario, BLEU scores of the M2M RNMT model for the largest language pair, i e , Ru INLINEFORM2 En, were lower than those of the uni- and bi-directional RNMT models as in TACL1081  In contrast, with the M2M Transformer model, Ru INLINEFORM3 En also benefited from multilingualism 
Standard PBSMT models (c1) achieved higher BLEU scores than uni-directional NMT models (a1) and (b1), as reported by koehn-knowles:2017:NMT, whereas they underperform the M2M Transformer NMT model (b3)  As shown in Table TABREF31 , pivot-based PBSMT systems always achieved higher BLEU scores than (c1)  The best model with three phrase tables, labeled “Synthesize / Triangulate / Gold,” brought visible BLEU gains with substantial reduction of OOV tokens (3047 INLINEFORM0 1180 for Ja INLINEFORM1 Ru, 4463 INLINEFORM2 1812 for Ru INLINEFORM3 Ja)  However, further extension with phrase tables induced from monolingual data did not push the limit, despite their high coverage; only 336 and 677 OOV tokens were left for the two translation directions, respectively  This is due to the poor quality of the bilingual word embeddings used to extract the phrase table, as envisaged in Section SECREF3  
None of pivot-based approaches with uni-directional NMT models could even remotely rival the M2M Transformer NMT model (b3) 
Table TABREF46 shows the results of our multistage fine-tuning, where the IDs of each row refer to those described in Section SECREF41   First of all, the final models of our multistage fine-tuning, i e , V and VII, achieved significantly higher BLEU scores than (b3) in Table TABREF27 , a weak baseline without using any monolingual data, and #10 in Table TABREF33 , a strong baseline established with monolingual data 
The performance of the initial model (I) depends on the language pair  For Ja INLINEFORM0 Ru pair, it cannot achieve minimum level of quality since the model has never seen parallel data for this pair  The performance on Ja INLINEFORM1 En pair was much lower than the two baseline models, reflecting the crucial mismatch between training and testing domains  In contrast, Ru INLINEFORM2 En pair benefited the most and achieved surprisingly high BLEU scores  The reason might be due to the proximity of out-of-domain training data and in-domain test data 
The first fine-tuning stage significantly pushed up the translation quality for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs, in both cases with fine-tuning (II) and mixed fine-tuning (III)  At this stage, both models performed only poorly for Ja INLINEFORM2 Ru pair as they have not yet seen Ja INLINEFORM3 Ru parallel data  Either model had a consistent advantage to the other 
When these models were further fine-tuned only on the in-domain Ja INLINEFORM0 Ru parallel data (IV and VI), we obtained translations of better quality than the two baselines for Ja INLINEFORM1 Ru pair  However, as a result of complete ignorance of Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs, the models only produced translations of poor quality for these language pairs  In contrast, mixed fine-tuning for the second fine-tuning stage (V and VII) resulted in consistently better models than conventional fine-tuning (IV and VI), irrespective of the choice at the first stage, thanks to the gradual shift of parameters realized by in-domain Ja INLINEFORM4 En and Ru INLINEFORM5 En parallel data  Unfortunately, the translation quality for Ja INLINEFORM6 En and Ru INLINEFORM7 En pairs sometimes degraded from II and III  Nevertheless, the BLEU scores still retain the large margin against two baselines 
The last three rows in Table TABREF46 present BLEU scores obtained by the methods with fewer fine-tuning steps  The most naive model I', trained on the balanced mixture of whole five types of corpora from scratch, and the model II', obtained through a single-step conventional fine-tuning of I on all the in-domain data, achieved only BLEU scores consistently worse than VII  In contrast, when we merged our two fine-tuning steps into a single mixed fine-tuning on I, we obtained a model III' which is better for the Ja INLINEFORM0 Ru pair than VII  Nevertheless, they are still comparable to those of VII and the BLEU scores for the other two language pairs are much lower than VII  As such, we conclude that our multistage fine-tuning leads to a more robust in-domain multilingual model 
Augmentation with Back-translation
Given that the M2M Transformer NMT model (b3) achieved best results for most of the translation directions and competitive results for the rest, we further explored it through back-translation 
We examined the utility of pseudo-parallel data for all the six translation directions, unlike the work of lakew2017improving and lakew2018comparison, which concentrate only on the zero-shot language pair, and the work of W18-2710, which compares only uni- or bi-directional models  We investigated whether each translation direction in M2M models will benefit from pseudo-parallel data and if so, what kind of improvement takes place 
First, we selected sentences to be back-translated from in-domain monolingual data (Table TABREF13 ), relying on the score proposed by moore:intelligent via the following procedure 
For each language, train two 4-gram language models, using KenLM: an in-domain one on all the Global Voices data, i e , both parallel and monolingual data, and a general-domain one on the concatenation of Global Voices, IWSLT, and Tatoeba data 
For each language, discard sentences containing OOVs according to the in-domain language model 
For each translation direction, select the INLINEFORM0 -best monolingual sentences in the news domain, according to the difference between cross-entropy scores given by the in-domain and general-domain language models 
Whereas W18-2710 exploited monolingual data much larger than parallel data, we maintained a 1:1 ratio between them BIBREF8 , setting INLINEFORM0 to the number of lines of parallel data of given language pair 
Selected monolingual sentences were then translated using the M2M Transformer NMT model (b3) to compose pseudo-parallel data  Then, the pseudo-parallel data were enlarged by over-sampling as summarized in Table TABREF32   Finally, new NMT models were trained on the concatenation of the original parallel and pseudo-parallel data from scratch in the same manner as the previous NMT models with the same hyper-parameters 
Table TABREF33 shows the BLEU scores achieved by several reasonable combinations of six-way pseudo-parallel data  We observed that the use of all six-way pseudo-parallel data (#10) significantly improved the base model for all the translation directions, except En INLINEFORM0 Ru  A translation direction often benefited when the pseudo-parallel data for that specific direction was used 
Summary
We have evaluated an extensive variation of MT models that rely only on in-domain parallel and monolingual data  However, the resulting BLEU scores for Ja INLINEFORM2 Ru and Ru INLINEFORM3 Ja tasks do not exceed 10 BLEU points, implying the inherent limitation of the in-domain data as well as the difficulty of these translation directions 
Exploiting Large Out-of-Domain Data Involving a Helping Language
The limitation of relying only on in-domain data demonstrated in Section SECREF4 motivates us to explore other types of parallel data  As raised in our second research question, [RQ2], we considered the effective ways to exploit out-of-domain data 
According to language pair and domain, parallel data can be classified into four categories in Table TABREF40   Among all the categories, out-of-domain data for the language pair of interest have been exploited in the domain adaptation scenarios (C INLINEFORM0 A) BIBREF9   However, for Ja INLINEFORM1 Ru, no out-of-domain data is available  To exploit out-of-domain parallel data for Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs instead, we propose a multistage fine-tuning method, which combines two types of transfer learning, i e , domain adaptation for Ja INLINEFORM4 En and Ru INLINEFORM5 En (D INLINEFORM6 B) and multilingual transfer (B INLINEFORM7 A), relying on the M2M model examined in Section SECREF4   We also examined the utility of fine-tuning for iteratively generating and using pseudo-parallel data 
Multistage Fine-tuning
Simply using NMT systems trained on out-of-domain data for in-domain translation is known to perform badly  In order to effectively use large-scale out-of-domain data for our extremely low-resource task, we propose to perform domain adaptation through either (a) conventional fine-tuning, where an NMT system trained on out-of-domain data is fine-tuned only on in-domain data, or (b) mixed fine-tuning BIBREF9 , where pre-trained out-of-domain NMT system is fine-tuned using a mixture of in-domain and out-of-domain data  The same options are available for transferring from Ja INLINEFORM0 En and Ru INLINEFORM1 En to Ja INLINEFORM2 Ru 
We inevitably involve two types of transfer learning, i e , domain adaptation for Ja INLINEFORM0 En and Ru INLINEFORM1 En and multilingual transfer for Ja INLINEFORM2 Ru pair  Among several conceivable options for managing these two problems, we examined the following multistage fine-tuning 
Pre-train a multilingual model only on the Ja INLINEFORM0 En and Ru INLINEFORM1 En out-of-domain parallel data (I), where the vocabulary of the model is determined on the basis of the in-domain parallel data in the same manner as the M2M NMT models examined in Section SECREF4  
Fine-tune the pre-trained model (I) on the in-domain Ja INLINEFORM0 En and Ru INLINEFORM1 En parallel data (fine-tuning, II) or on the mixture of in-domain and out-of-domain Ja INLINEFORM2 En and Ru INLINEFORM3 En parallel data (mixed fine-tuning, III) 
Further fine-tune the models (each of II and III) for Ja INLINEFORM0 Ru on in-domain parallel data for this language pair only (fine-tuning, IV and VI) or on all the in-domain parallel data (mixed fine-tuning, V and VII) 
We chose this way due to the following two reasons  First, we need to take a balance between several different parallel corpora sizes  The other reason is division of labor; we assume that solving each sub-problem one by one should enable gradual shift of parameters 
Data Selection
As an additional large-scale out-of-domain parallel data for Ja INLINEFORM0 En, we used the cleanest 1 5M sentences from the Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF24   As for Ru INLINEFORM1 En, we used the UN and Yandex corpora released for the WMT 2018 News Translation Task  We retained Ru INLINEFORM2 En sentence pairs that contain at least one OOV token in both sides, according to the in-domain language model trained in Section SECREF34   Table TABREF45 summarizes the statistics on the remaining out-of-domain parallel data 
Further Augmentation with Back-translation
Having obtained a better model, we examined again the utility of back-translation  More precisely, we investigated (a) whether the pseudo-parallel data generated by an improved NMT model leads to a further improvement, and (b) whether one more stage of fine-tuning on the mixture of original parallel and pseudo-parallel data will result in a model better than training a new model from scratch as examined in Section SECREF34  
Given an NMT model, we first generated six-way pseudo-parallel data by translating monolingual data  For the sake of comparability, we used the identical monolingual sentences sampled in Section SECREF34   Then, we further fine-tuned the given model on the mixture of the generated pseudo-parallel data and the original parallel data, following the same over-sampling procedure in Section SECREF34   We repeated these steps five times 
Table TABREF51 shows the results  “new #10” in the second row indicates an M2M Transformer model trained from scratch on the mixture of six-way pseudo-parallel data generated by VII and the original parallel data  It achieved higher BLEU scores than #10 in Table TABREF33 thanks to the pseudo-parallel data of better quality, but underperformed the base NMT model VII  In contrast, our fine-tuned model VIII successfully surpassed VII, and one more iteration (IX) further improved BLEU scores for all translation directions, except Ru INLINEFORM0 En  Although further iterations did not necessarily gain BLEU scores, we came to a much higher plateau compared to the results in Section SECREF4  
Conclusion
In this paper, we challenged the difficult task of Ja INLINEFORM0 Ru news domain translation in an extremely low-resource setting  We empirically confirmed the limited success of well-established solutions when restricted to in-domain data  Then, to incorporate out-of-domain data, we proposed a multilingual multistage fine-tuning approach and observed that it substantially improves Ja INLINEFORM1 Ru translation by over 3 7 BLEU points compared to a strong baseline, as summarized in Table TABREF53   This paper contains an empirical comparison of several existing approaches and hence we hope that our paper can act as a guideline to researchers attempting to tackle extremely low-resource translation 
In the future, we plan to confirm further fine-tuning for each of specific translation directions  We will also explore the way to exploit out-of-domain pseudo-parallel data, better domain-adaptation approaches, and additional challenging language pairs 
Acknowledgments
This work was carried out when Aizhan Imankulova was taking up an internship at NICT, Japan  We would like to thank the reviewers for their insightful comments  A part of this work was conducted under the program “Promotion of Global Communications Plan: Research, Development, and Social Demonstration of Multilingual Speech Translation Technology” of the Ministry of Internal Affairs and Communications (MIC), Japan 
Context 2: Introduction
Language model pretraining has advanced the state of the art in many NLP tasks ranging from sentiment analysis, to question answering, natural language inference, named entity recognition, and textual similarity  State-of-the-art pretrained models include ELMo BIBREF1, GPT BIBREF2, and more recently Bidirectional Encoder Representations from Transformers (Bert; BIBREF0)  Bert combines both word and sentence representations in a single very large Transformer BIBREF3; it is pretrained on vast amounts of text, with an unsupervised objective of masked language modeling and next-sentence prediction and can be fine-tuned with various task-specific objectives 
In most cases, pretrained language models have been employed as encoders for sentence- and paragraph-level natural language understanding problems BIBREF0 involving various classification tasks (e g , predicting whether any two sentences are in an entailment relationship; or determining the completion of a sentence among four alternative sentences)  In this paper, we examine the influence of language model pretraining on text summarization  Different from previous tasks, summarization requires wide-coverage natural language understanding going beyond the meaning of individual words and sentences  The aim is to condense a document into a shorter version while preserving most of its meaning  Furthermore, under abstractive modeling formulations, the task requires language generation capabilities in order to create summaries containing novel words and phrases not featured in the source text, while extractive summarization is often defined as a binary classification task with labels indicating whether a text span (typically a sentence) should be included in the summary 
We explore the potential of Bert for text summarization under a general framework encompassing both extractive and abstractive modeling paradigms  We propose a novel document-level encoder based on Bert which is able to encode a document and obtain representations for its sentences  Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers to capture document-level features for extracting sentences  Our abstractive model adopts an encoder-decoder architecture, combining the same pretrained Bert encoder with a randomly-initialized Transformer decoder BIBREF3  We design a new training schedule which separates the optimizers of the encoder and the decoder in order to accommodate the fact that the former is pretrained while the latter must be trained from scratch  Finally, motivated by previous work showing that the combination of extractive and abstractive objectives can help generate better summaries BIBREF4, we present a two-stage approach where the encoder is fine-tuned twice, first with an extractive objective and subsequently on the abstractive summarization task 
We evaluate the proposed approach on three single-document news summarization datasets representative of different writing conventions (e g , important information is concentrated at the beginning of the document or distributed more evenly throughout) and summary styles (e g , verbose vs  more telegraphic; extractive vs  abstractive)  Across datasets, we experimentally show that the proposed models achieve state-of-the-art results under both extractive and abstractive settings  Our contributions in this work are three-fold: a) we highlight the importance of document encoding for the summarization task; a variety of recently proposed techniques aim to enhance summarization performance via copying mechanisms BIBREF5, BIBREF6, BIBREF7, reinforcement learning BIBREF8, BIBREF9, BIBREF10, and multiple communicating encoders BIBREF11  We achieve better results with a minimum-requirement model without using any of these mechanisms; b) we showcase ways to effectively employ pretrained language models in summarization under both extractive and abstractive settings; we would expect any improvements in model pretraining to translate in better summarization in the future; and c) the proposed models can be used as a stepping stone to further improve summarization performance as well as baselines against which new proposals are tested 
Background ::: Pretrained Language Models
Pretrained language models BIBREF1, BIBREF2, BIBREF0, BIBREF12, BIBREF13 have recently emerged as a key technology for achieving impressive gains in a wide variety of natural language tasks  These models extend the idea of word embeddings by learning contextual representations from large-scale corpora using a language modeling objective  Bidirectional Encoder Representations from Transformers (Bert; BIBREF0) is a new language representation model which is trained with a masked language modeling and a “next sentence prediction” task on a corpus of 3,300M words 
The general architecture of Bert is shown in the left part of Figure FIGREF2  Input text is first preprocessed by inserting two special tokens  [cls] is appended to the beginning of the text; the output representation of this token is used to aggregate information from the whole sequence (e g , for classification tasks)  And token [sep] is inserted after each sentence as an indicator of sentence boundaries  The modified text is then represented as a sequence of tokens $X=[w_1,w_2,\cdots ,w_n]$  Each token $w_i$ is assigned three kinds of embeddings: token embeddings indicate the meaning of each token, segmentation embeddings are used to discriminate between two sentences (e g , during a sentence-pair classification task) and position embeddings indicate the position of each token within the text sequence  These three embeddings are summed to a single input vector $x_i$ and fed to a bidirectional Transformer with multiple layers:
where $h^0=x$ are the input vectors; $\mathrm {LN}$ is the layer normalization operation BIBREF14; $\mathrm {MHAtt}$ is the multi-head attention operation BIBREF3; superscript $l$ indicates the depth of the stacked layer  On the top layer, Bert will generate an output vector $t_i$ for each token with rich contextual information 
Pretrained language models are usually used to enhance performance in language understanding tasks  Very recently, there have been attempts to apply pretrained models to various generation problems BIBREF15, BIBREF16  When fine-tuning for a specific task, unlike ELMo whose parameters are usually fixed, parameters in Bert are jointly fine-tuned with additional task-specific parameters 
Background ::: Extractive Summarization
Extractive summarization systems create a summary by identifying (and subsequently concatenating) the most important sentences in a document  Neural models consider extractive summarization as a sentence classification problem: a neural encoder creates sentence representations and a classifier predicts which sentences should be selected as summaries  SummaRuNNer BIBREF7 is one of the earliest neural approaches adopting an encoder based on Recurrent Neural Networks  Refresh BIBREF8 is a reinforcement learning-based system trained by globally optimizing the ROUGE metric  More recent work achieves higher performance with more sophisticated model structures  Latent BIBREF17 frames extractive summarization as a latent variable inference problem; instead of maximizing the likelihood of “gold” standard labels, their latent model directly maximizes the likelihood of human summaries given selected sentences  Sumo BIBREF18 capitalizes on the notion of structured attention to induce a multi-root dependency tree representation of the document while predicting the output summary  NeuSum BIBREF19 scores and selects sentences jointly and represents the state of the art in extractive summarization 
Background ::: Abstractive Summarization
Neural approaches to abstractive summarization conceptualize the task as a sequence-to-sequence problem, where an encoder maps a sequence of tokens in the source document $\mathbf {x} = [x_1,    , x_n]$ to a sequence of continuous representations $\mathbf {z} = [z_1,    , z_n]$, and a decoder then generates the target summary $\mathbf {y} = [y_1,    , y_m]$ token-by-token, in an auto-regressive manner, hence modeling the conditional probability: $p(y_1,    , y_m|x_1,    , x_n)$ 
BIBREF20 and BIBREF21 were among the first to apply the neural encoder-decoder architecture to text summarization  BIBREF6 enhance this model with a pointer-generator network (PTgen) which allows it to copy words from the source text, and a coverage mechanism (Cov) which keeps track of words that have been summarized  BIBREF11 propose an abstractive system where multiple agents (encoders) represent the document together with a hierarchical attention mechanism (over the agents) for decoding  Their Deep Communicating Agents (DCA) model is trained end-to-end with reinforcement learning  BIBREF9 also present a deep reinforced model (DRM) for abstractive summarization which handles the coverage problem with an intra-attention mechanism where the decoder attends over previously generated words  BIBREF4 follow a bottom-up approach (BottomUp); a content selector first determines which phrases in the source document should be part of the summary, and a copy mechanism is applied only to preselected phrases during decoding  BIBREF22 propose an abstractive model which is particularly suited to extreme summarization (i e , single sentence summaries), based on convolutional neural networks and additionally conditioned on topic distributions (TConvS2S) 
Fine-tuning Bert for Summarization ::: Summarization Encoder
Although Bert has been used to fine-tune various NLP tasks, its application to summarization is not as straightforward  Since Bert is trained as a masked-language model, the output vectors are grounded to tokens instead of sentences, while in extractive summarization, most models manipulate sentence-level representations  Although segmentation embeddings represent different sentences in Bert, they only apply to sentence-pair inputs, while in summarization we must encode and manipulate multi-sentential inputs  Figure FIGREF2 illustrates our proposed Bert architecture for Summarization (which we call BertSum) 
In order to represent individual sentences, we insert external [cls] tokens at the start of each sentence, and each [cls] symbol collects features for the sentence preceding it  We also use interval segment embeddings to distinguish multiple sentences within a document  For $sent_i$ we assign segment embedding $E_A$ or $E_B$ depending on whether $i$ is odd or even  For example, for document $[sent_1, sent_2, sent_3, sent_4, sent_5]$, we would assign embeddings $[E_A, E_B, E_A,E_B, E_A]$  This way, document representations are learned hierarchically where lower Transformer layers represent adjacent sentences, while higher layers, in combination with self-attention, represent multi-sentence discourse 
Position embeddings in the original Bert model have a maximum length of 512; we overcome this limitation by adding more position embeddings that are initialized randomly and fine-tuned with other parameters in the encoder 
Fine-tuning Bert for Summarization ::: Extractive Summarization
Let $d$ denote a document containing sentences $[sent_1, sent_2, \cdots , sent_m]$, where $sent_i$ is the $i$-th sentence in the document  Extractive summarization can be defined as the task of assigning a label $y_i \in \lbrace 0, 1\rbrace $ to each $sent_i$, indicating whether the sentence should be included in the summary  It is assumed that summary sentences represent the most important content of the document 
With BertSum, vector $t_i$ which is the vector of the $i$-th [cls] symbol from the top layer can be used as the representation for $sent_i$  Several inter-sentence Transformer layers are then stacked on top of Bert outputs, to capture document-level features for extracting summaries:
where $h^0=\mathrm {PosEmb}(T)$; $T$ denotes the sentence vectors output by BertSum, and function $\mathrm {PosEmb}$ adds sinusoid positional embeddings BIBREF3 to $T$, indicating the position of each sentence 
The final output layer is a sigmoid classifier:
where $h^L_i$ is the vector for $sent_i$ from the top layer (the $L$-th layer ) of the Transformer  In experiments, we implemented Transformers with $L=1, 2, 3$ and found that a Transformer with $L=2$ performed best  We name this model BertSumExt 
The loss of the model is the binary classification entropy of prediction $\hat{y}_i$ against gold label $y_i$  Inter-sentence Transformer layers are jointly fine-tuned with BertSum  We use the Adam optimizer with $\beta _1=0 9$, and $\beta _2=0 999$)  Our learning rate schedule follows BIBREF3 with warming-up ($ \operatorname{\operatorname{warmup}}=10,000$):
Fine-tuning Bert for Summarization ::: Abstractive Summarization
We use a standard encoder-decoder framework for abstractive summarization BIBREF6  The encoder is the pretrained BertSum and the decoder is a 6-layered Transformer initialized randomly  It is conceivable that there is a mismatch between the encoder and the decoder, since the former is pretrained while the latter must be trained from scratch  This can make fine-tuning unstable; for example, the encoder might overfit the data while the decoder underfits, or vice versa  To circumvent this, we design a new fine-tuning schedule which separates the optimizers of the encoder and the decoder 
We use two Adam optimizers with $\beta _1=0 9$ and $\beta _2=0 999$ for the encoder and the decoder, respectively, each with different warmup-steps and learning rates:
where $\tilde{lr}_{\mathcal {E}}=2e^{-3}$, and $\operatorname{\operatorname{warmup}}_{\mathcal {E}}=20,000$ for the encoder and $\tilde{lr}_{\mathcal {D}}=0 1$, and $\operatorname{\operatorname{warmup}}_{\mathcal {D}}=10,000$ for the decoder  This is based on the assumption that the pretrained encoder should be fine-tuned with a smaller learning rate and smoother decay (so that the encoder can be trained with more accurate gradients when the decoder is becoming stable) 
In addition, we propose a two-stage fine-tuning approach, where we first fine-tune the encoder on the extractive summarization task (Section SECREF8) and then fine-tune it on the abstractive summarization task (Section SECREF13)  Previous work BIBREF4, BIBREF23 suggests that using extractive objectives can boost the performance of abstractive summarization  Also notice that this two-stage approach is conceptually very simple, the model can take advantage of information shared between these two tasks, without fundamentally changing its architecture  We name the default abstractive model BertSumAbs and the two-stage fine-tuned model BertSumExtAbs 
Experimental Setup
In this section, we describe the summarization datasets used in our experiments and discuss various implementation details 
Experimental Setup ::: Summarization Datasets
We evaluated our model on three benchmark datasets, namely the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), and XSum BIBREF22  These datasets represent different summary styles ranging from highlights to very brief one sentence summaries  The summaries also vary with respect to the type of rewriting operations they exemplify (e g , some showcase more cut and paste operations while others are genuinely abstractive)  Table TABREF12 presents statistics on these datasets (test set); example (gold-standard) summaries are provided in the supplementary material
Context 3: Introduction
End-to-end speech-to-text translation (ST) has attracted much attention recently BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 given its simplicity against cascading automatic speech recognition (ASR) and machine translation (MT) systems  The lack of labeled data, however, has become a major blocker for bridging the performance gaps between end-to-end models and cascading systems  Several corpora have been developed in recent years  post2013improved introduced a 38-hour Spanish-English ST corpus by augmenting the transcripts of the Fisher and Callhome corpora with English translations  di-gangi-etal-2019-must created the largest ST corpus to date from TED talks but the language pairs involved are out of English only  beilharz2019librivoxdeen created a 110-hour German-English ST corpus from LibriVox audiobooks  godard-etal-2018-low created a Moboshi-French ST corpus as part of a rare language documentation effort  woldeyohannis provided an Amharic-English ST corpus in the tourism domain  boito2019mass created a multilingual ST corpus involving 8 languages from a multilingual speech corpus based on Bible readings BIBREF7  Previous work either involves language pairs out of English, very specific domains, very low resource languages or a limited set of language pairs  This limits the scope of study, including the latest explorations on end-to-end multilingual ST BIBREF8, BIBREF9  Our work is mostly similar and concurrent to iranzosnchez2019europarlst who created a multilingual ST corpus from the European Parliament proceedings  The corpus we introduce has larger speech durations and more translation tokens  It is diversified with multiple speakers per transcript/translation  Finally, we provide additional out-of-domain test sets 
In this paper, we introduce CoVoST, a multilingual ST corpus based on Common Voice BIBREF10 for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora  We also collect an additional evaluation corpus from Tatoeba for French, German, Dutch, Russian and Spanish, resulting in a total of 9 3 hours of speech  Both corpora are created at the sentence level and do not require additional alignments or segmentation  Using the official Common Voice train-development-test split, we also provide baseline models, including, to our knowledge, the first end-to-end many-to-one multilingual ST models  CoVoST is released under CC0 license and free to use  The Tatoeba evaluation samples are also available under friendly CC licenses  All the data can be acquired at https://github com/facebookresearch/covost 
Data Collection and Processing ::: Common Voice (CoVo)
Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license  Contributors record voice clips by reading from a bank of donated sentences  Each voice clip was validated by at least two other users  Most of the sentences are covered by multiple speakers, with potentially different genders, age groups or accents 
Raw CoVo data contains samples that passed validation as well as those that did not  To build CoVoST, we only use the former one and reuse the official train-development-test partition of the validated data  As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages  CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese 
Validated transcripts were sent to professional translators  Note that the translators had access to the transcripts but not the corresponding voice clips since clips would not carry additional information  Since transcripts were duplicated due to multiple speakers, we deduplicated the transcripts before sending them to translators  As a result, different voice clips of the same content (transcript) will have identical translations in CoVoST for train, development and test splits 
In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11  1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU BIBREF12 with the NLTK BIBREF13 implementation between the human translations and the automatic translations produced by a state-of-the-art system BIBREF14 (the French-English system was a Transformer big BIBREF15 separately trained on WMT14)  We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems  Translations with a score that was too low were manually inspected and sent back to the translators when needed  2) We manually inspected examples where the source transcript was identical to the translation  3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data BIBREF14  We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly  4) We computed the ratio of English characters in the translations

ground_truth: pivot-based translation relying on a helping language BIBREF10, nduction of phrase tables from monolingual data BIBREF14 , attentional RNN-based model (RNMT) BIBREF2, Transformer model BIBREF18, bi-directional model BIBREF11, multi-to-multi (M2M) model BIBREF8, back-translation BIBREF17

M2M Transformer

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the datasets used for evaluation?
Context 2: 
Experimental Setup ::: Summarization Datasets ::: CNN/DailyMail
contains news articles and associated highlights, i e , a few bullet points giving a brief overview of the article  We used the standard splits of BIBREF24 for training, validation, and testing (90,266/1,220/1,093 CNN documents and 196,961/12,148/10,397 DailyMail documents)  We did not anonymize entities  We first split sentences with the Stanford CoreNLP toolkit BIBREF26 and pre-processed the dataset following BIBREF6  Input documents were truncated to 512 tokens 
Experimental Setup ::: Summarization Datasets ::: NYT
contains 110,540 articles with abstractive summaries  Following BIBREF27, we split these into 100,834/9,706 training/test examples, based on the date of publication (the test set contains all articles published from January 1, 2007 onward)  We used 4,000 examples from the training as validation set  We also followed their filtering procedure, documents with summaries less than 50 words were removed from the dataset  The filtered test set (NYT50) includes 3,452 examples  Sentences were split with the Stanford CoreNLP toolkit BIBREF26 and pre-processed following BIBREF27  Input documents were truncated to 800 tokens 
Experimental Setup ::: Summarization Datasets ::: XSum
contains 226,711 news articles accompanied with a one-sentence summary, answering the question “What is this article about?”  We used the splits of BIBREF22 for training, validation, and testing (204,045/11,332/11,334) and followed the pre-processing introduced in their work  Input documents were truncated to 512 tokens 
Aside from various statistics on the three datasets, Table TABREF12 also reports the proportion of novel bi-grams in gold summaries as a measure of their abstractiveness  We would expect models with extractive biases to perform better on datasets with (mostly) extractive summaries, and abstractive models to perform more rewrite operations on datasets with abstractive summaries  CNN/DailyMail and NYT are somewhat abstractive, while XSum is highly abstractive 
Experimental Setup ::: Implementation Details
For both extractive and abstractive settings, we used PyTorch, OpenNMT BIBREF28 and the `bert-base-uncased' version of Bert to implement BertSum  Both source and target texts were tokenized with Bert's subwords tokenizer 
Experimental Setup ::: Implementation Details ::: Extractive Summarization
All extractive models were trained for 50,000 steps on 3 GPUs (GTX 1080 Ti) with gradient accumulation every two steps  Model checkpoints were saved and evaluated on the validation set every 1,000 steps  We selected the top-3 checkpoints based on the evaluation loss on the validation set, and report the averaged results on the test set  We used a greedy algorithm similar to BIBREF7 to obtain an oracle summary for each document to train extractive models  The algorithm generates an oracle consisting of multiple sentences which maximize the ROUGE-2 score against the gold summary 
When predicting summaries for a new document, we first use the model to obtain the score for each sentence  We then rank these sentences by their scores from highest to lowest, and select the top-3 sentences as the summary 
During sentence selection we use Trigram Blocking to reduce redundancy BIBREF9  Given summary $S$ and candidate sentence $c$, we skip $c$ if there exists a trigram overlapping between $c$ and $S$  The intuition is similar to Maximal Marginal Relevance (MMR; BIBREF29); we wish to minimize the similarity between the sentence being considered and sentences which have been already selected as part of the summary 
Experimental Setup ::: Implementation Details ::: Abstractive Summarization
In all abstractive models, we applied dropout (with probability $0 1$) before all linear layers; label smoothing BIBREF30 with smoothing factor $0 1$ was also used  Our Transformer decoder has 768 hidden units and the hidden size for all feed-forward layers is 2,048  All models were trained for 200,000 steps on 4 GPUs (GTX 1080 Ti) with gradient accumulation every five steps  Model checkpoints were saved and evaluated on the validation set every 2,500 steps  We selected the top-3 checkpoints based on their evaluation loss on the validation set, and report the averaged results on the test set 
During decoding we used beam search (size 5), and tuned the $\alpha $ for the length penalty BIBREF31 between $0 6$ and 1 on the validation set; we decode until an end-of-sequence token is emitted and repeated trigrams are blocked BIBREF9  It is worth noting that our decoder applies neither a copy nor a coverage mechanism BIBREF6, despite their popularity in abstractive summarization  This is mainly because we focus on building a minimum-requirements model and these mechanisms may introduce additional hyper-parameters to tune  Thanks to the subwords tokenizer, we also rarely observe issues with out-of-vocabulary words in the output; moreover, trigram-blocking produces diverse summaries managing to reduce repetitions 
Results ::: Automatic Evaluation
We evaluated summarization quality automatically using ROUGE BIBREF32  We report unigram and bigram overlap (ROUGE-1 and ROUGE-2) as a means of assessing informativeness and the longest common subsequence (ROUGE-L) as a means of assessing fluency  Table TABREF23 summarizes our results on the CNN/DailyMail dataset  The first block in the table includes the results of an extractive Oracle system as an upper bound  We also present the Lead-3 baseline (which simply selects the first three sentences in a document)  The second block in the table includes various extractive models trained on the CNN/DailyMail dataset (see Section SECREF5 for an overview)  For comparison to our own model, we also implemented a non-pretrained Transformer baseline (TransformerExt) which uses the same architecture as BertSumExt, but with fewer parameters  It is randomly initialized and only trained on the summarization task  TransformerExt has 6 layers, the hidden size is 512, and the feed-forward filter size is 2,048  The model was trained with same settings as in BIBREF3  The third block in Table TABREF23 highlights the performance of several abstractive models on the CNN/DailyMail dataset (see Section SECREF6 for an overview)  We also include an abstractive Transformer baseline (TransformerAbs) which has the same decoder as our abstractive BertSum models; the encoder is a 6-layer Transformer with 768 hidden size and 2,048 feed-forward filter size  The fourth block reports results with fine-tuned Bert models: BertSumExt and its two variants (one without interval embeddings, and one with the large version of Bert), BertSumAbs, and BertSumExtAbs  Bert-based models outperform the Lead-3 baseline which is not a strawman; on the CNN/DailyMail corpus it is indeed superior to several extractive BIBREF7, BIBREF8, BIBREF19 and abstractive models BIBREF6  Bert models collectively outperform all previously proposed extractive and abstractive systems, only falling behind the Oracle upper bound  Among Bert variants, BertSumExt performs best which is not entirely surprising; CNN/DailyMail summaries are somewhat extractive and even abstractive models are prone to copying sentences from the source document when trained on this dataset BIBREF6  Perhaps unsurprisingly we observe that larger versions of Bert lead to performance improvements and that interval embeddings bring only slight gains  Table TABREF24 presents results on the NYT dataset  Following the evaluation protocol in BIBREF27, we use limited-length ROUGE Recall, where predicted summaries are truncated to the length of the gold summaries  Again, we report the performance of the Oracle upper bound and Lead-3 baseline  The second block in the table contains previously proposed extractive models as well as our own Transformer baseline  Compress BIBREF27 is an ILP-based model which combines compression and anaphoricity constraints  The third block includes abstractive models from the literature, and our Transformer baseline  Bert-based models are shown in the fourth block  Again, we observe that they outperform previously proposed approaches  On this dataset, abstractive Bert models generally perform better compared to BertSumExt, almost approaching Oracle performance 
Table TABREF26 summarizes our results on the XSum dataset  Recall that summaries in this dataset are highly abstractive (see Table TABREF12) consisting of a single sentence conveying the gist of the document  Extractive models here perform poorly as corroborated by the low performance of the Lead baseline (which simply selects the leading sentence from the document), and the Oracle (which selects a single-best sentence in each document) in Table TABREF26  As a result, we do not report results for extractive models on this dataset  The second block in Table TABREF26 presents the results of various abstractive models taken from BIBREF22 and also includes our own abstractive Transformer baseline  In the third block we show the results of our Bert summarizers which again are superior to all previously reported models (by a wide margin) 
Results ::: Model Analysis ::: Learning Rates
Recall that our abstractive model uses separate optimizers for the encoder and decoder  In Table TABREF27 we examine whether the combination of different learning rates ($\tilde{lr}_{\mathcal {E}}$ and $\tilde{lr}_{\mathcal {D}}$) is indeed beneficial  Specifically, we report model perplexity on the CNN/DailyMail validation set for varying encoder/decoder learning rates  We can see that the model performs best with $\tilde{lr}_{\mathcal {E}}=2e-3$ and $\tilde{lr}_{\mathcal {D}}=0 1$ 
Results ::: Model Analysis ::: Position of Extracted Sentences
In addition to the evaluation based on ROUGE, we also analyzed in more detail the summaries produced by our model  For the extractive setting, we looked at the position (in the source document) of the sentences which were selected to appear in the summary  Figure FIGREF31 shows the proportion of selected summary sentences which appear in the source document at positions 1, 2, and so on  The analysis was conducted on the CNN/DailyMail dataset for Oracle summaries, and those produced by BertSumExt and the TransformerExt  We can see that Oracle summary sentences are fairly smoothly distributed across documents, while summaries created by TransformerExt mostly concentrate on the first document sentences  BertSumExt outputs are more similar to Oracle summaries, indicating that with the pretrained encoder, the model relies less on shallow position features, and learns deeper document representations 
Results ::: Model Analysis ::: Novel N-grams
We also analyzed the output of abstractive systems by calculating the proportion of novel n-grams that appear in the summaries but not in the source texts  The results are shown in Figure FIGREF33  In the CNN/DailyMail dataset, the proportion of novel n-grams in automatically generated summaries is much lower compared to reference summaries, but in XSum, this gap is much smaller  We also observe that on CNN/DailyMail, BertExtAbs produces less novel n-ngrams than BertAbs, which is not surprising  BertExtAbs is more biased towards selecting sentences from the source document since it is initially trained as an extractive model  The supplementary material includes examples of system output and additional ablation studies 
Results ::: Human Evaluation
In addition to automatic evaluation, we also evaluated system output by eliciting human judgments  We report experiments following a question-answering (QA) paradigm BIBREF33, BIBREF8 which quantifies the degree to which summarization models retain key information from the document  Under this paradigm, a set of questions is created based on the gold summary under the assumption that it highlights the most important document content  Participants are then asked to answer these questions by reading system summaries alone without access to the article  The more questions a system can answer, the better it is at summarizing the document as a whole  Moreover, we also assessed the overall quality of the summaries produced by abstractive systems which due to their ability to rewrite content may produce disfluent or ungrammatical output  Specifically, we followed the Best-Worst Scaling BIBREF34 method where participants were presented with the output of two systems (and the original document) and asked to decide which one was better according to the criteria of Informativeness, Fluency, and Succinctness 
Both types of evaluation were conducted on the Amazon Mechanical Turk platform  For the CNN/DailyMail and NYT datasets we used the same documents (20 in total) and questions from previous work BIBREF8, BIBREF18  For XSum, we randomly selected 20 documents (and their questions) from the release of BIBREF22  We elicited 3 responses per HIT  With regard to QA evaluation, we adopted the scoring mechanism from BIBREF33; correct answers were marked with a score of one, partially correct answers with 0 5, and zero otherwise  For quality-based evaluation, the rating of each system was computed as the percentage of times it was chosen as better minus the times it was selected as worse  Ratings thus range from -1 (worst) to 1 (best) 
Results for extractive and abstractive systems are shown in Tables TABREF37 and TABREF38, respectively  We compared the best performing BertSum model in each setting (extractive or abstractive) against various state-of-the-art systems (whose output is publicly available), the Lead baseline, and the Gold standard as an upper bound  As shown in both tables participants overwhelmingly prefer the output of our model against comparison systems across datasets and evaluation paradigms  All differences between BertSum and comparison models are statistically significant ($p<0 05$), with the exception of TConvS2S (see Table TABREF38; XSum) in the QA evaluation setting 
Conclusions
In this paper, we showcased how pretrained Bert can be usefully applied in text summarization  We introduced a novel document-level encoder and proposed a general framework for both abstractive and extractive summarization  Experimental results across three datasets show that our model achieves state-of-the-art results across the board under automatic and human-based evaluation protocols  Although we mainly focused on document encoding for summarization, in the future, we would like to take advantage the capabilities of Bert for language generation 
Acknowledgments
This research is supported by a Google PhD Fellowship to the first author  We gratefully acknowledge the support of the European Research Council (Lapata, award number 681760, “Translating Multiple Modalities into Text”)  We would also like to thank Shashi Narayan for providing us with the XSum dataset 
Context 3:  We manually inspected examples with a low ratio and sent them back to translators accordingly  5) Finally, we used VizSeq BIBREF16 to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings BIBREF17  Samples with low scores were manually inspected and sent back for translation when needed 
We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint 
Data Collection and Processing ::: Tatoeba (TT)
Tatoeba (TT) is a community built language learning corpus having sentences aligned across multiple languages with the corresponding speech partially available  Its sentences are on average shorter than those in CoVoST (see also Table TABREF2) given the original purpose of language learning  Sentences in TT are licensed under CC BY 2 0 FR and part of the speeches are available under various CC licenses 
We construct an evaluation set from TT (for French, German, Dutch, Russian and Spanish) as a complement to CoVoST development and test sets  We collect (speech, transcript, English translation) triplets for the 5 languages and do not include those whose speech has a broken URL or is not CC licensed  We further filter these samples by sentence lengths (minimum 4 words including punctuations) to reduce the portion of short sentences  This makes the resulting evaluation set closer to real-world scenarios and more challenging 
We run the same quality checks for TT as for CoVoST but we do not find poor quality translations according to our criteria  Finally, we report the overlap between CoVo transcripts and TT sentences in Table TABREF5  We found a minimal overlap, which makes the TT evaluation set a suitable additional test set when training on CoVoST 
Data Analysis ::: Basic Statistics
Basic statistics for CoVoST and TT are listed in Table TABREF2 including (unique) sentence counts, speech durations, speaker demographics (partially available) as well as vocabulary and token statistics (based on Moses-tokenized sentences by sacreMoses) on both transcripts and translations  We see that CoVoST has over 327 hours of German speeches and over 171 hours of French speeches, which, to our knowledge, corresponds to the largest corpus among existing public ST corpora (the second largest is 110 hours BIBREF18 for German and 38 hours BIBREF19 for French)  Moreover, CoVoST has a total of 18 hours of Dutch speeches, to our knowledge, contributing the first public Dutch ST resource  CoVoST also has around 27-hour Russian speeches, 37-hour Italian speeches and 67-hour Persian speeches, which is 1 8 times, 2 5 times and 13 3 times of the previous largest public one BIBREF7  Most of the sentences (transcripts) in CoVoST are covered by multiple speakers with potentially different accents, resulting in a rich diversity in the speeches  For example, there are over 1,000 speakers and over 10 accents in the French and German development / test sets  This enables good coverage of speech variations in both model training and evaluation 
Data Analysis ::: Speaker Diversity
As we can see from Table TABREF2, CoVoST is diversified with a rich set of speakers and accents  We further inspect the speaker demographics in terms of sample distributions with respect to speaker counts, accent counts and age groups, which is shown in Figure FIGREF6, FIGREF7 and FIGREF8  We observe that for 8 of the 11 languages, at least 60% of the sentences (transcripts) are covered by multiple speakers  Over 80% of the French sentences have at least 3 speakers  And for German sentences, even over 90% of them have at least 5 speakers  Similarly, we see that a large portion of sentences are spoken in multiple accents for French, German, Dutch and Spanish  Speakers of each language also spread widely across different age groups (below 20, 20s, 30s, 40s, 50s, 60s and 70s) 
Baseline Results
We provide baselines using the official train-development-test split on the following tasks: automatic speech recognition (ASR), machine translation (MT) and speech translation (ST) 
Baseline Results ::: Experimental Settings ::: Data Preprocessing
We convert raw MP3 audio files from CoVo and TT into mono-channel waveforms, and downsample them to 16,000 Hz  For transcripts and translations, we normalize the punctuation, we tokenize the text with sacreMoses and lowercase it  For transcripts, we further remove all punctuation markers except for apostrophes  We use character vocabularies on all the tasks, with 100% coverage of all the characters  Preliminary experimentation showed that character vocabularies provided more stable training than BPE  For MT, the vocabulary is created jointly on both transcripts and translations  We extract 80-channel log-mel filterbank features, computed with a 25ms window size and 10ms window shift using torchaudio  The features are normalized to 0 mean and 1 0 standard deviation  We remove samples having more than 3,000 frames or more than 256 characters for GPU memory efficiency (less than 25 samples are removed for all languages) 
Baseline Results ::: Experimental Settings ::: Model Training
Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing  For MT, we use a Transformer base architecture BIBREF15, but with 3 encoder layers, 3 decoder layers and 0

ground_truth: CNN/DailyMail news highlights, New York Times Annotated Corpus, XSum

the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), XSum BIBREF22

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How is the quality of the data empirically evaluated? 
Context 3: 3 dropout  We use a batch size of 10,000 frames for ASR and ST, and a batch size of 4,000 tokens for MT  We train all models using Fairseq BIBREF20 for up to 200,000 updates  We use SpecAugment BIBREF21 for ASR and ST to alleviate overfitting 
Baseline Results ::: Experimental Settings ::: Inference and Evaluation
We use a beam size of 5 for all models  We use the best checkpoint by validation loss for MT, and average the last 5 checkpoints for ASR and ST  For MT and ST, we report case-insensitive tokenized BLEU BIBREF22 using sacreBLEU BIBREF23  For ASR, we report word error rate (WER) and character error rate (CER) using VizSeq 
Baseline Results ::: Automatic Speech Recognition (ASR)
For simplicity, we use the same model architecture for ASR and ST, although we do not leverage ASR models to pretrain ST model encoders later  Table TABREF18 shows the word error rate (WER) and character error rate (CER) for ASR models  We see that French and German perform the best given they are the two highest resource languages in CoVoST  The other languages are relatively low resource (especially Turkish and Swedish) and the ASR models are having difficulties to learn from this data 
Baseline Results ::: Machine Translation (MT)
MT models take transcripts (without punctuation) as inputs and outputs translations (with punctuation)  For simplicity, we do not change the text preprocessing methods for MT to correct this mismatch  Moreover, this mismatch also exists in cascading ST systems, where MT model inputs are the outputs of an ASR model  Table TABREF20 shows the BLEU scores of MT models  We notice that the results are consistent with what we see from ASR models  For example thanks to abundant training data, French has a decent BLEU score of 29 8/25 4  German doesn't perform well, because of less richness of content (transcripts)  The other languages are low resource in CoVoST and it is difficult to train decent models without additional data or pre-training techniques 
Baseline Results ::: Speech Translation (ST)
CoVoST is a many-to-one multilingual ST corpus  While end-to-end one-to-many and many-to-many multilingual ST models have been explored very recently BIBREF8, BIBREF9, many-to-one multilingual models, to our knowledge, have not  We hence use CoVoST to examine this setting  Table TABREF22 and TABREF23 show the BLEU scores for both bilingual and multilingual end-to-end ST models trained on CoVoST  We observe that combining speeches from multiple languages is consistently bringing gains to low-resource languages (all besides French and German)  This includes combinations of distant languages, such as Ru+Fr, Tr+Fr and Zh+Fr  Moreover, some combinations do bring gains to high-resource language (French) as well: Es+Fr, Tr+Fr and Mn+Fr  We simply provide the most basic many-to-one multilingual baselines here, and leave the full exploration of the best configurations to future work  Finally, we note that for some language pairs, absolute BLEU numbers are relatively low as we restrict model training to the supervised data  We encourage the community to improve upon those baselines, for example by leveraging semi-supervised training 
Baseline Results ::: Multi-Speaker Evaluation
In CoVoST, large portion of transcripts are covered by multiple speakers with different genders, accents and age groups  Besides the standard corpus-level BLEU scores, we also want to evaluate model output variance on the same content (transcript) but different speakers  We hence propose to group samples (and their sentence BLEU scores) by transcript, and then calculate average per-group mean and average coefficient of variation defined as follows:
and
where $G$ is the set of sentence BLEU scores grouped by transcript and $G^{\prime } = \lbrace g | g\in G, |g|>1, \textrm {Mean}(g) > 0 \rbrace $ 
$\textrm {BLEU}_{MS}$ provides a normalized quality score as oppose to corpus-level BLEU or unnormalized average of sentence BLEU  And $\textrm {CoefVar}_{MS}$ is a standardized measure of model stability against different speakers (the lower the better)  Table TABREF24 shows the $\textrm {BLEU}_{MS}$ and $\textrm {CoefVar}_{MS}$ of our ST models on CoVoST test set  We see that German and Persian have the worst $\textrm {CoefVar}_{MS}$ (least stable) given their rich speaker diversity in the test set and relatively small train set (see also Figure FIGREF6 and Table TABREF2)  Dutch also has poor $\textrm {CoefVar}_{MS}$ because of the lack of training data  Multilingual models are consistantly more stable on low-resource languages  Ru+Fr, Tr+Fr, Fa+Fr and Zh+Fr even have better $\textrm {CoefVar}_{MS}$ than all individual languages 
Conclusion
We introduce a multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  We also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation  CoVoST is free to use with a CC0 license, and the additional Tatoeba evaluation samples are also CC-licensed 

ground_truth: Validated transcripts were sent to professional translators., various sanity checks to the translations,  sanity check the overlaps of train, development and test sets

computed sentence-level BLEU, We manually inspected examples where the source transcript was identical to the translation, measured the perplexity of the translations, computed the ratio of English characters in the translations, calculate similarity scores between transcripts and translations