prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the datasets used for evaluation?
Context 1: Introduction
Language model pretraining has advanced the state of the art in many NLP tasks ranging from sentiment analysis, to question answering, natural language inference, named entity recognition, and textual similarity  State-of-the-art pretrained models include ELMo BIBREF1, GPT BIBREF2, and more recently Bidirectional Encoder Representations from Transformers (Bert; BIBREF0)  Bert combines both word and sentence representations in a single very large Transformer BIBREF3; it is pretrained on vast amounts of text, with an unsupervised objective of masked language modeling and next-sentence prediction and can be fine-tuned with various task-specific objectives 
In most cases, pretrained language models have been employed as encoders for sentence- and paragraph-level natural language understanding problems BIBREF0 involving various classification tasks (e g , predicting whether any two sentences are in an entailment relationship; or determining the completion of a sentence among four alternative sentences)  In this paper, we examine the influence of language model pretraining on text summarization  Different from previous tasks, summarization requires wide-coverage natural language understanding going beyond the meaning of individual words and sentences  The aim is to condense a document into a shorter version while preserving most of its meaning  Furthermore, under abstractive modeling formulations, the task requires language generation capabilities in order to create summaries containing novel words and phrases not featured in the source text, while extractive summarization is often defined as a binary classification task with labels indicating whether a text span (typically a sentence) should be included in the summary 
We explore the potential of Bert for text summarization under a general framework encompassing both extractive and abstractive modeling paradigms  We propose a novel document-level encoder based on Bert which is able to encode a document and obtain representations for its sentences  Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers to capture document-level features for extracting sentences  Our abstractive model adopts an encoder-decoder architecture, combining the same pretrained Bert encoder with a randomly-initialized Transformer decoder BIBREF3  We design a new training schedule which separates the optimizers of the encoder and the decoder in order to accommodate the fact that the former is pretrained while the latter must be trained from scratch  Finally, motivated by previous work showing that the combination of extractive and abstractive objectives can help generate better summaries BIBREF4, we present a two-stage approach where the encoder is fine-tuned twice, first with an extractive objective and subsequently on the abstractive summarization task 
We evaluate the proposed approach on three single-document news summarization datasets representative of different writing conventions (e g , important information is concentrated at the beginning of the document or distributed more evenly throughout) and summary styles (e g , verbose vs  more telegraphic; extractive vs  abstractive)  Across datasets, we experimentally show that the proposed models achieve state-of-the-art results under both extractive and abstractive settings  Our contributions in this work are three-fold: a) we highlight the importance of document encoding for the summarization task; a variety of recently proposed techniques aim to enhance summarization performance via copying mechanisms BIBREF5, BIBREF6, BIBREF7, reinforcement learning BIBREF8, BIBREF9, BIBREF10, and multiple communicating encoders BIBREF11  We achieve better results with a minimum-requirement model without using any of these mechanisms; b) we showcase ways to effectively employ pretrained language models in summarization under both extractive and abstractive settings; we would expect any improvements in model pretraining to translate in better summarization in the future; and c) the proposed models can be used as a stepping stone to further improve summarization performance as well as baselines against which new proposals are tested 
Background ::: Pretrained Language Models
Pretrained language models BIBREF1, BIBREF2, BIBREF0, BIBREF12, BIBREF13 have recently emerged as a key technology for achieving impressive gains in a wide variety of natural language tasks  These models extend the idea of word embeddings by learning contextual representations from large-scale corpora using a language modeling objective  Bidirectional Encoder Representations from Transformers (Bert; BIBREF0) is a new language representation model which is trained with a masked language modeling and a “next sentence prediction” task on a corpus of 3,300M words 
The general architecture of Bert is shown in the left part of Figure FIGREF2  Input text is first preprocessed by inserting two special tokens  [cls] is appended to the beginning of the text; the output representation of this token is used to aggregate information from the whole sequence (e g , for classification tasks)  And token [sep] is inserted after each sentence as an indicator of sentence boundaries  The modified text is then represented as a sequence of tokens $X=[w_1,w_2,\cdots ,w_n]$  Each token $w_i$ is assigned three kinds of embeddings: token embeddings indicate the meaning of each token, segmentation embeddings are used to discriminate between two sentences (e g , during a sentence-pair classification task) and position embeddings indicate the position of each token within the text sequence  These three embeddings are summed to a single input vector $x_i$ and fed to a bidirectional Transformer with multiple layers:
where $h^0=x$ are the input vectors; $\mathrm {LN}$ is the layer normalization operation BIBREF14; $\mathrm {MHAtt}$ is the multi-head attention operation BIBREF3; superscript $l$ indicates the depth of the stacked layer  On the top layer, Bert will generate an output vector $t_i$ for each token with rich contextual information 
Pretrained language models are usually used to enhance performance in language understanding tasks  Very recently, there have been attempts to apply pretrained models to various generation problems BIBREF15, BIBREF16  When fine-tuning for a specific task, unlike ELMo whose parameters are usually fixed, parameters in Bert are jointly fine-tuned with additional task-specific parameters 
Background ::: Extractive Summarization
Extractive summarization systems create a summary by identifying (and subsequently concatenating) the most important sentences in a document  Neural models consider extractive summarization as a sentence classification problem: a neural encoder creates sentence representations and a classifier predicts which sentences should be selected as summaries  SummaRuNNer BIBREF7 is one of the earliest neural approaches adopting an encoder based on Recurrent Neural Networks  Refresh BIBREF8 is a reinforcement learning-based system trained by globally optimizing the ROUGE metric  More recent work achieves higher performance with more sophisticated model structures  Latent BIBREF17 frames extractive summarization as a latent variable inference problem; instead of maximizing the likelihood of “gold” standard labels, their latent model directly maximizes the likelihood of human summaries given selected sentences  Sumo BIBREF18 capitalizes on the notion of structured attention to induce a multi-root dependency tree representation of the document while predicting the output summary  NeuSum BIBREF19 scores and selects sentences jointly and represents the state of the art in extractive summarization 
Background ::: Abstractive Summarization
Neural approaches to abstractive summarization conceptualize the task as a sequence-to-sequence problem, where an encoder maps a sequence of tokens in the source document $\mathbf {x} = [x_1,    , x_n]$ to a sequence of continuous representations $\mathbf {z} = [z_1,    , z_n]$, and a decoder then generates the target summary $\mathbf {y} = [y_1,    , y_m]$ token-by-token, in an auto-regressive manner, hence modeling the conditional probability: $p(y_1,    , y_m|x_1,    , x_n)$ 
BIBREF20 and BIBREF21 were among the first to apply the neural encoder-decoder architecture to text summarization  BIBREF6 enhance this model with a pointer-generator network (PTgen) which allows it to copy words from the source text, and a coverage mechanism (Cov) which keeps track of words that have been summarized  BIBREF11 propose an abstractive system where multiple agents (encoders) represent the document together with a hierarchical attention mechanism (over the agents) for decoding  Their Deep Communicating Agents (DCA) model is trained end-to-end with reinforcement learning  BIBREF9 also present a deep reinforced model (DRM) for abstractive summarization which handles the coverage problem with an intra-attention mechanism where the decoder attends over previously generated words  BIBREF4 follow a bottom-up approach (BottomUp); a content selector first determines which phrases in the source document should be part of the summary, and a copy mechanism is applied only to preselected phrases during decoding  BIBREF22 propose an abstractive model which is particularly suited to extreme summarization (i e , single sentence summaries), based on convolutional neural networks and additionally conditioned on topic distributions (TConvS2S) 
Fine-tuning Bert for Summarization ::: Summarization Encoder
Although Bert has been used to fine-tune various NLP tasks, its application to summarization is not as straightforward  Since Bert is trained as a masked-language model, the output vectors are grounded to tokens instead of sentences, while in extractive summarization, most models manipulate sentence-level representations  Although segmentation embeddings represent different sentences in Bert, they only apply to sentence-pair inputs, while in summarization we must encode and manipulate multi-sentential inputs  Figure FIGREF2 illustrates our proposed Bert architecture for Summarization (which we call BertSum) 
In order to represent individual sentences, we insert external [cls] tokens at the start of each sentence, and each [cls] symbol collects features for the sentence preceding it  We also use interval segment embeddings to distinguish multiple sentences within a document  For $sent_i$ we assign segment embedding $E_A$ or $E_B$ depending on whether $i$ is odd or even  For example, for document $[sent_1, sent_2, sent_3, sent_4, sent_5]$, we would assign embeddings $[E_A, E_B, E_A,E_B, E_A]$  This way, document representations are learned hierarchically where lower Transformer layers represent adjacent sentences, while higher layers, in combination with self-attention, represent multi-sentence discourse 
Position embeddings in the original Bert model have a maximum length of 512; we overcome this limitation by adding more position embeddings that are initialized randomly and fine-tuned with other parameters in the encoder 
Fine-tuning Bert for Summarization ::: Extractive Summarization
Let $d$ denote a document containing sentences $[sent_1, sent_2, \cdots , sent_m]$, where $sent_i$ is the $i$-th sentence in the document  Extractive summarization can be defined as the task of assigning a label $y_i \in \lbrace 0, 1\rbrace $ to each $sent_i$, indicating whether the sentence should be included in the summary  It is assumed that summary sentences represent the most important content of the document 
With BertSum, vector $t_i$ which is the vector of the $i$-th [cls] symbol from the top layer can be used as the representation for $sent_i$  Several inter-sentence Transformer layers are then stacked on top of Bert outputs, to capture document-level features for extracting summaries:
where $h^0=\mathrm {PosEmb}(T)$; $T$ denotes the sentence vectors output by BertSum, and function $\mathrm {PosEmb}$ adds sinusoid positional embeddings BIBREF3 to $T$, indicating the position of each sentence 
The final output layer is a sigmoid classifier:
where $h^L_i$ is the vector for $sent_i$ from the top layer (the $L$-th layer ) of the Transformer  In experiments, we implemented Transformers with $L=1, 2, 3$ and found that a Transformer with $L=2$ performed best  We name this model BertSumExt 
The loss of the model is the binary classification entropy of prediction $\hat{y}_i$ against gold label $y_i$  Inter-sentence Transformer layers are jointly fine-tuned with BertSum  We use the Adam optimizer with $\beta _1=0 9$, and $\beta _2=0 999$)  Our learning rate schedule follows BIBREF3 with warming-up ($ \operatorname{\operatorname{warmup}}=10,000$):
Fine-tuning Bert for Summarization ::: Abstractive Summarization
We use a standard encoder-decoder framework for abstractive summarization BIBREF6  The encoder is the pretrained BertSum and the decoder is a 6-layered Transformer initialized randomly  It is conceivable that there is a mismatch between the encoder and the decoder, since the former is pretrained while the latter must be trained from scratch  This can make fine-tuning unstable; for example, the encoder might overfit the data while the decoder underfits, or vice versa  To circumvent this, we design a new fine-tuning schedule which separates the optimizers of the encoder and the decoder 
We use two Adam optimizers with $\beta _1=0 9$ and $\beta _2=0 999$ for the encoder and the decoder, respectively, each with different warmup-steps and learning rates:
where $\tilde{lr}_{\mathcal {E}}=2e^{-3}$, and $\operatorname{\operatorname{warmup}}_{\mathcal {E}}=20,000$ for the encoder and $\tilde{lr}_{\mathcal {D}}=0 1$, and $\operatorname{\operatorname{warmup}}_{\mathcal {D}}=10,000$ for the decoder  This is based on the assumption that the pretrained encoder should be fine-tuned with a smaller learning rate and smoother decay (so that the encoder can be trained with more accurate gradients when the decoder is becoming stable) 
In addition, we propose a two-stage fine-tuning approach, where we first fine-tune the encoder on the extractive summarization task (Section SECREF8) and then fine-tune it on the abstractive summarization task (Section SECREF13)  Previous work BIBREF4, BIBREF23 suggests that using extractive objectives can boost the performance of abstractive summarization  Also notice that this two-stage approach is conceptually very simple, the model can take advantage of information shared between these two tasks, without fundamentally changing its architecture  We name the default abstractive model BertSumAbs and the two-stage fine-tuned model BertSumExtAbs 
Experimental Setup
In this section, we describe the summarization datasets used in our experiments and discuss various implementation details 
Experimental Setup ::: Summarization Datasets
We evaluated our model on three benchmark datasets, namely the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), and XSum BIBREF22  These datasets represent different summary styles ranging from highlights to very brief one sentence summaries  The summaries also vary with respect to the type of rewriting operations they exemplify (e g , some showcase more cut and paste operations while others are genuinely abstractive)  Table TABREF12 presents statistics on these datasets (test set); example (gold-standard) summaries are provided in the supplementary material 
Experimental Setup ::: Summarization Datasets ::: CNN/DailyMail
contains news articles and associated highlights, i e , a few bullet points giving a brief overview of the article  We used the standard splits of BIBREF24 for training, validation, and testing (90,266/1,220/1,093 CNN documents and 196,961/12,148/10,397 DailyMail documents)  We did not anonymize entities  We first split sentences with the Stanford CoreNLP toolkit BIBREF26 and pre-processed the dataset following BIBREF6  Input documents were truncated to 512 tokens 
Experimental Setup ::: Summarization Datasets ::: NYT
contains 110,540 articles with abstractive summaries  Following BIBREF27, we split these into 100,834/9,706 training/test examples, based on the date of publication (the test set contains all articles published from January 1, 2007 onward)  We used 4,000 examples from the training as validation set  We also followed their filtering procedure, documents with summaries less than 50 words were removed from the dataset  The filtered test set (NYT50) includes 3,452 examples  Sentences were split with the Stanford CoreNLP toolkit BIBREF26 and pre-processed following BIBREF27  Input documents were truncated to 800 tokens 
Experimental Setup ::: Summarization Datasets ::: XSum
contains 226,711 news articles accompanied with a one-sentence summary, answering the question “What is this article about?”  We used the splits of BIBREF22 for training, validation, and testing (204,045/11,332/11,334) and followed the pre-processing introduced in their work  Input documents were truncated to 512 tokens 
Aside from various statistics on the three datasets, Table TABREF12 also reports the proportion of novel bi-grams in gold summaries as a measure of their abstractiveness  We would expect models with extractive biases to perform better on datasets with (mostly) extractive summaries, and abstractive models to perform more rewrite operations on datasets with abstractive summaries  CNN/DailyMail and NYT are somewhat abstractive, while XSum is highly abstractive 
Experimental Setup ::: Implementation Details
For both extractive and abstractive settings, we used PyTorch, OpenNMT BIBREF28 and the `bert-base-uncased' version of Bert to implement BertSum  Both source and target texts were tokenized with Bert's subwords tokenizer 
Experimental Setup ::: Implementation Details ::: Extractive Summarization
All extractive models were trained for 50,000 steps on 3 GPUs (GTX 1080 Ti) with gradient accumulation every two steps  Model checkpoints were saved and evaluated on the validation set every 1,000 steps  We selected the top-3 checkpoints based on the evaluation loss on the validation set, and report the averaged results on the test set  We used a greedy algorithm similar to BIBREF7 to obtain an oracle summary for each document to train extractive models  The algorithm generates an oracle consisting of multiple sentences which maximize the ROUGE-2 score against the gold summary 
When predicting summaries for a new document, we first use the model to obtain the score for each sentence  We then rank these sentences by their scores from highest to lowest, and select the top-3 sentences as the summary 
During sentence selection we use Trigram Blocking to reduce redundancy BIBREF9  Given summary $S$ and candidate sentence $c$, we skip $c$ if there exists a trigram overlapping between $c$ and $S$  The intuition is similar to Maximal Marginal Relevance (MMR; BIBREF29); we wish to minimize the similarity between the sentence being considered and sentences which have been already selected as part of the summary 
Experimental Setup ::: Implementation Details ::: Abstractive Summarization
In all abstractive models, we applied dropout (with probability $0 1$) before all linear layers; label smoothing BIBREF30 with smoothing factor $0 1$ was also used  Our Transformer decoder has 768 hidden units and the hidden size for all feed-forward layers is 2,048  All models were trained for 200,000 steps on 4 GPUs (GTX 1080 Ti) with gradient accumulation every five steps  Model checkpoints were saved and evaluated on the validation set every 2,500 steps  We selected the top-3 checkpoints based on their evaluation loss on the validation set, and report the averaged results on the test set 
During decoding we used beam search (size 5), and tuned the $\alpha $ for the length penalty BIBREF31 between $0 6$ and 1 on the validation set; we decode until an end-of-sequence token is emitted and repeated trigrams are blocked BIBREF9  It is worth noting that our decoder applies neither a copy nor a coverage mechanism BIBREF6, despite their popularity in abstractive summarization  This is mainly because we focus on building a minimum-requirements model and these mechanisms may introduce additional hyper-parameters to tune  Thanks to the subwords tokenizer, we also rarely observe issues with out-of-vocabulary words in the output; moreover, trigram-blocking produces diverse summaries managing to reduce repetitions 
Results ::: Automatic Evaluation
We evaluated summarization quality automatically using ROUGE BIBREF32  We report unigram and bigram overlap (ROUGE-1 and ROUGE-2) as a means of assessing informativeness and the longest common subsequence (ROUGE-L) as a means of assessing fluency  Table TABREF23 summarizes our results on the CNN/DailyMail dataset  The first block in the table includes the results of an extractive Oracle system as an upper bound  We also present the Lead-3 baseline (which simply selects the first three sentences in a document)  The second block in the table includes various extractive models trained on the CNN/DailyMail dataset (see Section SECREF5 for an overview)  For comparison to our own model, we also implemented a non-pretrained Transformer baseline (TransformerExt) which uses the same architecture as BertSumExt, but with fewer parameters  It is randomly initialized and only trained on the summarization task  TransformerExt has 6 layers, the hidden size is 512, and the feed-forward filter size is 2,048  The model was trained with same settings as in BIBREF3  The third block in Table TABREF23 highlights the performance of several abstractive models on the CNN/DailyMail dataset (see Section SECREF6 for an overview)  We also include an abstractive Transformer baseline (TransformerAbs) which has the same decoder as our abstractive BertSum models; the encoder is a 6-layer Transformer with 768 hidden size and 2,048 feed-forward filter size  The fourth block reports results with fine-tuned Bert models: BertSumExt and its two variants (one without interval embeddings, and one with the large version of Bert), BertSumAbs, and BertSumExtAbs  Bert-based models outperform the Lead-3 baseline which is not a strawman; on the CNN/DailyMail corpus it is indeed superior to several extractive BIBREF7, BIBREF8, BIBREF19 and abstractive models BIBREF6  Bert models collectively outperform all previously proposed extractive and abstractive systems, only falling behind the Oracle upper bound  Among Bert variants, BertSumExt performs best which is not entirely surprising; CNN/DailyMail summaries are somewhat extractive and even abstractive models are prone to copying sentences from the source document when trained on this dataset BIBREF6  Perhaps unsurprisingly we observe that larger versions of Bert lead to performance improvements and that interval embeddings bring only slight gains  Table TABREF24 presents results on the NYT dataset  Following the evaluation protocol in BIBREF27, we use limited-length ROUGE Recall, where predicted summaries are truncated to the length of the gold summaries  Again, we report the performance of the Oracle upper bound and Lead-3 baseline  The second block in the table contains previously proposed extractive models as well as our own Transformer baseline  Compress BIBREF27 is an ILP-based model which combines compression and anaphoricity constraints  The third block includes abstractive models from the literature, and our Transformer baseline  Bert-based models are shown in the fourth block  Again, we observe that they outperform previously proposed approaches  On this dataset, abstractive Bert models generally perform better compared to BertSumExt, almost approaching Oracle performance 
Table TABREF26 summarizes our results on the XSum dataset  Recall that summaries in this dataset are highly abstractive (see Table TABREF12) consisting of a single sentence conveying the gist of the document  Extractive models here perform poorly as corroborated by the low performance of the Lead baseline (which simply selects the leading sentence from the document), and the Oracle (which selects a single-best sentence in each document) in Table TABREF26  As a result, we do not report results for extractive models on this dataset  The second block in Table TABREF26 presents the results of various abstractive models taken from BIBREF22 and also includes our own abstractive Transformer baseline  In the third block we show the results of our Bert summarizers which again are superior to all previously reported models (by a wide margin) 
Results ::: Model Analysis ::: Learning Rates
Recall that our abstractive model uses separate optimizers for the encoder and decoder  In Table TABREF27 we examine whether the combination of different learning rates ($\tilde{lr}_{\mathcal {E}}$ and $\tilde{lr}_{\mathcal {D}}$) is indeed beneficial  Specifically, we report model perplexity on the CNN/DailyMail validation set for varying encoder/decoder learning rates  We can see that the model performs best with $\tilde{lr}_{\mathcal {E}}=2e-3$ and $\tilde{lr}_{\mathcal {D}}=0 1$ 
Results ::: Model Analysis ::: Position of Extracted Sentences
In addition to the evaluation based on ROUGE, we also analyzed in more detail the summaries produced by our model  For the extractive setting, we looked at the position (in the source document) of the sentences which were selected to appear in the summary  Figure FIGREF31 shows the proportion of selected summary sentences which appear in the source document at positions 1, 2, and so on  The analysis was conducted on the CNN/DailyMail dataset for Oracle summaries, and those produced by BertSumExt and the TransformerExt  We can see that Oracle summary sentences are fairly smoothly distributed across documents, while summaries created by TransformerExt mostly concentrate on the first document sentences  BertSumExt outputs are more similar to Oracle summaries, indicating that with the pretrained encoder, the model relies less on shallow position features, and learns deeper document representations 
Results ::: Model Analysis ::: Novel N-grams
We also analyzed the output of abstractive systems by calculating the proportion of novel n-grams that appear in the summaries but not in the source texts  The results are shown in Figure FIGREF33  In the CNN/DailyMail dataset, the proportion of novel n-grams in automatically generated summaries is much lower compared to reference summaries, but in XSum, this gap is much smaller  We also observe that on CNN/DailyMail, BertExtAbs produces less novel n-ngrams than BertAbs, which is not surprising  BertExtAbs is more biased towards selecting sentences from the source document since it is initially trained as an extractive model  The supplementary material includes examples of system output and additional ablation studies 
Results ::: Human Evaluation
In addition to automatic evaluation, we also evaluated system output by eliciting human judgments  We report experiments following a question-answering (QA) paradigm BIBREF33, BIBREF8 which quantifies the degree to which summarization models retain key information from the document  Under this paradigm, a set of questions is created based on the gold summary under the assumption that it highlights the most important document content  Participants are then asked to answer these questions by reading system summaries alone without access to the article  The more questions a system can answer, the better it is at summarizing the document as a whole  Moreover, we also assessed the overall quality of the summaries produced by abstractive systems which due to their ability to rewrite content may produce disfluent or ungrammatical output  Specifically, we followed the Best-Worst Scaling BIBREF34 method where participants were presented with the output of two systems (and the original document) and asked to decide which one was better according to the criteria of Informativeness, Fluency, and Succinctness 
Both types of evaluation were conducted on the Amazon Mechanical Turk platform  For the CNN/DailyMail and NYT datasets we used the same documents (20 in total) and questions from previous work BIBREF8, BIBREF18  For XSum, we randomly selected 20 documents (and their questions) from the release of BIBREF22  We elicited 3 responses per HIT  With regard to QA evaluation, we adopted the scoring mechanism from BIBREF33; correct answers were marked with a score of one, partially correct answers with 0 5, and zero otherwise  For quality-based evaluation, the rating of each system was computed as the percentage of times it was chosen as better minus the times it was selected as worse  Ratings thus range from -1 (worst) to 1 (best) 
Results for extractive and abstractive systems are shown in Tables TABREF37 and TABREF38, respectively  We compared the best performing BertSum model in each setting (extractive or abstractive) against various state-of-the-art systems (whose output is publicly available), the Lead baseline, and the Gold standard as an upper bound  As shown in both tables participants overwhelmingly prefer the output of our model against comparison systems across datasets and evaluation paradigms  All differences between BertSum and comparison models are statistically significant ($p<0 05$), with the exception of TConvS2S (see Table TABREF38; XSum) in the QA evaluation setting 
Conclusions
In this paper, we showcased how pretrained Bert can be usefully applied in text summarization  We introduced a novel document-level encoder and proposed a general framework for both abstractive and extractive summarization  Experimental results across three datasets show that our model achieves state-of-the-art results across the board under automatic and human-based evaluation protocols  Although we mainly focused on document encoding for summarization, in the future, we would like to take advantage the capabilities of Bert for language generation 
Acknowledgments
This research is supported by a Google PhD Fellowship to the first author  We gratefully acknowledge the support of the European Research Council (Lapata, award number 681760, “Translating Multiple Modalities into Text”)  We would also like to thank Shashi Narayan for providing us with the XSum dataset 

ground_truth: CNN/DailyMail news highlights, New York Times Annotated Corpus, XSum

the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), XSum BIBREF22

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What metrics are used for evaluation?
Context 2: Introduction
Pre-trained models BIBREF0, BIBREF1 have received much of attention recently thanks to their impressive results in many down stream NLP tasks  Additionally, multilingual pre-trained models enable many NLP applications for other languages via zero-short cross-lingual transfer  Zero-shot cross-lingual transfer has shown promising results for rapidly building applications for low resource languages  BIBREF2 show the potential of multilingual-BERT BIBREF0 in zero-shot transfer for a large number of languages from different language families on five NLP tasks, namely, natural language inference, document classification, named entity recognition, part-of-speech tagging, and dependency parsing 
Although multilingual models are an important ingredient for enhancing language technology in many languages, recent research on improving pre-trained models puts much emphasis on English BIBREF3, BIBREF4, BIBREF5  The current state of affairs makes it difficult to translate advancements in pre-training from English to non-English languages  To our best knowledge, there are only three available multilingual pre-trained models to date: (1) the multilingual-BERT (mBERT) that supports 104 languages, (2) cross-lingual language model BIBREF6 that supports 100 languages, and (3) Language Agnostic SEntence Representations BIBREF7 that supports 93 languages  Among the three models, LASER is based on neural machine translation approach and strictly requires parallel data to train 
Do multilingual models always need to be trained from scratch? Can we transfer linguistic knowledge learned by English pre-trained models to other languages? In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way BIBREF8  As the first step, we focus on building a bilingual language model (LM) of English and a target language  Starting from a pre-trained English LM, we learn the target language specific parameters (i e , word embeddings), while keeping the encoder layers of the pre-trained English LM fixed  We then fine-tune both English and target model to obtain the bilingual LM  We apply our approach to autoencoding language models with masked language model objective and show the advantage of the proposed approach in zero-shot transfer  Our main contributions in this work are:
We propose a fast adaptation method for obtaining a bilingual BERT$_{\textsc {base}}$ of English and a target language within a day using one Tesla V100 16GB GPU 
We evaluate our bilingual LMs for six languages on two zero-shot cross-lingual transfer tasks, namely natural language inference BIBREF9 and universal dependency parsing  We show that our models offer competitive performance or even better that mBERT 
We illustrate that our bilingual LMs can serve as an excellent feature extractor in supervised dependency parsing task 
Bilingual Pre-trained LMs
We first provide some background of pre-trained language models  Let $_e$ be English word-embeddings and $\Psi ()$ be the Transformer BIBREF10 encoder with parameters $$  Let $_{w_i}$ denote the embedding of word $w_i$ (i e , $_{w_i} = _e[w_1]$)  We omit positional embeddings and bias for clarity  A pre-trained LM typically performs the following computations: (i) transform a sequence of input tokens to contextualized representations $[_{w_1},\dots ,_{w_n}] = \Psi (_{w_1}, \dots , _{w_n}; )$, and (ii) predict an output word $y_i$ at $i^{\text{th}}$ position $p(y_i | _{w_i}) \propto \exp (_{w_i}^\top _{y_i})$ 
Autoencoding LM BIBREF0 corrupts some input tokens $w_i$ by replacing them with a special token [MASK]  It then predicts the original tokens $y_i = w_i$ from the corrupted tokens  Autoregressive LM BIBREF3 predicts the next token ($y_i = w_{i+1}$) given all the previous tokens  The recently proposed XLNet model BIBREF5 is an autoregressive LM that factorizes output with all possible permutations, which shows empirical performance improvement over GPT-2 due to the ability to capture bidirectional context  Here we assume that the encoder performs necessary masking with respect to each training objective 
Given an English pre-trained LM, we wish to learn a bilingual LM for English and a given target language $f$ under a limited computational resource budget  To quickly build a bilingual LM, we directly adapt the English pre-traind model to the target model  Our approach consists of three steps  First, we initialize target language word-embeddings $_f$ in the English embedding space such that embeddings of a target word and its English equivalents are close together (§SECREF8)  Next, we create a target LM from the target embeddings and the English encoder $\Psi ()$  We then fine-tune target embeddings while keeping $\Psi ()$ fixed (§SECREF14)  Finally, we construct a bilingual LM of $_e$, $_f$, and $\Psi ()$ and fine-tune all the parameters (§SECREF15)  Figure FIGREF7 illustrates the last two steps in our approach 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings
Our approach to learn the initial foreign word embeddings $_f \in ^{|V_f| \times d}$ is based on the idea of mapping the trained English word embeddings $_e \in ^{|V_e| \times d}$ to $_f$ such that if a foreign word and an English word are similar in meaning then their embeddings are similar  Borrowing the idea of universal lexical sharing from BIBREF11, we represent each foreign word embedding $_f[i] \in ^d$ as a linear combination of English word embeddings $_e[j] \in ^d$
where $_i\in ^{|V_e|}$ is a sparse vector and $\sum _j^{|V_e|} \alpha _{ij} = 1$ 
In this step of initializing foreign embeddings, having a good estimation of $$ could speed of the convergence when tuning the foreign model and enable zero-shot transfer (§SECREF5)  In the following, we discuss how to estimate $_i\;\forall i\in \lbrace 1,2, \dots , |V_f|\rbrace $ under two scenarios: (i) we have parallel data of English-foreign, and (ii) we only rely on English and foreign monolingual data 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Parallel Corpus
Given an English-foreign parallel corpus, we can estimate word translation probability $p(e\,|\,f)$ for any (English-foreign) pair $(e, f)$ using popular word-alignment BIBREF12 toolkits such as fast-align BIBREF13  We then assign:
Since $_i$ is estimated from word alignment, it is a sparse vector 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Monolingual Corpus
For low resource languages, parallel data may not be available  In this case, we rely only on monolingual data (e g , Wikipedias)  We estimate word translation probabilities from word embeddings of the two languages  Word vectors of these languages can be learned using fastText BIBREF14 and then are aligned into a shared space with English BIBREF15, BIBREF16  Unlike learning contextualized representations, learning word vectors is fast and computationally cheap  Given the aligned vectors $\bar{}_f$ of foreign and $\bar{}_e$ of English, we calculate the word translation matrix $\in ^{|V_f|\times |V_e|}$ as
Here, we use $\mathrm {sparsemax}$ BIBREF17 instead of softmax  Sparsemax is a sparse version of softmax and it puts zero probabilities on most of the word in the English vocabulary except few English words that are similar to a given foreign word  This property is desirable in our approach since it leads to a better initialization of the foreign embeddings 
Bilingual Pre-trained LMs ::: Fine-tuning Target Embeddings
After initializing foreign word-embeddings, we replace English word-embeddings in the English pre-trained LM with foreign word-embeddings to obtain the foreign LM  We then fine-tune only foreign word-embeddings on monolingual data  The training objective is the same as the training objective of the English pre-trained LM (i e , masked LM for BERT)  Since the trained encoder $\Psi ()$ is good at capturing association, the purpose of this step is to further optimize target embeddings such that the target LM can utilized the trained encoder for association task  For example, if the words Albert Camus presented in a French input sequence, the self-attention in the encoder more likely attends to words absurde and existentialisme once their embeddings are tuned 
Bilingual Pre-trained LMs ::: Fine-tuning Bilingual LM
We create a bilingual LM by plugging foreign language specific parameters to the pre-trained English LM (Figure FIGREF7)  The new model has two separate embedding layers and output layers, one for English and one for foreign language  The encoder layer in between is shared  We then fine-tune this model using English and foreign monolingual data  Here, we keep tuning the model on English to ensure that it does not forget what it has learned in English and that we can use the resulting model for zero-shot transfer (§SECREF3)  In this step, the encoder parameters are also updated so that in can learn syntactic aspects (i e , word order, morphological agreement) of the target languages 
Zero-shot Experiments
We build our bilingual LMs, named RAMEN, starting from BERT$_{\textsc {base}}$, BERT$_{\textsc {large}}$, RoBERTa$_{\textsc {base}}$, and RoBERTa$_{\textsc {large}}$ pre-trained models  Using BERT$_{\textsc {base}}$ allows us to compare the results with mBERT model  Using BERT$_{\textsc {large}}$ and RoBERTa allows us to investigate whether the performance of the target LM correlates with the performance of the source LM  We evaluate our models on two cross-lingual zero-shot tasks: (1) Cross-lingual Natural Language Inference (XNLI) and (2) dependency parsing 
Zero-shot Experiments ::: Data
We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi)  These languages belong to four different language families  French, Russian, and Hindi are Indo-European languages, similar to English  Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan, and Austro-Asiatic family respectively  The choice of the six languages also reflects different training conditions depending on the amount of monolingual data  French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource 
For experiments that use parallel data to initialize foreign specific parameters, we use the same datasets in the work of BIBREF6  Specifically, we use United Nations Parallel Corpus BIBREF18 for en-ru, en-ar, en-zh, and en-fr  We collect en-hi parallel data from IIT Bombay corpus BIBREF19 and en-vi data from OpenSubtitles 2018  For experiments that use only monolingual data to initialize foreign parameters, instead of training word-vectors from the scratch, we use the pre-trained word vectors from fastText BIBREF14 to estimate word translation probabilities (Eq  DISPLAY_FORM13)  We align these vectors into a common space using orthogonal Procrustes BIBREF20, BIBREF15, BIBREF16  We only use identical words between the two languages as the supervised signal  We use WikiExtractor to extract extract raw sentences from Wikipedias as monolingual data for fine-tuning target embeddings and bilingual LMs (§SECREF15)  We do not lowercase or remove accents in our data preprocessing pipeline 
We tokenize English using the provided tokenizer from pre-trained models  For target languages, we use fastBPE to learn 30,000 BPE codes and 50,000 codes when transferring from BERT and RoBERTa respectively  We truncate the BPE vocabulary of foreign languages to match the size of the English vocabulary in the source models  Precisely, the size of foreign vocabulary is set to 32,000 when transferring from BERT and 50,000 when transferring from RoBERTa 
We use XNLI dataset BIBREF9 for classification task and Universal Dependencies v2 4 BIBREF21 for parsing task  Since a language might have more than one treebank in Universal Dependencies, we use the following treebanks: en_ewt (English), fr_gsd (French), ru_syntagrus (Russian) ar_padt (Arabic), vi_vtb (Vietnamese), hi_hdtb (Hindi), and zh_gsd (Chinese) 
Zero-shot Experiments ::: Data ::: Remark on BPE
BIBREF22 show that sharing subwords between languages improves alignments between embedding spaces  BIBREF2 observe a strong correlation between the percentage of overlapping subwords and mBERT's performances for cross-lingual zero-shot transfer  However, in our current approach, subwords between source and target are not shared  A subword that is in both English and foreign vocabulary has two different embeddings 
Zero-shot Experiments ::: Estimating translation probabilities
Since pre-trained models operate on subword level, we need to estimate subword translation probabilities  Therefore, we subsample 2M sentence pairs from each parallel corpus and tokenize the data into subwords before running fast-align BIBREF13 
Estimating subword translation probabilities from aligned word vectors requires an additional processing step since the provided vectors from fastText are not at subword level  We use the following approximation to obtain subword vectors: the vector $_s$ of subword $s$ is the weighted average of all the aligned word vectors $_{w_i}$ that have $s$ as an subword
where $p(w_j)$ is the unigram probability of word $w_j$ and $n_s = \sum _{w_j:\, s\in w_j} p(w_j)$  We take the top 50,000 words in each aligned word-vectors to compute subword vectors 
In both cases, not all the words in the foreign vocabulary can be initialized from the English word-embeddings  Those words are initialized randomly from a Gaussian $\mathcal {N}(0, {1}{d^2})$ 
Zero-shot Experiments ::: Hyper-parameters
In all the experiments, we tune RAMEN$_{\textsc {base}}$ for 175,000 updates and RAMEN$_{\textsc {large}}$ for 275,000 updates where the first 25,000 updates are for language specific parameters  The sequence length is set to 256  The mini-batch size are 64 and 24 when tuning language specific parameters using RAMEN$_{\textsc {base}}$ and RAMEN$_{\textsc {large}}$ respectively  For tuning bilingual LMs, we use a mini-batch size of 64 for RAMEN$_{\textsc {base}}$ and 24 for RAMEN$_{\textsc {large}}$ where half of the batch are English sequences and the other half are foreign sequences  This strategy of balancing mini-batch has been used in multilingual neural machine translation BIBREF23, BIBREF24 
We optimize RAMEN$_{\textsc {base}}$ using Lookahead optimizer BIBREF25 wrapped around Adam with the learning rate of $10^{-4}$, the number of fast weight updates $k=5$, and interpolation parameter $\alpha =0 5$  We choose Lookahead optimizer because it has been shown to be robust to the initial parameters of the based optimizer (Adam)  For Adam optimizer, we linearly increase the learning rate from $10^{-7}$ to $10^{-4}$ in the first 4000 updates and then follow an inverse square root decay  All RAMEN$_{\textsc {large}}$ models are optimized with Adam due to memory limit 
When fine-tuning RAMEN on XNLI and UD, we use a mini-batch size of 32, Adam's learning rate of $10^{-5}$  The number of epochs are set to 4 and 50 for XNLI and UD tasks respectively  All experiments are carried out on a single Tesla V100 16GB GPU  Each RAMEN$_{\textsc {base}}$ model is trained within a day and each RAMEN$_{\textsc {large}}$ is trained within two days 
Results
In this section, we present the results of out models for two zero-shot cross lingual transfer tasks: XNLI and dependency parsing 
Results ::: Cross-lingual Natural Language Inference
Table TABREF32 shows the XNLI test accuracy  For reference, we also include the scores from the previous work, notably the state-of-the-art system XLM BIBREF6  Before discussing the results, we spell out that the fairest comparison in this experiment is the comparison between mBERT and RAMEN$_{\textsc {base}}$+BERT trained with monolingual only 
We first discuss the transfer results from BERT  Initialized from fastText vectors, RAMEN$_{\textsc {base}}$ slightly outperforms mBERT by 1 9 points on average and widen the gap of 3 3 points on Arabic  RAMEN$_{\textsc {base}}$ gains extra 0 8 points on average when initialized from parallel data  With triple number of parameters, RAMEN$_{\textsc {large}}$ offers an additional boost in term of accuracy and initialization with parallel data consistently improves the performance  It has been shown that BERT$_{\textsc {large}}$ significantly outperforms BERT$_{\textsc {base}}$ on 11 English NLP tasks BIBREF0, the strength of BERT$_{\textsc {large}}$ also shows up when adapted to foreign languages 
Transferring from RoBERTa leads to better zero-shot accuracies  With the same initializing condition, RAMEN$_{\textsc {base}}$+RoBERTa outperforms RAMEN$_{\textsc {base}}$+BERT on average by 2 9 and 2 3 points when initializing from monolingual and parallel data respectively  This result show that with similar number of parameters, our approach benefits from a better English pre-trained model  When transferring from RoBERTa$_{\textsc {large}}$, we obtain state-of-the-art results for five languages 
Currently, RAMEN only uses parallel data to initialize foreign embeddings  RAMEN can also exploit parallel data through translation objective proposed in XLM  We believe that by utilizing parallel data during the fine-tuning of RAMEN would bring additional benefits for zero-shot tasks  We leave this exploration to future work  In summary, starting from BERT$_{\textsc {base}}$, our approach obtains competitive bilingual LMs with mBERT for zero-shot XNLI  Our approach shows the accuracy gains when adapting from a better pre-trained model 
Results ::: Universal Dependency Parsing
We build on top of RAMEN a graph-based dependency parser BIBREF27  For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags  Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores  Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing 
We first look at the fairest comparison between mBERT and monolingually initialized RAMEN$_{\textsc {base}}$+BERT  The latter outperforms the former on five languages except Arabic  We observe the largest gain of +5 2 LAS for French  Chinese enjoys +3 1 LAS from our approach  With similar architecture (12 or 24 layers) and initialization (using monolingual or parallel data), RAMEN+RoBERTa performs better than RAMEN+BERT for most of the languages  Arabic and Hindi benefit the most from bigger models  For the other four languages, RAMEN$_{\textsc {large}}$ renders a modest improvement over RAMEN$_{\textsc {base}}$ 
Analysis ::: Impact of initialization
Initializing foreign embeddings is the backbone of our approach  A good initialization leads to better zero-shot transfer results and enables fast adaptation  To verify the importance of a good initialization, we train a RAMEN$_{\textsc {base}}$+RoBERTa with foreign word-embeddings are initialized randomly from $\mathcal {N}(0, {1}{d^2})$  For a fair comparison, we use the same hyper-parameters in §SECREF27  Table TABREF36 shows the results of XNLI and UD parsing of random initialization  In comparison to the initialization using aligned fastText vectors, random initialization decreases the zero-shot performance of RAMEN$_{\textsc {base}}$ by 15 9% for XNLI and 27 8 points for UD parsing on average  We also see that zero-shot parsing of SOV languages (Arabic and Hindi) suffers random initialization 
Analysis ::: Are contextual representations from RAMEN also good for supervised parsing?
All the RAMEN models are built from English and tuned on English for zero-shot cross-lingual tasks  It is reasonable to expect RAMENs do well in those tasks as we have shown in our experiments  But are they also a good feature extractor for supervised tasks? We offer a partial answer to this question by evaluating our model for supervised dependency parsing on UD datasets 
We used train/dev/test splits provided in UD to train and evaluate our RAMEN-based parser  Table TABREF38 summarizes the results (LAS) of our supervised parser  For a fair comparison, we choose mBERT as the baseline and all the RAMEN models are initialized from aligned fastText vectors  With the same architecture of 12 Transformer layers, RAMEN$_{\textsc {base}}$+BERT performs competitive to mBERT and outshines mBERT by +1 2 points for Vietnamese  The best LAS results are obtained by RAMEN$_{\textsc {large}}$+RoBERTa with 24 Transformer layers  Overall, our results indicate the potential of using contextual representations from RAMEN for supervised tasks 
Analysis ::: How does linguistic knowledge transfer happen through each training stages?
We evaluate the performance of RAMEN+RoBERTa$_{\textsc {base}}$ (initialized from monolingual data) at each training steps: initialization of word embeddings (0K update), fine-tuning target embeddings (25K), and fine-tuning the model on both English and target language (at each 25K updates)  The results are presented in Figure FIGREF40 
Without fine-tuning, the average accuracy of XLNI is 39 7% for a three-ways classification task, and the average LAS score is 3 6 for dependency parsing  We see the biggest leap in the performance after 50K updates  While semantic similarity task profits significantly at 25K updates of the target embeddings, syntactic task benefits with further fine-tuning the encoder  This is expected since the target languages might exhibit different syntactic structures than English and fine-tuning encoder helps to capture language specific structures  We observe a substantial gain of 19-30 LAS for all languages except French after 50K updates 
Language similarities have more impact on transferring syntax than semantics  Without tuning the English encoder, French enjoys 50 3 LAS for being closely related to English, whereas Arabic and Hindi, SOV languages, modestly reach 4 2 and 6 4 points using the SVO encoder  Although Chinese has SVO order, it is often seen as head-final while English is strong head-initial  Perhaps, this explains the poor performance for Chinese 
Limitations
While we have successfully adapted autoencoding pre-trained LMs from English to other languages, the question whether our approach can also be applied for autoregressive LM such as XLNet still remains  We leave the investigation to future work 
Conclusions
In this work, we have presented a simple and effective approach for rapidly building a bilingual LM under a limited computational budget  Using BERT as the starting point, we demonstrate that our approach produces better than mBERT on two cross-lingual zero-shot sentence classification and dependency parsing  We find that the performance of our bilingual LM, RAMEN, correlates with the performance of the original pre-trained English models  We also find that RAMEN is also a powerful feature extractor in supervised dependency parsing  Finally, we hope that our work sparks of interest in developing fast and effective methods for transferring pre-trained English models to other languages 
Context 3: Introduction
Accurate language identification (LID) is the first step in many natural language processing and machine comprehension pipelines  If the language of a piece of text is known then the appropriate downstream models like parts of speech taggers and language models can be applied as required 
LID is further also an important step in harvesting scarce language resources  Harvested data can be used to bootstrap more accurate LID models and in doing so continually improve the quality of the harvested data  Availability of data is still one of the big roadblocks for applying data driven approaches like supervised machine learning in developing countries 
Having 11 official languages of South Africa has lead to initiatives (discussed in the next section) that have had positive effect on the availability of language resources for research  However, many of the South African languages are still under resourced from the point of view of building data driven models for machine comprehension and process automation 
Table TABREF2 shows the percentages of first language speakers for each of the official languages of South Africa  These are four conjunctively written Nguni languages (zul, xho, nbl, ssw), Afrikaans (afr) and English (eng), three disjunctively written Sotho languages (nso, sot, tsn), as well as tshiVenda (ven) and Xitsonga (tso)  The Nguni languages are similar to each other and harder to distinguish  The same is true of the Sotho languages 
This paper presents a hierarchical naive Bayesian and lexicon based classifier for LID of short pieces of text of 15-20 characters long  The algorithm is evaluated against recent approaches using existing test sets from previous works on South African languages as well as the Discriminating between Similar Languages (DSL) 2015 and 2017 shared tasks 
Section SECREF2 reviews existing works on the topic and summarises the remaining research problems  Section SECREF3 of the paper discusses the proposed algorithm and Section SECREF4 presents comparative results 
Related Works
The focus of this section is on recently published datasets and LID research applicable to the South African context  An in depth survey of algorithms, features, datasets, shared tasks and evaluation methods may be found in BIBREF0 
The datasets for the DSL 2015 & DSL 2017 shared tasks BIBREF1 are often used in LID benchmarks and also available on Kaggle   The DSL datasets, like other LID datasets, consists of text sentences labelled by language  The 2017 dataset, for example, contains 14 languages over 6 language groups with 18000 training samples and 1000 testing samples per language 
The recently published JW300 parallel corpus BIBREF2 covers over 300 languages with around 100 thousand parallel sentences per language pair on average  In South Africa, a multilingual corpus of academic texts produced by university students with different mother tongues is being developed BIBREF3  The WiLI-2018 benchmark dataset BIBREF4 for monolingual written natural language identification includes around 1000 paragraphs of 235 languages  A possibly useful link can also be made BIBREF5 between Native Language Identification (NLI) (determining the native language of the author of a text) and Language Variety Identification (LVI) (classification of different varieties of a single language) which opens up more datasets  The Leipzig Corpora Collection BIBREF6, the Universal Declaration of Human Rights and Tatoeba are also often used sources of data 
The NCHLT text corpora BIBREF7 is likely a good starting point for a shared LID task dataset for the South African languages BIBREF8  The NCHLT text corpora contains enough data to have 3500 training samples and 600 testing samples of 300+ character sentences per language  Researchers have recently started applying existing algorithms for tasks like neural machine translation in earnest to such South African language datasets BIBREF9 
Existing NLP datasets, models and services BIBREF10 are available for South African languages  These include an LID algorithm BIBREF11 that uses a character level n-gram language model  Multiple papers have shown that 'shallow' naive Bayes classifiers BIBREF12, BIBREF8, BIBREF13, BIBREF14, SVMs BIBREF15 and similar models work very well for doing LID  The DSL 2017 paper BIBREF1, for example, gives an overview of the solutions of all of the teams that competed on the shared task and the winning approach BIBREF16 used an SVM with character n-gram, parts of speech tag features and some other engineered features  The winning approach for DSL 2015 used an ensemble naive Bayes classifier  The fasttext classifier BIBREF17 is perhaps one of the best known efficient 'shallow' text classifiers that have been used for LID  
Multiple papers have proposed hierarchical stacked classifiers (including lexicons) that would for example first classify a piece of text by language group and then by exact language BIBREF18, BIBREF19, BIBREF8, BIBREF0  Some work has also been done on classifying surnames between Tshivenda, Xitsonga and Sepedi BIBREF20  Additionally, data augmentation BIBREF21 and adversarial training BIBREF22 approaches are potentially very useful to reduce the requirement for data 
Researchers have investigated deeper LID models like bidirectional recurrent neural networks BIBREF23 or ensembles of recurrent neural networks BIBREF24  The latter is reported to achieve 95 12% in the DSL 2015 shared task  In these models text features can include character and word n-grams as well as informative character and word-level features learnt BIBREF25 from the training data  The neural methods seem to work well in tasks where more training data is available 
In summary, LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched  Increased confusion can in general be expected between shorter pieces of text and languages that are more closely related  Shallow methods still seem to work well compared to deeper models for LID  Other remaining research opportunities seem to be data harvesting, building standardised datasets and creating shared tasks for South Africa and Africa  Support for language codes that include more languages seems to be growing and discoverability of research is improving with more survey papers coming out  Paywalls also seem to no longer be a problem; the references used in this paper was either openly published or available as preprint papers 
Methodology
The proposed LID algorithm builds on the work in BIBREF8 and BIBREF26  We apply a naive Bayesian classifier with character (2, 4 & 6)-grams, word unigram and word bigram features with a hierarchical lexicon based classifier 
The naive Bayesian classifier is trained to predict the specific language label of a piece of text, but used to first classify text as belonging to either the Nguni family, the Sotho family, English, Afrikaans, Xitsonga or Tshivenda  The scikit-learn multinomial naive Bayes classifier is used for the implementation with an alpha smoothing value of 0 01 and hashed text features 
The lexicon based classifier is then used to predict the specific language within a language group  For the South African languages this is done for the Nguni and Sotho groups  If the lexicon prediction of the specific language has high confidence then its result is used as the final label else the naive Bayesian classifier's specific language prediction is used as the final result  The lexicon is built over all the data and therefore includes the vocabulary from both the training and testing sets 
The lexicon based classifier is designed to trade higher precision for lower recall  The proposed implementation is considered confident if the number of words from the winning language is at least one more than the number of words considered to be from the language scored in second place 
The stacked classifier is tested against three public LID implementations BIBREF17, BIBREF23, BIBREF8  The LID implementation described in BIBREF17 is available on GitHub and is trained and tested according to a post on the fasttext blog  Character (5-6)-gram features with 16 dimensional vectors worked the best  The implementation discussed in BIBREF23 is available from https://github com/tomkocmi/LanideNN  Following the instructions for an OSX pip install of an old r0 8 release of TensorFlow, the LanideNN code could be executed in Python 3 7 4  Settings were left at their defaults and a learning rate of 0 001 was used followed by a refinement with learning rate of 0 0001  Only one code modification was applied to return the results from a method that previously just printed to screen  The LID algorithm described in BIBREF8 is also available on GitHub 
The stacked classifier is also tested against the results reported for four other algorithms BIBREF16, BIBREF26, BIBREF24, BIBREF15  All the comparisons are done using the NCHLT BIBREF7, DSL 2015 BIBREF19 and DSL 2017 BIBREF1 datasets discussed in Section SECREF2 
Results and Analysis
The average classification accuracy results are summarised in Table TABREF9  The accuracies reported are for classifying a piece of text by its specific language label  Classifying text only by language group or family is a much easier task as reported in BIBREF8 
Different variations of the proposed classifier were evaluated  A single NB classifier (NB), a stack of two NB classifiers (NB+NB), a stack of a NB classifier and lexicon (NB+Lex) and a lexicon (Lex) by itself  A lexicon with a 50% training token dropout is also listed to show the impact of the lexicon support on the accuracy 
From the results it seems that the DSL 2017 task might be harder than the DSL 2015 and NCHLT tasks  Also, the results for the implementation discussed in BIBREF23 might seem low, but the results reported in that paper is generated on longer pieces of text so lower scores on the shorter pieces of text derived from the NCHLT corpora is expected 
The accuracy of the proposed algorithm seems to be dependent on the support of the lexicon  Without a good lexicon a non-stacked naive Bayesian classifier might even perform better 
The execution performance of some of the LID implementations are shown in Table TABREF10  Results were generated on an early 2015 13-inch Retina MacBook Pro with a 2 9 GHz CPU (Turbo Boosted to 3 4 GHz) and 8GB RAM  The C++ implementation in BIBREF17 is the fastest  The implementation in BIBREF8 makes use of un-hashed feature representations which causes it to be slower than the proposed sklearn implementation  The execution performance of BIBREF23 might improve by a factor of five to ten when executed on a GPU 
Conclusion
LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched  The proposed algorithm was evaluated on three existing datasets and compared to the implementations of three public LID implementations as well as to reported results of four other algorithms  It performed well relative to the other methods beating their results  However, the performance is dependent on the support of the lexicon 
We would like to investigate the value of a lexicon in a production system and how to possibly maintain it using self-supervised learning  We are investigating the application of deeper language models some of which have been used in more recent DSL shared tasks  We would also like to investigate data augmentation strategies to reduce the amount of training data that is required 
Further research opportunities include data harvesting, building standardised datasets and shared tasks for South Africa as well as the rest of Africa  In general, the support for language codes that include more languages seems to be growing, discoverability of research is improving and paywalls seem to no longer be a big problem in getting access to published research 

ground_truth: translation probabilities, Labeled Attachment Scores (LAS)

accuracy, Labeled Attachment Scores (LAS)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which languages are similar to each other?
Context 4:  To construct the nbow representation, given the word embeddings dictionary where each word is associated with a vector, we apply the average compositional function that averages the embeddings of the words that compose a tweet  Simple compositional functions like average were shown to be robust and efficient in previous work BIBREF17   Instead of training embeddings from scratch, we use the pre-trained on tweets GloVe embeddings of BIBREF10   In terms of resources required, using only nbow is efficient as it does not require any domain knowledge  However, previous research on sentiment analysis showed that using extra resources, like sentiment lexicons, can benefit significantly the performance BIBREF3 , BIBREF2   To validate this and examine at which extent neural networks and multitask learning benefit from such features we evaluate the models using an augmented version of nbow, dubbed nbow+  The feature space of the latter, is augmented using 1,368 extra features consisting mostly of counts of punctuation symbols ('!?#@'), emoticons, elongated words and word membership features in several sentiment lexicons  Due to space limitations, for a complete presentation of these features, we refer the interested reader to BIBREF2 , whose open implementation we used to extract them 
Evaluation measure To reproduce the setting of the SemEval challenges BIBREF16 , we optimize our systems using as primary measure the macro-averaged Mean Absolute Error ( INLINEFORM0 ) given by: INLINEFORM1
where INLINEFORM0 is the number of categories, INLINEFORM1 is the set of instances whose true class is INLINEFORM2 , INLINEFORM3 is the true label of the instance INLINEFORM4 and INLINEFORM5 the predicted label  The measure penalizes decisions far from the true ones and is macro-averaged to account for the fact that the data are unbalanced  Complementary to INLINEFORM6 , we report the performance achieved on the micro-averaged INLINEFORM7 measure, which is a commonly used measure for classification 
The models To evaluate the multitask learning approach, we compared it with several other models  Support Vector Machines (SVMs) are maximum margin classification algorithms that have been shown to achieve competitive performance in several text classification problems BIBREF16   SVM INLINEFORM0 stands for an SVM with linear kernel and an one-vs-rest approach for the multi-class problem  Also, SVM INLINEFORM1 is an SVM with linear kernel that employs the crammer-singer strategy BIBREF18 for the multi-class problem  Logistic regression (LR) is another type of linear classification method, with probabilistic motivation  Again, we use two types of Logistic Regression depending on the multi-class strategy: LR INLINEFORM2 that uses an one-vs-rest approach and multinomial Logistic Regression also known as the MaxEnt classifier that uses a multinomial criterion 
Both SVMs and LRs as discussed above treat the problem as a multi-class one, without considering the ordering of the classes  For these four models, we tuned the hyper-parameter INLINEFORM0 that controls the importance of the L INLINEFORM1 regularization part in the optimization problem with grid-search over INLINEFORM2 using 10-fold cross-validation in the union of the training and development data and then retrained the models with the selected values  Also, to account for the unbalanced classification problem we used class weights to penalize more the errors made on the rare classes  These weights were inversely proportional to the frequency of each class  For the four models we used the implementations of Scikit-learn BIBREF19  
For multitask learning we use the architecture shown in Figure FIGREF2 , which we implemented with Keras BIBREF20   The embeddings are initialized with the 50-dimensional GloVe embeddings while the output of the biLSTM network is set to dimension 50  The activation function of the hidden layers is the hyperbolic tangent  The weights of the layers were initialized from a uniform distribution, scaled as described in BIBREF21   We used the Root Mean Square Propagation optimization method  We used dropout for regularizing the network  We trained the network using batches of 128 examples as follows: before selecting the batch, we perform a Bernoulli trial with probability INLINEFORM0 to select the task to train for  With probability INLINEFORM1 we pick a batch for the fine-grained sentiment classification problem, while with probability INLINEFORM2 we pick a batch for the ternary problem  As shown in Figure FIGREF2 , the error is backpropagated until the embeddings, that we fine-tune during the learning process  Notice also that the weights of the network until the layer INLINEFORM3 are shared and therefore affected by both tasks 
To tune the neural network hyper-parameters we used 5-fold cross validation  We tuned the probability INLINEFORM0 of dropout after the hidden layers INLINEFORM1 and for the biLSTM for INLINEFORM2 , the size of the hidden layer INLINEFORM3 and the probability INLINEFORM4 of the Bernoulli trials from INLINEFORM5   During training, we monitor the network's performance on the development set and apply early stopping if the performance on the validation set does not improve for 5 consecutive epochs 
Experimental results Table TABREF9 illustrates the performance of the models for the different data representations  The upper part of the Table summarizes the performance of the baselines  The entry “Balikas et al ” stands for the winning system of the 2016 edition of the challenge BIBREF2 , which to the best of our knowledge holds the state-of-the-art  Due to the stochasticity of training the biLSTM models, we repeat the experiment 10 times and report the average and the standard deviation of the performance achieved 
Several observations can be made from the table  First notice that, overall, the best performance is achieved by the neural network architecture that uses multitask learning  This entails that the system makes use of the available resources efficiently and improves the state-of-the-art performance  In conjunction with the fact that we found the optimal probability INLINEFORM0 , this highlights the benefits of multitask learning over single task learning  Furthermore, as described above, the neural network-based models have only access to the training data as the development are hold for early stopping  On the other hand, the baseline systems were retrained on the union of the train and development sets  Hence, even with fewer resources available for training on the fine-grained problem, the neural networks outperform the baselines  We also highlight the positive effect of the additional features that previous research proposed  Adding the features both in the baselines and in the biLSTM-based architectures improves the INLINEFORM1 scores by several points 
Lastly, we compare the performance of the baseline systems with the performance of the state-of-the-art system of BIBREF2   While BIBREF2 uses n-grams (and character-grams) with INLINEFORM0 , the baseline systems (SVMs, LRs) used in this work use the nbow+ representation, that relies on unigrams  Although they perform on par, the competitive performance of nbow highlights the potential of distributed representations for short-text classification  Further, incorporating structure and distributed representations leads to the gains of the biLSTM network, in the multitask and single task setting 
Similar observations can be drawn from Figure FIGREF10 that presents the INLINEFORM0 scores  Again, the biLSTM network with multitask learning achieves the best performance  It is also to be noted that although the two evaluation measures are correlated in the sense that the ranking of the models is the same, small differences in the INLINEFORM1 have large effect on the scores of the INLINEFORM2 measure 
Conclusion
In this paper, we showed that by jointly learning the tasks of ternary and fine-grained classification with a multitask learning model, one can greatly improve the performance on the second  This opens several avenues for future research  Since sentiment is expressed in different textual types like tweets and paragraph-sized reviews, in different languages (English, German,   ) and in different granularity levels (binary, ternary,  ) one can imagine multitask approaches that could benefit from combining such resources  Also, while we opted for biLSTM networks here, one could use convolutional neural networks or even try to combine different types of networks and tasks to investigate the performance effect of multitask learning  Lastly, while our approach mainly relied on the foundations of BIBREF4 , the internal mechanisms and the theoretical guarantees of multitask learning remain to be better understood 
Acknowledgements
This work is partially supported by the CIFRE N 28/2015 

ground_truth: Nguni languages (zul, xho, nbl, ssw), Sotho languages (nso, sot, tsn)

The Nguni languages are similar to each other, The same is true of the Sotho languages