prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What language model architectures are used?
Context 1: Introduction
Pre-training of language models has been shown to provide large improvements for a range of language understanding tasks BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   The key idea is to train a large generative model on vast corpora and use the resulting representations on tasks for which only limited amounts of labeled data is available  Pre-training of sequence to sequence models has been previously investigated for text classification BIBREF4 but not for text generation  In neural machine translation, there has been work on transferring representations from high-resource language pairs to low-resource settings BIBREF5  
In this paper, we apply pre-trained representations from language models to language generation tasks that can be modeled by sequence to sequence architectures  Previous work on integrating language models with sequence to sequence models focused on the decoder network and added language model representations right before the output of the decoder BIBREF6   We extend their study by investigating several other strategies such as inputting ELMo-style representations BIBREF0 or fine-tuning the language model (§ SECREF2 ) 
Our experiments rely on strong transformer-based language models trained on up to six billion tokens (§ SECREF3 )  We present a detailed study of various strategies in different simulated labeled training data scenarios and observe the largest improvements in low-resource settings but gains of over 1 BLEU are still possible when five million sentence-pairs are available  The most successful strategy to integrate pre-trained representations is as input to the encoder network (§ SECREF4 ) 
Strategies to add representations
We consider augmenting a standard sequence to sequence model with pre-trained representations following an ELMo-style regime (§ SECREF2 ) as well as by fine-tuning the language model (§ SECREF3 ) 
ELMo augmentation
The ELMo approach of BIBREF0 forms contextualized word embeddings based on language model representations without adjusting the actual language model parameters  Specifically, the ELMo module contains a set of parameters INLINEFORM0 to form a linear combination of the INLINEFORM1 layers of the language model: ELMo = INLINEFORM2 where INLINEFORM3 is a learned scalar, INLINEFORM4 is a constant to normalize the INLINEFORM5 to sum to one and INLINEFORM6 is the output of the INLINEFORM7 -th language model layer; the module also considers the input word embeddings of the language model  We also apply layer normalization BIBREF7 to each INLINEFORM8 before computing ELMo vectors 
We experiment with an ELMo module to input contextualized embeddings either to the encoder () or the decoder ()  This provides word representations specific to the current input sentence and these representations have been trained on much more data than is available for the text generation task 
Fine-tuning approach
Fine-tuning the pre-trained representations adjusts the language model parameters by the learning signal of the end-task BIBREF1 , BIBREF3   We replace learned input word embeddings in the encoder network with the output of the language model ()  Specifically, we use the language model representation of the layer before the softmax and feed it to the encoder  We also add dropout to the language model output  Tuning separate learning rates for the language model and the sequence to sequence model may lead to better performance but we leave this to future work  However, we do tune the number of encoder blocks INLINEFORM0 as we found this important to obtain good accuracy for this setting  We apply the same strategy to the decoder: we input language model representations to the decoder network and fine-tune the language model when training the sequence to sequence model () 
Datasets
We train language models on two languages: One model is estimated on the German newscrawl distributed by WMT'18 comprising 260M sentences or 6B tokens  Another model is trained on the English newscrawl data comprising 193M sentences or 5B tokens  We learn a joint Byte-Pair-Encoding (BPE; Sennrich et al , 2016) vocabulary of 37K types on the German and English newscrawl and train the language models with this vocabulary 
We consider two benchmarks: Most experiments are run on the WMT'18 English-German (en-de) news translation task and we validate our findings on the WMT'18 English-Turkish (en-tr) news task  For WMT'18 English-German, the training corpus consists of all available bitext excluding the ParaCrawl corpus and we remove sentences longer than 250 tokens as well as sentence-pairs with a source/target length ratio exceeding 1 5  This results in 5 18M sentence pairs  We tokenize all data with the Moses tokenizer BIBREF8 and apply the BPE vocabulary learned on the monolingual corpora 
For WMT'18 English-Turkish, we use all of the available bitext comprising 208K sentence-pairs without any filtering  We develop on newstest2017 and test on newstest2018  For en-tr we only experiment with adding representations to the encoder and therefore apply the language model vocabulary to the source side  For the target vocabulary we learn a BPE code with 32K merge operations on the Turkish side of the bitext  Both datasets are evaluated in terms of case-sensitive de-tokenized BLEU BIBREF9 , BIBREF10  
We consider the abstractive document summarization task comprising over 280K news articles paired with multi-sentence summaries  is a widely used dataset for abstractive text summarization  Following BIBREF11 , we report results on the non-anonymized version of rather than the entity-anonymized version BIBREF12 , BIBREF13 because the language model was trained on full text  Articles are truncated to 400 tokens BIBREF11 and we use a BPE vocabulary of 32K types BIBREF14   We evaluate in terms of F1-Rouge, that is Rouge-1, Rouge-2 and Rouge-L BIBREF15  
Language model pre-training
We consider two types of architectures: a bi-directional language model to augment the sequence to sequence encoder and a uni-directional model to augment the decoder  Both use self-attention BIBREF16 and the uni-directional model contains INLINEFORM0 transformer blocks, followed by a word classifier to predict the next word on the right  The bi-directional model solves a cloze-style token prediction task at training time BIBREF17   The model consists of two towers, the forward tower operates left-to-right and the tower operating right-to-left as backward tower; each tower contains INLINEFORM1 transformer blocks  The forward and backward representations are combined via a self-attention module and the output of this module is used to predict the token at position INLINEFORM2   The model has access to the entire input surrounding the current target token  Models use the standard settings for the Big Transformer BIBREF16   The bi-directional model contains 353M parameters and the uni-directional model 190M parameters  Both models were trained for 1M steps using Nesterov's accelerated gradient BIBREF18 with momentum INLINEFORM3 following BIBREF19   The learning rate is linearly warmed up from INLINEFORM4 to 1 for 16K steps and then annealed using a cosine learning rate schedule with a single phase to 0 0001 BIBREF20   We train on 32 Nvidia V100 SXM2 GPUs and use the NCCL2 library as well as the torch distributed package for inter-GPU communication  Training relies on 16-bit floating point operations BIBREF21 and it took six days for the bi-directional model and four days for the uni-directional model 
Sequence to sequence model
We use the transformer implementation of the fairseq toolkit BIBREF22   The WMT en-de and en-tr experiments are based on the Big Transformer sequence to sequence architecture with 6 blocks in the encoder and decoder  For abstractive summarization we use a base transformer model BIBREF16   We tune dropout values of between 0 1 and 0 4 on the validation set  Models are optimized with Adam BIBREF23 using INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 and we use the same learning rate schedule as BIBREF16 ; we perform 10K-200K depending on bitext size  All models use label smoothing with a uniform prior distribution over the vocabulary INLINEFORM3 BIBREF24 , BIBREF25   We run experiments on 8 GPUs and generate translations with a beam of size 5 
Machine translation
We first present a comparison of the various strategies in different simulated parallel corpus size settings  For each experiment, we tune the dropout applied to the language model representations, and we reduce the number of optimizer steps for smaller bitext setups as models converge faster; all other hyper-parameters are equal between setups  Our baseline is a Big Transformer model and we also consider a variant where we share token embeddings between the encoder and decoder (; Inan et al , 2016; Press & Wolf, 2016) 
Figure FIGREF10 shows results averaged over six test sets relative to the baseline which does not share source and target embeddings (Appendix SECREF6 shows a detailed breakdown)  performs very well with little labeled data but the gains erode to practically zero in large bitext settings  Pre-trained language model representations are most effective in low bitext setups  The best performing strategy is ELMo embeddings input to the encoder ()  This improves the baseline by 3 8 BLEU in the 160K bitext setting and it still improves the 5 2M setting by over 1 BLEU 
We further improve by sharing learned word representations in the decoder by tying input and output embeddings ()  This configuration performs even better than with a gain of 5 3 BLEU in the 160K setup  Sharing decoder embeddings is equally applicable to   Language model representations are much less effective in the decoder: improves the 160K bitext setup but yields no improvements thereafter and performs even worse  We conjecture that pre-trained representations give much easier wins in the encoder  Table TABREF14 shows additional results on newstest2018 
Pre-trained representations mostly impacts the training time of the sequence to sequence model (see Appendix SECREF7 ): slows throughput during training by about 5 3x and is even slower because of the need to backpropagate through the LM for fine-tuning (9 2x)  However, inference is only 12-14% slower than the baseline when adding pre-trained embeddings to the encoder (, )  This is because the LM computation can be paralelized for all input tokens  Inference is much slower when adding representations to the decoder because the LM needs to be invoked repeatedly  Our current implementation does not cache LM operations for the previous state and can be made much faster 
The baseline uses a BPE vocabulary estimated on the language model corpora (§ SECREF3 )  Appendix SECREF6 shows that this vocabulary actually leads to sligtly better performance than a joint BPE code learned on the bitext as is usual 
Next, we validate our findings on the WMT'18 English-Turkish task for which the bitext is truly limited (208K sentence-pairs)  We use the language model vocab for the the English side of the bitext and a BPE vocabulary learned on the Turkish side  Table TABREF15 shows that ELMo embeddings for the encoder improve English-Turkish translation 
Abstractive summarization
Following BIBREF11 , we experiment on the non-anonymized version of   When generating summaries, we follow standard practice of tuning the maximum output length and disallow repeating the same trigram BIBREF27 , BIBREF14   For this task we train language model representations on the combination of newscrawl and the training data  Table TABREF16 shows that pre-trained embeddings can significantly improve on top of a strong baseline transformer  We also compare to BIBREF26 who use a task-specific architecture compared to our generic sequence to sequence baseline  Pre-trained representations are complementary to their method 
Conclusion
We presented an analysis of different strategies to add pre-trained language model representations to sequence to sequence models for neural machine translation and abstractive document summarization  Adding pre-trained representations is very effective for the encoder network and while returns diminish when more labeled data becomes available, we still observe improvements when millions of examples are available  In future research we will investigate ways to improve the decoder with pre-trained representations 

ground_truth: uni-directional model to augment the decoder

bi-directional language model to augment the sequence to sequence encoder ,  uni-directional model to augment the decoder

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What does an individual model consist of?
Context 2: Introduction
Semantic Role Labeling (SRL) has emerged as an important task in Natural Language Processing (NLP) due to its applicability in information extraction, question answering, and other NLP tasks  SRL is the problem of finding predicate-argument structure in a sentence, as illustrated below:
INLINEFORM0
Here, the predicate WRITE has two arguments: `Mike' as A0 or the writer, and `a book' as A1 or the thing written  The labels A0 and A1 correspond to the PropBank annotations BIBREF0  
As the need for SRL arises in different domains and languages, the existing manually annotated corpora become insufficient to build supervised systems  This has motivated work on unsupervised SRL BIBREF1 , BIBREF2 , BIBREF3   Previous work has indicated that unsupervised systems could benefit from the word alignment information in parallel text in two or more languages BIBREF4 , BIBREF5 , BIBREF6   For example, consider the German translation of sentence INLINEFORM0 :
INLINEFORM0
If sentences INLINEFORM0 and INLINEFORM1 have the word alignments: Mike-Mike, written-geschrieben, and book-Buch, the system might be able to predict A1 for Buch, even if there is insufficient information in the monolingual German data to learn this assignment  Thus, in languages where the resources are sparse or not good enough, or the distributions are not informative, SRL systems could be made more accurate by using parallel data with resource rich or more amenable languages 
In this paper, we propose a joint Bayesian model for unsupervised semantic role induction in multiple languages  The model consists of individual Bayesian models for each language BIBREF3 , and crosslingual latent variables to incorporate soft role agreement between aligned constituents  This latent variable approach has been demonstrated to increase the performance in a multilingual unsupervised part-of-speech tagging model based on HMMs BIBREF4   We investigate the application of this approach to unsupervised SRL, presenting the performance improvements obtained in different settings involving labeled and unlabeled data, and analyzing the annotation effort required to obtain similar gains using labeled data 
We begin by briefly describing the unsupervised SRL pipeline and the monolingual semantic role induction model we use, and then describe our multilingual model 
Unsupervised SRL Pipeline
As established in previous work BIBREF7 , BIBREF8 , we use a standard unsupervised SRL setup, consisting of the following steps:
The task we model, unsupervised semantic role induction, is the step 4 of this pipeline 
Monolingual Model
We use the Bayesian model of garg2012unsupervised as our base monolingual model  The semantic roles are predicate-specific  To model the role ordering and repetition preferences, the role inventory for each predicate is divided into Primary and Secondary roles as follows:
For example, the complete role sequence in a frame could be: INLINEFORM0 INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 , INLINEFORM8 INLINEFORM9   The ordering is defined as the sequence of PRs, INLINEFORM10 INLINEFORM11 , INLINEFORM12 , INLINEFORM13 , INLINEFORM14 , INLINEFORM15 INLINEFORM16   Each pair of consecutive PRs in an ordering is called an interval  Thus, INLINEFORM17 is an interval that contains two SRs, INLINEFORM18 and INLINEFORM19   An interval could also be empty, for instance INLINEFORM20 contains no SRs  When we evaluate, these roles get mapped to gold roles  For instance, the PR INLINEFORM21 could get mapped to a core role like INLINEFORM22 , INLINEFORM23 , etc  or to a modifier role like INLINEFORM24 , INLINEFORM25 , etc  garg2012unsupervised reported that, in practice, PRs mostly get mapped to core roles and SRs to modifier roles, which conforms to the linguistic motivations for this distinction 
Figure FIGREF16 illustrates two copies of the monolingual model, on either side of the crosslingual latent variables  The generative process is as follows:
All the multinomial and binomial distributions have symmetric Dirichlet and beta priors respectively  Figure FIGREF7 gives the probability equations for the monolingual model  This formulation models the global role ordering and repetition preferences using PRs, and limited context for SRs using intervals  Ordering and repetition information was found to be helpful in supervised SRL as well BIBREF9 , BIBREF8 , BIBREF10   More details, including the motivations behind this model, are in BIBREF3  
Multilingual Model
The multilingual model uses word alignments between sentences in a parallel corpus to exploit role correspondences across languages  We make copies of the monolingual model for each language and add additional crosslingual latent variables (CLVs) to couple the monolingual models, capturing crosslingual semantic role patterns  Concretely, when training on parallel sentences, whenever the head words of the arguments are aligned, we add a CLV as a parent of the two corresponding role variables  Figure FIGREF16 illustrates this model  The generative process, as explained below, remains the same as the monolingual model for the most part, with the exception of aligned roles which are now generated by both the monolingual process as well as the CLV 
Every predicate-tuple has its own inventory of CLVs specific to that tuple  Each CLV INLINEFORM0 is a multi-valued variable where each value defines a distribution over role labels for each language (denoted by INLINEFORM1 above)  These distributions over labels are trained to be peaky, so that each value INLINEFORM2 for a CLV represents a correlation between the labels that INLINEFORM3 predicts in the two languages  For example, a value INLINEFORM4 for the CLV INLINEFORM5 might give high probabilities to INLINEFORM6 and INLINEFORM7 in language 1, and to INLINEFORM8 in language 2  If INLINEFORM9 is the only value for INLINEFORM10 that gives high probability to INLINEFORM11 in language 1, and the monolingual model in language 1 decides to assign INLINEFORM12 to the role for INLINEFORM13 , then INLINEFORM14 will predict INLINEFORM15 in language 2, with high probability  We generate the CLVs via a Chinese Restaurant Process BIBREF11 , a non-parametric Bayesian model, which allows us to induce the number of CLVs for every predicate-tuple from the data  We continue to train on the non-parallel sentences using the respective monolingual models 
The multilingual model is deficient, since the aligned roles are being generated twice  Ideally, we would like to add the CLV as additional conditioning variables in the monolingual models  The new joint probability can be written as equation UID11 (Figure FIGREF7 ), which can be further decomposed following the decomposition of the monolingual model in Figure FIGREF7   However, having this additional conditioning variable breaks the Dirichlet-multinomial conjugacy, which makes it intractable to marginalize out the parameters during inference  Hence, we use an approximation where we treat each of the aligned roles as being generated twice, once by the monolingual model and once by the corresponding CLV (equation ) 
This is the first work to incorporate the coupling of aligned arguments directly in a Bayesian SRL model  This makes it easier to see how to extend this model in a principled way to incorporate additional sources of information  First, the model scales gracefully to more than two languages  If there are a total of INLINEFORM0 languages, and there is an aligned argument in INLINEFORM1 of them, the multilingual latent variable is connected to only those INLINEFORM2 aligned arguments 
Second, having one joint Bayesian model allows us to use the same model in various semi-supervised learning settings, just by fixing the annotated variables during training  Section SECREF29 evaluates a setting where we have some labeled data in one language (called source), while no labeled data in the second language (called target)  Note that this is different from a classic annotation projection setting (e g  BIBREF12 ), where the role labels are mapped from source constituents to aligned target constituents 
Inference and Training
The inference problem consists of predicting the role labels and CLVs (the hidden variables) given the predicate, its voice, and syntactic features of all the identified arguments (the visible variables)  We use a collapsed Gibbs-sampling based approach to generate samples for the hidden variables (model parameters are integrated out)  The sample counts and the priors are then used to calculate the MAP estimate of the model parameters 
For the monolingual model, the role at a given position is sampled as:
DISPLAYFORM0
where the subscript INLINEFORM0 refers to all the variables except at position INLINEFORM1 , INLINEFORM2 refers to the variables in all the training instances except the current one, and INLINEFORM3 refers to all the model parameters  The above integral has a closed form solution due to Dirichlet-multinomial conjugacy 
For sampling roles in the multilingual model, we also need to consider the probabilities of roles being generated by the CLVs:
DISPLAYFORM0
For sampling CLVs, we need to consider three factors: two corresponding to probabilities of generating the aligned roles, and the third one corresponding to selecting the CLV according to CRP 
DISPLAYFORM0
where the aligned roles INLINEFORM0 and INLINEFORM1 are connected to INLINEFORM2 , and INLINEFORM3 refers to all the variables except INLINEFORM4 , INLINEFORM5 , and INLINEFORM6  
We use the trained parameters to parse the monolingual data using the monolingual model  The crosslingual parameters are ignored even if they were used during training  Thus, the information coming from the CLVs acts as a regularizer for the monolingual models 
Evaluation
Following the setting of titovcrosslingual, we evaluate only on the arguments that were correctly identified, as the incorrectly identified arguments do not have any gold semantic labels  Evaluation is done using the metric proposed by lang2011unsupervised, which has 3 components: (i) Purity (PU) measures how well an induced cluster corresponds to a single gold role, (ii) Collocation (CO) measures how well a gold role corresponds to a single induced cluster, and (iii) F1 is the harmonic mean of PU and CO  For each predicate, let INLINEFORM0 denote the total number of argument instances, INLINEFORM1 the instances in the induced cluster INLINEFORM2 , and INLINEFORM3 the instances having label INLINEFORM4 in gold annotations  INLINEFORM5 , INLINEFORM6 , and INLINEFORM7   The score for each predicate is weighted by the number of its argument instances, and a weighted average is computed over all the predicates 
Baseline
We use the same baseline as used by lang2011unsupervised which has been shown to be difficult to outperform  This baseline assigns a semantic role to a constituent based on its syntactic function, i e  the dependency relation to its head  If there is a total of INLINEFORM0 clusters, INLINEFORM1 most frequent syntactic functions get a cluster each, and the rest are assigned to the INLINEFORM2 th cluster 
Closest Previous Work
This work is closely related to the cross-lingual unsupervised SRL work of titovcrosslingual  Their model has separate monolingual models for each language and an extra penalty term which tries to maximize INLINEFORM0 and INLINEFORM1 i e  for all the aligned arguments with role label INLINEFORM2 in language 1, it tries to find a role label INLINEFORM3 in language 2 such that the given proportion is maximized and vice verse  However, there is no efficient way to optimize the objective with this penalty term and the authors used an inference method similar to annotation projection  Further, the method does not scale naturally to more than two languages  Their algorithm first does monolingual inference in one language ignoring the penalty and then does the inference in the second language taking into account the penalty term  In contrast, our model adds the latent variables as a part of the model itself, and not an external penalty, which enables us to use the standard Bayesian learning methods such as sampling 
The monolingual model we use BIBREF3 also has two main advantages over titovcrosslingual  First, the former incorporates a global role ordering probability that is missing in the latter  Secondly, the latter defines argument-keys as a tuple of four syntactic features and all the arguments having the same argument-keys are assigned the same role  This kind of hard clustering is avoided in the former model where two constituents having the same set of features might get assigned different roles if they appear in different contexts 
Data
Following titovcrosslingual, we run our experiments on the English (EN) and German (DE) sections of the CoNLL 2009 corpus BIBREF13 , and EN-DE section of the Europarl corpus BIBREF14   We get about 40k EN and 36k DE sentences from the CoNLL 2009 training set, and about 1 5M parallel EN-DE sentences from Europarl  For appropriate comparison, we keep the same setting as in BIBREF6 for automatic parses and argument identification, which we briefly describe here  The EN sentences are parsed syntactically using MaltParser BIBREF15 and DE using LTH parser BIBREF16   All the non-auxiliary verbs are selected as predicates  In CoNLL data, this gives us about 3k EN and 500 DE predicates  The total number of predicate instances are 3 4M in EN (89k CoNLL + 3 3M Europarl) and 2 62M in DE (17k CoNLL + 2 6M Europarl)  The arguments for EN are identified using the heuristics proposed by lang2011unsupervised  However, we get an F1 score of 85 1% for argument identification on CoNLL 2009 EN data as opposed to 80 7% reported by titovcrosslingual  This could be due to implementation differences, which unfortunately makes our EN results incomparable  For DE, the arguments are identified using the LTH system BIBREF16 , which gives an F1 score of 86 5% on the CoNLL 2009 DE data  The word alignments for the EN-DE parallel Europarl corpus are computed using GIZA++ BIBREF17   For high-precision, only the intersecting alignments in the two directions are kept  We define two semantic arguments as aligned if their head-words are aligned  In total we get 9 3M arguments for EN (240k CoNLL + 9 1M Europarl) and 4 43M for DE (32k CoNLL + 4 4M Europarl)  Out of these, 0 76M arguments are aligned 
Main Results
Since the CoNLL annotations have 21 semantic roles in total, we use 21 roles in our model as well as the baseline  Following garg2012unsupervised, we set the number of PRs to 2 (excluding INLINEFORM0 , INLINEFORM1 and INLINEFORM2 ), and SRs to 21-2=19  Table TABREF27 shows the results 
In the first setting (Line 1), we train and test the monolingual model on the CoNLL data  We observe significant improvements in F1 score over the Baseline (Line 0) in both languages  Using the CoNLL 2009 dataset alone, titovcrosslingual report an F1 score of 80 9% (PU=86 8%, CO=75 7%) for German  Thus, our monolingual model outperforms their monolingual model in German  For English, they report an F1 score of 83 6% (PU=87 5%, CO=80 1%), but note that our English results are not directly comparable to theirs due to differences argument identification, as discussed in section SECREF25   As their argument identification score is lower, perhaps their system is discarding “difficult” arguments which leads to a higher clustering score 
In the second setting (Line 2), we use the additional monolingual Europarl (EP) data for training  We get equivalent results in English and a significant improvement in German compared to our previous setting (Line 1)  The German dataset in CoNLL is quite small and benefits from the additional EP training data  In contrast, the English model is already quite good due to a relatively big dataset from CoNLL, and good accuracy syntactic parsers  Unfortunately, titovcrosslingual do not report results with this setting 
The third setting (Line 3) gives the results of our multilingual model, which adds the word alignments in the EP data  Comparing with Line 2, we get non-significant improvements in both languages  titovcrosslingual obtain an F1 score of 82 7% (PU=85 0%, CO=80 6%) for German, and 83 7% (PU=86 8%, CO=80 7%) for English  Thus, for German, our multilingual Bayesian model is able to capture the cross-lingual patterns at least as well as the external penalty term in BIBREF6   We cannot compare the English results unfortunately due to differences in argument identification 
We also compared monolingual and bilingual training data using a setting that emulates the standard supervised setup of separate training and test data sets  We train only on the EP dataset and test on the CoNLL dataset  Lines 4 and 5 of Table TABREF27 give the results  The multilingual model obtains small improvements in both languages, which confirms the results from the standard unsupervised setup, comparing lines 2 to 3 
These results indicate that little information can be learned about semantic roles from this parallel data setup  One possible explanation for this result is that the setup itself is inadequate  Given the definition of aligned arguments, only 8% of English arguments and 17% of German arguments are aligned  This plus our experiments suggest that improving the alignment model is a necessary step to making effective use of parallel data in multilingual SRI, for example by joint modeling with SRI  We leave this exploration to future work 
Multilingual Training with Labeled Data for One Language
Another motivation for jointly modeling SRL in multiple languages is the transfer of information from a resource rich language to a resource poor language  We evaluated our model in a very general annotation transfer scenario, where we have a small labeled dataset for one language (source), and a large parallel unlabeled dataset for the source and another (target) language  We investigate whether this setting improves the parameter estimates for the target language  To this end, we clamp the role annotations of the source language in the CoNLL dataset using a predefined mapping, and do not sample them during training  This data gives us good parameters for the source language, which are used to sample the roles of the source language in the unlabeled Europarl data  The CLVs aim to capture this improvement and thereby improve sampling and parameter estimates for the target language  Table TABREF28 shows the results of this experiment  We obtain small improvements in the target languages  As in the unsupervised setting, the small percentage of aligned roles probably limits the impact of the cross-lingual information 
Labeled Data in Monolingual Model
We explored the improvement in the monolingual model in a semi-supervised setting  To this end, we randomly selected INLINEFORM0 of the sentences in the CoNLL dataset as “supervised sentences” and the rest INLINEFORM1 were kept unsupervised  Next, we clamped the role labels of the supervised sentences using the predefined mapping from Section SECREF29   Sampling was done on the unsupervised sentences as usual  We then measured the clustering performance using the trained parameters 
To access the contribution of partial supervision better, we constructed a “supervised baseline” as follows  For predicates seen in the supervised sentences, a MAP estimate of the parameters was calculated using the predefined mapping  For the unseen predicates, the standard baseline was used 
Figures FIGREF33 and FIGREF33 show the performance variation with INLINEFORM0   We make the following observations:
[leftmargin=*]
In both languages, at around INLINEFORM0 , the supervised baseline starts outperforming the semi-supervised model, which suggests that manually labeling about 10% of the sentences is a good enough alternative to our training procedure  Note that 10% amounts to about 3 6k sentences in German and 4k in English  We noticed that the proportion of seen predicates increases dramatically as we increase the proportion of supervised sentences  At 10% supervised sentences, the model has already seen 63% of predicates in German and 44% in English  This explains to some extent why only 10% labeled sentences are enough 
For German, it takes about 3 5% or 1260 supervised sentences to have the same performance increase as 1 5M unlabeled sentences (Line 1 to Line 2 in Table TABREF27 )  Adding about 180 more supervised sentences also covers the benefit obtained by alignments in the multilingual model (Line 2 to Line 3 in Table TABREF27 )  There is no noticeable performance difference in English 
We also evaluated the performance variation on a completely unseen CoNLL test set  Since the test set is very small compared to the training set, the clustering evaluation is not as reliable  Nonetheless, we broadly obtained the same pattern 
Related Work
As discussed in section SECREF24 , our work is closely related to the crosslingual unsupervised SRL work of titovcrosslingual  The idea of using superlingual latent variables to capture cross-lingual information was proposed for POS tagging by naseem2009multilingual, which we use here for SRL  In a semi-supervised setting, pado2009cross used a graph based approach to transfer semantic role annotations from English to German  furstenau2009graph used a graph alignment method to measure the semantic and syntactic similarity between dependency tree arguments of known and unknown verbs 
For monolingual unsupervised SRL, swier2004unsupervised presented the first work on a domain-general corpus, the British National Corpus, using 54 verbs taken from VerbNet  garg2012unsupervised proposed a Bayesian model for this problem that we use here  titov2012bayesian also proposed a closely related Bayesian model  grenager2006unsupervised proposed a generative model but their parameter space consisted of all possible linkings of syntactic constituents and semantic roles, which made unsupervised learning difficult and a separate language-specific rule based method had to be used to constrain this space  Other proposed models include an iterative split-merge algorithm BIBREF18 and a graph-partitioning based approach BIBREF1   marquez2008semantic provide a good overview of the supervised SRL systems 
Conclusions
We propose a Bayesian model of semantic role induction (SRI) that uses crosslingual latent variables to capture role alignments in parallel corpora  The crosslingual latent variables capture correlations between roles in different languages, and regularize the parameter estimates of the monolingual models  Because this is a joint Bayesian model of multilingual SRI, we can apply the same model to a variety of training scenarios just by changing the inference procedure appropriately  We evaluate monolingual SRI with a large unlabeled dataset, bilingual SRI with a parallel corpus, bilingual SRI with annotations available for the source language, and monolingual SRI with a small labeled dataset  Increasing the amount of monolingual unlabeled data significantly improves SRI in German but not in English  Adding word alignments in parallel sentences results in small, non significant improvements, even if there is some labeled data available in the source language  This difficulty in showing the usefulness of parallel corpora for SRI may be due to the current assumptions about role alignments, which mean that only a small percentage of roles are aligned  Further analyses reveals that annotating small amounts of data can easily outperform the performance gains obtained by adding large unlabeled dataset as well as adding parallel corpora 
Future work includes training on different language pairs, on more than two languages, and with more inclusive models of role alignment 
Acknowledgments
This work was funded by the Swiss NSF grant 200021_125137 and EC FP7 grant PARLANCE 
Context 3: Introduction
Machine translation has made remarkable progress, and studies claiming it to reach a human parity are starting to appear BIBREF0  However, when evaluating translations of the whole documents rather than isolated sentences, human raters show a stronger preference for human over machine translation BIBREF1  These findings emphasize the need to shift towards context-aware machine translation both from modeling and evaluation perspective 
Most previous work on context-aware NMT assumed that either all the bilingual data is available at the document level BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10 or at least its fraction BIBREF11  But in practical scenarios, document-level parallel data is often scarce, which is one of the challenges when building a context-aware system 
We introduce an approach to context-aware machine translation using only monolingual document-level data  In our setting, a separate monolingual sequence-to-sequence model (DocRepair) is used to correct sentence-level translations of adjacent sentences  The key idea is to use monolingual data to imitate typical inconsistencies between context-agnostic translations of isolated sentences  The DocRepair model is trained to map inconsistent groups of sentences into consistent ones  The consistent groups come from the original training data; the inconsistent groups are obtained by sampling round-trip translations for each isolated sentence 
To validate the performance of our model, we use three kinds of evaluation: the BLEU score, contrastive evaluation of translation of several discourse phenomena BIBREF11, and human evaluation  We show strong improvements for all metrics 
We analyze which discourse phenomena are hard to capture using monolingual data only  Using contrastive test sets for targeted evaluation of several contextual phenomena, we compare the performance of the models trained on round-trip translations and genuine document-level parallel data  Among the four phenomena in the test sets we use (deixis, lexical cohesion, VP ellipsis and ellipsis which affects NP inflection) we find VP ellipsis to be the hardest phenomenon to be captured using round-trip translations 
Our key contributions are as follows:
we introduce the first approach to context-aware machine translation using only monolingual document-level data;
our approach shows substantial improvements in translation quality as measured by BLEU, targeted contrastive evaluation of several discourse phenomena and human evaluation;
we show which discourse phenomena are hard to capture using monolingual data only 
Our Approach: Document-level Repair
We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations of a context-agnostic MT system  It does not use any states of a trained MT model whose outputs it corrects and therefore can in principle be trained to correct translations from any black-box MT system 
The DocRepair model requires only monolingual document-level data in the target language  It is a monolingual sequence-to-sequence model that maps inconsistent groups of sentences into consistent ones  Consistent groups come from monolingual document-level data  To obtain inconsistent groups, each sentence in a group is replaced with its round-trip translation produced in isolation from context  More formally, forming a training minibatch for the DocRepair model involves the following steps (see also Figure FIGREF9):
sample several groups of sentences from the monolingual data;
for each sentence in a group, (i) translate it using a target-to-source MT model, (ii) sample a translation of this back-translated sentence in the source language using a source-to-target MT model;
using these round-trip translations of isolated sentences, form an inconsistent version of the initial groups;
use inconsistent groups as input for the DocRepair model, consistent ones as output 
At test time, the process of getting document-level translations is two-step (Figure FIGREF10):
produce translations of isolated sentences using a context-agnostic MT model;
apply the DocRepair model to a sequence of context-agnostic translations to correct inconsistencies between translations 
In the scope of the current work, the DocRepair model is the standard sequence-to-sequence Transformer  Sentences in a group are concatenated using a reserved token-separator between sentences  The Transformer is trained to correct these long inconsistent pseudo-sentences into consistent ones  The token-separator is then removed from corrected translations 
Evaluation of Contextual Phenomena
We use contrastive test sets for evaluation of discourse phenomena for English-Russian by BIBREF11  These test sets allow for testing different kinds of phenomena which, as we show, can be captured from monolingual data with varying success  In this section, we provide test sets statistics and briefly describe the tested phenomena  For more details, the reader is referred to BIBREF11 
Evaluation of Contextual Phenomena ::: Test sets
There are four test sets in the suite  Each test set contains contrastive examples  It is specifically designed to test the ability of a system to adapt to contextual information and handle the phenomenon under consideration  Each test instance consists of a true example (a sequence of sentences and their reference translation from the data) and several contrastive translations which differ from the true one only in one specific aspect  All contrastive translations are correct and plausible translations at the sentence level, and only context reveals the inconsistencies between them  The system is asked to score each candidate translation, and we compute the system accuracy as the proportion of times the true translation is preferred to the contrastive ones  Test set statistics are shown in Table TABREF15  The suites for deixis and lexical cohesion are split into development and test sets, with 500 examples from each used for validation purposes and the rest for testing  Convergence of both consistency scores on these development sets and BLEU score on a general development set are used as early stopping criteria in models training  For ellipsis, there is no dedicated development set, so we evaluate on all the ellipsis data and do not use it for development 
Evaluation of Contextual Phenomena ::: Phenomena overview
Deixis Deictic words or phrases, are referential expressions whose denotation depends on context  This includes personal deixis (“I”, “you”), place deixis (“here”, “there”), and discourse deixis, where parts of the discourse are referenced (“that's a good question”)  The test set examples are all related to person deixis, specifically the T-V distinction between informal and formal you (Latin “tu” and “vos”) in the Russian translations, and test for consistency in this respect 
Ellipsis Ellipsis is the omission from a clause of one or more words that are nevertheless understood in the context of the remaining elements  In machine translation, elliptical constructions in the source language pose a problem in two situations  First, if the target language does not allow the same types of ellipsis, requiring the elided material to be predicted from context  Second, if the elided material affects the syntax of the sentence  For example, in Russian the grammatical function of a noun phrase, and thus its inflection, may depend on the elided verb, or, conversely, the verb inflection may depend on the elided subject 
There are two different test sets for ellipsis  One contains examples where a morphological form of a noun group in the last sentence can not be understood without context beyond the sentence level (“ellipsis (infl )” in Table TABREF15)  Another includes cases of verb phrase ellipsis in English, which does not exist in Russian, thus requires predicting the verb when translating into Russian (“ellipsis (VP)” in Table TABREF15) 
Lexical cohesion The test set focuses on reiteration of named entities  Where several translations of a named entity are possible, a model has to prefer consistent translations over inconsistent ones 
Experimental Setup ::: Data preprocessing
We use the publicly available OpenSubtitles2018 corpus BIBREF12 for English and Russian  For a fair comparison with previous work, we train the baseline MT system on the data released by BIBREF11  Namely, our MT system is trained on 6m instances  These are sentence pairs with a relative time overlap of subtitle frames between source and target language subtitles of at least $0 9$ 
We gathered 30m groups of 4 consecutive sentences as our monolingual data  We used only documents not containing groups of sentences from general development and test sets as well as from contrastive test sets  The main results we report are for the model trained on all 30m fragments 
We use the tokenization provided by the corpus and use multi-bleu perl on lowercased data to compute BLEU score  We use beam search with a beam of 4 
Sentences were encoded using byte-pair encoding BIBREF13, with source and target vocabularies of about 32000 tokens  Translation pairs were batched together by approximate sequence length  Each training batch contained a set of translation pairs containing approximately 15000 source tokens  It has been shown that Transformer's performance depends heavily on batch size BIBREF14, and we chose a large batch size to ensure the best performance  In training context-aware models, for early stopping we use both convergence in BLEU score on the general development set and scores on the consistency development sets  After training, we average the 5 latest checkpoints 
Experimental Setup ::: Models
The baseline model, the model used for back-translation, and the DocRepair model are all Transformer base models BIBREF15  More precisely, the number of layers is $N=6$ with $h = 8$ parallel attention layers, or heads  The dimensionality of input and output is $d_{model} = 512$, and the inner-layer of a feed-forward networks has dimensionality $d_{ff}=2048$  We use regularization as described in BIBREF15 
As a second baseline, we use the two-pass CADec model BIBREF11  The first pass produces sentence-level translations  The second pass takes both the first-pass translation and representations of the context sentences as input and returns contextualized translations  CADec requires document-level parallel training data, while DocRepair only needs monolingual training data 
Experimental Setup ::: Generating round-trip translations
On the selected 6m instances we train sentence-level translation models in both directions  To create training data for DocRepair, we proceed as follows  The Russian monolingual data is first translated into English, using the Russian$\rightarrow $English model and beam search with beam size of 4  Then, we use the English$\rightarrow $Russian model to sample translations with temperature of $0{ }5$  For each sentence, we precompute 20 sampled translations and randomly choose one of them when forming a training minibatch for DocRepair  Also, in training, we replace each token in the input with a random one with the probability of $10\%$ 
Experimental Setup ::: Optimizer
As in BIBREF15, we use the Adam optimizer BIBREF16, the parameters are $\beta _1 = 0{ }9$, $\beta _2 = 0{ }98$ and $\varepsilon = 10^{-9}$  We vary the learning rate over the course of training using the formula:
where $warmup\_steps = 16000$ and $scale=4$ 
Results ::: General results
The BLEU scores are provided in Table TABREF24 (we evaluate translations of 4-sentence fragments)  To see which part of the improvement is due to fixing agreement between sentences rather than simply sentence-level post-editing, we train the same repair model at the sentence level  Each sentence in a group is now corrected separately, then they are put back together in a group  One can see that most of the improvement comes from accounting for extra-sentential dependencies  DocRepair outperforms the baseline and CADec by 0 7 BLEU, and its sentence-level repair version by 0 5 BLEU 
Results ::: Consistency results
Scores on the phenomena test sets are provided in Table TABREF26  For deixis, lexical cohesion and ellipsis (infl ) we see substantial improvements over both the baseline and CADec  The largest improvement over CADec (22 5 percentage points) is for lexical cohesion  However, there is a drop of almost 5 percentage points for VP ellipsis  We hypothesize that this is because it is hard to learn to correct inconsistencies in translations caused by VP ellipsis relying on monolingual data alone  Figure FIGREF27(a) shows an example of inconsistency caused by VP ellipsis in English  There is no VP ellipsis in Russian, and when translating auxiliary “did” the model has to guess the main verb  Figure FIGREF27(b) shows steps of generating round-trip translations for the target side of the previous example  When translating from Russian, main verbs are unlikely to be translated as the auxiliary “do” in English, and hence the VP ellipsis is rarely present on the English side  This implies the model trained using the round-trip translations will not be exposed to many VP ellipsis examples in training  We discuss this further in Section SECREF34 
Table TABREF28 provides scores for deixis and lexical cohesion separately for different distances between sentences requiring consistency  It can be seen, that the performance of DocRepair degrades less than that of CADec when the distance between sentences requiring consistency gets larger 
Results ::: Human evaluation
We conduct a human evaluation on random 700 examples from our general test set  We picked only examples where a DocRepair translation is not a full copy of the baseline one 
The annotators were provided an original group of sentences in English and two translations: baseline context-agnostic one and the one corrected by the DocRepair model  Translations were presented in random order with no indication which model they came from  The task is to pick one of the three options: (1) the first translation is better, (2) the second translation is better, (3) the translations are of equal quality  The annotators were asked to avoid the third answer if they are able to give preference to one of the translations  No other guidelines were given 
The results are provided in Table TABREF30  In about $52\%$ of the cases annotators marked translations as having equal quality  Among the cases where one of the translations was marked better than the other, the DocRepair translation was marked better in $73\%$ of the cases  This shows a strong preference of the annotators for corrected translations over the baseline ones 
Varying Training Data
In this section, we discuss the influence of the training data chosen for document-level models  In all experiments, we used the DocRepair model 
Varying Training Data ::: The amount of training data
Table TABREF33 provides BLEU and consistency scores for the DocRepair model trained on different amount of data  We see that even when using a dataset of moderate size (e g , 5m fragments) we can achieve performance comparable to the model trained on a large amount of data (30m fragments)  Moreover, we notice that deixis scores are less sensitive to the amount of training data than lexical cohesion and ellipsis scores  The reason might be that, as we observed in our previous work BIBREF11, inconsistencies in translations due to the presence of deictic words and phrases are more frequent in this dataset than other types of inconsistencies  Also, as we show in Section SECREF7, this is the phenomenon the model learns faster in training 
Varying Training Data ::: One-way vs round-trip translations
In this section, we discuss the limitations of using only monolingual data to model inconsistencies between sentence-level translations  In Section SECREF25 we observed a drop in performance on VP ellipsis for DocRepair compared to CADec, which was trained on parallel data  We hypothesized that this is due to the differences between one-way and round-trip translations, and now we test this hypothesis  To do so, we fix the dataset and vary the way in which the input for DocRepair is generated: round-trip or one-way translations  The latter assumes that document-level data is parallel, and translations are sampled from the source side of the sentences in a group rather than from their back-translations  For parallel data, we take 1 5m parallel instances which were used for CADec training and add 1m instances from our monolingual data  For segments in the parallel part, we either sample translations from the source side or use round-trip translations  The results are provided in Table TABREF35 
The model trained on one-way translations is slightly better than the one trained on round-trip translations  As expected, VP ellipsis is the hardest phenomena to be captured using round-trip translations, and the DocRepair model trained on one-way translated data gains 6% accuracy on this test set  This shows that the DocRepair model benefits from having access to non-synthetic English data  This results in exposing DocRepair at training time to Russian translations which suffer from the same inconsistencies as the ones it will have to correct at test time 
Varying Training Data ::: Filtering: monolingual (no filtering) or parallel
Note that the scores of the DocRepair model trained on 2 5m instances randomly chosen from monolingual data (Table TABREF33) are different from the ones for the model trained on 2 5m instances combined from parallel and monolingual data (Table TABREF35)  For convenience, we show these two in Table TABREF36 
The domain, the dataset these two data samples were gathered from, and the way we generated training data for DocRepair (round-trip translations) are all the same  The only difference lies in how the data was filtered  For parallel data, as in the previous work BIBREF6, we picked only sentence pairs with large relative time overlap of subtitle frames between source-language and target-language subtitles  This is necessary to ensure the quality of translation data: one needs groups of consecutive sentences in the target language where every sentence has a reliable translation 
Table TABREF36 shows that the quality of the model trained on data which came from the parallel part is worse than the one trained on monolingual data  This indicates that requiring each sentence in a group to have a reliable translation changes the distribution of the data, which might be not beneficial for translation quality and provides extra motivation for using monolingual data 
Learning Dynamics
Let us now look into how the process of DocRepair training progresses  Figure FIGREF38 shows how the BLEU scores with the reference translation and with the baseline context-agnostic translation (i e  the input for the DocRepair model) are changing during training  First, the model quickly learns to copy baseline translations: the BLEU score with the baseline is very high  Then it gradually learns to change them, which leads to an improvement in BLEU with the reference translation and a drop in BLEU with the baseline  Importantly, the model is reluctant to make changes: the BLEU score between translations of the converged model and the baseline is 82 5  We count the number of changed sentences in every 4-sentence fragment in the test set and plot the histogram in Figure FIGREF38  In over than 20$\%$ of the cases the model has not changed base translations at all  In almost $40\%$, it modified only one sentence and left the remaining 3 sentences unchanged  The model changed more than half sentences in a group in only $14\%$ of the cases  Several examples of the DocRepair translations are shown in Figure FIGREF43 
Figure FIGREF42 shows how consistency scores are changing in training  For deixis, the model achieves the final quality quite quickly; for the rest, it needs a large number of training steps to converge 
Related Work
Our work is most closely related to two lines of research: automatic post-editing (APE) and document-level machine translation 
Related Work ::: Automatic post-editing
Our model can be regarded as an automatic post-editing system – a system designed to fix systematic MT errors that is decoupled from the main MT system  Automatic post-editing has a long history, including rule-based BIBREF17, statistical BIBREF18 and neural approaches BIBREF19, BIBREF20, BIBREF21 
In terms of architectures, modern approaches use neural sequence-to-sequence models, either multi-source architectures that consider both the original source and the baseline translation BIBREF19, BIBREF20, or monolingual repair systems, as in BIBREF21, which is concurrent work to ours  True post-editing datasets are typically small and expensive to create BIBREF22, hence synthetic training data has been created that uses original monolingual data as output for the sequence-to-sequence model, paired with an automatic back-translation BIBREF23 and/or round-trip translation as its input(s) BIBREF19, BIBREF21 
While previous work on automatic post-editing operated on the sentence level, the main novelty of this work is that our DocRepair model operates on groups of sentences and is thus able to fix consistency errors caused by the context-agnostic baseline MT system  We consider this strategy of sentence-level baseline translation and context-aware monolingual repair attractive when parallel document-level data is scarce 
For training, the DocRepair model only requires monolingual document-level data  While we create synthetic training data via round-trip translation similarly to earlier work BIBREF19, BIBREF21, note that we purposefully use sentence-level MT systems for this to create the types of consistency errors that we aim to fix with the context-aware DocRepair model  Not all types of consistency errors that we want to fix emerge from a round-trip translation, so access to parallel document-level data can be useful (Section SECREF34) 
Related Work ::: Document-level NMT
Neural models of MT that go beyond the sentence-level are an active research area BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF10, BIBREF9, BIBREF11  Typically, the main MT system is modified to take additional context as its input  One limitation of these approaches is that they assume that parallel document-level training data is available 
Closest to our work are two-pass models for document-level NMT BIBREF24, BIBREF11, where a second, context-aware model takes the translation and hidden representations of the sentence-level first-pass model as its input  The second-pass model can in principle be trained on a subset of the parallel training data BIBREF11, somewhat relaxing the assumption that all training data is at the document level 
Our work is different from this previous work in two main respects  Firstly, we show that consistency can be improved with only monolingual document-level training data  Secondly, the DocRepair model is decoupled from the first-pass MT system, which improves its portability 
Conclusions
We introduce the first approach to context-aware machine translation using only monolingual document-level data  We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations  The model performs automatic post-editing on a sequence of sentence-level translations, refining translations of sentences in context of each other  Our approach results in substantial improvements in translation quality as measured by BLEU, targeted contrastive evaluation of several discourse phenomena and human evaluation  Moreover, we perform error analysis and detect which discourse phenomena are hard to capture using only monolingual document-level data  While in the current work we used text fragments of 4 sentences, in future work we would like to consider longer contexts 
Acknowledgments
We would like to thank the anonymous reviewers for their comments  The authors also thank David Talbot and Yandex Machine Translation team for helpful discussions and inspiration  Ivan Titov acknowledges support of the European Research Council (ERC StG BroadSem 678254) and the Dutch National Science Foundation (NWO VIDI 639 022 518)  Rico Sennrich acknowledges support from the Swiss National Science Foundation (105212_169888), the European Union’s Horizon 2020 research and innovation programme (grant agreement no 825460), and the Royal Society (NAF\R1\180122) 

ground_truth: Bayesian model of garg2012unsupervised as our base monolingual model

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what was the baseline?
Context 4:  Table TABREF2 describes most hyper-parameters explored for each dataset  In all, 80 runs (of about 160 minutes) were conducted for the 15MB Wiki Abstract dataset with 80 serialized models totaling 15 136GB while 80 runs (for over 320 hours) were conducted for the 711MB SW dataset, with 80 serialized models totaling over 145GB  Experiments for all combinations for 300 dimensions were conducted on the 3 9GB training set of the BW corpus and additional runs for other dimensions for the window 8 + skipgram + heirarchical softmax combination to verify the trend of quality of word vectors as dimensions are increased 
Google (semantic and syntactic) analogy tests and WordSimilarity-353 (with Spearman correlation) by BIBREF20 were chosen for intrinsic evaluations  They measure the quality of word vectors  The analogy scores are averages of both semantic and syntactic tests  NER and SA were chosen for extrinsic evaluations  The GMB dataset for NER was trained in an LSTM network, which had an embedding layer for input  The network diagram is shown in fig  FIGREF4  The IMDb dataset for SA was trained in a BiLSTM network, which also used an embedding layer for input  Its network diagram is given in fig  FIGREF4  It includes an additional hidden linear layer  Hyper-parameter details of the two networks for the downstream tasks are given in table TABREF3  The metrics for extrinsic evaluation include F1, precision, recall and accuracy scores  In both tasks, the default pytorch embedding was tested before being replaced by pre-trained embeddings released by BIBREF0 and ours  In each case, the dataset was shuffled before training and split in the ratio 70:15:15 for training, validation (dev) and test sets  Batch size of 64 was used  For each task, experiments for each embedding was conducted four times and an average value calculated and reported in the next section
Results and Discussion
Table TABREF5 summarizes key results from the intrinsic evaluations for 300 dimensions  Table TABREF6 reveals the training time (in hours) and average embedding loading time (in seconds) representative of the various models used  Tables TABREF11 and TABREF12 summarize key results for the extrinsic evaluations  Figures FIGREF7, FIGREF9, FIGREF10, FIGREF13 and FIGREF14 present line graph of the eight combinations for different dimension sizes for Simple Wiki, trend of Simple Wiki and Billion Word corpora over several dimension sizes, analogy score comparison for models across datasets, NER mean F1 scores on the GMB dataset and SA mean F1 scores on the IMDb dataset, respectively  Combination of the skipgram using hierarchical softmax and window size of 8 for 300 dimensions outperformed others in analogy scores for the Wiki Abstract  However, its results are so poor, because of the tiny file size, they're not worth reporting here  Hence, we'll focus on results from the Simple Wiki and Billion Word corpora 
Best combination changes when corpus size increases, as will be noticed from table TABREF5  In terms of analogy score, for 10 epochs, w8s0h0 performs best while w8s1h0 performs best in terms of WordSim and corresponding Spearman correlation  Meanwhile, increasing the corpus size to BW, w4s1h0 performs best in terms of analogy score while w8s1h0 maintains its position as the best in terms of WordSim and Spearman correlation  Besides considering quality metrics, it can be observed from table TABREF6 that comparative ratio of values between the models is not commensurate with the results in intrinsic or extrinsic values, especially when we consider the amount of time and energy spent, since more training time results in more energy consumption (BIBREF23) 
Information on the length of training time for the released Mikolov model is not readily available  However, it's interesting to note that their presumed best model, which was released is also s1h0  Its analogy score, which we tested and report, is confirmed in their paper  It beats our best models in only analogy score (even for Simple Wiki), performing worse in others  This is inspite of using a much bigger corpus of 3,000,000 vocabulary size and 100 billion words while Simple Wiki had vocabulary size of 367,811 and is 711MB  It is very likely our analogy scores will improve when we use a much larger corpus, as can be observed from table TABREF5, which involves just one billion words 
Although the two best combinations in analogy (w8s0h0 & w4s0h0) for SW, as shown in fig  FIGREF7, decreased only slightly compared to others with increasing dimensions, the increased training time and much larger serialized model size render any possible minimal score advantage over higher dimensions undesirable  As can be observed in fig  FIGREF9, from 100 dimensions, scores improve but start to drop after over 300 dimensions for SW and after over 400 dimensions for BW  More becomes worse! This trend is true for all combinations for all tests  Polynomial interpolation may be used to determine the optimal dimension in both corpora  Our models are available for confirmation and source codes are available on github 
With regards to NER, most pretrained embeddings outperformed the default pytorch embedding, with our BW w4s1h0 model (which is best in BW analogy score) performing best in F1 score and closely followed by BIBREF0 model  On the other hand, with regards to SA, pytorch embedding outperformed the pretrained embeddings but was closely followed by our SW w8s0h0 model (which also had the best SW analogy score)  BIBREF0 performed second worst of all, despite originating from a very huge corpus  The combinations w8s0h0 & w4s0h0 of SW performed reasonably well in both extrinsic tasks, just as the default pytorch embedding did 
Conclusion
This work analyses, empirically, optimal combinations of hyper-parameters for embeddings, specifically for word2vec  It further shows that for downstream tasks, like NER and SA, there's no silver bullet! However, some combinations show strong performance across tasks  Performance of embeddings is task-specific and high analogy scores do not necessarily correlate positively with performance on downstream tasks  This point on correlation is somewhat similar to results by BIBREF24 and BIBREF14  It was discovered that increasing dimension size depreciates performance after a point  If strong considerations of saving time, energy and the environment are made, then reasonably smaller corpora may suffice or even be better in some cases  The on-going drive by many researchers to use ever-growing data to train deep neural networks can benefit from the findings of this work  Indeed, hyper-parameter choices are very important in neural network systems (BIBREF19) 
Future work that may be investigated are performance of other architectures of word or sub-word embeddings, the performance and comparison of embeddings applied to languages other than English and how embeddings perform in other downstream tasks  In addition, since the actual reason for the changes in best model as corpus size increases is not clear, this will also be suitable for further research 
The work on this project is partially funded by Vinnova under the project number 2019-02996 "Språkmodeller för svenska myndigheter"
Acronyms

ground_truth:  MT system on the data released by BIBREF11

Transformer base, two-pass CADec model