prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Were any of these tasks evaluated in any previous work?
Context 1: Introduction
The recently introduced BERT model BIBREF0 exhibits strong performance on several language understanding benchmarks  To what extent does it capture syntax-sensitive structures?
Recent work examines the extent to which RNN-based models capture syntax-sensitive phenomena that are traditionally taken as evidence for the existence in hierarchical structure  In particular, in BIBREF1 we assess the ability of LSTMs to learn subject-verb agreement patterns in English, and evaluate on naturally occurring wikipedia sentences  BIBREF2 also consider subject-verb agreement, but in a “colorless green ideas” setting in which content words in naturally occurring sentences are replaced with random words with the same part-of-speech and inflection, thus ensuring a focus on syntax rather than on selectional-preferences based cues  BIBREF3 consider a wider range of syntactic phenomena (subject-verb agreement, reflexive anaphora, negative polarity items) using manually constructed stimuli, allowing for greater coverage and control than in the naturally occurring setting 
The BERT model is based on the “Transformer” architecture BIBREF4 , which—in contrast to RNNs—relies purely on attention mechanisms, and does not have an explicit notion of word order beyond marking each word with its absolute-position embedding  This reliance on attention may lead one to expect decreased performance on syntax-sensitive tasks compared to RNN (LSTM) models that do model word order directly, and explicitly track states across the sentence  Indeed, BIBREF5 finds that transformer-based models perform worse than LSTM models on the BIBREF1 agreement prediction dataset  In contrast, BIBREF6 find that self-attention performs on par with LSTM for syntax sensitive dependencies in the context of machine-translation, and performance on syntactic tasks is correlated with the number of attention heads in multi-head attention 
I adapt the evaluation protocol and stimuli of BIBREF1 , BIBREF2 and BIBREF3 to the bidirectional setting required by BERT, and evaluate the pre-trained BERT models (both the Large and the Base models)  Surprisingly (at least to me), the out-of-the-box models (without any task-specific fine-tuning) perform very well on all the syntactic tasks 
Methodology
I use the stimuli provided by BIBREF1 , BIBREF2 , BIBREF3 , but change the experimental protocol to adapt it to the bidirectional nature of the BERT model  This requires discarding some of the stimuli, as described below  Thus, the numbers are not strictly comparable to those reported in previous work 
Previous setups
All three previous work use uni-directional language-model-like models 
BIBREF1 start with existing sentences from wikipedia that contain a present-tense verb  They feed each sentence word by word into an LSTM, stop right before the focus verb, and ask the model to predict a binary plural/singular decision (supervised setup) or compare the probability assigned by a pre-trained language model (LM) to the plural vs singular forms of the verb (LM setup)  The evaluation is then performed on sentences with “agreement attractors” in which at there is at least one noun between the verb and its subject, and all of the nouns between the verb and subject are of the opposite number from the subject 
BIBREF2 also start with existing sentences  However, in order to control for the possibillity of the model learning to rely on “semantic” selectional-preferences cues rather than syntactic ones, they replace each content word with random words from the same part-of-speech and inflection  This results in “coloreless green ideas” nonce sentences  The evaluation is then performed similarly to the LM setup of BIBREF1 : the sentence is fed into a pre-traiend LSTM LM up to the focus verb, and the model is considered correct if the probability assigned to the correct inflection of the original verb form given the prefix is larger than that assigned to the incorrect inflection 
BIBREF3 focus on manually constructed and controlled stimuli, that also emphasizes linguistic structure over selectional preferences  They construct minimal pairs of grammatical and ungrammatical sentences, feed each one in its entirety into a pre-trained LSTM-LM, and compare the perplexity assigned by the model to the grammatical and ungrammatical sentences  The model is “correct” if it assigns the grammatical sentence a higher probability than to the ungrammatical one  Since the minimal pairs for most phenomena differ only in a single word (the focus verb), this scoring is very similar to the one used in the two previous works  However, it does consider the continuation of the sentence after the focus verb, and also allows for assessing phenomena that require change into two or more words (like negative polarity items) 
Adaptation to the BERT model
In contrast to these works, the BERT model is bi-directional: it is trained to predict the identity of masked words based on both the prefix and suffix surrounding these words  I adapt the uni-directional setup by feeding into BERT the complete sentence, while masking out the single focus verb  I then ask BERT for its word predictions for the masked position, and compare the score assigned to the original correct verb to the score assigned to the incorrect one 
For example, for the sentence:
a 2002 systemic review of herbal products found that several herbs , including peppermint and caraway , have anti-dyspeptic effects for non-ulcer dyspepsia with “ encouraging safety profiles ”   (from BIBREF1 )
I feed into BERT:
[CLS] a 2002 systemic review of herbal products found that several herbs , including peppermint and caraway , [MASK] anti-dyspeptic effects for non-ulcer dyspepsia with “ encouraging safety profiles ”   and look for the score assigned to the words have and has at the masked position 
Similarly, for the pair
the game that the guard hates is bad  
the game that the guard hates are bad  
(from BIBREF3 ), I feed into BERT:
[CLS] the game that the guard hates [MASK] bad  
and compare the scores predicted for is and are 
This differs from BIBREF1 and BIBREF2 by considering the entire sentence (excluding the verb) and not just its prefix leading to the verb, and differs from BIBREF3 by conditioning the focus verb on bidirectional context 
I use the PyTorch implementation of BERT, with the pre-trained models supplied by Google  I experiment with the bert-large-uncased and bert-base-uncased models 
The bi-directional setup precludes using using the NPI stimuli of BIBREF3 , in which the minimal pair differs in two words position, which I discard from the evaluation  I also discard the agreement cases involving the verbs is or are in BIBREF1 and in BIBREF2 , because some of them are copular construction, in which strong agreement hints can be found also on the object following the verb  This is not an issue in the manually constructed BIBREF3 stimuli due to the patterns they chose 
Finally, I discard stimuli in which the focus verb or its plural/singular inflection does not appear as a single word in the BERT word-piece-based vocabulary (and hence cannot be predicted by the model)  This include discarding BIBREF3 stimuli involving the words swims or admires, resulting in 23,368 discarded pairs (out of 152,300)  I similarly discard 680 sentences from BIBREF1 where the focus verb or its inflection were one of 108 out-of-vocabulary tokens, and 28 sentence-pairs (8 tokens) from BIBREF2  
The BERT results are not directly comparable to the numbers reported in previous work  Beyond the differences due to bidirectionality and the discarded stimuli, the BERT models are also trained on a different and larger corpus (covering both wikipedia and books) 
Code is available at https://github com/yoavg/bert-syntax 
Results
Tables 1 , 2 and 3 show the results  All cases exhibit high scores—in the vast majority of the cases substantially higher than reported in previous work  As discussed above, the results are not directly comparable to previous work: the BERT models are trained on different (and larger) data, are allowed to access the suffix of the sentence in addition to its prefix, and are evaluated on somewhat different data due to discarding OOV items  Still, taken together, the high performance numbers indicate that the purely attention-based BERT models are likely capable of capturing the same kind of syntactic regularities that LSTM-based models are capable of capturing, at least as well as the LSTM models and probably better 
Another noticeable and interesting trend is that larger is not necessarily better: the BERT-Base model outperforms the BERT-Large model on many of the syntactic conditions 
Discussion
The BERT models perform remarkably well on all the syntactic test cases  I expected the attention-based mechanism to fail on these (compared to the LSTM-based models), and am surprised by these results  The BIBREF2 and BIBREF3 conditions rule out the possibility of overly relying on selectional preference cues or memorizing the wikipedia training data, and suggest real syntactic generalization is taking place  Exploring the extent to which deep purely-attention-based architectures such as BERT are capable of capturing hierarchy-sensitive and syntactic dependencies—as well as the mechanisms by which this is achieved—is a fascinating area for future research 
Context 3: Introduction
Word Sense Disambiguation (WSD) is a fundamental task and long-standing challenge in Natural Language Processing (NLP), which aims to find the exact sense of an ambiguous word in a particular context BIBREF0  Previous WSD approaches can be grouped into two main categories: knowledge-based and supervised methods 
Knowledge-based WSD methods rely on lexical resources like WordNet BIBREF1 and usually exploit two kinds of lexical knowledge  The gloss, which defines a word sense meaning, is first utilized in Lesk algorithm BIBREF2 and then widely taken into account in many other approaches BIBREF3, BIBREF4  Besides, structural properties of semantic graphs are mainly used in graph-based algorithms BIBREF5, BIBREF6 
Traditional supervised WSD methods BIBREF7, BIBREF8, BIBREF9 focus on extracting manually designed features and then train a dedicated classifier (word expert) for every target lemma 
Although word expert supervised WSD methods perform better, they are less flexible than knowledge-based methods in the all-words WSD task BIBREF10  Recent neural-based methods are devoted to dealing with this problem  BIBREF11 present a supervised classifier based on Bi-LSTM, which shares parameters among all word types except the last layer  BIBREF10 convert WSD task to a sequence labeling task, thus building a unified model for all polysemous words  However, neither of them can totally beat the best word expert supervised methods 
More recently, BIBREF12 propose to leverage the gloss information from WordNet and model the semantic relationship between the context and gloss in an improved memory network  Similarly, BIBREF13 introduce a (hierarchical) co-attention mechanism to generate co-dependent representations for the context and gloss  Their attempts prove that incorporating gloss knowledge into supervised WSD approach is helpful, but they still have not achieved much improvement, because they may not make full use of gloss knowledge 
In this paper, we focus on how to better leverage gloss information in a supervised neural WSD system  Recently, the pre-trained language models, such as ELMo BIBREF14 and BERT BIBREF15, have shown their effectiveness to alleviate the effort of feature engineering  Especially, BERT has achieved excellent results in question answering (QA) and natural language inference (NLI)  We construct context-gloss pairs from glosses of all possible senses (in WordNet) of the target word, thus treating WSD task as a sentence-pair classification problem  We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task  In particular, our contribution is two-fold:
1  We construct context-gloss pairs and propose three BERT-based models for WSD 
2  We fine-tune the pre-trained BERT model, and the experimental results on several English all-words WSD benchmark datasets show that our approach significantly outperforms the state-of-the-art systems 
Methodology
In this section, we describe our method in detail 
Methodology ::: Task Definition
In WSD, a sentence $s$ usually consists of a series of words: $\lbrace w_1,\cdots ,w_m\rbrace $, and some of the words $\lbrace w_{i_1},\cdots ,w_{i_k}\rbrace $ are targets $\lbrace t_1,\cdots ,t_k\rbrace $ need to be disambiguated  For each target $t$, its candidate senses $\lbrace c_1,\cdots ,c_n\rbrace $ come from entries of its lemma in a pre-defined sense inventory (usually WordNet)  Therefore, WSD task aims to find the most suitable entry (symbolized as unique sense key) for each target in a sentence  See a sentence example in Table TABREF1 
Methodology ::: BERT
BERT BIBREF15 is a new language representation model, and its architecture is a multi-layer bidirectional Transformer encoder  BERT model is pre-trained on a large corpus and two novel unsupervised prediction tasks, i e , masked language model and next sentence prediction tasks are used in pre-training  When incorporating BERT into downstream tasks, the fine-tuning procedure is recommended  We fine-tune the pre-trained BERT model on WSD task 
Methodology ::: BERT ::: BERT(Token-CLS)
Since every target in a sentence needs to be disambiguated to find its exact sense, WSD task can be regarded as a token-level classification task  To incorporate BERT to WSD task, we take the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer for every target lemma, which is the same as the last layer of the Bi-LSTM model BIBREF11 
Methodology ::: GlossBERT
BERT can explicitly model the relationship of a pair of texts, which has shown to be beneficial to many pair-wise natural language understanding tasks  In order to fully leverage gloss information, we propose GlossBERT to construct context-gloss pairs from all possible senses of the target word in WordNet, thus treating WSD task as a sentence-pair classification problem 
We describe our construction method with an example (See Table TABREF1)  There are four targets in this sentence, and here we take target word research as an example:
Methodology ::: GlossBERT ::: Context-Gloss Pairs
The sentence containing target words is denoted as context sentence  For each target word, we extract glosses of all $N$ possible senses (here $N=4$) of the target word (research) in WordNet to obtain the gloss sentence  [CLS] and [SEP] marks are added to the context-gloss pairs to make it suitable for the input of BERT model  A similar idea is also used in aspect-based sentiment analysis BIBREF16 
Methodology ::: GlossBERT ::: Context-Gloss Pairs with Weak Supervision
Based on the previous construction method, we add weak supervised signals to the context-gloss pairs (see the highlighted part in Table TABREF1)  The signal in the gloss sentence aims to point out the target word, and the signal in the context sentence aims to emphasize the target word considering the situation that a target word may occur more than one time in the same sentence 
Therefore, each target word has $N$ context-gloss pair training instances ($label\in \lbrace yes, no\rbrace $)  When testing, we output the probability of $label=yes$ of each context-gloss pair and choose the sense corresponding to the highest probability as the prediction label of the target word  We experiment with three GlossBERT models:
Methodology ::: GlossBERT ::: GlossBERT(Token-CLS)
We use context-gloss pairs as input  We highlight the target word by taking the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer ($label\in \lbrace yes, no\rbrace $) 
Methodology ::: GlossBERT ::: GlossBERT(Sent-CLS)
We use context-gloss pairs as input  We take the final hidden state of the first token [CLS] as the representation of the whole sequence and add a classification layer ($label\in \lbrace yes, no\rbrace $), which does not highlight the target word 
Methodology ::: GlossBERT ::: GlossBERT(Sent-CLS-WS)
We use context-gloss pairs with weak supervision as input  We take the final hidden state of the first token [CLS] and add a classification layer ($label\in \lbrace yes, no\rbrace $), which weekly highlight the target word by the weak supervision

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the improvement in performance for Estonian in the NER task?
Context 2: Introduction
Word embeddings are representations of words in numerical form, as vectors of typically several hundred dimensions  The vectors are used as an input to machine learning models; for complex language processing tasks these are typically deep neural networks  The embedding vectors are obtained from specialized learning tasks, based on neural networks, e g , word2vec BIBREF0, GloVe BIBREF1, FastText BIBREF2, ELMo BIBREF3, and BERT BIBREF4  For training, the embeddings algorithms use large monolingual corpora that encode important information about word meaning as distances between vectors  In order to enable downstream machine learning on text understanding tasks, the embeddings shall preserve semantic relations between words, and this is true even across languages 
Probably the best known word embeddings are produced by the word2vec method BIBREF5  The problem with word2vec embeddings is their failure to express polysemous words  During training of an embedding, all senses of a given word (e g , paper as a material, as a newspaper, as a scientific work, and as an exam) contribute relevant information in proportion to their frequency in the training corpus  This causes the final vector to be placed somewhere in the weighted middle of all words' meanings  Consequently, rare meanings of words are poorly expressed with word2vec and the resulting vectors do not offer good semantic representations  For example, none of the 50 closest vectors of the word paper is related to science 
The idea of contextual embeddings is to generate a different vector for each context a word appears in and the context is typically defined sentence-wise  To a large extent, this solves the problems with word polysemy, i e  the context of a sentence is typically enough to disambiguate different meanings of a word for humans and so it is for the learning algorithms  In this work, we describe high-quality models for contextual embeddings, called ELMo BIBREF3, precomputed for seven morphologically rich, less-resourced languages: Slovenian, Croatian, Finnish, Estonian, Latvian, Lithuanian, and Swedish  ELMo is one of the most successful approaches to contextual word embeddings  At time of its creation, ELMo has been shown to outperform previous word embeddings BIBREF3 like word2vec and GloVe on many NLP tasks, e g , question answering, named entity extraction, sentiment analysis, textual entailment, semantic role labeling, and coreference resolution 
This report is split into further five sections  In section SECREF2, we describe the contextual embeddings ELMo  In Section SECREF3, we describe the datasets used and in Section SECREF4 we describe preprocessing and training of the embeddings  We describe the methodology for evaluation of created vectors and results in Section SECREF5  We present conclusion in Section SECREF6 where we also outline plans for further work 
ELMo
Typical word embeddings models or representations, such as word2vec BIBREF0, GloVe BIBREF1, or FastText BIBREF2, are fast to train and have been pre-trained for a number of different languages  They do not capture the context, though, so each word is always given the same vector, regardless of its context or meaning  This is especially problematic for polysemous words  ELMo (Embeddings from Language Models) embedding BIBREF3 is one of the state-of-the-art pretrained transfer learning models, that remedies the problem and introduces a contextual component 
ELMo model`s architecture consists of three neural network layers  The output of the model after each layer gives one set of embeddings, altogether three sets  The first layer is a CNN layer, which operates on a character level  It is context independent, so each word always gets the same embedding, regardless of its context  It is followed by two biLM layers  A biLM layer consists of two concatenated LSTMs  In the first LSTM, we try to predict the following word, based on the given past words, where each word is represented by the embeddings from the CNN layer  In the second LSTM, we try to predict the preceding word, based on the given following words  It is equivalent to the first LSTM, just reading the text in reverse 
In NLP tasks, any set of these embeddings may be used; however, a weighted average is usually used  The weights of the average are learned during the training of the model for the specific task  Additionally, an entire ELMo model can be fine-tuned on a specific end task 
Although ELMo is trained on character level and is able to handle out-of-vocabulary words, a vocabulary file containing most common tokens is used for efficiency during training and embedding generation  The original ELMo model was trained on a one billion word large English corpus, with a given vocabulary file of about 800,000 words  Later, ELMo models for other languages were trained as well, but limited to larger languages with many resources, like German and Japanese 
ELMo ::: ELMoForManyLangs
Recently, ELMoForManyLangs BIBREF6 project released pre-trained ELMo models for a number of different languages BIBREF7  These models, however, were trained on a significantly smaller datasets  They used 20-million-words data randomly sampled from the raw text released by the CoNLL 2017 Shared Task - Automatically Annotated Raw Texts and Word Embeddings BIBREF8, which is a combination of Wikipedia dump and common crawl  The quality of these models is questionable  For example, we compared the Latvian model by ELMoForManyLangs with a model we trained on a complete (wikidump + common crawl) Latvian corpus, which has about 280 million tokens  The difference of each model on the word analogy task is shown in Figure FIGREF16 in Section SECREF5  As the results of the ELMoForManyLangs embeddings are significantly worse than using the full corpus, we can conclude that these embeddings are not of sufficient quality  For that reason, we computed ELMo embeddings for seven languages on much larger corpora  As this effort requires access to large amount of textual data and considerable computational resources, we made the precomputed models publicly available by depositing them to Clarin repository 
Training Data
We trained ELMo models for seven languages: Slovenian, Croatian, Finnish, Estonian, Latvian, Lithuanian and Swedish  To obtain high-quality embeddings, we used large monolingual corpora from various sources for each language  Some corpora are available online under permissive licences, others are available only for research purposes or have limited availability  The corpora used in training datasets are a mix of news articles and general web crawl, which we preprocessed and deduplicated  Below we shortly describe the used corpora in alphabetical order of the involved languages  Their names and sizes are summarized in Table TABREF3 
Croatian dataset include hrWaC 2 1 corpus BIBREF9, Riznica BIBREF10, and articles of Croatian branch of Styria media house, made available to us through partnership in a joint project  hrWaC was built by crawling the  hr internet domain in 2011 and 2014  Riznica is composed of Croatian fiction and non-fiction prose, poetry, drama, textbooks, manuals, etc  The Styria dataset consists of 570,219 news articles published on the Croatian 24sata news portal and niche portals related to 24sata 
Estonian dataset contains texts from two sources, CoNLL 2017 Shared Task - Automatically Annotated Raw Texts and Word Embeddings BIBREF8, and news articles made available to us by Ekspress Meedia due to partnership in the project  Ekspress Meedia dataset is composed of Estonian news articles between years 2009 and 2019  The CoNLL 2017 corpus is composed of Estonian Wikipedia and webcrawl 
Finnish dataset contains articles by Finnish news agency STT, Finnish part of the CoNLL 2017 dataset, and Ylilauta downloadable version BIBREF11  STT news articles were published between years 1992 and 2018  Ylilauta is a Finnish online discussion board; the corpus contains parts of the discussions from 2012 to 2014 
Latvian dataset consists only of the Latvian portion of the ConLL 2017 corpus 
Lithuanian dataset is composed of Lithuanian Wikipedia articles from 2018, DGT-UD corpus, and LtTenTen  DGT-UD is a parallel corpus of 23 official languages of the EU, composed of JRC DGT translation memory of European law, automatically annotated with UD-Pipe 1 2  LtTenTen is Lithuanian web corpus made up of texts collected from the internet in April 2014 BIBREF12 
Slovene dataset is formed from the Gigafida 2 0 corpus BIBREF13  It is a general language corpus composed of various sources, mostly newspapers, internet pages, and magazines, but also fiction and non-fiction prose, textbooks, etc 
Swedish dataset is composed of STT Swedish articles and Swedish part of CoNLL 2017  The Finnish news agency STT publishes some of its articles in Swedish language  They were made available to us through partnership in a joint project  The corpus contains those articles from 1992 to 2017 
Preprocessing and Training
Prior to training the ELMo models, we sentence and word tokenized all the datasets  The text was formatted in such a way that each sentence was in its own line with tokens separated by white spaces  CoNLL 2017, DGT-UD and LtTenTen14 corpora were already pre-tokenized  We tokenized the others using the NLTK library and its tokenizers for each of the languages  There is no tokenizer for Croatian in NLTK library, so we used Slovene tokenizer instead  After tokenization, we deduplicated the datasets for each language separately, using the Onion (ONe Instance ONly) tool for text deduplication  We applied the tool on paragraph level for corpora that did not have sentences shuffled and on sentence level for the rest  We considered 9-grams with duplicate content threshold of 0 9 
For each language we prepared a vocabulary file, containing roughly one million most common tokens, i e  tokens that appear at least $n$ times in the corpus, where $n$ is between 15 and 25, depending on the dataset size  We included the punctuation marks among the tokens  We trained each ELMo model using default values used to train the original English ELMo (large) model 
Evaluation
We evaluated the produced ELMo models for all languages using two evaluation tasks: a word analogy task and named entity recognition (NER) task  Below, we first shortly describe each task, followed by the evaluation results 
Evaluation ::: Word Analogy Task
The word analogy task was popularized by mikolov2013distributed  The goal is to find a term $y$ for a given term $x$ so that the relationship between $x$ and $y$ best resembles the given relationship $a : b$  There are two main groups of categories: 5 semantic and 10 syntactic  To illustrate a semantic relationship, consider for example that the word pair $a : b$ is given as “Finland : Helsinki”  The task is to find the term $y$ corresponding to the relationship “Sweden : $y$”, with the expected answer being $y=$ Stockholm  In syntactic categories, the two words in a pair have a common stem (in some cases even same lemma), with all the pairs in a given category having the same morphological relationship  For example, given the word pair “long : longer”, we see that we have an adjective in its base form and the same adjective in a comparative form  That task is then to find the term $y$ corresponding to the relationship “dark : $y$”, with the expected answer being $y=$ darker, that is a comparative form of the adjective dark 
In the vector space, the analogy task is transformed into vector arithmetic and search for nearest neighbours, i e  we compute the distance between vectors: d(vec(Finland), vec(Helsinki)) and search for word $y$ which would give the closest result in distance d(vec(Sweden), vec($y$))  In the analogy dataset the analogies are already pre-specified, so we are measuring how close are the given pairs  In the evaluation below, we use analogy datasets for all tested languages based on the English dataset by BIBREF14   Due to English-centered bias of this dataset, we used a modified dataset which was first written in Slovene language and then translated into other languages BIBREF15 
As each instance of analogy contains only four words, without any context, the contextual models (such as ELMo) do not have enough context to generate sensible embeddings  We therefore used some additional text to form simple sentences using the four analogy words, while taking care that their noun case stays the same  For example, for the words "Rome", "Italy", "Paris" and "France" (forming the analogy Rome is to Italy as Paris is to $x$, where the correct answer is $x=$France), we formed the sentence "If the word Rome corresponds to the word Italy, then the word Paris corresponds to the word France"  We generated embeddings for those four words in the constructed sentence, substituted the last word with each word in our vocabulary and generated the embeddings again  As typical for non-contextual analogy task, we measure the cosine distance ($d$) between the last word ($w_4$) and the combination of the first three words ($w_2-w_1+w_3$)  We use the CSLS metric BIBREF16 to find the closest candidate word ($w_4$)  If we find the correct word among the five closest words, we consider that entry as successfully identified  The proportion of correctly identified words forms a statistic called accuracy@5, which we report as the result 
We first compare existing Latvian ELMo embeddings from ELMoForManyLangs project with our Latvian embeddings, followed by the detailed analysis of our ELMo embeddings  We trained Latvian ELMo using only CoNLL 2017 corpora  Since this is the only language, where we trained the embedding model on exactly the same corpora as ELMoForManyLangs models, we chose it for comparison between our ELMo model with ELMoForManyLangs  In other languages, additional or other corpora were used, so a direct comparison would also reflect the quality of the corpora used for training  In Latvian, however, only the size of the training dataset is different  ELMoForManyLangs uses only 20 million tokens and we use the whole corpus of 270 million tokens 
The Latvian ELMo model from ELMoForManyLangs project performs significantly worse than EMBEDDIA ELMo Latvian model on all categories of word analogy task (Figure FIGREF16)  We also include the comparison with our Estonian ELMo embeddings in the same figure  This comparison shows that while differences between our Latvian and Estonian embeddings can be significant for certain categories, the accuracy score of ELMoForManyLangs is always worse than either of our models  The comparison of Estonian and Latvian models leads us to believe that a few hundred million tokens is a sufficiently large corpus to train ELMo models (at least for word analogy task), but 20-million token corpora used in ELMoForManyLangs are too small 
The results for all languages and all ELMo layers, averaged over semantic and syntactic categories, are shown in Table TABREF17  The embeddings after the first LSTM layer perform best in semantic categories  In syntactic categories, the non-contextual CNN layer performs the best  Syntactic categories are less context dependent and much more morphology and syntax based, so it is not surprising that the non-contextual layer performs well  The second LSTM layer embeddings perform the worst in syntactic categories, though still outperforming CNN layer embeddings in semantic categories  Latvian ELMo performs worse compared to other languages we trained, especially in semantic categories, presumably due to smaller training data size  Surprisingly, the original English ELMo performs very poorly in syntactic categories and only outperforms Latvian in semantic categories  The low score can be partially explained by English model scoring $0 00$ in one syntactic category “opposite adjective”, which we have not been able to explain 
Evaluation ::: Named Entity Recognition
For evaluation of ELMo models on a relevant downstream task, we used named entity recognition (NER) task  NER is an information extraction task that seeks to locate and classify named entity mentions in unstructured text into pre-defined categories such as the person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc  To allow comparison of results between languages, we used an adapted version of this task, which uses a reduced set of labels, available in NER datasets for all processed languages  The labels in the used NER datasets are simplified to a common label set of three labels (person - PER, location - LOC, organization - ORG)  Each word in the NER dataset is labeled with one of the three mentioned labels or a label 'O' (other, i e  not a named entity) if it does not fit any of the other three labels  The number of words having each label is shown in Table TABREF19 
To measure the performance of ELMo embeddings on the NER task we proceeded as follows  We embedded the text in the datasets sentence by sentence, producing three vectors (one from each ELMo layer) for each token in a sentence  We calculated the average of the three vectors and used it as the input of our recognition model  The input layer was followed by a single LSTM layer with 128 LSTM cells and a dropout layer, randomly dropping 10% of the neurons on both the output and the recurrent branch  The final layer of our model was a time distributed softmax layer with 4 neurons 
We used ADAM optimiser BIBREF17 with the learning rate 0 01 and $10^{-5}$ learning rate decay  We used categorical cross-entropy as a loss function and trained the model for 3 epochs  We present the results using the Macro $F_1$ score, that is the average of $F_1$-scores for each of the three NE classes (the class Other is excluded) 
Since the differences between the tested languages depend more on the properties of the NER datasets than on the quality of embeddings, we can not directly compare ELMo models  For this reason, we take the non-contextual fastText embeddings as a baseline and predict named entities using them  The architecture of the model using fastText embeddings is the same as the one using ELMo embeddings, except that the input uses 300 dimensional fastText embedding vectors, and the model was trained for 5 epochs (instead of 3 as for ELMo)  In both cases (ELMo and fastText) we trained and evaluated the model five times, because there is some random component involved in initialization of the neural network model  By training and evaluating multiple times, we minimise this random component 
The results are presented in Table TABREF21  We included the evaluation of the original ELMo English model in the same table  NER models have little difficulty distinguishing between types of named entities, but recognizing whether a word is a named entity or not is more difficult  For languages with the smallest NER datasets, Croatian and Lithuanian, ELMo embeddings show the largest improvement over fastText embeddings  However, we can observe significant improvements with ELMo also on English and Finnish, which are among the largest datasets (English being by far the largest)  Only on Slovenian dataset did ELMo perform slightly worse than fastText, on all other EMBEDDIA languages, the ELMo embeddings improve the results 
Conclusion
We prepared precomputed ELMo contextual embeddings for seven languages: Croatian, Estonian, Finnish, Latvian, Lithuanian, Slovenian, and Swedish  We present the necessary background on embeddings and contextual embeddings, the details of training the embedding models, and their evaluation  We show that the size of used training sets importantly affects the quality of produced embeddings, and therefore the existing publicly available ELMo embeddings for the processed languages are inadequate  We trained new ELMo embeddings on larger training sets and analysed their properties on the analogy task and on the NER task  The results show that the newly produced contextual embeddings produce substantially better results compared to the non-contextual fastText baseline  In future work, we plan to use the produced contextual embeddings on the problems of news media industry  The pretrained ELMo models will be deposited to the CLARIN repository by the time of the final version of this paper 
Acknowledgments
The work was partially supported by the Slovenian Research Agency (ARRS) core research programme P6-0411  This paper is supported by European Union's Horizon 2020 research and innovation programme under grant agreement No 825153, project EMBEDDIA (Cross-Lingual Embeddings for Less-Represented Languages in European News Media)  The results of this publication reflects only the authors' view and the EU Commission is not responsible for any use that may be made of the information it contains 
Context 3: 
Experiments ::: Datasets
The statistics of the WSD datasets are shown in Table TABREF12 
Experiments ::: Datasets ::: Training Dataset
Following previous work BIBREF13, BIBREF12, BIBREF10, BIBREF17, BIBREF9, BIBREF7, we choose SemCor3 0 as training corpus, which is the largest corpus manually annotated with WordNet sense for WSD 
Experiments ::: Datasets ::: Evaluation Datasets
We evaluate our method on several English all-words WSD datasets  For a fair comparison, we use the benchmark datasets proposed by BIBREF17 which include five standard all-words fine-grained WSD datasets from the Senseval and SemEval competitions: Senseval-2 (SE2), Senseval-3 (SE3), SemEval-2007 (SE07), SemEval-2013 (SE13) and SemEval-2015 (SE15)  Following BIBREF13, BIBREF12 and BIBREF10, we choose SE07, the smallest among these test sets, as the development set 
Experiments ::: Datasets ::: WordNet
Since BIBREF17 map all the sense annotations in these datasets from their original versions to WordNet 3 0, we extract word sense glosses from WordNet 3 0 
Experiments ::: Settings
We use the pre-trained uncased BERT$_\mathrm {BASE}$ model for fine-tuning, because we find that BERT$_\mathrm {LARGE}$ model performs slightly worse than BERT$_\mathrm {BASE}$ in this task  The number of Transformer blocks is 12, the number of the hidden layer is 768, the number of self-attention heads is 12, and the total number of parameters of the pre-trained model is 110M  When fine-tuning, we use the development set (SE07) to find the optimal settings for our experiments  We keep the dropout probability at 0 1, set the number of epochs to 4  The initial learning rate is 2e-5, and the batch size is 64 
Experiments ::: Results
Table TABREF19 shows the performance of our method on the English all-words WSD benchmark datasets  We compare our approach with previous methods 
The first block shows the MFS baseline, which selects the most frequent sense in the training corpus for each target word 
The second block shows two knowledge-based systems  Lesk$_{ext+emb}$ BIBREF4 is a variant of Lesk algorithm BIBREF2 by calculating the gloss-context overlap of the target word  Babelfy BIBREF6 is a unified graph-based approach which exploits the semantic network structure from BabelNet 
The third block shows two word expert traditional supervised systems  IMS BIBREF7 is a flexible framework which trains SVM classifiers and uses local features  And IMS$_{+emb}$ BIBREF9 is the best configuration of the IMS framework, which also integrates word embeddings as features 
The fourth block shows several recent neural-based methods  Bi-LSTM BIBREF11 is a baseline for neural models  Bi-LSTM$_{+ att  + LEX + POS}$ BIBREF10 is a multi-task learning framework for WSD, POS tagging, and LEX with self-attention mechanism, which converts WSD to a sequence learning task  GAS$_{ext}$ BIBREF12 is a variant of GAS which is a gloss-augmented variant of the memory network by extending gloss knowledge  CAN$^s$ and HCAN BIBREF13 are sentence-level and hierarchical co-attention neural network models which leverage gloss knowledge 
In the last block, we report the performance of our method  BERT(Token-CLS) is our baseline, which does not incorporate gloss information, and it performs slightly worse than previous traditional supervised methods and recent neural-based methods  It proves that directly using BERT cannot obtain performance growth  The other three methods outperform other models by a substantial margin, which proves that the improvements come from leveraging BERT to better exploit gloss information  It is worth noting that our method achieves significant improvements in SE07 and Verb over previous methods, which have the highest ambiguity level among all datasets and all POS tags respectively according to BIBREF17 
Moreover, GlossBERT(Token-CLS) performs better than GlossBERT(Sent-CLS), which proves that highlighting the target word in the sentence is important  However, the weakly highlighting method GlossBERT(Sent-CLS-WS) performs best in most circumstances, which may result from its combination of the advantages of the other two methods 
Experiments ::: Discussion
There are two main reasons for the great improvements of our experimental results  First, we construct context-gloss pairs and convert WSD problem to a sentence-pair classification task which is similar to NLI tasks and train only one classifier, which is equivalent to expanding the corpus  Second, we leverage BERT BIBREF15 to better exploit the gloss information  BERT model shows its advantage in dealing with sentence-pair classification tasks by its amazing improvement on QA and NLI tasks  This advantage comes from both of its two novel unsupervised prediction tasks 
Compared with traditional word expert supervised methods, our GlossBERT shows its effectiveness to alleviate the effort of feature engineering and does not require training a dedicated classifier for every target lemma  Up to now, it can be said that the neural network method can totally beat the traditional word expert method  Compared with recent neural-based methods, our solution is more intuitive and can make better use of gloss knowledge  Besides, our approach demonstrates that when we fine-tune BERT on a downstream task, converting it into a sentence-pair classification task may be a good choice 
Conclusion
In this paper, we seek to better leverage gloss knowledge in a supervised neural WSD system  We propose a new solution to WSD by constructing context-gloss pairs and then converting WSD to a sentence-pair classification task  We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task 
Acknowledgments
We would like to thank the anonymous reviewers for their valuable comments  The research work is supported by National Natural Science Foundation of China (No  61751201 and 61672162), Shanghai Municipal Science and Technology Commission (16JC1420401 and 17JC1404100), Shanghai Municipal Science and Technology Major Project (No 2018SHZDZX01) and ZJLab 

ground_truth: 5 percent points.

0.05 F1

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they use large or small BERT?
Context 4:  Interestingly, we found that while masking before convolution gives a lower WER, using masks for both convolutions and batch norm results in worse performance 
As a final note, we found that training with weight norm was very unstable leading to exploding activations 
Residual Connections
For models deeper than Jasper 5x3, we observe consistently that residual connections are necessary for training to converge  In addition to the simple residual and dense residual model described above, we investigated DenseNet BIBREF15 and DenseRNet BIBREF16 variants of Jasper  Both connect the outputs of each sub-block to the inputs of following sub-blocks within a block  DenseRNet, similar to Dense Residual, connects the output of each output of each block to the input of all following blocks  DenseNet and DenseRNet combine residual connections using concatenation whereas Residual and Dense Residual use addition  We found that Dense Residual and DenseRNet perform similarly with each performing better on specific subsets of LibriSpeech  We decided to use Dense Residual for subsequent experiments  The main reason is that due to concatenation, the growth factor for DenseNet and DenseRNet requires tuning for deeper models whereas Dense Residual simply just repeats a sub-blocks 
Language Model
A language model (LM) is a probability distribution over arbitrary symbol sequences INLINEFORM0 such that more likely sequences are assigned high probabilities  LMs are frequently used to condition beam search  During decoding, candidates are evaluated using both acoustic scores and LM scores  Traditional N-gram LMs have been augmented with neural LMs in recent work BIBREF20 , BIBREF21 , BIBREF22  
We experiment with statistical N-gram language models BIBREF23 and neural Transformer-XL BIBREF11 models  Our best results use acoustic and word-level N-gram language models to generate a candidate list using beam search with a width of 2048  Next, an external Transformer-XL LM rescores the final list  All LMs were trained on datasets independently from acoustic models  We show results with the neural LM in our Results section  We observed a strong correlation between the quality of the neural LM (measured by perplexity) and WER as shown in Figure FIGREF20  
NovoGrad
For training, we use either Stochastic Gradient Descent (SGD) with momentum or our own NovoGrad, an optimizer similar to Adam BIBREF14 , except that its second moments are computed per layer instead of per weight  Compared to Adam, it reduces memory consumption and we find it to be more numerically stable 
At each step INLINEFORM0 , NovoGrad computes the stochastic gradient INLINEFORM1 following the regular forward-backward pass  Then the second-order moment INLINEFORM2 is computed for each layer INLINEFORM3 similar to ND-Adam BIBREF27 : DISPLAYFORM0
The second-order moment INLINEFORM0 is used to re-scale gradients INLINEFORM1 before calculating the first-order moment INLINEFORM2 : DISPLAYFORM0
If L2-regularization is used, a weight decay INLINEFORM0 is added to the re-scaled gradient (as in AdamW BIBREF28 ): DISPLAYFORM0
Finally, new weights are computed using the learning rate INLINEFORM0 : DISPLAYFORM0
Using NovoGrad instead of SGD with momentum, we decreased the WER on dev-clean LibriSpeech from 4 00% to 3 64%, a relative improvement of 9% for Jasper DR 10x5  We will further analyze NovoGrad in forthcoming work 
Results
We evaluate Jasper across a number of datasets in various domains  In all experiments, we use dropout and weight decay as regularization  At training time, we use speed perturbation with fixed +/-10% BIBREF29 for LibriSpeech  For WSJ and Hub5'00, we use a random speed perturbation factor between [-10%, 10%] as each utterance is fed into the model  All models have been trained on NVIDIA DGX-1 in mixed precision BIBREF30 using OpenSeq2Seq BIBREF31   Source code, training configurations, and pretrained models are available 
Read Speech
We evaluated the performance of Jasper on two read speech datasets: LibriSpeech and Wall Street Journal (WSJ)  For LibriSpeech, we trained Jasper DR 10x5 using our NovoGrad optimizer for 400 epochs  We achieve SOTA performance on the test-clean subset and SOTA among end-to-end speech recognition models on test-other 
We trained a smaller Jasper 10x3 model with SGD with momentum optimizer for 400 epochs on a combined WSJ dataset (80 hours): LDC93S6A (WSJ0) and LDC94S13A (WSJ1)  The results are provided in Table TABREF29  
Conversational Speech
We also evaluate the Jasper model's performance on a conversational English corpus  The Hub5 Year 2000 (Hub5'00) evaluation (LDC2002S09, LDC2005S13) is widely used in academia  It is divided into two subsets: Switchboard (SWB) and Callhome (CHM)  The training data for both the acoustic and language models consisted of the 2000hr Fisher+Switchboard training data (LDC2004S13, LDC2005S13, LDC97S62)  Jasper DR 10x5 was trained using SGD with momentum for 50 epochs  We compare to other models trained using the same data and report Hub5'00 results in Table TABREF31  
We obtain good results for SWB  However, there is work to be done to improve WER on harder tasks such as CHM 
Conclusions
We have presented a new family of neural architectures for end-to-end speech recognition  Inspired by wav2letter's convolutional approach, we build a deep and scalable model, which requires a well-designed residual topology, effective regularization, and a strong optimizer  As our architecture studies demonstrated, a combination of standard components leads to SOTA results on LibriSpeech and competitive results on other benchmarks  Our Jasper architecture is highly efficient for training and inference, and serves as a good baseline approach on top of which to explore more sophisticated regularization, data augmentation, loss functions, language models, and optimization strategies  We are interested to see if our approach can continue to scale to deeper models and larger datasets 

ground_truth: small BERT

small BERT