prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is best performing model among author's submissions, what performance it had?
Context 1: Introduction
In the age of information dissemination without quality control, it has enabled malicious users to spread misinformation via social media and aim individual users with propaganda campaigns to achieve political and financial gains as well as advance a specific agenda  Often disinformation is complied in the two major forms: fake news and propaganda, where they differ in the sense that the propaganda is possibly built upon true information (e g , biased, loaded language, repetition, etc ) 
Prior works BIBREF0, BIBREF1, BIBREF2 in detecting propaganda have focused primarily at document level, typically labeling all articles from a propagandistic news outlet as propaganda and thus, often non-propagandistic articles from the outlet are mislabeled  To this end, EMNLP19DaSanMartino focuses on analyzing the use of propaganda and detecting specific propagandistic techniques in news articles at sentence and fragment level, respectively and thus, promotes explainable AI  For instance, the following text is a propaganda of type `slogan' 
Trump tweeted: $\underbrace{\text{`}`{\texttt {BUILD THE WALL!}"}}_{\text{slogan}}$
Shared Task: This work addresses the two tasks in propaganda detection BIBREF3 of different granularities: (1) Sentence-level Classification (SLC), a binary classification that predicts whether a sentence contains at least one propaganda technique, and (2) Fragment-level Classification (FLC), a token-level (multi-label) classification that identifies both the spans and the type of propaganda technique(s) 
Contributions: (1) To address SLC, we design an ensemble of different classifiers based on Logistic Regression, CNN and BERT, and leverage transfer learning benefits using the pre-trained embeddings/models from FastText and BERT  We also employed different features such as linguistic (sentiment, readability, emotion, part-of-speech and named entity tags, etc ), layout, topics, etc  (2) To address FLC, we design a multi-task neural sequence tagger based on LSTM-CRF and linguistic features to jointly detect propagandistic fragments and its type  Moreover, we investigate performing FLC and SLC jointly in a multi-granularity network based on LSTM-CRF and BERT  (3) Our system (MIC-CIS) is ranked 3rd (out of 12 participants) and 4th (out of 25 participants) in FLC and SLC tasks, respectively 
System Description ::: Linguistic, Layout and Topical Features
Some of the propaganda techniques BIBREF3 involve word and phrases that express strong emotional implications, exaggeration, minimization, doubt, national feeling, labeling , stereotyping, etc  This inspires us in extracting different features (Table TABREF1) including the complexity of text, sentiment, emotion, lexical (POS, NER, etc ), layout, etc  To further investigate, we use topical features (e g , document-topic proportion) BIBREF4, BIBREF5, BIBREF6 at sentence and document levels in order to determine irrelevant themes, if introduced to the issue being discussed (e g , Red Herring) 
For word and sentence representations, we use pre-trained vectors from FastText BIBREF7 and BERT BIBREF8 
System Description ::: Sentence-level Propaganda Detection
Figure FIGREF2 (left) describes the three components of our system for SLC task: features, classifiers and ensemble  The arrows from features-to-classifier indicate that we investigate linguistic, layout and topical features in the two binary classifiers: LogisticRegression and CNN  For CNN, we follow the architecture of DBLP:conf/emnlp/Kim14 for sentence-level classification, initializing the word vectors by FastText or BERT  We concatenate features in the last hidden layer before classification 
One of our strong classifiers includes BERT that has achieved state-of-the-art performance on multiple NLP benchmarks  Following DBLP:conf/naacl/DevlinCLT19, we fine-tune BERT for binary classification, initializing with a pre-trained model (i e , BERT-base, Cased)  Additionally, we apply a decision function such that a sentence is tagged as propaganda if prediction probability of the classifier is greater than a threshold ($\tau $)  We relax the binary decision boundary to boost recall, similar to pankajgupta:CrossRE2019 
Ensemble of Logistic Regression, CNN and BERT: In the final component, we collect predictions (i e , propaganda label) for each sentence from the three ($\mathcal {M}=3$) classifiers and thus, obtain $\mathcal {M}$ number of predictions for each sentence  We explore two ensemble strategies (Table TABREF1): majority-voting and relax-voting to boost precision and recall, respectively 
System Description ::: Fragment-level Propaganda Detection
Figure FIGREF2 (right) describes our system for FLC task, where we design sequence taggers BIBREF9, BIBREF10 in three modes: (1) LSTM-CRF BIBREF11 with word embeddings ($w\_e$) and character embeddings $c\_e$, token-level features ($t\_f$) such as polarity, POS, NER, etc  (2) LSTM-CRF+Multi-grain that jointly performs FLC and SLC with FastTextWordEmb and BERTSentEmb, respectively  Here, we add binary sentence classification loss to sequence tagging weighted by a factor of $\alpha $  (3) LSTM-CRF+Multi-task that performs propagandistic span/fragment detection (PFD) and FLC (fragment detection + 19-way classification) 
Ensemble of Multi-grain, Multi-task LSTM-CRF with BERT: Here, we build an ensemble by considering propagandistic fragments (and its type) from each of the sequence taggers  In doing so, we first perform majority voting at the fragment level for the fragment where their spans exactly overlap  In case of non-overlapping fragments, we consider all  However, when the spans overlap (though with the same label), we consider the fragment with the largest span 
Experiments and Evaluation
Data: While the SLC task is binary, the FLC consists of 18 propaganda techniques BIBREF3  We split (80-20%) the annotated corpus into 5-folds and 3-folds for SLC and FLC tasks, respectively  The development set of each the folds is represented by dev (internal); however, the un-annotated corpus used in leaderboard comparisons by dev (external)  We remove empty and single token sentences after tokenization  Experimental Setup: We use PyTorch framework for the pre-trained BERT model (Bert-base-cased), fine-tuned for SLC task  In the multi-granularity loss, we set $\alpha = 0 1$ for sentence classification based on dev (internal, fold1) scores  We use BIO tagging scheme of NER in FLC task  For CNN, we follow DBLP:conf/emnlp/Kim14 with filter-sizes of [2, 3, 4, 5, 6], 128 filters and 16 batch-size  We compute binary-F1and macro-F1 BIBREF12 in SLC and FLC, respectively on dev (internal) 
Experiments and Evaluation ::: Results: Sentence-Level Propaganda
Table TABREF10 shows the scores on dev (internal and external) for SLC task  Observe that the pre-trained embeddings (FastText or BERT) outperform TF-IDF vector representation  In row r2, we apply logistic regression classifier with BERTSentEmb that leads to improved scores over FastTextSentEmb  Subsequently, we augment the sentence vector with additional features that improves F1 on dev (external), however not dev (internal)  Next, we initialize CNN by FastTextWordEmb or BERTWordEmb and augment the last hidden layer (before classification) with BERTSentEmb and feature vectors, leading to gains in F1 for both the dev sets  Further, we fine-tune BERT and apply different thresholds in relaxing the decision boundary, where $\tau \ge 0 35$ is found optimal 
We choose the three different models in the ensemble: Logistic Regression, CNN and BERT on fold1 and subsequently an ensemble+ of r3, r6 and r12 from each fold1-5 (i e , 15 models) to obtain predictions for dev (external)  We investigate different ensemble schemes (r17-r19), where we observe that the relax-voting improves recall and therefore, the higher F1 (i e , 0 673)  In postprocess step, we check for repetition propaganda technique by computing cosine similarity between the current sentence and its preceding $w=10$ sentence vectors (i e , BERTSentEmb) in the document  If the cosine-similarity is greater than $\lambda \in \lbrace  99,  95\rbrace $, then the current sentence is labeled as propaganda due to repetition  Comparing r19 and r21, we observe a gain in recall, however an overall decrease in F1 applying postprocess 
Finally, we use the configuration of r19 on the test set  The ensemble+ of (r4, r7 r12) was analyzed after test submission  Table TABREF9 (SLC) shows that our submission is ranked at 4th position 
Experiments and Evaluation ::: Results: Fragment-Level Propaganda
Table TABREF11 shows the scores on dev (internal and external) for FLC task  Observe that the features (i e , polarity, POS and NER in row II) when introduced in LSTM-CRF improves F1  We run multi-grained LSTM-CRF without BERTSentEmb (i e , row III) and with it (i e , row IV), where the latter improves scores on dev (internal), however not on dev (external)  Finally, we perform multi-tasking with another auxiliary task of PFD  Given the scores on dev (internal and external) using different configurations (rows I-V), it is difficult to infer the optimal configuration  Thus, we choose the two best configurations (II and IV) on dev (internal) set and build an ensemble+ of predictions (discussed in section SECREF6), leading to a boost in recall and thus an improved F1 on dev (external) 
Finally, we use the ensemble+ of (II and IV) from each of the folds 1-3, i e , $|{\mathcal {M}}|=6$ models to obtain predictions on test  Table TABREF9 (FLC) shows that our submission is ranked at 3rd position 
Conclusion and Future Work
Our system (Team: MIC-CIS) explores different neural architectures (CNN, BERT and LSTM-CRF) with linguistic, layout and topical features to address the tasks of fine-grained propaganda detection  We have demonstrated gains in performance due to the features, ensemble schemes, multi-tasking and multi-granularity architectures  Compared to the other participating systems, our submissions are ranked 3rd and 4th in FLC and SLC tasks, respectively 
In future, we would like to enrich BERT models with linguistic, layout and topical features during their fine-tuning  Further, we would also be interested in understanding and analyzing the neural network learning, i e , extracting salient fragments (or key-phrases) in the sentence that generate propaganda, similar to pankajgupta:2018LISA in order to promote explainable AI 
Context 2: Introduction
Word embeddings are representations of words in numerical form, as vectors of typically several hundred dimensions  The vectors are used as an input to machine learning models; for complex language processing tasks these are typically deep neural networks  The embedding vectors are obtained from specialized learning tasks, based on neural networks, e g , word2vec BIBREF0, GloVe BIBREF1, FastText BIBREF2, ELMo BIBREF3, and BERT BIBREF4  For training, the embeddings algorithms use large monolingual corpora that encode important information about word meaning as distances between vectors  In order to enable downstream machine learning on text understanding tasks, the embeddings shall preserve semantic relations between words, and this is true even across languages 
Probably the best known word embeddings are produced by the word2vec method BIBREF5  The problem with word2vec embeddings is their failure to express polysemous words  During training of an embedding, all senses of a given word (e g , paper as a material, as a newspaper, as a scientific work, and as an exam) contribute relevant information in proportion to their frequency in the training corpus  This causes the final vector to be placed somewhere in the weighted middle of all words' meanings  Consequently, rare meanings of words are poorly expressed with word2vec and the resulting vectors do not offer good semantic representations  For example, none of the 50 closest vectors of the word paper is related to science 
The idea of contextual embeddings is to generate a different vector for each context a word appears in and the context is typically defined sentence-wise  To a large extent, this solves the problems with word polysemy, i e  the context of a sentence is typically enough to disambiguate different meanings of a word for humans and so it is for the learning algorithms  In this work, we describe high-quality models for contextual embeddings, called ELMo BIBREF3, precomputed for seven morphologically rich, less-resourced languages: Slovenian, Croatian, Finnish, Estonian, Latvian, Lithuanian, and Swedish  ELMo is one of the most successful approaches to contextual word embeddings  At time of its creation, ELMo has been shown to outperform previous word embeddings BIBREF3 like word2vec and GloVe on many NLP tasks, e g , question answering, named entity extraction, sentiment analysis, textual entailment, semantic role labeling, and coreference resolution 
This report is split into further five sections  In section SECREF2, we describe the contextual embeddings ELMo  In Section SECREF3, we describe the datasets used and in Section SECREF4 we describe preprocessing and training of the embeddings  We describe the methodology for evaluation of created vectors and results in Section SECREF5  We present conclusion in Section SECREF6 where we also outline plans for further work 
ELMo
Typical word embeddings models or representations, such as word2vec BIBREF0, GloVe BIBREF1, or FastText BIBREF2, are fast to train and have been pre-trained for a number of different languages  They do not capture the context, though, so each word is always given the same vector, regardless of its context or meaning  This is especially problematic for polysemous words  ELMo (Embeddings from Language Models) embedding BIBREF3 is one of the state-of-the-art pretrained transfer learning models, that remedies the problem and introduces a contextual component 
ELMo model`s architecture consists of three neural network layers  The output of the model after each layer gives one set of embeddings, altogether three sets  The first layer is a CNN layer, which operates on a character level  It is context independent, so each word always gets the same embedding, regardless of its context  It is followed by two biLM layers  A biLM layer consists of two concatenated LSTMs  In the first LSTM, we try to predict the following word, based on the given past words, where each word is represented by the embeddings from the CNN layer  In the second LSTM, we try to predict the preceding word, based on the given following words  It is equivalent to the first LSTM, just reading the text in reverse 
In NLP tasks, any set of these embeddings may be used; however, a weighted average is usually used  The weights of the average are learned during the training of the model for the specific task  Additionally, an entire ELMo model can be fine-tuned on a specific end task 
Although ELMo is trained on character level and is able to handle out-of-vocabulary words, a vocabulary file containing most common tokens is used for efficiency during training and embedding generation  The original ELMo model was trained on a one billion word large English corpus, with a given vocabulary file of about 800,000 words  Later, ELMo models for other languages were trained as well, but limited to larger languages with many resources, like German and Japanese 
ELMo ::: ELMoForManyLangs
Recently, ELMoForManyLangs BIBREF6 project released pre-trained ELMo models for a number of different languages BIBREF7  These models, however, were trained on a significantly smaller datasets  They used 20-million-words data randomly sampled from the raw text released by the CoNLL 2017 Shared Task - Automatically Annotated Raw Texts and Word Embeddings BIBREF8, which is a combination of Wikipedia dump and common crawl  The quality of these models is questionable  For example, we compared the Latvian model by ELMoForManyLangs with a model we trained on a complete (wikidump + common crawl) Latvian corpus, which has about 280 million tokens  The difference of each model on the word analogy task is shown in Figure FIGREF16 in Section SECREF5  As the results of the ELMoForManyLangs embeddings are significantly worse than using the full corpus, we can conclude that these embeddings are not of sufficient quality  For that reason, we computed ELMo embeddings for seven languages on much larger corpora  As this effort requires access to large amount of textual data and considerable computational resources, we made the precomputed models publicly available by depositing them to Clarin repository 
Training Data
We trained ELMo models for seven languages: Slovenian, Croatian, Finnish, Estonian, Latvian, Lithuanian and Swedish  To obtain high-quality embeddings, we used large monolingual corpora from various sources for each language  Some corpora are available online under permissive licences, others are available only for research purposes or have limited availability  The corpora used in training datasets are a mix of news articles and general web crawl, which we preprocessed and deduplicated  Below we shortly describe the used corpora in alphabetical order of the involved languages  Their names and sizes are summarized in Table TABREF3 
Croatian dataset include hrWaC 2 1 corpus BIBREF9, Riznica BIBREF10, and articles of Croatian branch of Styria media house, made available to us through partnership in a joint project  hrWaC was built by crawling the  hr internet domain in 2011 and 2014  Riznica is composed of Croatian fiction and non-fiction prose, poetry, drama, textbooks, manuals, etc  The Styria dataset consists of 570,219 news articles published on the Croatian 24sata news portal and niche portals related to 24sata 
Estonian dataset contains texts from two sources, CoNLL 2017 Shared Task - Automatically Annotated Raw Texts and Word Embeddings BIBREF8, and news articles made available to us by Ekspress Meedia due to partnership in the project  Ekspress Meedia dataset is composed of Estonian news articles between years 2009 and 2019  The CoNLL 2017 corpus is composed of Estonian Wikipedia and webcrawl 
Finnish dataset contains articles by Finnish news agency STT, Finnish part of the CoNLL 2017 dataset, and Ylilauta downloadable version BIBREF11  STT news articles were published between years 1992 and 2018  Ylilauta is a Finnish online discussion board; the corpus contains parts of the discussions from 2012 to 2014 
Latvian dataset consists only of the Latvian portion of the ConLL 2017 corpus 
Lithuanian dataset is composed of Lithuanian Wikipedia articles from 2018, DGT-UD corpus, and LtTenTen  DGT-UD is a parallel corpus of 23 official languages of the EU, composed of JRC DGT translation memory of European law, automatically annotated with UD-Pipe 1 2  LtTenTen is Lithuanian web corpus made up of texts collected from the internet in April 2014 BIBREF12 
Slovene dataset is formed from the Gigafida 2 0 corpus BIBREF13  It is a general language corpus composed of various sources, mostly newspapers, internet pages, and magazines, but also fiction and non-fiction prose, textbooks, etc 
Swedish dataset is composed of STT Swedish articles and Swedish part of CoNLL 2017  The Finnish news agency STT publishes some of its articles in Swedish language  They were made available to us through partnership in a joint project  The corpus contains those articles from 1992 to 2017 
Preprocessing and Training
Prior to training the ELMo models, we sentence and word tokenized all the datasets  The text was formatted in such a way that each sentence was in its own line with tokens separated by white spaces  CoNLL 2017, DGT-UD and LtTenTen14 corpora were already pre-tokenized  We tokenized the others using the NLTK library and its tokenizers for each of the languages
Context 3: Introduction
Conventional automatic speech recognition (ASR) systems typically consist of several independently learned components: an acoustic model to predict context-dependent sub-phoneme states (senones) from audio, a graph structure to map senones to phonemes, and a pronunciation model to map phonemes to words  Hybrid systems combine hidden Markov models to model state dependencies with neural networks to predict states BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Newer approaches such as end-to-end (E2E) systems reduce the overall complexity of the final system 
Our research builds on prior work that has explored using time-delay neural networks (TDNN), other forms of convolutional neural networks, and Connectionist Temporal Classification (CTC) loss BIBREF4 , BIBREF5 , BIBREF6   We took inspiration from wav2letter BIBREF6 , which uses 1D-convolution layers  Liptchinsky et al  BIBREF7 improved wav2letter by increasing the model depth to 19 convolutional layers and adding Gated Linear Units (GLU) BIBREF8 , weight normalization BIBREF9 and dropout 
By building a deeper and larger capacity network, we aim to demonstrate that we can match or outperform non end-to-end models on the LibriSpeech and 2000hr Fisher+Switchboard tasks  Like wav2letter, our architecture, Jasper, uses a stack of 1D-convolution layers, but with ReLU and batch normalization BIBREF10   We find that ReLU and batch normalization outperform other activation and normalization schemes that we tested for convolutional ASR  As a result, Jasper's architecture contains only 1D convolution, batch normalization, ReLU, and dropout layers – operators highly optimized for training and inference on GPUs 
It is possible to increase the capacity of the Jasper model by stacking these operations  Our largest version uses 54 convolutional layers (333M parameters), while our small model uses 34 (201M parameters)  We use residual connections to enable this level of depth  We investigate a number of residual options and propose a new residual connection topology we call Dense Residual (DR) 
Integrating our best acoustic model with a Transformer-XL BIBREF11 language model allows us to obtain new state-of-the-art (SOTA) results on LibriSpeech BIBREF12 test-clean of 2 95% WER and SOTA results among end-to-end models on LibriSpeech test-other  We show competitive results on Wall Street Journal (WSJ), and 2000hr Fisher+Switchboard (F+S)  Using only greedy decoding without a language model we achieve 3 86% WER on LibriSpeech test-clean 
This paper makes the following contributions:
Jasper Architecture
Jasper is a family of end-to-end ASR models that replace acoustic and pronunciation models with a convolutional neural network  Jasper uses mel-filterbank features calculated from 20ms windows with a 10ms overlap, and outputs a probability distribution over characters per frame  Jasper has a block architecture: a Jasper INLINEFORM0 x INLINEFORM1 model has INLINEFORM2 blocks, each with INLINEFORM3 sub-blocks  Each sub-block applies the following operations: a 1D-convolution, batch norm, ReLU, and dropout  All sub-blocks in a block have the same number of output channels 
Each block input is connected directly into the last sub-block via a residual connection  The residual connection is first projected through a 1x1 convolution to account for different numbers of input and output channels, then through a batch norm layer  The output of this batch norm layer is added to the output of the batch norm layer in the last sub-block  The result of this sum is passed through the activation function and dropout to produce the output of the sub-block 
The sub-block architecture of Jasper was designed to facilitate fast GPU inference  Each sub-block can be fused into a single GPU kernel: dropout is not used at inference-time and is eliminated, batch norm can be fused with the preceding convolution, ReLU clamps the result, and residual summation can be treated as a modified bias term in this fused operation 
All Jasper models have four additional convolutional blocks: one pre-processing and three post-processing  See Figure FIGREF7 and Table TABREF8 for details 
We also build a variant of Jasper, Jasper Dense Residual (DR)  Jasper DR follows DenseNet BIBREF15 and DenseRNet BIBREF16 , but instead of having dense connections within a block, the output of a convolution block is added to the inputs of all the following blocks  While DenseNet and DenseRNet concatenates the outputs of different layers, Jasper DR adds them in the same way that residuals are added in ResNet  As explained below, we find addition to be as effective as concatenation 
Normalization and Activation
In our study, we evaluate performance of models with:
3 types of normalization: batch norm BIBREF10 , weight norm BIBREF9 , and layer norm BIBREF17
3 types of rectified linear units: ReLU, clipped ReLU (cReLU), and leaky ReLU (lReLU)
2 types of gated units: gated linear units (GLU) BIBREF8 , and gated activation units (GAU) BIBREF18
All experiment results are shown in Table TABREF15   We first experimented with a smaller Jasper5x3 model to pick the top 3 settings before training on larger Jasper models  We found that layer norm with GAU performed the best on the smaller model  Layer norm with ReLU and batch norm with ReLU came second and third in our tests  Using these 3, we conducted further experiments on a larger Jasper10x4  For larger models, we noticed that batch norm with ReLU outperformed other choices  Thus, leading us to decide on batch normalization and ReLU for our architecture 
During batching, all sequences are padded to match the longest sequence  These padded values caused issues when using layer norm  We applied a sequence mask to exclude padding values from the mean and variance calculation  Further, we computed mean and variance over both the time dimension and channels similar to the sequence-wise normalization proposed by Laurent et al  BIBREF19   In addition to masking layer norm, we additionally applied masking prior to the convolution operation, and masking the mean and variance calculation in batch norm  These results are shown in Table TABREF16 

ground_truth: For SLC task, the "ltuorp" team  has the best performing  model (0.6323/0.6028/0.6649 for F1/P/R  respectively) and for FLC task the "newspeak" team  has the best performing  model (0.2488/0.2863/0.2201 for F1/P/R respectively).

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the improvement in performance for Estonian in the NER task?
Context 2:  There is no tokenizer for Croatian in NLTK library, so we used Slovene tokenizer instead  After tokenization, we deduplicated the datasets for each language separately, using the Onion (ONe Instance ONly) tool for text deduplication  We applied the tool on paragraph level for corpora that did not have sentences shuffled and on sentence level for the rest  We considered 9-grams with duplicate content threshold of 0 9 
For each language we prepared a vocabulary file, containing roughly one million most common tokens, i e  tokens that appear at least $n$ times in the corpus, where $n$ is between 15 and 25, depending on the dataset size  We included the punctuation marks among the tokens  We trained each ELMo model using default values used to train the original English ELMo (large) model 
Evaluation
We evaluated the produced ELMo models for all languages using two evaluation tasks: a word analogy task and named entity recognition (NER) task  Below, we first shortly describe each task, followed by the evaluation results 
Evaluation ::: Word Analogy Task
The word analogy task was popularized by mikolov2013distributed  The goal is to find a term $y$ for a given term $x$ so that the relationship between $x$ and $y$ best resembles the given relationship $a : b$  There are two main groups of categories: 5 semantic and 10 syntactic  To illustrate a semantic relationship, consider for example that the word pair $a : b$ is given as “Finland : Helsinki”  The task is to find the term $y$ corresponding to the relationship “Sweden : $y$”, with the expected answer being $y=$ Stockholm  In syntactic categories, the two words in a pair have a common stem (in some cases even same lemma), with all the pairs in a given category having the same morphological relationship  For example, given the word pair “long : longer”, we see that we have an adjective in its base form and the same adjective in a comparative form  That task is then to find the term $y$ corresponding to the relationship “dark : $y$”, with the expected answer being $y=$ darker, that is a comparative form of the adjective dark 
In the vector space, the analogy task is transformed into vector arithmetic and search for nearest neighbours, i e  we compute the distance between vectors: d(vec(Finland), vec(Helsinki)) and search for word $y$ which would give the closest result in distance d(vec(Sweden), vec($y$))  In the analogy dataset the analogies are already pre-specified, so we are measuring how close are the given pairs  In the evaluation below, we use analogy datasets for all tested languages based on the English dataset by BIBREF14   Due to English-centered bias of this dataset, we used a modified dataset which was first written in Slovene language and then translated into other languages BIBREF15 
As each instance of analogy contains only four words, without any context, the contextual models (such as ELMo) do not have enough context to generate sensible embeddings  We therefore used some additional text to form simple sentences using the four analogy words, while taking care that their noun case stays the same  For example, for the words "Rome", "Italy", "Paris" and "France" (forming the analogy Rome is to Italy as Paris is to $x$, where the correct answer is $x=$France), we formed the sentence "If the word Rome corresponds to the word Italy, then the word Paris corresponds to the word France"  We generated embeddings for those four words in the constructed sentence, substituted the last word with each word in our vocabulary and generated the embeddings again  As typical for non-contextual analogy task, we measure the cosine distance ($d$) between the last word ($w_4$) and the combination of the first three words ($w_2-w_1+w_3$)  We use the CSLS metric BIBREF16 to find the closest candidate word ($w_4$)  If we find the correct word among the five closest words, we consider that entry as successfully identified  The proportion of correctly identified words forms a statistic called accuracy@5, which we report as the result 
We first compare existing Latvian ELMo embeddings from ELMoForManyLangs project with our Latvian embeddings, followed by the detailed analysis of our ELMo embeddings  We trained Latvian ELMo using only CoNLL 2017 corpora  Since this is the only language, where we trained the embedding model on exactly the same corpora as ELMoForManyLangs models, we chose it for comparison between our ELMo model with ELMoForManyLangs  In other languages, additional or other corpora were used, so a direct comparison would also reflect the quality of the corpora used for training  In Latvian, however, only the size of the training dataset is different  ELMoForManyLangs uses only 20 million tokens and we use the whole corpus of 270 million tokens 
The Latvian ELMo model from ELMoForManyLangs project performs significantly worse than EMBEDDIA ELMo Latvian model on all categories of word analogy task (Figure FIGREF16)  We also include the comparison with our Estonian ELMo embeddings in the same figure  This comparison shows that while differences between our Latvian and Estonian embeddings can be significant for certain categories, the accuracy score of ELMoForManyLangs is always worse than either of our models  The comparison of Estonian and Latvian models leads us to believe that a few hundred million tokens is a sufficiently large corpus to train ELMo models (at least for word analogy task), but 20-million token corpora used in ELMoForManyLangs are too small 
The results for all languages and all ELMo layers, averaged over semantic and syntactic categories, are shown in Table TABREF17  The embeddings after the first LSTM layer perform best in semantic categories  In syntactic categories, the non-contextual CNN layer performs the best  Syntactic categories are less context dependent and much more morphology and syntax based, so it is not surprising that the non-contextual layer performs well  The second LSTM layer embeddings perform the worst in syntactic categories, though still outperforming CNN layer embeddings in semantic categories  Latvian ELMo performs worse compared to other languages we trained, especially in semantic categories, presumably due to smaller training data size  Surprisingly, the original English ELMo performs very poorly in syntactic categories and only outperforms Latvian in semantic categories  The low score can be partially explained by English model scoring $0 00$ in one syntactic category “opposite adjective”, which we have not been able to explain 
Evaluation ::: Named Entity Recognition
For evaluation of ELMo models on a relevant downstream task, we used named entity recognition (NER) task  NER is an information extraction task that seeks to locate and classify named entity mentions in unstructured text into pre-defined categories such as the person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc  To allow comparison of results between languages, we used an adapted version of this task, which uses a reduced set of labels, available in NER datasets for all processed languages  The labels in the used NER datasets are simplified to a common label set of three labels (person - PER, location - LOC, organization - ORG)  Each word in the NER dataset is labeled with one of the three mentioned labels or a label 'O' (other, i e  not a named entity) if it does not fit any of the other three labels  The number of words having each label is shown in Table TABREF19 
To measure the performance of ELMo embeddings on the NER task we proceeded as follows  We embedded the text in the datasets sentence by sentence, producing three vectors (one from each ELMo layer) for each token in a sentence  We calculated the average of the three vectors and used it as the input of our recognition model  The input layer was followed by a single LSTM layer with 128 LSTM cells and a dropout layer, randomly dropping 10% of the neurons on both the output and the recurrent branch  The final layer of our model was a time distributed softmax layer with 4 neurons 
We used ADAM optimiser BIBREF17 with the learning rate 0 01 and $10^{-5}$ learning rate decay  We used categorical cross-entropy as a loss function and trained the model for 3 epochs  We present the results using the Macro $F_1$ score, that is the average of $F_1$-scores for each of the three NE classes (the class Other is excluded) 
Since the differences between the tested languages depend more on the properties of the NER datasets than on the quality of embeddings, we can not directly compare ELMo models  For this reason, we take the non-contextual fastText embeddings as a baseline and predict named entities using them  The architecture of the model using fastText embeddings is the same as the one using ELMo embeddings, except that the input uses 300 dimensional fastText embedding vectors, and the model was trained for 5 epochs (instead of 3 as for ELMo)  In both cases (ELMo and fastText) we trained and evaluated the model five times, because there is some random component involved in initialization of the neural network model  By training and evaluating multiple times, we minimise this random component 
The results are presented in Table TABREF21  We included the evaluation of the original ELMo English model in the same table  NER models have little difficulty distinguishing between types of named entities, but recognizing whether a word is a named entity or not is more difficult  For languages with the smallest NER datasets, Croatian and Lithuanian, ELMo embeddings show the largest improvement over fastText embeddings  However, we can observe significant improvements with ELMo also on English and Finnish, which are among the largest datasets (English being by far the largest)  Only on Slovenian dataset did ELMo perform slightly worse than fastText, on all other EMBEDDIA languages, the ELMo embeddings improve the results 
Conclusion
We prepared precomputed ELMo contextual embeddings for seven languages: Croatian, Estonian, Finnish, Latvian, Lithuanian, Slovenian, and Swedish  We present the necessary background on embeddings and contextual embeddings, the details of training the embedding models, and their evaluation  We show that the size of used training sets importantly affects the quality of produced embeddings, and therefore the existing publicly available ELMo embeddings for the processed languages are inadequate  We trained new ELMo embeddings on larger training sets and analysed their properties on the analogy task and on the NER task  The results show that the newly produced contextual embeddings produce substantially better results compared to the non-contextual fastText baseline  In future work, we plan to use the produced contextual embeddings on the problems of news media industry  The pretrained ELMo models will be deposited to the CLARIN repository by the time of the final version of this paper 
Acknowledgments
The work was partially supported by the Slovenian Research Agency (ARRS) core research programme P6-0411  This paper is supported by European Union's Horizon 2020 research and innovation programme under grant agreement No 825153, project EMBEDDIA (Cross-Lingual Embeddings for Less-Represented Languages in European News Media)  The results of this publication reflects only the authors' view and the EU Commission is not responsible for any use that may be made of the information it contains 

ground_truth: 5 percent points.

0.05 F1

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what were the baselines?
Context 3:  Interestingly, we found that while masking before convolution gives a lower WER, using masks for both convolutions and batch norm results in worse performance 
As a final note, we found that training with weight norm was very unstable leading to exploding activations 
Residual Connections
For models deeper than Jasper 5x3, we observe consistently that residual connections are necessary for training to converge  In addition to the simple residual and dense residual model described above, we investigated DenseNet BIBREF15 and DenseRNet BIBREF16 variants of Jasper  Both connect the outputs of each sub-block to the inputs of following sub-blocks within a block  DenseRNet, similar to Dense Residual, connects the output of each output of each block to the input of all following blocks  DenseNet and DenseRNet combine residual connections using concatenation whereas Residual and Dense Residual use addition  We found that Dense Residual and DenseRNet perform similarly with each performing better on specific subsets of LibriSpeech  We decided to use Dense Residual for subsequent experiments  The main reason is that due to concatenation, the growth factor for DenseNet and DenseRNet requires tuning for deeper models whereas Dense Residual simply just repeats a sub-blocks 
Language Model
A language model (LM) is a probability distribution over arbitrary symbol sequences INLINEFORM0 such that more likely sequences are assigned high probabilities  LMs are frequently used to condition beam search  During decoding, candidates are evaluated using both acoustic scores and LM scores  Traditional N-gram LMs have been augmented with neural LMs in recent work BIBREF20 , BIBREF21 , BIBREF22  
We experiment with statistical N-gram language models BIBREF23 and neural Transformer-XL BIBREF11 models  Our best results use acoustic and word-level N-gram language models to generate a candidate list using beam search with a width of 2048  Next, an external Transformer-XL LM rescores the final list  All LMs were trained on datasets independently from acoustic models  We show results with the neural LM in our Results section  We observed a strong correlation between the quality of the neural LM (measured by perplexity) and WER as shown in Figure FIGREF20  
NovoGrad
For training, we use either Stochastic Gradient Descent (SGD) with momentum or our own NovoGrad, an optimizer similar to Adam BIBREF14 , except that its second moments are computed per layer instead of per weight  Compared to Adam, it reduces memory consumption and we find it to be more numerically stable 
At each step INLINEFORM0 , NovoGrad computes the stochastic gradient INLINEFORM1 following the regular forward-backward pass  Then the second-order moment INLINEFORM2 is computed for each layer INLINEFORM3 similar to ND-Adam BIBREF27 : DISPLAYFORM0
The second-order moment INLINEFORM0 is used to re-scale gradients INLINEFORM1 before calculating the first-order moment INLINEFORM2 : DISPLAYFORM0
If L2-regularization is used, a weight decay INLINEFORM0 is added to the re-scaled gradient (as in AdamW BIBREF28 ): DISPLAYFORM0
Finally, new weights are computed using the learning rate INLINEFORM0 : DISPLAYFORM0
Using NovoGrad instead of SGD with momentum, we decreased the WER on dev-clean LibriSpeech from 4 00% to 3 64%, a relative improvement of 9% for Jasper DR 10x5  We will further analyze NovoGrad in forthcoming work 
Results
We evaluate Jasper across a number of datasets in various domains  In all experiments, we use dropout and weight decay as regularization  At training time, we use speed perturbation with fixed +/-10% BIBREF29 for LibriSpeech  For WSJ and Hub5'00, we use a random speed perturbation factor between [-10%, 10%] as each utterance is fed into the model  All models have been trained on NVIDIA DGX-1 in mixed precision BIBREF30 using OpenSeq2Seq BIBREF31   Source code, training configurations, and pretrained models are available 
Read Speech
We evaluated the performance of Jasper on two read speech datasets: LibriSpeech and Wall Street Journal (WSJ)  For LibriSpeech, we trained Jasper DR 10x5 using our NovoGrad optimizer for 400 epochs  We achieve SOTA performance on the test-clean subset and SOTA among end-to-end speech recognition models on test-other 
We trained a smaller Jasper 10x3 model with SGD with momentum optimizer for 400 epochs on a combined WSJ dataset (80 hours): LDC93S6A (WSJ0) and LDC94S13A (WSJ1)  The results are provided in Table TABREF29  
Conversational Speech
We also evaluate the Jasper model's performance on a conversational English corpus  The Hub5 Year 2000 (Hub5'00) evaluation (LDC2002S09, LDC2005S13) is widely used in academia  It is divided into two subsets: Switchboard (SWB) and Callhome (CHM)  The training data for both the acoustic and language models consisted of the 2000hr Fisher+Switchboard training data (LDC2004S13, LDC2005S13, LDC97S62)  Jasper DR 10x5 was trained using SGD with momentum for 50 epochs  We compare to other models trained using the same data and report Hub5'00 results in Table TABREF31  
We obtain good results for SWB  However, there is work to be done to improve WER on harder tasks such as CHM 
Conclusions
We have presented a new family of neural architectures for end-to-end speech recognition  Inspired by wav2letter's convolutional approach, we build a deep and scalable model, which requires a well-designed residual topology, effective regularization, and a strong optimizer  As our architecture studies demonstrated, a combination of standard components leads to SOTA results on LibriSpeech and competitive results on other benchmarks  Our Jasper architecture is highly efficient for training and inference, and serves as a good baseline approach on top of which to explore more sophisticated regularization, data augmentation, loss functions, language models, and optimization strategies  We are interested to see if our approach can continue to scale to deeper models and larger datasets 

ground_truth: Unanswerable

LF-MMI Attention
Seq2Seq 
RNN-T 
Char E2E LF-MMI 
Phone E2E LF-MMI 
CTC + Gram-CTC