prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what were the baselines?
Context 1: Introduction
Conventional automatic speech recognition (ASR) systems typically consist of several independently learned components: an acoustic model to predict context-dependent sub-phoneme states (senones) from audio, a graph structure to map senones to phonemes, and a pronunciation model to map phonemes to words  Hybrid systems combine hidden Markov models to model state dependencies with neural networks to predict states BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Newer approaches such as end-to-end (E2E) systems reduce the overall complexity of the final system 
Our research builds on prior work that has explored using time-delay neural networks (TDNN), other forms of convolutional neural networks, and Connectionist Temporal Classification (CTC) loss BIBREF4 , BIBREF5 , BIBREF6   We took inspiration from wav2letter BIBREF6 , which uses 1D-convolution layers  Liptchinsky et al  BIBREF7 improved wav2letter by increasing the model depth to 19 convolutional layers and adding Gated Linear Units (GLU) BIBREF8 , weight normalization BIBREF9 and dropout 
By building a deeper and larger capacity network, we aim to demonstrate that we can match or outperform non end-to-end models on the LibriSpeech and 2000hr Fisher+Switchboard tasks  Like wav2letter, our architecture, Jasper, uses a stack of 1D-convolution layers, but with ReLU and batch normalization BIBREF10   We find that ReLU and batch normalization outperform other activation and normalization schemes that we tested for convolutional ASR  As a result, Jasper's architecture contains only 1D convolution, batch normalization, ReLU, and dropout layers – operators highly optimized for training and inference on GPUs 
It is possible to increase the capacity of the Jasper model by stacking these operations  Our largest version uses 54 convolutional layers (333M parameters), while our small model uses 34 (201M parameters)  We use residual connections to enable this level of depth  We investigate a number of residual options and propose a new residual connection topology we call Dense Residual (DR) 
Integrating our best acoustic model with a Transformer-XL BIBREF11 language model allows us to obtain new state-of-the-art (SOTA) results on LibriSpeech BIBREF12 test-clean of 2 95% WER and SOTA results among end-to-end models on LibriSpeech test-other  We show competitive results on Wall Street Journal (WSJ), and 2000hr Fisher+Switchboard (F+S)  Using only greedy decoding without a language model we achieve 3 86% WER on LibriSpeech test-clean 
This paper makes the following contributions:
Jasper Architecture
Jasper is a family of end-to-end ASR models that replace acoustic and pronunciation models with a convolutional neural network  Jasper uses mel-filterbank features calculated from 20ms windows with a 10ms overlap, and outputs a probability distribution over characters per frame  Jasper has a block architecture: a Jasper INLINEFORM0 x INLINEFORM1 model has INLINEFORM2 blocks, each with INLINEFORM3 sub-blocks  Each sub-block applies the following operations: a 1D-convolution, batch norm, ReLU, and dropout  All sub-blocks in a block have the same number of output channels 
Each block input is connected directly into the last sub-block via a residual connection  The residual connection is first projected through a 1x1 convolution to account for different numbers of input and output channels, then through a batch norm layer  The output of this batch norm layer is added to the output of the batch norm layer in the last sub-block  The result of this sum is passed through the activation function and dropout to produce the output of the sub-block 
The sub-block architecture of Jasper was designed to facilitate fast GPU inference  Each sub-block can be fused into a single GPU kernel: dropout is not used at inference-time and is eliminated, batch norm can be fused with the preceding convolution, ReLU clamps the result, and residual summation can be treated as a modified bias term in this fused operation 
All Jasper models have four additional convolutional blocks: one pre-processing and three post-processing  See Figure FIGREF7 and Table TABREF8 for details 
We also build a variant of Jasper, Jasper Dense Residual (DR)  Jasper DR follows DenseNet BIBREF15 and DenseRNet BIBREF16 , but instead of having dense connections within a block, the output of a convolution block is added to the inputs of all the following blocks  While DenseNet and DenseRNet concatenates the outputs of different layers, Jasper DR adds them in the same way that residuals are added in ResNet  As explained below, we find addition to be as effective as concatenation 
Normalization and Activation
In our study, we evaluate performance of models with:
3 types of normalization: batch norm BIBREF10 , weight norm BIBREF9 , and layer norm BIBREF17
3 types of rectified linear units: ReLU, clipped ReLU (cReLU), and leaky ReLU (lReLU)
2 types of gated units: gated linear units (GLU) BIBREF8 , and gated activation units (GAU) BIBREF18
All experiment results are shown in Table TABREF15   We first experimented with a smaller Jasper5x3 model to pick the top 3 settings before training on larger Jasper models  We found that layer norm with GAU performed the best on the smaller model  Layer norm with ReLU and batch norm with ReLU came second and third in our tests  Using these 3, we conducted further experiments on a larger Jasper10x4  For larger models, we noticed that batch norm with ReLU outperformed other choices  Thus, leading us to decide on batch normalization and ReLU for our architecture 
During batching, all sequences are padded to match the longest sequence  These padded values caused issues when using layer norm  We applied a sequence mask to exclude padding values from the mean and variance calculation  Further, we computed mean and variance over both the time dimension and channels similar to the sequence-wise normalization proposed by Laurent et al  BIBREF19   In addition to masking layer norm, we additionally applied masking prior to the convolution operation, and masking the mean and variance calculation in batch norm  These results are shown in Table TABREF16   Interestingly, we found that while masking before convolution gives a lower WER, using masks for both convolutions and batch norm results in worse performance 
As a final note, we found that training with weight norm was very unstable leading to exploding activations 
Residual Connections
For models deeper than Jasper 5x3, we observe consistently that residual connections are necessary for training to converge  In addition to the simple residual and dense residual model described above, we investigated DenseNet BIBREF15 and DenseRNet BIBREF16 variants of Jasper  Both connect the outputs of each sub-block to the inputs of following sub-blocks within a block  DenseRNet, similar to Dense Residual, connects the output of each output of each block to the input of all following blocks  DenseNet and DenseRNet combine residual connections using concatenation whereas Residual and Dense Residual use addition  We found that Dense Residual and DenseRNet perform similarly with each performing better on specific subsets of LibriSpeech  We decided to use Dense Residual for subsequent experiments  The main reason is that due to concatenation, the growth factor for DenseNet and DenseRNet requires tuning for deeper models whereas Dense Residual simply just repeats a sub-blocks 
Language Model
A language model (LM) is a probability distribution over arbitrary symbol sequences INLINEFORM0 such that more likely sequences are assigned high probabilities  LMs are frequently used to condition beam search  During decoding, candidates are evaluated using both acoustic scores and LM scores  Traditional N-gram LMs have been augmented with neural LMs in recent work BIBREF20 , BIBREF21 , BIBREF22  
We experiment with statistical N-gram language models BIBREF23 and neural Transformer-XL BIBREF11 models  Our best results use acoustic and word-level N-gram language models to generate a candidate list using beam search with a width of 2048  Next, an external Transformer-XL LM rescores the final list  All LMs were trained on datasets independently from acoustic models  We show results with the neural LM in our Results section  We observed a strong correlation between the quality of the neural LM (measured by perplexity) and WER as shown in Figure FIGREF20  
NovoGrad
For training, we use either Stochastic Gradient Descent (SGD) with momentum or our own NovoGrad, an optimizer similar to Adam BIBREF14 , except that its second moments are computed per layer instead of per weight  Compared to Adam, it reduces memory consumption and we find it to be more numerically stable 
At each step INLINEFORM0 , NovoGrad computes the stochastic gradient INLINEFORM1 following the regular forward-backward pass  Then the second-order moment INLINEFORM2 is computed for each layer INLINEFORM3 similar to ND-Adam BIBREF27 : DISPLAYFORM0
The second-order moment INLINEFORM0 is used to re-scale gradients INLINEFORM1 before calculating the first-order moment INLINEFORM2 : DISPLAYFORM0
If L2-regularization is used, a weight decay INLINEFORM0 is added to the re-scaled gradient (as in AdamW BIBREF28 ): DISPLAYFORM0
Finally, new weights are computed using the learning rate INLINEFORM0 : DISPLAYFORM0
Using NovoGrad instead of SGD with momentum, we decreased the WER on dev-clean LibriSpeech from 4 00% to 3 64%, a relative improvement of 9% for Jasper DR 10x5  We will further analyze NovoGrad in forthcoming work 
Results
We evaluate Jasper across a number of datasets in various domains  In all experiments, we use dropout and weight decay as regularization  At training time, we use speed perturbation with fixed +/-10% BIBREF29 for LibriSpeech  For WSJ and Hub5'00, we use a random speed perturbation factor between [-10%, 10%] as each utterance is fed into the model  All models have been trained on NVIDIA DGX-1 in mixed precision BIBREF30 using OpenSeq2Seq BIBREF31   Source code, training configurations, and pretrained models are available 
Read Speech
We evaluated the performance of Jasper on two read speech datasets: LibriSpeech and Wall Street Journal (WSJ)  For LibriSpeech, we trained Jasper DR 10x5 using our NovoGrad optimizer for 400 epochs  We achieve SOTA performance on the test-clean subset and SOTA among end-to-end speech recognition models on test-other 
We trained a smaller Jasper 10x3 model with SGD with momentum optimizer for 400 epochs on a combined WSJ dataset (80 hours): LDC93S6A (WSJ0) and LDC94S13A (WSJ1)  The results are provided in Table TABREF29  
Conversational Speech
We also evaluate the Jasper model's performance on a conversational English corpus  The Hub5 Year 2000 (Hub5'00) evaluation (LDC2002S09, LDC2005S13) is widely used in academia  It is divided into two subsets: Switchboard (SWB) and Callhome (CHM)  The training data for both the acoustic and language models consisted of the 2000hr Fisher+Switchboard training data (LDC2004S13, LDC2005S13, LDC97S62)  Jasper DR 10x5 was trained using SGD with momentum for 50 epochs  We compare to other models trained using the same data and report Hub5'00 results in Table TABREF31  
We obtain good results for SWB  However, there is work to be done to improve WER on harder tasks such as CHM 
Conclusions
We have presented a new family of neural architectures for end-to-end speech recognition  Inspired by wav2letter's convolutional approach, we build a deep and scalable model, which requires a well-designed residual topology, effective regularization, and a strong optimizer  As our architecture studies demonstrated, a combination of standard components leads to SOTA results on LibriSpeech and competitive results on other benchmarks  Our Jasper architecture is highly efficient for training and inference, and serves as a good baseline approach on top of which to explore more sophisticated regularization, data augmentation, loss functions, language models, and optimization strategies  We are interested to see if our approach can continue to scale to deeper models and larger datasets 
Context 2: Introduction
The automatic processing of medical texts and documents plays an increasingly important role in the recent development of the digital health area  To enable dedicated Natural Language Processing (NLP) that is highly accurate with respect to medically relevant categories, manually annotated data from this domain is needed  One category of high interest and relevance are medical entities  Only very few annotated corpora in the medical domain exist  Many of them focus on the relation between chemicals and diseases or proteins and diseases, such as the BC5CDR corpus BIBREF0, the Comparative Toxicogenomics Database BIBREF1, the FSU PRotein GEne corpus BIBREF2 or the ADE (adverse drug effect) corpus BIBREF3  The NCBI Disease Corpus BIBREF4 contains condition mention annotations along with annotations of symptoms  Several new corpora of annotated case reports were made available recently  grouin-etal-2019-clinical presented a corpus with medical entity annotations of clinical cases written in French, copdPhenotype presented a corpus focusing on phenotypic information for chronic obstructive pulmonary disease while 10 1093/database/bay143 presented a corpus focusing on identifying main finding sentences in case reports 
The corpus most comparable to ours is the French corpus of clinical case reports by grouin-etal-2019-clinical  Their annotations are based on UMLS semantic types  Even though there is an overlap in annotated entities, semantic classes are not the same  Lab results are subsumed under findings in our corpus and are not annotated as their own class  Factors extend beyond gender and age and describe any kind of risk factor that contributes to a higher probability of having a certain disease  Our corpus includes additional entity types  We annotate conditions, findings (including medical findings such as blood values), factors, and also modifiers which indicate the negation of other entities as well as case entities, i  e , entities specific to one case report  An overview is available in Table TABREF3 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation tasks
Case reports are standardized in the CARE guidelines BIBREF5  They represent a detailed description of the symptoms, signs, diagnosis, treatment, and follow-up of an individual patient  We focus on documents freely available through PubMed Central (PMC)  The presentation of the patient's case can usually be found in a dedicated section or the abstract  We perform a manual annotation of all mentions of case entities, conditions, findings, factors and modifiers  The scope of our manual annotation is limited to the presentation of a patient's signs and symptoms  In addition, we annotate the title of the case report 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation Guidelines
We annotate the following entities:
case entity marks the mention of a patient  A case report can contain more than one case description  Therefore, all the findings, factors and conditions related to one patient are linked to the respective case entity  Within the text, this entity is often represented by the first mention of the patient and overlaps with the factor annotations which can, e  g , mark sex and age (cf  Figure FIGREF12) 
condition marks a medical disease such as pneumothorax or dislocation of the shoulder 
factor marks a feature of a patient which might influence the probability for a specific diagnosis  It can be immutable (e  g , sex and age), describe a specific medical history (e  g , diabetes mellitus) or a behaviour (e  g , smoking) 
finding marks a sign or symptom a patient shows  This can be visible (e  g , rash), described by a patient (e  g , headache) or measurable (e  g , decreased blood glucose level) 
negation modifier explicitly negate the presence of a certain finding usually setting the case apart from common cases 
We also annotate relations between these entities, where applicable  Since we work on case descriptions, the anchor point of these relations is the case that is described  The following relations are annotated:
has relations exist between a case entity and factor, finding or condition entities 
modifies relations exist between negation modifiers and findings 
causes relations exist between conditions and findings 
Example annotations are shown in Figure FIGREF16 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotators
We asked medical doctors experienced in extracting knowledge related to medical entities from texts to annotate the entities described above  Initially, we asked four annotators to test our guidelines on two texts  Subsequently, identified issues were discussed and resolved  Following this pilot annotation phase, we asked two different annotators to annotate two case reports according to our guidelines  The same annotators annotated an overall collection of 53 case reports 
Inter-annotator agreement is calculated based on two case reports  We reach a Cohen's kappa BIBREF6 of 0 68  Disagreements mainly appear for findings that are rather unspecific such as She no longer eats out with friends which can be seen as a finding referring to “avoidance behaviour” 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation Tools and Format
The annotation was performed using WebAnno BIBREF7, a web-based tool for linguistic annotation  The annotators could choose between a pre-annotated version or a blank version of each text  The pre-annotated versions contained suggested entity spans based on string matches from lists of conditions and findings synonym lists  Their quality varied widely throughout the corpus  The blank version was preferred by the annotators  We distribute the corpus in BioC JSON format  BioC was chosen as it allows us to capture the complexities of the annotations in the biomedical domain  It represented each documents properties ranging from full text, individual passages/sentences along with captured annotations and relationships in an organized manner  BioC is based on character offsets of annotations and allows the stacking of different layers 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Corpus Overview
The corpus consists of 53 documents, which contain an average number of 156 1 sentences per document, each with 19 55 tokens on average  The corpus comprises 8,275 sentences and 167,739 words in total  However, as mentioned above, only case presentation sections, headings and abstracts are annotated  The numbers of annotated entities are summarized in Table TABREF24 
Findings are the most frequently annotated type of entity  This makes sense given that findings paint a clinical picture of the patient's condition  The number of tokens per entity ranges from one token for all types to 5 tokens for cases (average length 3 1), nine tokens for conditions (average length 2 0), 16 tokens for factors (average length 2 5), 25 tokens for findings (average length 2 6) and 18 tokens for modifiers (average length 1 4) (cf  Table TABREF24)  Examples of rather long entities are given in Table TABREF25 
Entities can appear in a discontinuous way  We model this as a relation between two spans which we call “discontinuous” (cf  Figure FIGREF26)
Context 3: Introduction
End-to-end speech-to-text translation (ST) has attracted much attention recently BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 given its simplicity against cascading automatic speech recognition (ASR) and machine translation (MT) systems  The lack of labeled data, however, has become a major blocker for bridging the performance gaps between end-to-end models and cascading systems  Several corpora have been developed in recent years  post2013improved introduced a 38-hour Spanish-English ST corpus by augmenting the transcripts of the Fisher and Callhome corpora with English translations  di-gangi-etal-2019-must created the largest ST corpus to date from TED talks but the language pairs involved are out of English only  beilharz2019librivoxdeen created a 110-hour German-English ST corpus from LibriVox audiobooks  godard-etal-2018-low created a Moboshi-French ST corpus as part of a rare language documentation effort  woldeyohannis provided an Amharic-English ST corpus in the tourism domain  boito2019mass created a multilingual ST corpus involving 8 languages from a multilingual speech corpus based on Bible readings BIBREF7  Previous work either involves language pairs out of English, very specific domains, very low resource languages or a limited set of language pairs  This limits the scope of study, including the latest explorations on end-to-end multilingual ST BIBREF8, BIBREF9  Our work is mostly similar and concurrent to iranzosnchez2019europarlst who created a multilingual ST corpus from the European Parliament proceedings  The corpus we introduce has larger speech durations and more translation tokens  It is diversified with multiple speakers per transcript/translation  Finally, we provide additional out-of-domain test sets 
In this paper, we introduce CoVoST, a multilingual ST corpus based on Common Voice BIBREF10 for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora  We also collect an additional evaluation corpus from Tatoeba for French, German, Dutch, Russian and Spanish, resulting in a total of 9 3 hours of speech  Both corpora are created at the sentence level and do not require additional alignments or segmentation  Using the official Common Voice train-development-test split, we also provide baseline models, including, to our knowledge, the first end-to-end many-to-one multilingual ST models  CoVoST is released under CC0 license and free to use  The Tatoeba evaluation samples are also available under friendly CC licenses  All the data can be acquired at https://github com/facebookresearch/covost 
Data Collection and Processing ::: Common Voice (CoVo)
Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license  Contributors record voice clips by reading from a bank of donated sentences  Each voice clip was validated by at least two other users  Most of the sentences are covered by multiple speakers, with potentially different genders, age groups or accents 
Raw CoVo data contains samples that passed validation as well as those that did not  To build CoVoST, we only use the former one and reuse the official train-development-test partition of the validated data  As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages  CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese 
Validated transcripts were sent to professional translators  Note that the translators had access to the transcripts but not the corresponding voice clips since clips would not carry additional information  Since transcripts were duplicated due to multiple speakers, we deduplicated the transcripts before sending them to translators  As a result, different voice clips of the same content (transcript) will have identical translations in CoVoST for train, development and test splits 
In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11  1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU BIBREF12 with the NLTK BIBREF13 implementation between the human translations and the automatic translations produced by a state-of-the-art system BIBREF14 (the French-English system was a Transformer big BIBREF15 separately trained on WMT14)  We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems  Translations with a score that was too low were manually inspected and sent back to the translators when needed  2) We manually inspected examples where the source transcript was identical to the translation  3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data BIBREF14  We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly  4) We computed the ratio of English characters in the translations

ground_truth: Unanswerable

LF-MMI Attention
Seq2Seq 
RNN-T 
Char E2E LF-MMI 
Phone E2E LF-MMI 
CTC + Gram-CTC

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How large is the corpus?
Context 2:  Especially findings often appear as discontinuous entities, we found 543 discontinuous finding relations  The numbers for conditions and factors are lower with seven and two, respectively  Entities can also be nested within one another  This happens either when the span of one annotation is completely embedded in the span of another annotation (fully-nested; cf  Figure FIGREF12), or when there is a partial overlapping between the spans of two different entities (partially-nested; cf  Figure FIGREF12)  There is a high number of inter-sentential relations in the corpus (cf  Table TABREF27)  This can be explained by the fact that the case entity occurs early in each document; furthermore, it is related to finding and factor annotations that are distributed across different sentences 
The most frequently annotated relation in our corpus is the has-relation between a case entity and the findings related to that case  This correlates with the high number of finding entities  The relations contained in our corpus are summarized in Table TABREF27 
Baseline systems for Named Entity Recognition in medical case reports
We evaluate the corpus using Named Entity Recognition (NER), i  e , the task of finding mentions of concepts of interest in unstructured text  We focus on detecting cases, conditions, factors, findings and modifiers in case reports (cf  Section SECREF6)  We approach this as a sequence labeling problem  Four systems were developed to offer comparable robust baselines 
The original documents are pre-processed (sentence splitting and tokenization with ScispaCy)  We do not perform stop word removal or lower-casing of the tokens  The BIO labeling scheme is used to capture the order of tokens belonging to the same entity type and enable span-level detection of entities  Detection of nested and/or discontinuous entities is not supported  The annotated corpus is randomized and split in five folds using scikit-learn BIBREF9  Each fold has a train, test and dev split with the test split defined as  15% of the train split  This ensures comparability between the presented systems 
Baseline systems for Named Entity Recognition in medical case reports ::: Conditional Random Fields
Conditional Random Fields (CRF) BIBREF10 are a standard approach when dealing with sequential data in the context of sequence labeling  We use a combination of linguistic and semantic features, with a context window of size five, to describe each of the tokens and the dependencies between them  Hyper-parameter optimization is performed using randomized search and cross validation  Span-based F1 score is used as the optimization metric 
Baseline systems for Named Entity Recognition in medical case reports ::: BiLSTM-CRF
Prior to the emergence of deep neural language models, BiLSTM-CRF models BIBREF11 had achieved state-of-the-art results for the task of sequence labeling  We use a BiLSTM-CRF model with both word-level and character-level input  BioWordVec BIBREF12 pre-trained word embeddings are used in the embedding layer for the input representation  A bidirectional LSTM layer is applied to a multiplication of the two input representations  Finally, a CRF layer is applied to predict the sequence of labels  Dropout and L1/L2 regularization is used where applicable  He (uniform) initialization BIBREF13 is used to initialize the kernels of the individual layers  As the loss metric, CRF-based loss is used, while optimizing the model based on the CRF Viterbi accuracy  Additionally, span-based F1 score is used to serialize the best performing model  We train for a maximum of 100 epochs, or until an early stopping criterion is reached (no change in validation loss value grater than 0 01 for ten consecutive epochs)  Furthermore, Adam BIBREF14 is used as the optimizer  The learning rate is reduced by a factor of 0 3 in case no significant increase of the optimization metric is achieved in three consecutive epochs 
Baseline systems for Named Entity Recognition in medical case reports ::: Multi-Task Learning
Multi-Task Learning (MTL) BIBREF15 has become popular with the progress in deep learning  This model family is characterized by simultaneous optimization of multiple loss functions and transfer of knowledge achieved this way  The knowledge is transferred through the use of one or multiple shared layers  Through finding supporting patterns in related tasks, MTL provides better generalization on unseen cases and the main tasks we are trying to solve 
We rely on the model presented by bekoulis2018joint and reuse the implementation provided by the authors  The model jointly trains two objectives supported by the dataset: the main task of NER and a supporting task of Relation Extraction (RE)  Two separate models are developed for each of the tasks  The NER task is solved with the help of a BiLSTM-CRF model, similar to the one presented in Section SECREF32 The RE task is solved by using a multi-head selection approach, where each token can have none or more relationships to in-sentence tokens  Additionally, this model also leverages the output of the NER branch model (the CRF prediction) to learn label embeddings  Shared layers consist of a concatenation of word and character embeddings followed by two bidirectional LSTM layers  We keep most of the parameters suggested by the authors and change (1) the number of training epochs to 100 to allow the comparison to other deep learning approaches in this work, (2) use label embeddings of size 64, (3) allow gradient clipping and (4) use $d=0 8$ as the pre-trained word embedding dropout and $d=0 5$ for all other dropouts  $\eta =1^{-3}$ is used as the learning rate with the Adam optimizer and tanh activation functions across layers  Although it is possible to use adversarial training BIBREF16, we omit from using it  We also omit the publication of results for the task of RE as we consider it to be a supporting task and no other competing approaches have been developed 
Baseline systems for Named Entity Recognition in medical case reports ::: BioBERT
Deep neural language models have recently evolved to a successful method for representing text  In particular, Bidirectional Encoder Representations from Transformers (BERT) outperformed previous state-of-the-art methods by a large margin on various NLP tasks BIBREF17  For our experiments, we use BioBERT, an adaptation of BERT for the biomedical domain, pre-trained on PubMed abstracts and PMC full-text articles BIBREF18  The BERT architecture for deriving text representations uses 12 hidden layers, consisting of 768 units each  For NER, token level BIO-tag probabilities are computed with a single output layer based on the representations from the last layer of BERT  We fine-tune the model on the entity recognition task during four training epochs with batch size $b=32$, dropout probability $d=0 1$ and learning rate $\eta =2^{-5}$  These hyper-parameters are proposed by Devlin2018 for BERT fine-tuning 
Baseline systems for Named Entity Recognition in medical case reports ::: Evaluation
To evaluate the performance of the four systems, we calculate the span-level precision (P), recall (R) and F1 scores, along with corresponding micro and macro scores  The reported values are shown in Table TABREF29 and are averaged over five folds, utilising the seqeval framework 
With a macro avg  F1-score of 0 59, MTL achieves the best result with a significant margin compared to CRF, BiLSTM-CRF and BERT  This confirms the usefulness of jointly training multiple objectives (minimizing multiple loss functions), and enabling knowledge transfer, especially in a setting with limited data (which is usually the case in the biomedical NLP domain)  This result also suggest the usefulness of BioBERT for other biomedical datasets as reported by Lee2019  Despite being a rather standard approach, CRF outperforms the more elaborated BiLSTM-CRF, presumably due to data scarcity and class imbalance  We hypothesize that an increase in training data would yield better results for BiLSTM-CRF but not outperform transfer learning approach of MTL (or even BioBERT)  In contrast to other common NER corpora, like CoNLL 2003, even the best baseline system only achieves relatively low scores  This outcome is due to the inherent difficulty of the task (annotators are experienced medical doctors) and the small number of training samples 
Conclusion
We present a new corpus, developed to facilitate the processing of case reports  The corpus focuses on five distinct entity types: cases, conditions, factors, findings and modifiers  Where applicable, relationships between entities are also annotated  Additionally, we annotate discontinuous entities with a special relationship type (discontinuous)  The corpus presented in this paper is the very first of its kind and a valuable addition to the scarce number of corpora available in the field of biomedical NLP  Its complexity, given the discontinuous nature of entities and a high number of nested and multi-label entities, poses new challenges for NLP methods applied for NER and can, hence, be a valuable source for insights into what entities “look like in the wild”  Moreover, it can serve as a playground for new modelling techniques such as the resolution of discontinuous entities as well as multi-task learning given the combination of entities and their relations  We provide an evaluation of four distinct NER systems that will serve as robust baselines for future work but which are, as of yet, unable to solve all the complex challenges this dataset holds  A functional service based on the presented corpus is currently being integrated, as a NER service, in the QURATOR platform BIBREF20 
Acknowledgments
The research presented in this article is funded by the German Federal Ministry of Education and Research (BMBF) through the project QURATOR (Unternehmen Region, Wachstumskern, grant no  03WKDA1A), see http://qurator ai  We want to thank our medical experts for their help annotating the data set, especially Ashlee Finckh and Sophie Klopfenstein 
Context 3:  We manually inspected examples with a low ratio and sent them back to translators accordingly  5) Finally, we used VizSeq BIBREF16 to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings BIBREF17  Samples with low scores were manually inspected and sent back for translation when needed 
We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint 
Data Collection and Processing ::: Tatoeba (TT)
Tatoeba (TT) is a community built language learning corpus having sentences aligned across multiple languages with the corresponding speech partially available  Its sentences are on average shorter than those in CoVoST (see also Table TABREF2) given the original purpose of language learning  Sentences in TT are licensed under CC BY 2 0 FR and part of the speeches are available under various CC licenses 
We construct an evaluation set from TT (for French, German, Dutch, Russian and Spanish) as a complement to CoVoST development and test sets  We collect (speech, transcript, English translation) triplets for the 5 languages and do not include those whose speech has a broken URL or is not CC licensed  We further filter these samples by sentence lengths (minimum 4 words including punctuations) to reduce the portion of short sentences  This makes the resulting evaluation set closer to real-world scenarios and more challenging 
We run the same quality checks for TT as for CoVoST but we do not find poor quality translations according to our criteria  Finally, we report the overlap between CoVo transcripts and TT sentences in Table TABREF5  We found a minimal overlap, which makes the TT evaluation set a suitable additional test set when training on CoVoST 
Data Analysis ::: Basic Statistics
Basic statistics for CoVoST and TT are listed in Table TABREF2 including (unique) sentence counts, speech durations, speaker demographics (partially available) as well as vocabulary and token statistics (based on Moses-tokenized sentences by sacreMoses) on both transcripts and translations  We see that CoVoST has over 327 hours of German speeches and over 171 hours of French speeches, which, to our knowledge, corresponds to the largest corpus among existing public ST corpora (the second largest is 110 hours BIBREF18 for German and 38 hours BIBREF19 for French)  Moreover, CoVoST has a total of 18 hours of Dutch speeches, to our knowledge, contributing the first public Dutch ST resource  CoVoST also has around 27-hour Russian speeches, 37-hour Italian speeches and 67-hour Persian speeches, which is 1 8 times, 2 5 times and 13 3 times of the previous largest public one BIBREF7  Most of the sentences (transcripts) in CoVoST are covered by multiple speakers with potentially different accents, resulting in a rich diversity in the speeches  For example, there are over 1,000 speakers and over 10 accents in the French and German development / test sets  This enables good coverage of speech variations in both model training and evaluation 
Data Analysis ::: Speaker Diversity
As we can see from Table TABREF2, CoVoST is diversified with a rich set of speakers and accents  We further inspect the speaker demographics in terms of sample distributions with respect to speaker counts, accent counts and age groups, which is shown in Figure FIGREF6, FIGREF7 and FIGREF8  We observe that for 8 of the 11 languages, at least 60% of the sentences (transcripts) are covered by multiple speakers  Over 80% of the French sentences have at least 3 speakers  And for German sentences, even over 90% of them have at least 5 speakers  Similarly, we see that a large portion of sentences are spoken in multiple accents for French, German, Dutch and Spanish  Speakers of each language also spread widely across different age groups (below 20, 20s, 30s, 40s, 50s, 60s and 70s) 
Baseline Results
We provide baselines using the official train-development-test split on the following tasks: automatic speech recognition (ASR), machine translation (MT) and speech translation (ST) 
Baseline Results ::: Experimental Settings ::: Data Preprocessing
We convert raw MP3 audio files from CoVo and TT into mono-channel waveforms, and downsample them to 16,000 Hz  For transcripts and translations, we normalize the punctuation, we tokenize the text with sacreMoses and lowercase it  For transcripts, we further remove all punctuation markers except for apostrophes  We use character vocabularies on all the tasks, with 100% coverage of all the characters  Preliminary experimentation showed that character vocabularies provided more stable training than BPE  For MT, the vocabulary is created jointly on both transcripts and translations  We extract 80-channel log-mel filterbank features, computed with a 25ms window size and 10ms window shift using torchaudio  The features are normalized to 0 mean and 1 0 standard deviation  We remove samples having more than 3,000 frames or more than 256 characters for GPU memory efficiency (less than 25 samples are removed for all languages) 
Baseline Results ::: Experimental Settings ::: Model Training
Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing  For MT, we use a Transformer base architecture BIBREF15, but with 3 encoder layers, 3 decoder layers and 0

ground_truth: 8,275 sentences and 167,739 words in total

The corpus comprises 8,275 sentences and 167,739 words in total.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is Arabic one of the 11 languages in CoVost?
Context 3: 3 dropout  We use a batch size of 10,000 frames for ASR and ST, and a batch size of 4,000 tokens for MT  We train all models using Fairseq BIBREF20 for up to 200,000 updates  We use SpecAugment BIBREF21 for ASR and ST to alleviate overfitting 
Baseline Results ::: Experimental Settings ::: Inference and Evaluation
We use a beam size of 5 for all models  We use the best checkpoint by validation loss for MT, and average the last 5 checkpoints for ASR and ST  For MT and ST, we report case-insensitive tokenized BLEU BIBREF22 using sacreBLEU BIBREF23  For ASR, we report word error rate (WER) and character error rate (CER) using VizSeq 
Baseline Results ::: Automatic Speech Recognition (ASR)
For simplicity, we use the same model architecture for ASR and ST, although we do not leverage ASR models to pretrain ST model encoders later  Table TABREF18 shows the word error rate (WER) and character error rate (CER) for ASR models  We see that French and German perform the best given they are the two highest resource languages in CoVoST  The other languages are relatively low resource (especially Turkish and Swedish) and the ASR models are having difficulties to learn from this data 
Baseline Results ::: Machine Translation (MT)
MT models take transcripts (without punctuation) as inputs and outputs translations (with punctuation)  For simplicity, we do not change the text preprocessing methods for MT to correct this mismatch  Moreover, this mismatch also exists in cascading ST systems, where MT model inputs are the outputs of an ASR model  Table TABREF20 shows the BLEU scores of MT models  We notice that the results are consistent with what we see from ASR models  For example thanks to abundant training data, French has a decent BLEU score of 29 8/25 4  German doesn't perform well, because of less richness of content (transcripts)  The other languages are low resource in CoVoST and it is difficult to train decent models without additional data or pre-training techniques 
Baseline Results ::: Speech Translation (ST)
CoVoST is a many-to-one multilingual ST corpus  While end-to-end one-to-many and many-to-many multilingual ST models have been explored very recently BIBREF8, BIBREF9, many-to-one multilingual models, to our knowledge, have not  We hence use CoVoST to examine this setting  Table TABREF22 and TABREF23 show the BLEU scores for both bilingual and multilingual end-to-end ST models trained on CoVoST  We observe that combining speeches from multiple languages is consistently bringing gains to low-resource languages (all besides French and German)  This includes combinations of distant languages, such as Ru+Fr, Tr+Fr and Zh+Fr  Moreover, some combinations do bring gains to high-resource language (French) as well: Es+Fr, Tr+Fr and Mn+Fr  We simply provide the most basic many-to-one multilingual baselines here, and leave the full exploration of the best configurations to future work  Finally, we note that for some language pairs, absolute BLEU numbers are relatively low as we restrict model training to the supervised data  We encourage the community to improve upon those baselines, for example by leveraging semi-supervised training 
Baseline Results ::: Multi-Speaker Evaluation
In CoVoST, large portion of transcripts are covered by multiple speakers with different genders, accents and age groups  Besides the standard corpus-level BLEU scores, we also want to evaluate model output variance on the same content (transcript) but different speakers  We hence propose to group samples (and their sentence BLEU scores) by transcript, and then calculate average per-group mean and average coefficient of variation defined as follows:
and
where $G$ is the set of sentence BLEU scores grouped by transcript and $G^{\prime } = \lbrace g | g\in G, |g|>1, \textrm {Mean}(g) > 0 \rbrace $ 
$\textrm {BLEU}_{MS}$ provides a normalized quality score as oppose to corpus-level BLEU or unnormalized average of sentence BLEU  And $\textrm {CoefVar}_{MS}$ is a standardized measure of model stability against different speakers (the lower the better)  Table TABREF24 shows the $\textrm {BLEU}_{MS}$ and $\textrm {CoefVar}_{MS}$ of our ST models on CoVoST test set  We see that German and Persian have the worst $\textrm {CoefVar}_{MS}$ (least stable) given their rich speaker diversity in the test set and relatively small train set (see also Figure FIGREF6 and Table TABREF2)  Dutch also has poor $\textrm {CoefVar}_{MS}$ because of the lack of training data  Multilingual models are consistantly more stable on low-resource languages  Ru+Fr, Tr+Fr, Fa+Fr and Zh+Fr even have better $\textrm {CoefVar}_{MS}$ than all individual languages 
Conclusion
We introduce a multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  We also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation  CoVoST is free to use with a CC0 license, and the additional Tatoeba evaluation samples are also CC-licensed 

ground_truth: No

No