prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Where does the ancient Chinese dataset come from?
Context 1: Introduction
Ancient Chinese is the writing language in ancient China  It is a treasure of Chinese culture which brings together the wisdom and ideas of the Chinese nation and chronicles the ancient cultural heritage of China  Learning ancient Chinese not only helps people to understand and inherit the wisdom of the ancients, but also promotes people to absorb and develop Chinese culture 
However, it is difficult for modern people to read ancient Chinese  Firstly, compared with modern Chinese, ancient Chinese is more concise and shorter  The grammatical order of modern Chinese is also quite different from that of ancient Chinese  Secondly, most modern Chinese words are double syllables, while the most of the ancient Chinese words are monosyllabic  Thirdly, there is more than one polysemous phenomenon in ancient Chinese  In addition, manual translation has a high cost  Therefore, it is meaningful and useful to study the automatic translation from ancient Chinese to modern Chinese  Through ancient-modern Chinese translation, the wisdom, talent and accumulated experience of the predecessors can be passed on to more people 
Neural machine translation (NMT) BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 has achieved remarkable performance on many bilingual translation tasks  It is an end-to-end learning approach for machine translation, with the potential to show great advantages over the statistic machine translation (SMT) systems  However, NMT approach has not been widely applied to the ancient-modern Chinese translation task  One of the main reasons is the limited high-quality parallel data resource 
The most popular method of acquiring translation examples is bilingual text alignment BIBREF5   This kind of method can be classified into two types: lexical-based and statistical-based  The lexical-based approaches BIBREF6 , BIBREF7 focus on lexical information, which utilize the bilingual dictionary BIBREF8 , BIBREF9 or lexical features  Meanwhile, the statistical-based approaches BIBREF10 , BIBREF11 rely on statistical information, such as sentence length ratio in two languages and align mode probability 
However, these methods are designed for other bilingual language pairs that are written in different language characters (e g  English-French, Chinese-Japanese)  The ancient-modern Chinese has some characteristics that are quite different from other language pairs  For example, ancient and modern Chinese are both written in Chinese characters, but ancient Chinese is highly concise and its syntactical structure is different from modern Chinese  The traditional methods do not take these characteristics into account  In this paper, we propose an effective ancient-modern Chinese text alignment method at the level of clause based on the characteristics of these two languages  The proposed method combines both lexical-based information and statistical-based information, which achieves 94 2 F1-score on Test set  Recently, a simple longest common subsequence based approach for ancient-modern Chinese sentence alignment is proposed in BIBREF12   Our experiments showed that our proposed alignment approach performs much better than their method 
We apply the proposed method to create a large translation parallel corpus which contains INLINEFORM0 1 24M bilingual sentence pairs  To our best knowledge, this is the first large high-quality ancient-modern Chinese dataset  Furthermore, we test SMT models and various NMT models on the created dataset and provide a strong baseline for this task 
Overview
There are four steps to build the ancient-modern Chinese translation dataset: (i) The parallel corpus crawling and cleaning  (ii) The paragraph alignment  (iii) The clause alignment based on aligned paragraphs  (iv) Augmenting data by merging aligned adjacent clauses  The most critical step is the third step 
Clause Alignment
In the clause alignment step, we combine both statistical-based and lexical-based information to measure the score for each possible clause alignment between ancient and modern Chinese strings  The dynamic programming is employed to further find overall optimal alignment paragraph by paragraph  According to the characteristics of the ancient and modern Chinese languages, we consider the following factors to measure the alignment score INLINEFORM0 between a bilingual clause pair:
Lexical Matching  The lexical matching score is used to calculate the matching coverage of the ancient clause INLINEFORM0   It contains two parts: exact matching and dictionary matching  An ancient Chinese character usually corresponds to one or more modern Chinese words  In the first part, we carry out Chinese Word segmentation to the modern Chinese clause INLINEFORM1   Then we match the ancient characters and modern words in the order from left to right  In further matching, the words that have been matched will be deleted from the original clauses 
However, some ancient characters do not appear in its corresponding modern Chinese words  An ancient Chinese dictionary is employed to address this issue  We preprocess the ancient Chinese dictionary and remove the stop words  In this dictionary matching step, we retrieve the dictionary definition of each unmatched ancient character and use it to match the remaining modern Chinese words  To reduce the impact of universal word matching, we use Inverse Document Frequency (IDF) to weight the matching words  The lexical matching score is calculated as: DISPLAYFORM0
The above equation is used to calculate the matching coverage of the ancient clause INLINEFORM0   The first term of equation ( EQREF8 ) represents exact matching score  INLINEFORM1 denotes the length of INLINEFORM2 , INLINEFORM3 denotes each ancient character in INLINEFORM4 , and the indicator function INLINEFORM5 indicates whether the character INLINEFORM6 can match the words in the clause INLINEFORM7   The second term is dictionary matching score  Here INLINEFORM8 and INLINEFORM9 represent the remaining unmatched strings of INLINEFORM10 and INLINEFORM11 , respectively  INLINEFORM12 denotes the INLINEFORM13 -th character in the dictionary definition of the INLINEFORM14 and its IDF score is denoted as INLINEFORM15   The INLINEFORM16 is a predefined parameter which is used to normalize the IDF score  We tuned the value of this parameter on the Dev set 
Statistical Information  Similar to BIBREF11 and BIBREF6 , the statistical information contains alignment mode and length information  There are many alignment modes between ancient and modern Chinese languages  If one ancient Chinese clause aligns two adjacent modern Chinese clauses, we call this alignment as 1-2 alignment mode  We show some examples of different alignment modes in Figure FIGREF9   In this paper, we only consider 1-0, 0-1, 1-1, 1-2, 2-1 and 2-2 alignment modes which account for INLINEFORM0 of the Dev set  We estimate the probability Pr INLINEFORM1 n-m INLINEFORM2 of each alignment mode n-m on the Dev set  To utilize length information, we make an investigation on length correlation between these two languages  Based on the assumption of BIBREF11 that each character in one language gives rise to a random number of characters in the other language and those random variables INLINEFORM3 are independent and identically distributed with a normal distribution, we estimate the mean INLINEFORM4 and standard deviation INLINEFORM5 from the paragraph aligned parallel corpus  Given a clause pair INLINEFORM6 , the statistical information score can be calculated by: DISPLAYFORM0
where INLINEFORM0 denotes the normal distribution probability density function 
Edit Distance  Because ancient and modern Chinese are both written in Chinese characters, we also consider using the edit distance  It is a way of quantifying the dissimilarity between two strings by counting the minimum number of operations (insertion, deletion, and substitution) required to transform one string into the other  Here we define the edit distance score as: DISPLAYFORM0
Dynamic Programming  The overall alignment score for each possible clause alignment is as follows: DISPLAYFORM0
Here INLINEFORM0 and INLINEFORM1 are pre-defined interpolation factors  We use dynamic programming to find the overall optimal alignment paragraph by paragraph  Let INLINEFORM2 be total alignment scores of aligning the first to INLINEFORM3 -th ancient Chinese clauses with the first to to INLINEFORM4 -th modern Chinese clauses, and the recurrence then can be described as follows: DISPLAYFORM0
Where INLINEFORM0 denotes concatenate clause INLINEFORM1 to clause INLINEFORM2   As we discussed above, here we only consider 1-0, 0-1, 1-1, 1-2, 2-1 and 2-2 alignment modes 
Ancient-Modern Chinese Dataset
Data Collection  To build the large ancient-modern Chinese dataset, we collected 1 7K bilingual ancient-modern Chinese articles from the internet  More specifically, a large part of the ancient Chinese data we used come from ancient Chinese history records in several dynasties (about 1000BC-200BC) and articles written by celebrities of that era  They used plain and accurate words to express what happened at that time, and thus ensure the generality of the translated materials 
Paragraph Alignment  To further ensure the quality of the new dataset, the work of paragraph alignment is manually completed  After data cleaning and manual paragraph alignment, we obtained 35K aligned bilingual paragraphs 
Clause Alignment  We applied our clause alignment algorithm on the 35K aligned bilingual paragraphs and obtained 517K aligned bilingual clauses  The reason we use clause alignment algorithm instead of sentence alignment is because we can construct more aligned sentences more flexibly and conveniently  To be specific, we can get multiple additional sentence level bilingual pairs by “data augmentation” 
Data Augmentation  We augmented the data in the following way: Given an aligned clause pair, we merged its adjacent clause pairs as a new sample pair  For example, suppose we have three adjacent clause level bilingual pairs: ( INLINEFORM0 , INLINEFORM1 ), ( INLINEFORM2 , INLINEFORM3 ), and ( INLINEFORM4 , INLINEFORM5 )  We can get some additional sentence level bilingual pairs, such as: ( INLINEFORM6 , INLINEFORM7 ) and ( INLINEFORM8 , INLINEFORM9 )  Here INLINEFORM10 , INLINEFORM11 , and INLINEFORM12 are adjacent clauses in the original paragraph, and INLINEFORM13 denotes concatenate clause INLINEFORM14 to clause INLINEFORM15   The advantage of using this data augmentation method is that compared with only using ( INLINEFORM16 , INLINEFORM17 ) as the training data, we can also use ( INLINEFORM18 , INLINEFORM19 ) and ( INLINEFORM20 , INLINEFORM21 ) as the training data, which can provide richer supervision information for the model and make the model learn the align information between the source language and the target language better  After the data augmentation, we filtered the sentences which are longer than 50 or contain more than four clause pairs 
Dataset Creation  Finally, we split the dataset into three sets: training (Train), development (Dev) and testing (Test)  Note that the unaugmented dataset contains 517K aligned bilingual clause pairs from 35K aligned bilingual paragraphs  To keep all the sentences in different sets come from different articles, we split the 35K aligned bilingual paragraphs into Train, Dev and Test sets following these ratios respectively: 80%, 10%, 10%  Before data augmentation, the unaugmented Train set contains INLINEFORM0 aligned bilingual clause pairs from 28K aligned bilingual paragraphs  Then we augmented the Train, Dev and Test sets respectively  Note that the augmented Train, Dev and Test sets also contain the unaugmented data  The statistical information of the three data sets is shown in Table TABREF17   We show some examples of data in Figure FIGREF14  
RNN-based NMT model
We first briefly introduce the RNN based Neural Machine Translation (RNN-based NMT) model  The RNN-based NMT with attention mechanism BIBREF0 has achieved remarkable performance on many translation tasks  It consists of encoder and decoder part 
We firstly introduce the encoder part  The input word sequence of source language are individually mapped into a INLINEFORM0 -dimensional vector space INLINEFORM1   Then a bi-directional RNN BIBREF15 with GRU BIBREF16 or LSTM BIBREF17 cell converts these vectors into a sequences of hidden states INLINEFORM2  
For the decoder part, another RNN is used to generate target sequence INLINEFORM0   The attention mechanism BIBREF0 , BIBREF18 is employed to allow the decoder to refer back to the hidden state sequence and focus on a particular segment  The INLINEFORM1 -th hidden state INLINEFORM2 of decoder part is calculated as: DISPLAYFORM0
Here g INLINEFORM0 is a linear combination of attended context vector c INLINEFORM1 and INLINEFORM2 is the word embedding of (i-1)-th target word: DISPLAYFORM0
The attended context vector c INLINEFORM0 is computed as a weighted sum of the hidden states of the encoder: DISPLAYFORM0
The probability distribution vector of the next word INLINEFORM0 is generated according to the following: DISPLAYFORM0
We take this model as the basic RNN-based NMT model in the following experiments 
Transformer-NMT
Recently, the Transformer model BIBREF4 has made remarkable progress in machine translation  This model contains a multi-head self-attention encoder and a multi-head self-attention decoder 
As proposed by BIBREF4 , an attention function maps a query and a set of key-value pairs to an output, where the queries INLINEFORM0 , keys INLINEFORM1 , and values INLINEFORM2 are all vectors  The input consists of queries and keys of dimension INLINEFORM3 , and values of dimension INLINEFORM4   The attention function is given by: DISPLAYFORM0
Multi-head attention mechanism projects queries, keys and values to INLINEFORM0 different representation subspaces and calculates corresponding attention  The attention function outputs are concatenated and projected again before giving the final output  Multi-head attention allows the model to attend to multiple features at different positions 
The encoder is composed of a stack of INLINEFORM0 identical layers  Each layer has two sub-layers: multi-head self-attention mechanism and position-wise fully connected feed-forward network  Similarly, the decoder is also composed of a stack of INLINEFORM1 identical layers  In addition to the two sub-layers in each encoder layer, the decoder contains a third sub-layer which performs multi-head attention over the output of the encoder stack (see more details in BIBREF4 ) 
Experiments
Our experiments revolve around the following questions: Q1: As we consider three factors for clause alignment, do all these factors help? How does our method compare with previous methods? Q2: How does the NMT and SMT models perform on this new dataset we build?
Clause Alignment Results (Q1)
In order to evaluate our clause alignment algorithm, we manually aligned bilingual clauses from 37 bilingual ancient-modern Chinese articles, and finally got 4K aligned bilingual clauses as the Test set and 2K clauses as the Dev set 
Metrics  We used F1-score and precision score as the evaluation metrics  Suppose that we get INLINEFORM0 bilingual clause pairs after running the algorithm on the Test set, and there are INLINEFORM1 bilingual clause pairs of these INLINEFORM2 pairs are in the ground truth of the Test set, the precision score is defined as INLINEFORM3 (the algorithm gives INLINEFORM4 outputs, INLINEFORM5 of which are correct)  And suppose that the ground truth of the Test set contains INLINEFORM6 bilingual clause pairs, the recall score is INLINEFORM7 (there are INLINEFORM8 ground truth samples, INLINEFORM9 of which are output by the algorithm), then the F1-score is INLINEFORM10  
Baselines  Since the related work BIBREF10 , BIBREF11 can be seen as the ablation cases of our method (only statistical score INLINEFORM0 with dynamic programming), we compared the full proposed method with its variants on the Test set for ablation study  In addition, we also compared our method with the longest common subsequence (LCS) based approach proposed by BIBREF12   To the best of our knowledge, BIBREF12 is the latest related work which are designed for Ancient-Modern Chinese alignment 
Hyper-parameters  For the proposed method, we estimated INLINEFORM0 and INLINEFORM1 on all aligned paragraphs  The probability Pr INLINEFORM2 n-m INLINEFORM3 of each alignment mode n-m was estimated on the Dev set  For the hyper-parameters INLINEFORM4 , INLINEFORM5 and INLINEFORM6 , the grid search was applied to tune them on the Dev set  In order to show the effect of hyper-parameters INLINEFORM7 , INLINEFORM8 , and INLINEFORM9 , we reported the results of various hyper-parameters on the Dev set in Table TABREF26   Based on the results of grid search on the Dev set, we set INLINEFORM10 , INLINEFORM11 , and INLINEFORM12 in the following experiment  The Jieba Chinese text segmentation is employed for modern Chinese word segmentation 
Results  The results on the Test set are shown in Table TABREF28 , the abbreviation w/o means removing a particular part from the setting  From the results, we can see that the lexical matching score is the most important among these three factors, and statistical information score is more important than edit distance score  Moreover, the dictionary term in lexical matching score significantly improves the performance  From these results, we obtain the best setting that involves all these three factors  We used this setting for dataset creation  Furthermore, the proposed method performs much better than LCS BIBREF12  
Translation Results (Q2)
In this experiment, we analyzed and compared the performance of the SMT and various NMT models on our built dataset  To verify the effectiveness of our data augmented method  We trained the NMT and SMT models on both unaugmented dataset (including 0 46M training pairs) and augmented dataset, and test all the models on the same Test set which is augmented  The models to be tested and their configurations are as follows:
SMT  The state-of-art Moses toolkit BIBREF19 was used to train SMT model  We used KenLM BIBREF20 to train a 5-gram language model, and the GIZA++ toolkit to align the data 
RNN-based NMT  The basic RNN-based NMT model is based on BIBREF0 which is introduced above  Both the encoder and decoder used 2-layer RNN with 1024 LSTM cells, and the encoder is a bi-directional RNN  The batch size, threshold of element-wise gradient clipping and initial learning rate of Adam optimizer BIBREF21 were set to 128, 5 0 and 0 001  When trained the model on augmented dataset, we used 4-layer RNN  Several techniques were investigated to train the model, including layer-normalization BIBREF22 , RNN-dropout BIBREF23 , and learning rate decay BIBREF1   The hyper-parameters were chosen empirically and adjusted in the Dev set  Furthermore, we tested the basic NMT model with several techniques, such as target language reversal BIBREF24 (reversing the order of the words in all target sentences, but not source sentences), residual connection BIBREF25 and pre-trained word2vec BIBREF26   For word embedding pre-training, we collected an external ancient corpus which contains INLINEFORM0 134M tokens 
Transformer-NMT  We also trained the Transformer model BIBREF4 which is a strong baseline of NMT on both augmented and unaugmented parallel corpus  The training configuration of the Transformer model is shown in Table TABREF32   The hyper-parameters are set based on the settings in the paper BIBREF4 and the sizes of our training sets 
For the evaluation, we used the average of 1 to 4 gram BLEUs multiplied by a brevity penalty BIBREF27 which computed by multi-bleu perl in Moses as metrics  The results are reported in Table TABREF34   For RNN-based NMT, we can see that target language reversal, residual connection, and word2vec can further improve the performance of the basic RNN-based NMT model  However, we find that word2vec and reversal tricks seem no obvious improvement when trained the RNN-based NMT and Transformer models on augmented parallel corpus  For SMT, it performs better than NMT models when they were trained on the unaugmented dataset  Nevertheless, when trained on the augmented dataset, both the RNN-based NMT model and Transformer based NMT model outperform the SMT model  In addition, as with other translation tasks BIBREF4 , the Transformer also performs better than RNN-based NMT 
Because the Test set contains both augmented and unaugmented data, it is not surprising that the RNN-based NMT model and Transformer based NMT model trained on unaugmented data would perform poorly  In order to further verify the effect of data augmentation, we report the test results of the models on only unaugmented test data (including 48K test pairs) in Table TABREF35   From the results, it can be seen that the data augmentation can still improve the models 
Analysis
The generated samples of various models are shown in Figure FIGREF36   Besides BLEU scores, we analyze these examples from a human perspective and draw some conclusions  At the same time, we design different metrics and evaluate on the whole Test set to support our conclusions as follows:
On the one hand, we further compare the translation results from the perspective of people  We find that although the original meaning can be basically translated by SMT, its translation results are less smooth when compared with the other two NMT models (RNN-based NMT and Transformer)  For example, the translations of SMT are usually lack of auxiliary words, conjunctions and function words, which is not consistent with human translation habits  To further confirm this conclusion, the average length of the translation results of the three models are measured (RNN-based NMT:17 12, SMT:15 50, Transformer:16 78, Reference:16 47)  We can see that the average length of the SMT outputs is shortest, and the length gaps between the SMT outputs and the references are largest  Meanwhile, the average length of the sentences translated by Transformer is closest to the average length of references  These results indirectly verify our point of view, and show that the NMT models perform better than SMT in this task 
On the other hand, there still exists some problems to be solved  We observe that translating proper nouns and personal pronouns (such as names, place names and ancient-specific appellations) is very difficult for all of these models  For instance, the ancient Chinese appellation `Zhen' should be translated into `Wo' in modern Chinese  Unfortunately, we calculate the accurate rate of some special words (such as `Zhen',`Chen' and `Gua'), and find that this rate is very low (the accurate rate of translating `Zhen' are: RNN-based NMT:0 14, SMT:0 16, Transformer:0 05)  We will focus on this issue in the future 
Conclusion and Future Work
We propose an effective ancient-modern Chinese clause alignment method which achieves 94 2 F1-score on Test set  Based on it, we build a large scale parallel corpus which contains INLINEFORM0 1 24M bilingual sentence pairs  To our best knowledge, this is the first large high-quality ancient-modern Chinese dataset  In addition, we test the performance of the SMT and various NMT models on our built dataset and provide a strong NMT baseline for this task which achieves 27 16 BLEU score (4-gram)  We further analyze the performance of the SMT and various NMT models and summarize some specific problems that machine translation models will encounter when translating ancient Chinese 
For the future work, firstly, we are going to expand the dataset using the proposed method continually  Secondly, we will focus on solving the problem of proper noun translation and improve the translation system according to the features of ancient Chinese translation  Finally, we plan to introduce some techniques of statistical translation into neural machine translation to improve the performance 
This work is supported by National Natural Science Fund for Distinguished Young Scholar (Grant No  61625204) and partially supported by the State Key Program of National Science Foundation of China (Grant Nos  61836006 and 61432014) 
Context 2: Introduction
Assembling training corpora of annotated natural language examples in specialized domains such as biomedicine poses considerable challenges  Experts with the requisite domain knowledge to perform high-quality annotation tend to be expensive, while lay annotators may not have the necessary knowledge to provide high-quality annotations  A practical approach for collecting a sufficiently large corpus would be to use crowdsourcing platforms like Amazon Mechanical Turk (MTurk)  However, crowd workers in general are likely to provide noisy annotations BIBREF0 , BIBREF1 , BIBREF2 , an issue exacerbated by the technical nature of specialized content  Some of this noise may reflect worker quality and can be modeled BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4 , but for some instances lay people may simply lack the domain knowledge to provide useful annotation 
In this paper we report experiments on the EBM-NLP corpus comprising crowdsourced annotations of medical literature BIBREF5   We operationalize the concept of annotation difficulty and show how it can be exploited during training to improve information extraction models  We then obtain expert annotations for the abstracts predicted to be most difficult, as well as for a similar number of randomly selected abstracts  The annotation of highly specialized data and the use of lay and expert annotators allow us to examine the following key questions related to lay and expert annotations in specialized domains:
Can we predict item difficulty? We define a training instance as difficult if a lay annotator or an automated model disagree on its labeling  We show that difficulty can be predicted, and that it is distinct from inter-annotator agreement  Further, such predictions can be used during training to improve information extraction models 
Are there systematic differences between expert and lay annotations? We observe decidedly lower agreement between lay workers as compared to domain experts  Lay annotations have high precision but low recall with respect to expert annotations in the new data that we collected  More generally, we expect lay annotations to be lower quality, which may translate to lower precision, recall, or both, compared to expert annotations  Can one rely solely on lay annotations? Reasonable models can be trained using lay annotations alone, but similar performance can be achieved using markedly less expert data  This suggests that the optimal ratio of expert to crowd annotations for specialized tasks will depend on the cost and availability of domain experts  Expert annotations are preferable whenever its collection is practical  But in real-world settings, a combination of expert and lay annotations is better than using lay data alone 
Does it matter what data is annotated by experts? We demonstrate that a system trained on combined data achieves better predictive performance when experts annotate difficult examples rather than instances selected at i i d  random 
Our contributions in this work are summarized as follows  We define a task difficulty prediction task and show how this is related to, but distinct from, inter-worker agreement  We introduce a new model for difficulty prediction combining learned representations induced via a pre-trained `universal' sentence encoder BIBREF6 , and a sentence encoder learned from scratch for this task  We show that predicting annotation difficulty can be used to improve the task routing and model performance for a biomedical information extraction task  Our results open up a new direction for ensuring corpus quality  We believe that item difficulty prediction will likely be useful in other, non-specialized tasks as well, and that the most effective data collection in specialized domains requires research addressing the fundamental questions we examine here 
Related Work
Crowdsourcing annotation is now a well-studied problem BIBREF7 , BIBREF0 , BIBREF1 , BIBREF2   Due to the noise inherent in such annotations, there have also been considerable efforts to develop aggregation models that minimize noise BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4  
There are also several surveys of crowdsourcing in biomedicine specifically BIBREF8 , BIBREF9 , BIBREF10   Some work in this space has contrasted model performance achieved using expert vs  crowd annotated training data BIBREF11 , BIBREF12 , BIBREF13   Dumitrache et al  Dumitrache:2018:CGT:3232718 3152889 concluded that performance is similar under these supervision types, finding no clear advantage from using expert annotators  This differs from our findings, perhaps owing to differences in design  The experts we used already hold advanced medical degrees, for instance, while those in prior work were medical students  Furthermore, the task considered here would appear to be of greater difficulty: even a system trained on $\sim $ 5k instances performs reasonably, but far from perfect  By contrast, in some of the prior work where experts and crowd annotations were deemed equivalent, a classifier trained on 300 examples can achieve very high accuracy BIBREF12  
More relevant to this paper, prior work has investigated methods for `task routing' in active learning scenarios in which supervision is provided by heterogeneous labelers with varying levels of expertise BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF14   The related question of whether effort is better spent collecting additional annotations for already labeled (but potentially noisily so) examples or novel instances has also been addressed BIBREF18   What distinguishes the work here is our focus on providing an operational definition of instance difficulty, showing that this can be predicted, and then using this to inform task routing 
Application Domain
Our specific application concerns annotating abstracts of articles that describe the conduct and results of randomized controlled trials (RCTs)  Experimentation in this domain has become easy with the recent release of the EBM-NLP BIBREF5 corpus, which includes a reasonably large training dataset annotated via crowdsourcing, and a modest test set labeled by individuals with advanced medical training  More specifically, the training set comprises 4,741 medical article abstracts with crowdsourced annotations indicating snippets (sequences) that describe the Participants (p), Interventions (i), and Outcome (o) elements of the respective RCT, and the test set is composed of 191 abstracts with p, i, o sequence annotations from three medical experts 
Table 1 shows an example of difficult and easy examples according to our definition of difficulty  The underlined text demarcates the (consensus) reference label provided by domain experts  In the difficult examples, crowd workers marked text distinct from these reference annotations; whereas in the easy cases they reproduced them with reasonable fidelity  The difficult sentences usually exhibit complicated structure and feature jargon 
An abstract may contain some `easy' and some `difficult' sentences  We thus perform our analysis at the sentence level  We split abstracts into sentences using spaCy  We excluded sentences that comprise fewer than two tokens, as these are likely an artifact of errors in sentence splitting  In total, this resulted in 57,505 and 2,428 sentences in the train and test set abstracts, respectively 
Quantifying Task Difficulty
The test set includes annotations from both crowd workers and domain experts  We treat the latter as ground truth and then define the difficulty of sentences in terms of the observed agreement between expert and lay annotators  Formally, for annotation task $t$ and instance $i$ :
$$\text{Difficulty}_{ti} = \frac{\sum _{j=1}^n{f(\text{label}_{ij}, y_i})}{n}$$   (Eq  3)
where $f$ is a scoring function that measures the quality of the label from worker $j$ for sentence $i$ , as compared to a ground truth annotation, $y_i$   The difficulty score of sentence $i$ is taken as an average over the scores for all $n$ layworkers  We use Spearmans' correlation coefficient as a scoring function  Specifically, for each sentence we create two vectors comprising counts of how many times each token was annotated by crowd and expert workers, respectively, and calculate the correlation between these  Sentences with no labels are treated as maximally easy; those with only either crowd worker or expert label(s) are assumed maximally difficult 
The training set contains only crowdsourced annotations  To label the training data, we use a 10-fold validation like setting  We iteratively retrain the LSTM-CRF-Pattern sequence tagger of Patel et al  patel2018syntactic on 9 folds of the training data and use that trained model to predict labels for the 10th  In this way we obtain predictions on the full training set  We then use predicted spans as proxy `ground truth' annotations to calculate the difficulty score of sentences as described above; we normalize these to the [ $0, 1$ ] interval  We validate this approximation by comparing the proxy scores against reference scores over the test set, the Pearson's correlation coefficients are 0 57 for Population, 0 71 for Intervention and 0 68 for Outcome 
There exist many sentences that contain neither manual nor predicted annotations  We treat these as maximally easy sentences (with difficulty scores of 0)  Such sentences comprise 51%, 42% and 36% for Population, Interventions and Outcomes data respectively, indicating that it is easier to identify sentences that have no Population spans, but harder to identify sentences that have no Interventions or Outcomes spans  This is intuitive as descriptions of the latter two tend to be more technical and dense with medical jargon 
We show the distribution of the automatically labeled scores for sentences that do contain spans in Figure 1   The mean of the Population (p) sentence scores is significantly lower than that for other types of sentences (i and o), again indicating that they are easier on average to annotate  This aligns with a previous finding that annotating Interventions and Outcomes is more difficult than annotating Participants BIBREF5  
Many sentences contain spans tagged by the LSTM-CRF-Pattern model, but missed by all crowd workers, resulting in a maximally difficult score (1)  Inspection of such sentences revealed that some are truly difficult examples, but others are tagging model errors  In either case, such sentences have confused workers and/or the model, and so we retain them all as `difficult' sentences 
Content describing the p, i and o, respectively, is quite different  As such, one sentence usually contains (at most) only one of these three content types  We thus treat difficulty prediction for the respective label types as separate tasks 
Difficulty is not Worker Agreement
Our definition of difficulty is derived from agreement between expert and crowd annotations for the test data, and agreement between a predictive model and crowd annotations in the training data  It is reasonable to ask if these measures are related to inter-annotator agreement, a metric often used in language technology research to identify ambiguous or difficult items  Here we explicitly verify that our definition of difficulty only weakly correlates with inter-annotator agreement 
We calculate inter-worker agreement between crowd and expert annotators using Spearman's correlation coefficient  As shown in Table 2 , average agreement between domain experts are considerably higher than agreements between crowd workers for all three label types  This is a clear indication that the crowd annotations are noisier 
Furthermore, we compare the correlation between inter-annotator agreement and difficulty scores in the training data  Given that the majority of sentences do not contain a PICO span, we only include in these calculations those that contain a reference label  Pearson's r are 0 34, 0 30 and 0 31 for p, i and o, respectively, confirming that inter-worker agreement and our proposed difficulty score are quite distinct 
Predicting Annotation Difficulty
We treat difficulty prediction as a regression problem, and propose and evaluate neural model variants for the task  We first train RNN BIBREF19 and CNN BIBREF20 models 
We also use the universal sentence encoder (USE) BIBREF6 to induce sentence representations, and train a model using these as features  Following BIBREF6 , we then experiment with an ensemble model that combines the `universal' and task-specific representations to predict annotation difficulty  We expect these universal embeddings to capture general, high-level semantics, and the task specific representations to capture more granular information  Figure 2 depicts the model architecture  Sentences are fed into both the universal sentence encoder and, separately, a task specific neural encoder, yielding two representations  We concatenate these and pass the combined vector to the regression layer 
Experimental Setup and Results
We trained models for each label type separately  Word embeddings were initialized to 300d GloVe vectors BIBREF21 trained on common crawl data; these are fine-tuned during training  We used the Adam optimizer BIBREF22 with learning rate and decay set to 0 001 and 0 99, respectively  We used batch sizes of 16 
We used the large version of the universal sentence encoder with a transformer BIBREF23   We did not update the pretrained sentence encoder parameters during training  All hyperparamaters for all models (including hidden layers, hidden sizes, and dropout) were tuned using Vizier BIBREF24 via 10-fold cross validation on the training set maximizing for F1 
As a baseline, we also trained a linear Support-Vector Regression BIBREF25 model on $n$ -gram features ( $n$ ranges from 1 to 3) 
Table 3 reports Pearson correlation coefficients between the predictions with each of the neural models and the ground truth difficulty scores  Rows 1-4 correspond to individual models, and row 5 reports the ensemble performance  Columns correspond to label type  Results from all models outperform the baseline SVR model: Pearson's correlation coefficients range from 0 550 to 0 622  The regression correlations are the lowest 
The RNN model realizes the strongest performance among the stand-alone (non-ensemble) models, outperforming variants that exploit CNN and USE representations  Combining the RNN and USE further improves results  We hypothesize that this is due to complementary sentence information encoded in universal representations 
For all models, correlations for Intervention and Outcomes are higher than for Population, which is expected given the difficulty distributions in Figure 1   In these, the sentences are more uniformly distributed, with a fair number of difficult and easier sentences  By contrast, in Population there are a greater number of easy sentences and considerably fewer difficult sentences, which makes the difficulty ranking task particularly challenging
Context 3: Introduction
Word embeddings are representations of words in numerical form, as vectors of typically several hundred dimensions  The vectors are used as an input to machine learning models; for complex language processing tasks these are typically deep neural networks  The embedding vectors are obtained from specialized learning tasks, based on neural networks, e g , word2vec BIBREF0, GloVe BIBREF1, FastText BIBREF2, ELMo BIBREF3, and BERT BIBREF4  For training, the embeddings algorithms use large monolingual corpora that encode important information about word meaning as distances between vectors  In order to enable downstream machine learning on text understanding tasks, the embeddings shall preserve semantic relations between words, and this is true even across languages 
Probably the best known word embeddings are produced by the word2vec method BIBREF5  The problem with word2vec embeddings is their failure to express polysemous words  During training of an embedding, all senses of a given word (e g , paper as a material, as a newspaper, as a scientific work, and as an exam) contribute relevant information in proportion to their frequency in the training corpus  This causes the final vector to be placed somewhere in the weighted middle of all words' meanings  Consequently, rare meanings of words are poorly expressed with word2vec and the resulting vectors do not offer good semantic representations  For example, none of the 50 closest vectors of the word paper is related to science 
The idea of contextual embeddings is to generate a different vector for each context a word appears in and the context is typically defined sentence-wise  To a large extent, this solves the problems with word polysemy, i e  the context of a sentence is typically enough to disambiguate different meanings of a word for humans and so it is for the learning algorithms  In this work, we describe high-quality models for contextual embeddings, called ELMo BIBREF3, precomputed for seven morphologically rich, less-resourced languages: Slovenian, Croatian, Finnish, Estonian, Latvian, Lithuanian, and Swedish  ELMo is one of the most successful approaches to contextual word embeddings  At time of its creation, ELMo has been shown to outperform previous word embeddings BIBREF3 like word2vec and GloVe on many NLP tasks, e g , question answering, named entity extraction, sentiment analysis, textual entailment, semantic role labeling, and coreference resolution 
This report is split into further five sections  In section SECREF2, we describe the contextual embeddings ELMo  In Section SECREF3, we describe the datasets used and in Section SECREF4 we describe preprocessing and training of the embeddings  We describe the methodology for evaluation of created vectors and results in Section SECREF5  We present conclusion in Section SECREF6 where we also outline plans for further work 
ELMo
Typical word embeddings models or representations, such as word2vec BIBREF0, GloVe BIBREF1, or FastText BIBREF2, are fast to train and have been pre-trained for a number of different languages  They do not capture the context, though, so each word is always given the same vector, regardless of its context or meaning  This is especially problematic for polysemous words  ELMo (Embeddings from Language Models) embedding BIBREF3 is one of the state-of-the-art pretrained transfer learning models, that remedies the problem and introduces a contextual component 
ELMo model`s architecture consists of three neural network layers  The output of the model after each layer gives one set of embeddings, altogether three sets  The first layer is a CNN layer, which operates on a character level  It is context independent, so each word always gets the same embedding, regardless of its context  It is followed by two biLM layers  A biLM layer consists of two concatenated LSTMs  In the first LSTM, we try to predict the following word, based on the given past words, where each word is represented by the embeddings from the CNN layer  In the second LSTM, we try to predict the preceding word, based on the given following words  It is equivalent to the first LSTM, just reading the text in reverse 
In NLP tasks, any set of these embeddings may be used; however, a weighted average is usually used  The weights of the average are learned during the training of the model for the specific task  Additionally, an entire ELMo model can be fine-tuned on a specific end task 
Although ELMo is trained on character level and is able to handle out-of-vocabulary words, a vocabulary file containing most common tokens is used for efficiency during training and embedding generation  The original ELMo model was trained on a one billion word large English corpus, with a given vocabulary file of about 800,000 words  Later, ELMo models for other languages were trained as well, but limited to larger languages with many resources, like German and Japanese 
ELMo ::: ELMoForManyLangs
Recently, ELMoForManyLangs BIBREF6 project released pre-trained ELMo models for a number of different languages BIBREF7  These models, however, were trained on a significantly smaller datasets  They used 20-million-words data randomly sampled from the raw text released by the CoNLL 2017 Shared Task - Automatically Annotated Raw Texts and Word Embeddings BIBREF8, which is a combination of Wikipedia dump and common crawl  The quality of these models is questionable  For example, we compared the Latvian model by ELMoForManyLangs with a model we trained on a complete (wikidump + common crawl) Latvian corpus, which has about 280 million tokens  The difference of each model on the word analogy task is shown in Figure FIGREF16 in Section SECREF5  As the results of the ELMoForManyLangs embeddings are significantly worse than using the full corpus, we can conclude that these embeddings are not of sufficient quality  For that reason, we computed ELMo embeddings for seven languages on much larger corpora  As this effort requires access to large amount of textual data and considerable computational resources, we made the precomputed models publicly available by depositing them to Clarin repository 
Training Data
We trained ELMo models for seven languages: Slovenian, Croatian, Finnish, Estonian, Latvian, Lithuanian and Swedish  To obtain high-quality embeddings, we used large monolingual corpora from various sources for each language  Some corpora are available online under permissive licences, others are available only for research purposes or have limited availability  The corpora used in training datasets are a mix of news articles and general web crawl, which we preprocessed and deduplicated  Below we shortly describe the used corpora in alphabetical order of the involved languages  Their names and sizes are summarized in Table TABREF3 
Croatian dataset include hrWaC 2 1 corpus BIBREF9, Riznica BIBREF10, and articles of Croatian branch of Styria media house, made available to us through partnership in a joint project  hrWaC was built by crawling the  hr internet domain in 2011 and 2014  Riznica is composed of Croatian fiction and non-fiction prose, poetry, drama, textbooks, manuals, etc  The Styria dataset consists of 570,219 news articles published on the Croatian 24sata news portal and niche portals related to 24sata 
Estonian dataset contains texts from two sources, CoNLL 2017 Shared Task - Automatically Annotated Raw Texts and Word Embeddings BIBREF8, and news articles made available to us by Ekspress Meedia due to partnership in the project  Ekspress Meedia dataset is composed of Estonian news articles between years 2009 and 2019  The CoNLL 2017 corpus is composed of Estonian Wikipedia and webcrawl 
Finnish dataset contains articles by Finnish news agency STT, Finnish part of the CoNLL 2017 dataset, and Ylilauta downloadable version BIBREF11  STT news articles were published between years 1992 and 2018  Ylilauta is a Finnish online discussion board; the corpus contains parts of the discussions from 2012 to 2014 
Latvian dataset consists only of the Latvian portion of the ConLL 2017 corpus 
Lithuanian dataset is composed of Lithuanian Wikipedia articles from 2018, DGT-UD corpus, and LtTenTen  DGT-UD is a parallel corpus of 23 official languages of the EU, composed of JRC DGT translation memory of European law, automatically annotated with UD-Pipe 1 2  LtTenTen is Lithuanian web corpus made up of texts collected from the internet in April 2014 BIBREF12 
Slovene dataset is formed from the Gigafida 2 0 corpus BIBREF13  It is a general language corpus composed of various sources, mostly newspapers, internet pages, and magazines, but also fiction and non-fiction prose, textbooks, etc 
Swedish dataset is composed of STT Swedish articles and Swedish part of CoNLL 2017  The Finnish news agency STT publishes some of its articles in Swedish language  They were made available to us through partnership in a joint project  The corpus contains those articles from 1992 to 2017 
Preprocessing and Training
Prior to training the ELMo models, we sentence and word tokenized all the datasets  The text was formatted in such a way that each sentence was in its own line with tokens separated by white spaces  CoNLL 2017, DGT-UD and LtTenTen14 corpora were already pre-tokenized  We tokenized the others using the NLTK library and its tokenizers for each of the languages

ground_truth: ancient Chinese history records in several dynasties (about 1000BC-200BC) and articles written by celebrities of that era

Ancient Chinese history records in several dynasties and articles written by celebrities during 1000BC-200BC collected from the internet 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How much data is needed to train the task-specific encoder?
Context 2: 
Better IE with Difficulty Prediction
We next present experiments in which we attempt to use the predicted difficulty during training to improve models for information extraction of descriptions of Population, Interventions and Outcomes from medical article abstracts  We investigate two uses: (1) simply removing the most difficult sentences from the training set, and, (2) re-weighting the most difficult sentences 
We again use LSTM-CRF-Pattern as the base model and experimenting on the EBM-NLP corpus BIBREF5   This is trained on either (1) the training set with difficult sentences removed, or (2) the full training set but with instances re-weighted in proportion to their predicted difficulty score  Following BIBREF5 , we use the Adam optimizer with learning rate of 0 001, decay 0 9, batch size 20 and dropout 0 5  We use pretrained 200d GloVe vectors BIBREF21 to initialize word embeddings, and use 100d hidden char representations  Each word is thus represented with 300 dimensions in total  The hidden size is 100 for the LSTM in the character representation component, and 200 for the LSTM in the information extraction component  We train for 15 epochs, saving parameters that achieve the best F1 score on a nested development set 
Removing Difficult Examples
We first evaluate changes in performance induced by training the sequence labeling model using less data by removing difficult sentences prior to training  The hypothesis here is that these difficult instances are likely to introduce more noise than signal  We used a cross-fold approach to predict sentence difficulties, training on 9/10ths of the data and scoring the remaining 1/10th at a time  We then sorted sentences by predicted difficulty scores, and experimented with removing increasing numbers of these (in order of difficulty) prior to training the LSTM-CRF-Pattern model 
Figure 3 shows the results achieved by the LSTM-CRF-Pattern model after discarding increasing amounts of the training data: the $x$ and $y$ axes correspond to the the percentage of data removed and F1 scores, respectively  We contrast removing sentences predicted to be difficult with removing them (a) randomly (i i d ), and, (b) in inverse order of predicted inter-annotator agreement  The agreement prediction model is trained exactly the same like difficult prediction model, with simply changing the difficult score to annotation agreement  F1 scores actually improve (marginally) when we remove the most difficult sentences, up until we drop 4% of the data for Population and Interventions, and 6% for Outcomes  Removing training points at i i d  random degrades performance, as expected  Removing sentences in order of disagreement seems to have similar effect as removing them by difficulty score when removing small amount of the data, but the F1 scores drop much faster when removing more data  These findings indicate that sentences predicted to be difficult are indeed noisy, to the extent that they do not seem to provide the model useful signal 
Re-weighting by Difficulty
We showed above that removing a small number of the most difficult sentences does not harm, and in fact modestly improves, medical IE model performance  However, using the available data we are unable to test if this will be useful in practice, as we would need additional data to determine how many difficult sentences should be dropped 
We instead explore an alternative, practical means of exploiting difficulty predictions: we re-weight sentences during training inversely to their predicted difficulty  Formally, we weight sentence $i$ with difficulty scores above $\tau $ according to: $1-a\cdot (d_i-\tau )/(1-\tau )$ , where $d_i$ is the difficulty score for sentence $i$ , and $a$ is a parameter codifying the minimum weight value  We set $\tau $ to 0 8 so as to only re-weight sentences with difficulty in the top 20th percentile, and we set $a$ to 0 5  The re-weighting is equivalent to down-sampling the difficult sentences  LSTM-CRF-Pattern is our base model 
Table 4 reports the precision, recall and F1 achieved both with and without sentence re-weighting  Re-weighting improves all metrics modestly but consistently  All F1 differences are statistically significant under a sign test ( $p<0 01$ )  The model with best precision is different for Patient, Intervention and Outcome labels  However re-weighting by difficulty does consistently yield the best recall for all three extraction types, with the most notable improvement for i and o, where recall improved by 10 percentage points  This performance increase translated to improvements in F1 across all types, as compared to the base model and to re-weighting by agreement 
Involving Expert Annotators
The preceding experiments demonstrate that re-weighting difficult sentences annotated by the crowd generally improves the extraction models  Presumably the performance is influenced by the annotation quality 
We now examine the possibility that the higher quality and more consistent annotations of domain experts on the difficult instances will benefit the extraction model  This simulates an annotation strategy in which we route difficult instances to domain experts and easier ones to crowd annotators  We also contrast the value of difficult data to that of an i i d  random sample of the same size, both annotated by experts 
Expert annotations of Random and Difficult Instances
We re-annotate by experts a subset of most difficult instances and the same number of random instances  As collecting annotations from experts is slow and expensive, we only re-annotate the difficult instances for the interventions extraction task  We re-annotate the abstracts which cover the sentences with predicted difficulty scores in the top 5 percentile  We rank the abstracts from the training set by the count of difficult sentences, and re-annotate the abstracts that contain the most difficult sentences  Constrained by time and budget, we select only 2000 abstracts for re-annotation; 1000 of these are top-ranked, and 1000 are randomly sampled  This re-annotation cost $3,000  We have released the new annotation data at: https://github com/bepnye/EBM-NLP 
Following BIBREF5 , we recruited five medical experts via Up-work with advanced medical training and strong technical reading/writing skills  The expert annotator were asked to read the entire abstract and highlight, using the BRAT toolkit BIBREF26 , all spans describing medical Interventions  Each abstract is only annotated by one expert  We examined 30 re-annotated abstracts to ensure the annotation quality before hiring the annotator 
Table 5 presents the results of LSTM-CRF-Pattern model trained on the reannotated difficult subset and the random subset  The first two rows show the results for models trained with expert annotations  The model trained on random data has a slightly better F1 than that trained on the same amount of difficult data  The model trained on random data has higher precision but lower recall 
Rows 3 and 4 list the results for models trained on the same data but with crowd annotation  Models trained with expert-annotated data are clearly superior to those trained with crowd labels with respect to F1, indicating that the experts produced higher quality annotations  For crowdsourced annotations, training the model with data sampled at i i d  random achieves 2% higher F1 than when difficult instances are used  When expert annotations are used, this difference is less than 1%  This trend in performance may be explained by differences in annotation quality: the randomly sampled set was more consistently annotated by both experts and crowd because the difficult set is harder  However, in both cases expert annotations are better, with a bigger difference between the expert and crowd models on the difficult set 
The last row is the model trained on all 5k abstracts with crowd annotations  Its F1 score is lower than either expert model trained on only 20% of data, suggesting that expert annotations should be collected whenever possible  Again the crowd model on complete data has higher precision than expert models but its recall is much lower 
Routing To Experts or Crowd
So far a system was trained on one type of data, either labeled by crowd or experts  We now examine the performance of a system trained on data that was routed to either experts or crowd annotators depending on their predicted difficult  Given the results presented so far mixing annotators may be beneficial given their respective trade-offs of precision and recall  We use the annotations from experts for an abstract if it exists otherwise use crowd annotations  The results are presented in Table 6  
Rows 1 and 2 repeat the performance of the models trained on difficult subset and random subset with expert annotations only respectively  The third row is the model trained by combining difficult and random subsets with expert annotations  There are around 250 abstracts in the overlap of these two sets, so there are total 1 75k abstracts used for training the D+R model  Rows 4 to 6 are the models trained on all 5k abstracts with mixed annotations, where Other means the rest of the abstracts with crowd annotation only 
The results show adding more training data with crowd annotation still improves at least 1 point F1 score in all three extraction tasks  The improvement when the difficult subset with expert annotations is mixed with the remaining crowd annotation is 3 5 F1 score, much larger than when a random set of expert annotations are added  The model trained with re-annotating the difficult subset (D+Other) also outperforms the model with re-annotating the random subset (R+Other) by 2 points in F1  The model trained with re-annotating both of difficult and random subsets (D+R+Other), however, achieves only marginally higher F1 than the model trained with the re-annotated difficult subset (D+Other)  In sum, the results clearly indicate that mixing expert and crowd annotations leads to better models than using solely crowd data, and better than using expert data alone  More importantly, there is greater gain in performance when instances are routed according to difficulty, as compared to randomly selecting the data for expert annotators  These findings align with our motivating hypothesis that annotation quality for difficult instances is important for final model performance  They also indicate that mixing annotations from expert and crowd could be an effective way to achieve acceptable model performance given a limited budget 
How Many Expert Annotations?
We established that crowd annotation are still useful in supplementing expert annotations for medical IE  Obtaining expert annotations for the one thousand most difficult instances greatly improved the model performance  However the choice of how many difficult instances to annotate was an uninformed choice  Here we check if less expert data would have yielded similar gains  Future work will need to address how best to choose this parameter for a routing system 
We simulate a routing scenario in which we send consecutive batches of the most difficult examples to the experts for annotation  We track changes in performance as we increase the number of most-difficult-articles sent to domain experts  As shown in Figure 4 , adding expert annotations for difficult articles consistently increases F1 scores  The performance gain is mostly from increased recall; the precision changes only a bit with higher quality annotation  This observation implies that crowd workers often fail to mark target tokens, but do not tend to produce large numbers of false positives  We suspect such failures to identify relevant spans/tokens are due to insufficient domain knowledge possessed by crowd workers 
The F1 score achieved after re-annotating the 600 most-difficult articles reaches 68 1%, which is close to the performance when re-annotating 1000 random articles  This demonstrates the effectiveness of recognizing difficult instances  The trend when we use up all expert data is still upward, so adding even more expert data is likely to further improve performance  Unfortunately we exhausted our budget and were not able to obtain additional expert annotations  It is likely that as the size of the expert annotations increases, the value of crowd annotations will diminish  This investigation is left for future work 
Conclusions
We have introduced the task of predicting annotation difficulty for biomedical information extraction (IE)  We trained neural models using different learned representations to score texts in terms of their difficulty  Results from all models were strong with Pearson’s correlation coefficients higher than 0 45 in almost all evaluations, indicating the feasibility of this task  An ensemble model combining universal and task specific feature sentence vectors yielded the best results 
Experiments on biomedical IE tasks show that removing up to $\sim $ 10% of the sentences predicted to be most difficult did not decrease model performance, and that re-weighting sentences inversely to their difficulty score during training improves predictive performance  Simulations in which difficult examples are routed to experts and other instances to crowd annotators yields the best results, outperforming the strategy of randomly selecting data for expert annotation, and substantially improving upon the approach of relying exclusively on crowd annotations  In future work, routing strategies based on instance difficulty could be further investigated for budget-quality trade-off 
Acknowledgements
This work has been partially supported by NSF1748771 grant  Wallace was support in part by NIH/NLM R01LM012086 

ground_truth: 57,505 sentences

57,505 sentences

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the improvement in performance for Estonian in the NER task?
Context 3:  There is no tokenizer for Croatian in NLTK library, so we used Slovene tokenizer instead  After tokenization, we deduplicated the datasets for each language separately, using the Onion (ONe Instance ONly) tool for text deduplication  We applied the tool on paragraph level for corpora that did not have sentences shuffled and on sentence level for the rest  We considered 9-grams with duplicate content threshold of 0 9 
For each language we prepared a vocabulary file, containing roughly one million most common tokens, i e  tokens that appear at least $n$ times in the corpus, where $n$ is between 15 and 25, depending on the dataset size  We included the punctuation marks among the tokens  We trained each ELMo model using default values used to train the original English ELMo (large) model 
Evaluation
We evaluated the produced ELMo models for all languages using two evaluation tasks: a word analogy task and named entity recognition (NER) task  Below, we first shortly describe each task, followed by the evaluation results 
Evaluation ::: Word Analogy Task
The word analogy task was popularized by mikolov2013distributed  The goal is to find a term $y$ for a given term $x$ so that the relationship between $x$ and $y$ best resembles the given relationship $a : b$  There are two main groups of categories: 5 semantic and 10 syntactic  To illustrate a semantic relationship, consider for example that the word pair $a : b$ is given as “Finland : Helsinki”  The task is to find the term $y$ corresponding to the relationship “Sweden : $y$”, with the expected answer being $y=$ Stockholm  In syntactic categories, the two words in a pair have a common stem (in some cases even same lemma), with all the pairs in a given category having the same morphological relationship  For example, given the word pair “long : longer”, we see that we have an adjective in its base form and the same adjective in a comparative form  That task is then to find the term $y$ corresponding to the relationship “dark : $y$”, with the expected answer being $y=$ darker, that is a comparative form of the adjective dark 
In the vector space, the analogy task is transformed into vector arithmetic and search for nearest neighbours, i e  we compute the distance between vectors: d(vec(Finland), vec(Helsinki)) and search for word $y$ which would give the closest result in distance d(vec(Sweden), vec($y$))  In the analogy dataset the analogies are already pre-specified, so we are measuring how close are the given pairs  In the evaluation below, we use analogy datasets for all tested languages based on the English dataset by BIBREF14   Due to English-centered bias of this dataset, we used a modified dataset which was first written in Slovene language and then translated into other languages BIBREF15 
As each instance of analogy contains only four words, without any context, the contextual models (such as ELMo) do not have enough context to generate sensible embeddings  We therefore used some additional text to form simple sentences using the four analogy words, while taking care that their noun case stays the same  For example, for the words "Rome", "Italy", "Paris" and "France" (forming the analogy Rome is to Italy as Paris is to $x$, where the correct answer is $x=$France), we formed the sentence "If the word Rome corresponds to the word Italy, then the word Paris corresponds to the word France"  We generated embeddings for those four words in the constructed sentence, substituted the last word with each word in our vocabulary and generated the embeddings again  As typical for non-contextual analogy task, we measure the cosine distance ($d$) between the last word ($w_4$) and the combination of the first three words ($w_2-w_1+w_3$)  We use the CSLS metric BIBREF16 to find the closest candidate word ($w_4$)  If we find the correct word among the five closest words, we consider that entry as successfully identified  The proportion of correctly identified words forms a statistic called accuracy@5, which we report as the result 
We first compare existing Latvian ELMo embeddings from ELMoForManyLangs project with our Latvian embeddings, followed by the detailed analysis of our ELMo embeddings  We trained Latvian ELMo using only CoNLL 2017 corpora  Since this is the only language, where we trained the embedding model on exactly the same corpora as ELMoForManyLangs models, we chose it for comparison between our ELMo model with ELMoForManyLangs  In other languages, additional or other corpora were used, so a direct comparison would also reflect the quality of the corpora used for training  In Latvian, however, only the size of the training dataset is different  ELMoForManyLangs uses only 20 million tokens and we use the whole corpus of 270 million tokens 
The Latvian ELMo model from ELMoForManyLangs project performs significantly worse than EMBEDDIA ELMo Latvian model on all categories of word analogy task (Figure FIGREF16)  We also include the comparison with our Estonian ELMo embeddings in the same figure  This comparison shows that while differences between our Latvian and Estonian embeddings can be significant for certain categories, the accuracy score of ELMoForManyLangs is always worse than either of our models  The comparison of Estonian and Latvian models leads us to believe that a few hundred million tokens is a sufficiently large corpus to train ELMo models (at least for word analogy task), but 20-million token corpora used in ELMoForManyLangs are too small 
The results for all languages and all ELMo layers, averaged over semantic and syntactic categories, are shown in Table TABREF17  The embeddings after the first LSTM layer perform best in semantic categories  In syntactic categories, the non-contextual CNN layer performs the best  Syntactic categories are less context dependent and much more morphology and syntax based, so it is not surprising that the non-contextual layer performs well  The second LSTM layer embeddings perform the worst in syntactic categories, though still outperforming CNN layer embeddings in semantic categories  Latvian ELMo performs worse compared to other languages we trained, especially in semantic categories, presumably due to smaller training data size  Surprisingly, the original English ELMo performs very poorly in syntactic categories and only outperforms Latvian in semantic categories  The low score can be partially explained by English model scoring $0 00$ in one syntactic category “opposite adjective”, which we have not been able to explain 
Evaluation ::: Named Entity Recognition
For evaluation of ELMo models on a relevant downstream task, we used named entity recognition (NER) task  NER is an information extraction task that seeks to locate and classify named entity mentions in unstructured text into pre-defined categories such as the person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc  To allow comparison of results between languages, we used an adapted version of this task, which uses a reduced set of labels, available in NER datasets for all processed languages  The labels in the used NER datasets are simplified to a common label set of three labels (person - PER, location - LOC, organization - ORG)  Each word in the NER dataset is labeled with one of the three mentioned labels or a label 'O' (other, i e  not a named entity) if it does not fit any of the other three labels  The number of words having each label is shown in Table TABREF19 
To measure the performance of ELMo embeddings on the NER task we proceeded as follows  We embedded the text in the datasets sentence by sentence, producing three vectors (one from each ELMo layer) for each token in a sentence  We calculated the average of the three vectors and used it as the input of our recognition model  The input layer was followed by a single LSTM layer with 128 LSTM cells and a dropout layer, randomly dropping 10% of the neurons on both the output and the recurrent branch  The final layer of our model was a time distributed softmax layer with 4 neurons 
We used ADAM optimiser BIBREF17 with the learning rate 0 01 and $10^{-5}$ learning rate decay  We used categorical cross-entropy as a loss function and trained the model for 3 epochs  We present the results using the Macro $F_1$ score, that is the average of $F_1$-scores for each of the three NE classes (the class Other is excluded) 
Since the differences between the tested languages depend more on the properties of the NER datasets than on the quality of embeddings, we can not directly compare ELMo models  For this reason, we take the non-contextual fastText embeddings as a baseline and predict named entities using them  The architecture of the model using fastText embeddings is the same as the one using ELMo embeddings, except that the input uses 300 dimensional fastText embedding vectors, and the model was trained for 5 epochs (instead of 3 as for ELMo)  In both cases (ELMo and fastText) we trained and evaluated the model five times, because there is some random component involved in initialization of the neural network model  By training and evaluating multiple times, we minimise this random component 
The results are presented in Table TABREF21  We included the evaluation of the original ELMo English model in the same table  NER models have little difficulty distinguishing between types of named entities, but recognizing whether a word is a named entity or not is more difficult  For languages with the smallest NER datasets, Croatian and Lithuanian, ELMo embeddings show the largest improvement over fastText embeddings  However, we can observe significant improvements with ELMo also on English and Finnish, which are among the largest datasets (English being by far the largest)  Only on Slovenian dataset did ELMo perform slightly worse than fastText, on all other EMBEDDIA languages, the ELMo embeddings improve the results 
Conclusion
We prepared precomputed ELMo contextual embeddings for seven languages: Croatian, Estonian, Finnish, Latvian, Lithuanian, Slovenian, and Swedish  We present the necessary background on embeddings and contextual embeddings, the details of training the embedding models, and their evaluation  We show that the size of used training sets importantly affects the quality of produced embeddings, and therefore the existing publicly available ELMo embeddings for the processed languages are inadequate  We trained new ELMo embeddings on larger training sets and analysed their properties on the analogy task and on the NER task  The results show that the newly produced contextual embeddings produce substantially better results compared to the non-contextual fastText baseline  In future work, we plan to use the produced contextual embeddings on the problems of news media industry  The pretrained ELMo models will be deposited to the CLARIN repository by the time of the final version of this paper 
Acknowledgments
The work was partially supported by the Slovenian Research Agency (ARRS) core research programme P6-0411  This paper is supported by European Union's Horizon 2020 research and innovation programme under grant agreement No 825153, project EMBEDDIA (Cross-Lingual Embeddings for Less-Represented Languages in European News Media)  The results of this publication reflects only the authors' view and the EU Commission is not responsible for any use that may be made of the information it contains 

ground_truth: 5 percent points.

0.05 F1