prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How large is the corpus?
Context 1: Introduction
The automatic processing of medical texts and documents plays an increasingly important role in the recent development of the digital health area  To enable dedicated Natural Language Processing (NLP) that is highly accurate with respect to medically relevant categories, manually annotated data from this domain is needed  One category of high interest and relevance are medical entities  Only very few annotated corpora in the medical domain exist  Many of them focus on the relation between chemicals and diseases or proteins and diseases, such as the BC5CDR corpus BIBREF0, the Comparative Toxicogenomics Database BIBREF1, the FSU PRotein GEne corpus BIBREF2 or the ADE (adverse drug effect) corpus BIBREF3  The NCBI Disease Corpus BIBREF4 contains condition mention annotations along with annotations of symptoms  Several new corpora of annotated case reports were made available recently  grouin-etal-2019-clinical presented a corpus with medical entity annotations of clinical cases written in French, copdPhenotype presented a corpus focusing on phenotypic information for chronic obstructive pulmonary disease while 10 1093/database/bay143 presented a corpus focusing on identifying main finding sentences in case reports 
The corpus most comparable to ours is the French corpus of clinical case reports by grouin-etal-2019-clinical  Their annotations are based on UMLS semantic types  Even though there is an overlap in annotated entities, semantic classes are not the same  Lab results are subsumed under findings in our corpus and are not annotated as their own class  Factors extend beyond gender and age and describe any kind of risk factor that contributes to a higher probability of having a certain disease  Our corpus includes additional entity types  We annotate conditions, findings (including medical findings such as blood values), factors, and also modifiers which indicate the negation of other entities as well as case entities, i  e , entities specific to one case report  An overview is available in Table TABREF3 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation tasks
Case reports are standardized in the CARE guidelines BIBREF5  They represent a detailed description of the symptoms, signs, diagnosis, treatment, and follow-up of an individual patient  We focus on documents freely available through PubMed Central (PMC)  The presentation of the patient's case can usually be found in a dedicated section or the abstract  We perform a manual annotation of all mentions of case entities, conditions, findings, factors and modifiers  The scope of our manual annotation is limited to the presentation of a patient's signs and symptoms  In addition, we annotate the title of the case report 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation Guidelines
We annotate the following entities:
case entity marks the mention of a patient  A case report can contain more than one case description  Therefore, all the findings, factors and conditions related to one patient are linked to the respective case entity  Within the text, this entity is often represented by the first mention of the patient and overlaps with the factor annotations which can, e  g , mark sex and age (cf  Figure FIGREF12) 
condition marks a medical disease such as pneumothorax or dislocation of the shoulder 
factor marks a feature of a patient which might influence the probability for a specific diagnosis  It can be immutable (e  g , sex and age), describe a specific medical history (e  g , diabetes mellitus) or a behaviour (e  g , smoking) 
finding marks a sign or symptom a patient shows  This can be visible (e  g , rash), described by a patient (e  g , headache) or measurable (e  g , decreased blood glucose level) 
negation modifier explicitly negate the presence of a certain finding usually setting the case apart from common cases 
We also annotate relations between these entities, where applicable  Since we work on case descriptions, the anchor point of these relations is the case that is described  The following relations are annotated:
has relations exist between a case entity and factor, finding or condition entities 
modifies relations exist between negation modifiers and findings 
causes relations exist between conditions and findings 
Example annotations are shown in Figure FIGREF16 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotators
We asked medical doctors experienced in extracting knowledge related to medical entities from texts to annotate the entities described above  Initially, we asked four annotators to test our guidelines on two texts  Subsequently, identified issues were discussed and resolved  Following this pilot annotation phase, we asked two different annotators to annotate two case reports according to our guidelines  The same annotators annotated an overall collection of 53 case reports 
Inter-annotator agreement is calculated based on two case reports  We reach a Cohen's kappa BIBREF6 of 0 68  Disagreements mainly appear for findings that are rather unspecific such as She no longer eats out with friends which can be seen as a finding referring to “avoidance behaviour” 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation Tools and Format
The annotation was performed using WebAnno BIBREF7, a web-based tool for linguistic annotation  The annotators could choose between a pre-annotated version or a blank version of each text  The pre-annotated versions contained suggested entity spans based on string matches from lists of conditions and findings synonym lists  Their quality varied widely throughout the corpus  The blank version was preferred by the annotators  We distribute the corpus in BioC JSON format  BioC was chosen as it allows us to capture the complexities of the annotations in the biomedical domain  It represented each documents properties ranging from full text, individual passages/sentences along with captured annotations and relationships in an organized manner  BioC is based on character offsets of annotations and allows the stacking of different layers 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Corpus Overview
The corpus consists of 53 documents, which contain an average number of 156 1 sentences per document, each with 19 55 tokens on average  The corpus comprises 8,275 sentences and 167,739 words in total  However, as mentioned above, only case presentation sections, headings and abstracts are annotated  The numbers of annotated entities are summarized in Table TABREF24 
Findings are the most frequently annotated type of entity  This makes sense given that findings paint a clinical picture of the patient's condition  The number of tokens per entity ranges from one token for all types to 5 tokens for cases (average length 3 1), nine tokens for conditions (average length 2 0), 16 tokens for factors (average length 2 5), 25 tokens for findings (average length 2 6) and 18 tokens for modifiers (average length 1 4) (cf  Table TABREF24)  Examples of rather long entities are given in Table TABREF25 
Entities can appear in a discontinuous way  We model this as a relation between two spans which we call “discontinuous” (cf  Figure FIGREF26)  Especially findings often appear as discontinuous entities, we found 543 discontinuous finding relations  The numbers for conditions and factors are lower with seven and two, respectively  Entities can also be nested within one another  This happens either when the span of one annotation is completely embedded in the span of another annotation (fully-nested; cf  Figure FIGREF12), or when there is a partial overlapping between the spans of two different entities (partially-nested; cf  Figure FIGREF12)  There is a high number of inter-sentential relations in the corpus (cf  Table TABREF27)  This can be explained by the fact that the case entity occurs early in each document; furthermore, it is related to finding and factor annotations that are distributed across different sentences 
The most frequently annotated relation in our corpus is the has-relation between a case entity and the findings related to that case  This correlates with the high number of finding entities  The relations contained in our corpus are summarized in Table TABREF27 
Baseline systems for Named Entity Recognition in medical case reports
We evaluate the corpus using Named Entity Recognition (NER), i  e , the task of finding mentions of concepts of interest in unstructured text  We focus on detecting cases, conditions, factors, findings and modifiers in case reports (cf  Section SECREF6)  We approach this as a sequence labeling problem  Four systems were developed to offer comparable robust baselines 
The original documents are pre-processed (sentence splitting and tokenization with ScispaCy)  We do not perform stop word removal or lower-casing of the tokens  The BIO labeling scheme is used to capture the order of tokens belonging to the same entity type and enable span-level detection of entities  Detection of nested and/or discontinuous entities is not supported  The annotated corpus is randomized and split in five folds using scikit-learn BIBREF9  Each fold has a train, test and dev split with the test split defined as  15% of the train split  This ensures comparability between the presented systems 
Baseline systems for Named Entity Recognition in medical case reports ::: Conditional Random Fields
Conditional Random Fields (CRF) BIBREF10 are a standard approach when dealing with sequential data in the context of sequence labeling  We use a combination of linguistic and semantic features, with a context window of size five, to describe each of the tokens and the dependencies between them  Hyper-parameter optimization is performed using randomized search and cross validation  Span-based F1 score is used as the optimization metric 
Baseline systems for Named Entity Recognition in medical case reports ::: BiLSTM-CRF
Prior to the emergence of deep neural language models, BiLSTM-CRF models BIBREF11 had achieved state-of-the-art results for the task of sequence labeling  We use a BiLSTM-CRF model with both word-level and character-level input  BioWordVec BIBREF12 pre-trained word embeddings are used in the embedding layer for the input representation  A bidirectional LSTM layer is applied to a multiplication of the two input representations  Finally, a CRF layer is applied to predict the sequence of labels  Dropout and L1/L2 regularization is used where applicable  He (uniform) initialization BIBREF13 is used to initialize the kernels of the individual layers  As the loss metric, CRF-based loss is used, while optimizing the model based on the CRF Viterbi accuracy  Additionally, span-based F1 score is used to serialize the best performing model  We train for a maximum of 100 epochs, or until an early stopping criterion is reached (no change in validation loss value grater than 0 01 for ten consecutive epochs)  Furthermore, Adam BIBREF14 is used as the optimizer  The learning rate is reduced by a factor of 0 3 in case no significant increase of the optimization metric is achieved in three consecutive epochs 
Baseline systems for Named Entity Recognition in medical case reports ::: Multi-Task Learning
Multi-Task Learning (MTL) BIBREF15 has become popular with the progress in deep learning  This model family is characterized by simultaneous optimization of multiple loss functions and transfer of knowledge achieved this way  The knowledge is transferred through the use of one or multiple shared layers  Through finding supporting patterns in related tasks, MTL provides better generalization on unseen cases and the main tasks we are trying to solve 
We rely on the model presented by bekoulis2018joint and reuse the implementation provided by the authors  The model jointly trains two objectives supported by the dataset: the main task of NER and a supporting task of Relation Extraction (RE)  Two separate models are developed for each of the tasks  The NER task is solved with the help of a BiLSTM-CRF model, similar to the one presented in Section SECREF32 The RE task is solved by using a multi-head selection approach, where each token can have none or more relationships to in-sentence tokens  Additionally, this model also leverages the output of the NER branch model (the CRF prediction) to learn label embeddings  Shared layers consist of a concatenation of word and character embeddings followed by two bidirectional LSTM layers  We keep most of the parameters suggested by the authors and change (1) the number of training epochs to 100 to allow the comparison to other deep learning approaches in this work, (2) use label embeddings of size 64, (3) allow gradient clipping and (4) use $d=0 8$ as the pre-trained word embedding dropout and $d=0 5$ for all other dropouts  $\eta =1^{-3}$ is used as the learning rate with the Adam optimizer and tanh activation functions across layers  Although it is possible to use adversarial training BIBREF16, we omit from using it  We also omit the publication of results for the task of RE as we consider it to be a supporting task and no other competing approaches have been developed 
Baseline systems for Named Entity Recognition in medical case reports ::: BioBERT
Deep neural language models have recently evolved to a successful method for representing text  In particular, Bidirectional Encoder Representations from Transformers (BERT) outperformed previous state-of-the-art methods by a large margin on various NLP tasks BIBREF17  For our experiments, we use BioBERT, an adaptation of BERT for the biomedical domain, pre-trained on PubMed abstracts and PMC full-text articles BIBREF18  The BERT architecture for deriving text representations uses 12 hidden layers, consisting of 768 units each  For NER, token level BIO-tag probabilities are computed with a single output layer based on the representations from the last layer of BERT  We fine-tune the model on the entity recognition task during four training epochs with batch size $b=32$, dropout probability $d=0 1$ and learning rate $\eta =2^{-5}$  These hyper-parameters are proposed by Devlin2018 for BERT fine-tuning 
Baseline systems for Named Entity Recognition in medical case reports ::: Evaluation
To evaluate the performance of the four systems, we calculate the span-level precision (P), recall (R) and F1 scores, along with corresponding micro and macro scores  The reported values are shown in Table TABREF29 and are averaged over five folds, utilising the seqeval framework 
With a macro avg  F1-score of 0 59, MTL achieves the best result with a significant margin compared to CRF, BiLSTM-CRF and BERT  This confirms the usefulness of jointly training multiple objectives (minimizing multiple loss functions), and enabling knowledge transfer, especially in a setting with limited data (which is usually the case in the biomedical NLP domain)  This result also suggest the usefulness of BioBERT for other biomedical datasets as reported by Lee2019  Despite being a rather standard approach, CRF outperforms the more elaborated BiLSTM-CRF, presumably due to data scarcity and class imbalance  We hypothesize that an increase in training data would yield better results for BiLSTM-CRF but not outperform transfer learning approach of MTL (or even BioBERT)  In contrast to other common NER corpora, like CoNLL 2003, even the best baseline system only achieves relatively low scores  This outcome is due to the inherent difficulty of the task (annotators are experienced medical doctors) and the small number of training samples 
Conclusion
We present a new corpus, developed to facilitate the processing of case reports  The corpus focuses on five distinct entity types: cases, conditions, factors, findings and modifiers  Where applicable, relationships between entities are also annotated  Additionally, we annotate discontinuous entities with a special relationship type (discontinuous)  The corpus presented in this paper is the very first of its kind and a valuable addition to the scarce number of corpora available in the field of biomedical NLP  Its complexity, given the discontinuous nature of entities and a high number of nested and multi-label entities, poses new challenges for NLP methods applied for NER and can, hence, be a valuable source for insights into what entities “look like in the wild”  Moreover, it can serve as a playground for new modelling techniques such as the resolution of discontinuous entities as well as multi-task learning given the combination of entities and their relations  We provide an evaluation of four distinct NER systems that will serve as robust baselines for future work but which are, as of yet, unable to solve all the complex challenges this dataset holds  A functional service based on the presented corpus is currently being integrated, as a NER service, in the QURATOR platform BIBREF20 
Acknowledgments
The research presented in this article is funded by the German Federal Ministry of Education and Research (BMBF) through the project QURATOR (Unternehmen Region, Wachstumskern, grant no  03WKDA1A), see http://qurator ai  We want to thank our medical experts for their help annotating the data set, especially Ashlee Finckh and Sophie Klopfenstein 
Context 2: Introduction
Assembling training corpora of annotated natural language examples in specialized domains such as biomedicine poses considerable challenges  Experts with the requisite domain knowledge to perform high-quality annotation tend to be expensive, while lay annotators may not have the necessary knowledge to provide high-quality annotations  A practical approach for collecting a sufficiently large corpus would be to use crowdsourcing platforms like Amazon Mechanical Turk (MTurk)  However, crowd workers in general are likely to provide noisy annotations BIBREF0 , BIBREF1 , BIBREF2 , an issue exacerbated by the technical nature of specialized content  Some of this noise may reflect worker quality and can be modeled BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4 , but for some instances lay people may simply lack the domain knowledge to provide useful annotation 
In this paper we report experiments on the EBM-NLP corpus comprising crowdsourced annotations of medical literature BIBREF5   We operationalize the concept of annotation difficulty and show how it can be exploited during training to improve information extraction models  We then obtain expert annotations for the abstracts predicted to be most difficult, as well as for a similar number of randomly selected abstracts  The annotation of highly specialized data and the use of lay and expert annotators allow us to examine the following key questions related to lay and expert annotations in specialized domains:
Can we predict item difficulty? We define a training instance as difficult if a lay annotator or an automated model disagree on its labeling  We show that difficulty can be predicted, and that it is distinct from inter-annotator agreement  Further, such predictions can be used during training to improve information extraction models 
Are there systematic differences between expert and lay annotations? We observe decidedly lower agreement between lay workers as compared to domain experts  Lay annotations have high precision but low recall with respect to expert annotations in the new data that we collected  More generally, we expect lay annotations to be lower quality, which may translate to lower precision, recall, or both, compared to expert annotations  Can one rely solely on lay annotations? Reasonable models can be trained using lay annotations alone, but similar performance can be achieved using markedly less expert data  This suggests that the optimal ratio of expert to crowd annotations for specialized tasks will depend on the cost and availability of domain experts  Expert annotations are preferable whenever its collection is practical  But in real-world settings, a combination of expert and lay annotations is better than using lay data alone 
Does it matter what data is annotated by experts? We demonstrate that a system trained on combined data achieves better predictive performance when experts annotate difficult examples rather than instances selected at i i d  random 
Our contributions in this work are summarized as follows  We define a task difficulty prediction task and show how this is related to, but distinct from, inter-worker agreement  We introduce a new model for difficulty prediction combining learned representations induced via a pre-trained `universal' sentence encoder BIBREF6 , and a sentence encoder learned from scratch for this task  We show that predicting annotation difficulty can be used to improve the task routing and model performance for a biomedical information extraction task  Our results open up a new direction for ensuring corpus quality  We believe that item difficulty prediction will likely be useful in other, non-specialized tasks as well, and that the most effective data collection in specialized domains requires research addressing the fundamental questions we examine here 
Related Work
Crowdsourcing annotation is now a well-studied problem BIBREF7 , BIBREF0 , BIBREF1 , BIBREF2   Due to the noise inherent in such annotations, there have also been considerable efforts to develop aggregation models that minimize noise BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4  
There are also several surveys of crowdsourcing in biomedicine specifically BIBREF8 , BIBREF9 , BIBREF10   Some work in this space has contrasted model performance achieved using expert vs  crowd annotated training data BIBREF11 , BIBREF12 , BIBREF13   Dumitrache et al  Dumitrache:2018:CGT:3232718 3152889 concluded that performance is similar under these supervision types, finding no clear advantage from using expert annotators  This differs from our findings, perhaps owing to differences in design  The experts we used already hold advanced medical degrees, for instance, while those in prior work were medical students  Furthermore, the task considered here would appear to be of greater difficulty: even a system trained on $\sim $ 5k instances performs reasonably, but far from perfect  By contrast, in some of the prior work where experts and crowd annotations were deemed equivalent, a classifier trained on 300 examples can achieve very high accuracy BIBREF12  
More relevant to this paper, prior work has investigated methods for `task routing' in active learning scenarios in which supervision is provided by heterogeneous labelers with varying levels of expertise BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF14   The related question of whether effort is better spent collecting additional annotations for already labeled (but potentially noisily so) examples or novel instances has also been addressed BIBREF18   What distinguishes the work here is our focus on providing an operational definition of instance difficulty, showing that this can be predicted, and then using this to inform task routing 
Application Domain
Our specific application concerns annotating abstracts of articles that describe the conduct and results of randomized controlled trials (RCTs)  Experimentation in this domain has become easy with the recent release of the EBM-NLP BIBREF5 corpus, which includes a reasonably large training dataset annotated via crowdsourcing, and a modest test set labeled by individuals with advanced medical training  More specifically, the training set comprises 4,741 medical article abstracts with crowdsourced annotations indicating snippets (sequences) that describe the Participants (p), Interventions (i), and Outcome (o) elements of the respective RCT, and the test set is composed of 191 abstracts with p, i, o sequence annotations from three medical experts 
Table 1 shows an example of difficult and easy examples according to our definition of difficulty  The underlined text demarcates the (consensus) reference label provided by domain experts  In the difficult examples, crowd workers marked text distinct from these reference annotations; whereas in the easy cases they reproduced them with reasonable fidelity  The difficult sentences usually exhibit complicated structure and feature jargon 
An abstract may contain some `easy' and some `difficult' sentences  We thus perform our analysis at the sentence level  We split abstracts into sentences using spaCy  We excluded sentences that comprise fewer than two tokens, as these are likely an artifact of errors in sentence splitting  In total, this resulted in 57,505 and 2,428 sentences in the train and test set abstracts, respectively 
Quantifying Task Difficulty
The test set includes annotations from both crowd workers and domain experts  We treat the latter as ground truth and then define the difficulty of sentences in terms of the observed agreement between expert and lay annotators  Formally, for annotation task $t$ and instance $i$ :
$$\text{Difficulty}_{ti} = \frac{\sum _{j=1}^n{f(\text{label}_{ij}, y_i})}{n}$$   (Eq  3)
where $f$ is a scoring function that measures the quality of the label from worker $j$ for sentence $i$ , as compared to a ground truth annotation, $y_i$   The difficulty score of sentence $i$ is taken as an average over the scores for all $n$ layworkers  We use Spearmans' correlation coefficient as a scoring function  Specifically, for each sentence we create two vectors comprising counts of how many times each token was annotated by crowd and expert workers, respectively, and calculate the correlation between these  Sentences with no labels are treated as maximally easy; those with only either crowd worker or expert label(s) are assumed maximally difficult 
The training set contains only crowdsourced annotations  To label the training data, we use a 10-fold validation like setting  We iteratively retrain the LSTM-CRF-Pattern sequence tagger of Patel et al  patel2018syntactic on 9 folds of the training data and use that trained model to predict labels for the 10th  In this way we obtain predictions on the full training set  We then use predicted spans as proxy `ground truth' annotations to calculate the difficulty score of sentences as described above; we normalize these to the [ $0, 1$ ] interval  We validate this approximation by comparing the proxy scores against reference scores over the test set, the Pearson's correlation coefficients are 0 57 for Population, 0 71 for Intervention and 0 68 for Outcome 
There exist many sentences that contain neither manual nor predicted annotations  We treat these as maximally easy sentences (with difficulty scores of 0)  Such sentences comprise 51%, 42% and 36% for Population, Interventions and Outcomes data respectively, indicating that it is easier to identify sentences that have no Population spans, but harder to identify sentences that have no Interventions or Outcomes spans  This is intuitive as descriptions of the latter two tend to be more technical and dense with medical jargon 
We show the distribution of the automatically labeled scores for sentences that do contain spans in Figure 1   The mean of the Population (p) sentence scores is significantly lower than that for other types of sentences (i and o), again indicating that they are easier on average to annotate  This aligns with a previous finding that annotating Interventions and Outcomes is more difficult than annotating Participants BIBREF5  
Many sentences contain spans tagged by the LSTM-CRF-Pattern model, but missed by all crowd workers, resulting in a maximally difficult score (1)  Inspection of such sentences revealed that some are truly difficult examples, but others are tagging model errors  In either case, such sentences have confused workers and/or the model, and so we retain them all as `difficult' sentences 
Content describing the p, i and o, respectively, is quite different  As such, one sentence usually contains (at most) only one of these three content types  We thus treat difficulty prediction for the respective label types as separate tasks 
Difficulty is not Worker Agreement
Our definition of difficulty is derived from agreement between expert and crowd annotations for the test data, and agreement between a predictive model and crowd annotations in the training data  It is reasonable to ask if these measures are related to inter-annotator agreement, a metric often used in language technology research to identify ambiguous or difficult items  Here we explicitly verify that our definition of difficulty only weakly correlates with inter-annotator agreement 
We calculate inter-worker agreement between crowd and expert annotators using Spearman's correlation coefficient  As shown in Table 2 , average agreement between domain experts are considerably higher than agreements between crowd workers for all three label types  This is a clear indication that the crowd annotations are noisier 
Furthermore, we compare the correlation between inter-annotator agreement and difficulty scores in the training data  Given that the majority of sentences do not contain a PICO span, we only include in these calculations those that contain a reference label  Pearson's r are 0 34, 0 30 and 0 31 for p, i and o, respectively, confirming that inter-worker agreement and our proposed difficulty score are quite distinct 
Predicting Annotation Difficulty
We treat difficulty prediction as a regression problem, and propose and evaluate neural model variants for the task  We first train RNN BIBREF19 and CNN BIBREF20 models 
We also use the universal sentence encoder (USE) BIBREF6 to induce sentence representations, and train a model using these as features  Following BIBREF6 , we then experiment with an ensemble model that combines the `universal' and task-specific representations to predict annotation difficulty  We expect these universal embeddings to capture general, high-level semantics, and the task specific representations to capture more granular information  Figure 2 depicts the model architecture  Sentences are fed into both the universal sentence encoder and, separately, a task specific neural encoder, yielding two representations  We concatenate these and pass the combined vector to the regression layer 
Experimental Setup and Results
We trained models for each label type separately  Word embeddings were initialized to 300d GloVe vectors BIBREF21 trained on common crawl data; these are fine-tuned during training  We used the Adam optimizer BIBREF22 with learning rate and decay set to 0 001 and 0 99, respectively  We used batch sizes of 16 
We used the large version of the universal sentence encoder with a transformer BIBREF23   We did not update the pretrained sentence encoder parameters during training  All hyperparamaters for all models (including hidden layers, hidden sizes, and dropout) were tuned using Vizier BIBREF24 via 10-fold cross validation on the training set maximizing for F1 
As a baseline, we also trained a linear Support-Vector Regression BIBREF25 model on $n$ -gram features ( $n$ ranges from 1 to 3) 
Table 3 reports Pearson correlation coefficients between the predictions with each of the neural models and the ground truth difficulty scores  Rows 1-4 correspond to individual models, and row 5 reports the ensemble performance  Columns correspond to label type  Results from all models outperform the baseline SVR model: Pearson's correlation coefficients range from 0 550 to 0 622  The regression correlations are the lowest 
The RNN model realizes the strongest performance among the stand-alone (non-ensemble) models, outperforming variants that exploit CNN and USE representations  Combining the RNN and USE further improves results  We hypothesize that this is due to complementary sentence information encoded in universal representations 
For all models, correlations for Intervention and Outcomes are higher than for Population, which is expected given the difficulty distributions in Figure 1   In these, the sentences are more uniformly distributed, with a fair number of difficult and easier sentences  By contrast, in Population there are a greater number of easy sentences and considerably fewer difficult sentences, which makes the difficulty ranking task particularly challenging
Context 3: Introduction
Neural networks have been successfully used to describe images with text using sequence-to-sequence models BIBREF0  However, the results are simple and dry captions which are one or two phrases long  Humans looking at a painting see more than just objects  Paintings stimulate sentiments, metaphors and stories as well  Therefore, our goal is to have a neural network describe the painting artistically in a style of choice  As a proof of concept, we present a model which generates Shakespearean prose for a given painting as shown in Figure FIGREF1  Accomplishing this task is difficult with traditional sequence to sequence models since there does not exist a large collection of Shakespearean prose which describes paintings: Shakespeare's works describes a single painting shown in Figure FIGREF3  Fortunately we have a dataset of modern English poems which describe images BIBREF1 and line-by-line modern paraphrases of Shakespeare's plays BIBREF2  Our solution is therefore to combine two separately trained models to synthesize Shakespearean prose for a given painting 
Introduction ::: Related work
A general end-to-end approach to sequence learning BIBREF3 makes minimal assumptions on the sequence structure  This model is widely used in tasks such as machine translation, text summarization, conversational modeling, and image captioning  A generative model using a deep recurrent architecture BIBREF0 has also beeen used for generating phrases describing an image  The task of synthesizing multiple lines of poetry for a given image BIBREF1 is accomplished by extracting poetic clues from images  Given the context image, the network associates image attributes with poetic descriptions using a convolutional neural net  The poem is generated using a recurrent neural net which is trained using multi-adversarial training via policy gradient 
Transforming text from modern English to Shakespearean English using text "style transfer" is challenging  An end to end approach using a sequence-to-sequence model over a parallel text corpus BIBREF2 has been proposed based on machine translation  In the absence of a parallel text corpus, generative adversarial networks (GANs) have been used, which simultaneously train two models: a generative model which captures the data distribution, and a discriminative model which evaluates the performance of the generator  Using a target domain language model as a discriminator has also been employed BIBREF4, providing richer and more stable token-level feedback during the learning process  A key challenge in both image and text style transfer is separating content from style BIBREF5, BIBREF6, BIBREF7  Cross-aligned auto-encoder models have focused on style transfer using non-parallel text BIBREF7  Recently, a fine grained model for text style transfer has been proposed BIBREF8 which controls several factors of variation in textual data by using back-translation  This allows control over multiple attributes, such as gender and sentiment, and fine-grained control over the trade-off between content and style 
Methods
We use a total three datasets: two datasets for generating an English poem from an image, and Shakespeare plays and their English translations for text style transfer 
We train a model for generating poems from images based on two datasets BIBREF1  The first dataset consists of image and poem pairs, namely a multi-modal poem dataset (MultiM-Poem), and the second dataset is a large poem corpus, namely a uni-modal poem dataset (UniM-Poem)  The image and poem pairs are extended by adding the nearest three neighbor poems from the poem corpus without redundancy, and an extended image and poem pair dataset is constructed and denoted as MultiM-Poem(Ex)BIBREF1

ground_truth: 8,275 sentences and 167,739 words in total

The corpus comprises 8,275 sentences and 167,739 words in total.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How much higher quality is the resulting annotated data?
Context 2: 
Better IE with Difficulty Prediction
We next present experiments in which we attempt to use the predicted difficulty during training to improve models for information extraction of descriptions of Population, Interventions and Outcomes from medical article abstracts  We investigate two uses: (1) simply removing the most difficult sentences from the training set, and, (2) re-weighting the most difficult sentences 
We again use LSTM-CRF-Pattern as the base model and experimenting on the EBM-NLP corpus BIBREF5   This is trained on either (1) the training set with difficult sentences removed, or (2) the full training set but with instances re-weighted in proportion to their predicted difficulty score  Following BIBREF5 , we use the Adam optimizer with learning rate of 0 001, decay 0 9, batch size 20 and dropout 0 5  We use pretrained 200d GloVe vectors BIBREF21 to initialize word embeddings, and use 100d hidden char representations  Each word is thus represented with 300 dimensions in total  The hidden size is 100 for the LSTM in the character representation component, and 200 for the LSTM in the information extraction component  We train for 15 epochs, saving parameters that achieve the best F1 score on a nested development set 
Removing Difficult Examples
We first evaluate changes in performance induced by training the sequence labeling model using less data by removing difficult sentences prior to training  The hypothesis here is that these difficult instances are likely to introduce more noise than signal  We used a cross-fold approach to predict sentence difficulties, training on 9/10ths of the data and scoring the remaining 1/10th at a time  We then sorted sentences by predicted difficulty scores, and experimented with removing increasing numbers of these (in order of difficulty) prior to training the LSTM-CRF-Pattern model 
Figure 3 shows the results achieved by the LSTM-CRF-Pattern model after discarding increasing amounts of the training data: the $x$ and $y$ axes correspond to the the percentage of data removed and F1 scores, respectively  We contrast removing sentences predicted to be difficult with removing them (a) randomly (i i d ), and, (b) in inverse order of predicted inter-annotator agreement  The agreement prediction model is trained exactly the same like difficult prediction model, with simply changing the difficult score to annotation agreement  F1 scores actually improve (marginally) when we remove the most difficult sentences, up until we drop 4% of the data for Population and Interventions, and 6% for Outcomes  Removing training points at i i d  random degrades performance, as expected  Removing sentences in order of disagreement seems to have similar effect as removing them by difficulty score when removing small amount of the data, but the F1 scores drop much faster when removing more data  These findings indicate that sentences predicted to be difficult are indeed noisy, to the extent that they do not seem to provide the model useful signal 
Re-weighting by Difficulty
We showed above that removing a small number of the most difficult sentences does not harm, and in fact modestly improves, medical IE model performance  However, using the available data we are unable to test if this will be useful in practice, as we would need additional data to determine how many difficult sentences should be dropped 
We instead explore an alternative, practical means of exploiting difficulty predictions: we re-weight sentences during training inversely to their predicted difficulty  Formally, we weight sentence $i$ with difficulty scores above $\tau $ according to: $1-a\cdot (d_i-\tau )/(1-\tau )$ , where $d_i$ is the difficulty score for sentence $i$ , and $a$ is a parameter codifying the minimum weight value  We set $\tau $ to 0 8 so as to only re-weight sentences with difficulty in the top 20th percentile, and we set $a$ to 0 5  The re-weighting is equivalent to down-sampling the difficult sentences  LSTM-CRF-Pattern is our base model 
Table 4 reports the precision, recall and F1 achieved both with and without sentence re-weighting  Re-weighting improves all metrics modestly but consistently  All F1 differences are statistically significant under a sign test ( $p<0 01$ )  The model with best precision is different for Patient, Intervention and Outcome labels  However re-weighting by difficulty does consistently yield the best recall for all three extraction types, with the most notable improvement for i and o, where recall improved by 10 percentage points  This performance increase translated to improvements in F1 across all types, as compared to the base model and to re-weighting by agreement 
Involving Expert Annotators
The preceding experiments demonstrate that re-weighting difficult sentences annotated by the crowd generally improves the extraction models  Presumably the performance is influenced by the annotation quality 
We now examine the possibility that the higher quality and more consistent annotations of domain experts on the difficult instances will benefit the extraction model  This simulates an annotation strategy in which we route difficult instances to domain experts and easier ones to crowd annotators  We also contrast the value of difficult data to that of an i i d  random sample of the same size, both annotated by experts 
Expert annotations of Random and Difficult Instances
We re-annotate by experts a subset of most difficult instances and the same number of random instances  As collecting annotations from experts is slow and expensive, we only re-annotate the difficult instances for the interventions extraction task  We re-annotate the abstracts which cover the sentences with predicted difficulty scores in the top 5 percentile  We rank the abstracts from the training set by the count of difficult sentences, and re-annotate the abstracts that contain the most difficult sentences  Constrained by time and budget, we select only 2000 abstracts for re-annotation; 1000 of these are top-ranked, and 1000 are randomly sampled  This re-annotation cost $3,000  We have released the new annotation data at: https://github com/bepnye/EBM-NLP 
Following BIBREF5 , we recruited five medical experts via Up-work with advanced medical training and strong technical reading/writing skills  The expert annotator were asked to read the entire abstract and highlight, using the BRAT toolkit BIBREF26 , all spans describing medical Interventions  Each abstract is only annotated by one expert  We examined 30 re-annotated abstracts to ensure the annotation quality before hiring the annotator 
Table 5 presents the results of LSTM-CRF-Pattern model trained on the reannotated difficult subset and the random subset  The first two rows show the results for models trained with expert annotations  The model trained on random data has a slightly better F1 than that trained on the same amount of difficult data  The model trained on random data has higher precision but lower recall 
Rows 3 and 4 list the results for models trained on the same data but with crowd annotation  Models trained with expert-annotated data are clearly superior to those trained with crowd labels with respect to F1, indicating that the experts produced higher quality annotations  For crowdsourced annotations, training the model with data sampled at i i d  random achieves 2% higher F1 than when difficult instances are used  When expert annotations are used, this difference is less than 1%  This trend in performance may be explained by differences in annotation quality: the randomly sampled set was more consistently annotated by both experts and crowd because the difficult set is harder  However, in both cases expert annotations are better, with a bigger difference between the expert and crowd models on the difficult set 
The last row is the model trained on all 5k abstracts with crowd annotations  Its F1 score is lower than either expert model trained on only 20% of data, suggesting that expert annotations should be collected whenever possible  Again the crowd model on complete data has higher precision than expert models but its recall is much lower 
Routing To Experts or Crowd
So far a system was trained on one type of data, either labeled by crowd or experts  We now examine the performance of a system trained on data that was routed to either experts or crowd annotators depending on their predicted difficult  Given the results presented so far mixing annotators may be beneficial given their respective trade-offs of precision and recall  We use the annotations from experts for an abstract if it exists otherwise use crowd annotations  The results are presented in Table 6  
Rows 1 and 2 repeat the performance of the models trained on difficult subset and random subset with expert annotations only respectively  The third row is the model trained by combining difficult and random subsets with expert annotations  There are around 250 abstracts in the overlap of these two sets, so there are total 1 75k abstracts used for training the D+R model  Rows 4 to 6 are the models trained on all 5k abstracts with mixed annotations, where Other means the rest of the abstracts with crowd annotation only 
The results show adding more training data with crowd annotation still improves at least 1 point F1 score in all three extraction tasks  The improvement when the difficult subset with expert annotations is mixed with the remaining crowd annotation is 3 5 F1 score, much larger than when a random set of expert annotations are added  The model trained with re-annotating the difficult subset (D+Other) also outperforms the model with re-annotating the random subset (R+Other) by 2 points in F1  The model trained with re-annotating both of difficult and random subsets (D+R+Other), however, achieves only marginally higher F1 than the model trained with the re-annotated difficult subset (D+Other)  In sum, the results clearly indicate that mixing expert and crowd annotations leads to better models than using solely crowd data, and better than using expert data alone  More importantly, there is greater gain in performance when instances are routed according to difficulty, as compared to randomly selecting the data for expert annotators  These findings align with our motivating hypothesis that annotation quality for difficult instances is important for final model performance  They also indicate that mixing annotations from expert and crowd could be an effective way to achieve acceptable model performance given a limited budget 
How Many Expert Annotations?
We established that crowd annotation are still useful in supplementing expert annotations for medical IE  Obtaining expert annotations for the one thousand most difficult instances greatly improved the model performance  However the choice of how many difficult instances to annotate was an uninformed choice  Here we check if less expert data would have yielded similar gains  Future work will need to address how best to choose this parameter for a routing system 
We simulate a routing scenario in which we send consecutive batches of the most difficult examples to the experts for annotation  We track changes in performance as we increase the number of most-difficult-articles sent to domain experts  As shown in Figure 4 , adding expert annotations for difficult articles consistently increases F1 scores  The performance gain is mostly from increased recall; the precision changes only a bit with higher quality annotation  This observation implies that crowd workers often fail to mark target tokens, but do not tend to produce large numbers of false positives  We suspect such failures to identify relevant spans/tokens are due to insufficient domain knowledge possessed by crowd workers 
The F1 score achieved after re-annotating the 600 most-difficult articles reaches 68 1%, which is close to the performance when re-annotating 1000 random articles  This demonstrates the effectiveness of recognizing difficult instances  The trend when we use up all expert data is still upward, so adding even more expert data is likely to further improve performance  Unfortunately we exhausted our budget and were not able to obtain additional expert annotations  It is likely that as the size of the expert annotations increases, the value of crowd annotations will diminish  This investigation is left for future work 
Conclusions
We have introduced the task of predicting annotation difficulty for biomedical information extraction (IE)  We trained neural models using different learned representations to score texts in terms of their difficulty  Results from all models were strong with Pearson’s correlation coefficients higher than 0 45 in almost all evaluations, indicating the feasibility of this task  An ensemble model combining universal and task specific feature sentence vectors yielded the best results 
Experiments on biomedical IE tasks show that removing up to $\sim $ 10% of the sentences predicted to be most difficult did not decrease model performance, and that re-weighting sentences inversely to their difficulty score during training improves predictive performance  Simulations in which difficult examples are routed to experts and other instances to crowd annotators yields the best results, outperforming the strategy of randomly selecting data for expert annotation, and substantially improving upon the approach of relying exclusively on crowd annotations  In future work, routing strategies based on instance difficulty could be further investigated for budget-quality trade-off 
Acknowledgements
This work has been partially supported by NSF1748771 grant  Wallace was support in part by NIH/NLM R01LM012086 
Context 3: 
We use a collection of line-by-line modern paraphrases for 16 of Shakespeare’s plays BIBREF2, for training a style transfer network from English poems to Shakespearean prose  We use 18,395 sentences from the training data split  We keep 1,218 sentences in the validation data set and 1,462 sentences in our test set 
Methods ::: Image To Poem Actor-Critic Model
For generating a poem from images we use an existing actor-critic architecture BIBREF1  This involves 3 parallel CNNs: an object CNN, sentiment CNN, and scene CNN, for feature extraction  These features are combined with a skip-thought model which provides poetic clues, which are then fed into a sequence-to-sequence model trained by policy gradient with 2 discriminator networks for rewards  This as a whole forms a pipeline that takes in an image and outputs a poem as shown on the top left of Figure FIGREF4  A CNN-RNN generative model acts as an agent  The parameters of this agent define a policy whose execution determines which word is selected as an action  When the agent selects all words in a poem, it receives a reward  Two discriminative networks, shown on the top right of Figure FIGREF4, are defined to serve as rewards concerning whether the generated poem properly describes the input image and whether the generated poem is poetic  The goal of the poem generation model is to generate a sequence of words as a poem for an image to maximize the expected return 
Methods ::: Shakespearizing Poetic Captions
For Shakespearizing modern English texts, we experimented with various types of sequence to sequence models  Since the size of the parallel translation data available is small, we leverage a dictionary providing a mapping between Shakespearean words and modern English words to retrofit pre-trained word embeddings  Incorporating this extra information improves the translation task  The large number of shared word types between the source and target sentences indicates that sharing the representation between them is beneficial 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with Attention
We use a sequence-to-sequence model which consists of a single layer unidrectional LSTM encoder and a single layer LSTM decoder and pre-trained retrofitted word embeddings shared between source and target sentences  We experimented with two different types of attention: global attention BIBREF9, in which the model makes use of the output from the encoder and decoder for the current time step only, and Bahdanau attention BIBREF10, where computing attention requires the output of the decoder from the prior time step  We found that global attention performs better in practice for our task of text style transfer 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with a Pointer Network
Since a pair of corresponding Shakespeare and modern English sentences have significant vocabulary overlap we extend the sequence-to-sequence model mentioned above using pointer networks BIBREF11 that provide location based attention and have been used to enable copying of tokens directly from the input  Moreover, there are lot of proper nouns and rare words which might not be predicted by a vanilla sequence to sequence model 
Methods ::: Shakespearizing Poetic Captions ::: Prediction
For both seq2seq models, we use the attention matrices returned at each decoder time step during inference, to compute the next word in the translated sequence if the decoder output at the current time step is the UNK token  We replace the UNKs in the target output with the highest aligned, maximum attention, source word  The seq2seq model with global attention gives the best results with an average target BLEU score of 29 65 on the style transfer dataset, compared with an average target BLEU score of 26 97 using the seq2seq model with pointer networks 
Results
We perform a qualitative analysis of the Shakespearean prose generated for the input paintings

ground_truth: improvement when the difficult subset with expert annotations is mixed with the remaining crowd annotation is 3.5 F1 score, much larger than when a random set of expert annotations are added

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What limitations do the authors demnostrate of their model?
Context 3:  We conducted a survey, in which we presented famous paintings including those shown in Figures FIGREF1 and FIGREF10 and the corresponding Shakespearean prose generated by the model, and asked 32 students to rate them on the basis of content, creativity and similarity to Shakespearean style on a Likert scale of 1-5  Figure FIGREF12 shows the result of our human evaluation 
The average content score across the paintings is 3 7 which demonstrates that the prose generated is relevant to the painting  The average creativity score is 3 9 which demonstrates that the model captures more than basic objects in the painting successfully using poetic clues in the scene  The average style score is 3 9 which demonstrates that the prose generated is perceived to be in the style of Shakespeare 
We also perform a quantitative analysis of style transfer by generating BLEU scores for the model output using the style transfer dataset  The variation of the BLEU scores with the source sentence lengths is shown in Figure FIGREF11  As expected, the BLEU scores decrease with increase in source sentence lengths 
Results ::: Implementation
All models were trained on Google Colab with a single GPU using Python 3 6 and Tensorflow 2 0  The number of hidden units for the encoder and decoder is 1,576 and 256 for seq2seq with global attention and seq2seq with pointer networks respectively  Adam optimizer was used with the default learning rate of 0 001  The model was trained for 25 epochs  We use pre-trained retrofitted word embeddings of dimension 192 
Results ::: Limitations
Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score  This happens when the style transfer dataset does not have similar words in the training set of sentences  A solution would be to expand the style transfer dataset, for a better representation of the poem data 
Results ::: Conclusions and Future Work
In conclusion, combining two pipelines with an intermediate representation works well in practice  We observe that a CNN-RNN based image-to-poem net combined with a seq2seq model with parallel text corpus for text style transfer synthesizes Shakespeare-style prose for a given painting  For the seq2seq model used, we observe that it performs better in practice using global attention as compared with local attention  We make our models and code publicly available BIBREF12  In future work we would like to experiment with GANs in the absence of non-parallel datasets, so that we can use varied styles for text style transfer  We would also like to experiment with cross aligned auto-encoders, which form a latent content representation, to efficiently separate style and content 

ground_truth: Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer

we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score