prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How much higher quality is the resulting annotated data?
Context 1: Introduction
Assembling training corpora of annotated natural language examples in specialized domains such as biomedicine poses considerable challenges  Experts with the requisite domain knowledge to perform high-quality annotation tend to be expensive, while lay annotators may not have the necessary knowledge to provide high-quality annotations  A practical approach for collecting a sufficiently large corpus would be to use crowdsourcing platforms like Amazon Mechanical Turk (MTurk)  However, crowd workers in general are likely to provide noisy annotations BIBREF0 , BIBREF1 , BIBREF2 , an issue exacerbated by the technical nature of specialized content  Some of this noise may reflect worker quality and can be modeled BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4 , but for some instances lay people may simply lack the domain knowledge to provide useful annotation 
In this paper we report experiments on the EBM-NLP corpus comprising crowdsourced annotations of medical literature BIBREF5   We operationalize the concept of annotation difficulty and show how it can be exploited during training to improve information extraction models  We then obtain expert annotations for the abstracts predicted to be most difficult, as well as for a similar number of randomly selected abstracts  The annotation of highly specialized data and the use of lay and expert annotators allow us to examine the following key questions related to lay and expert annotations in specialized domains:
Can we predict item difficulty? We define a training instance as difficult if a lay annotator or an automated model disagree on its labeling  We show that difficulty can be predicted, and that it is distinct from inter-annotator agreement  Further, such predictions can be used during training to improve information extraction models 
Are there systematic differences between expert and lay annotations? We observe decidedly lower agreement between lay workers as compared to domain experts  Lay annotations have high precision but low recall with respect to expert annotations in the new data that we collected  More generally, we expect lay annotations to be lower quality, which may translate to lower precision, recall, or both, compared to expert annotations  Can one rely solely on lay annotations? Reasonable models can be trained using lay annotations alone, but similar performance can be achieved using markedly less expert data  This suggests that the optimal ratio of expert to crowd annotations for specialized tasks will depend on the cost and availability of domain experts  Expert annotations are preferable whenever its collection is practical  But in real-world settings, a combination of expert and lay annotations is better than using lay data alone 
Does it matter what data is annotated by experts? We demonstrate that a system trained on combined data achieves better predictive performance when experts annotate difficult examples rather than instances selected at i i d  random 
Our contributions in this work are summarized as follows  We define a task difficulty prediction task and show how this is related to, but distinct from, inter-worker agreement  We introduce a new model for difficulty prediction combining learned representations induced via a pre-trained `universal' sentence encoder BIBREF6 , and a sentence encoder learned from scratch for this task  We show that predicting annotation difficulty can be used to improve the task routing and model performance for a biomedical information extraction task  Our results open up a new direction for ensuring corpus quality  We believe that item difficulty prediction will likely be useful in other, non-specialized tasks as well, and that the most effective data collection in specialized domains requires research addressing the fundamental questions we examine here 
Related Work
Crowdsourcing annotation is now a well-studied problem BIBREF7 , BIBREF0 , BIBREF1 , BIBREF2   Due to the noise inherent in such annotations, there have also been considerable efforts to develop aggregation models that minimize noise BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4  
There are also several surveys of crowdsourcing in biomedicine specifically BIBREF8 , BIBREF9 , BIBREF10   Some work in this space has contrasted model performance achieved using expert vs  crowd annotated training data BIBREF11 , BIBREF12 , BIBREF13   Dumitrache et al  Dumitrache:2018:CGT:3232718 3152889 concluded that performance is similar under these supervision types, finding no clear advantage from using expert annotators  This differs from our findings, perhaps owing to differences in design  The experts we used already hold advanced medical degrees, for instance, while those in prior work were medical students  Furthermore, the task considered here would appear to be of greater difficulty: even a system trained on $\sim $ 5k instances performs reasonably, but far from perfect  By contrast, in some of the prior work where experts and crowd annotations were deemed equivalent, a classifier trained on 300 examples can achieve very high accuracy BIBREF12  
More relevant to this paper, prior work has investigated methods for `task routing' in active learning scenarios in which supervision is provided by heterogeneous labelers with varying levels of expertise BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF14   The related question of whether effort is better spent collecting additional annotations for already labeled (but potentially noisily so) examples or novel instances has also been addressed BIBREF18   What distinguishes the work here is our focus on providing an operational definition of instance difficulty, showing that this can be predicted, and then using this to inform task routing 
Application Domain
Our specific application concerns annotating abstracts of articles that describe the conduct and results of randomized controlled trials (RCTs)  Experimentation in this domain has become easy with the recent release of the EBM-NLP BIBREF5 corpus, which includes a reasonably large training dataset annotated via crowdsourcing, and a modest test set labeled by individuals with advanced medical training  More specifically, the training set comprises 4,741 medical article abstracts with crowdsourced annotations indicating snippets (sequences) that describe the Participants (p), Interventions (i), and Outcome (o) elements of the respective RCT, and the test set is composed of 191 abstracts with p, i, o sequence annotations from three medical experts 
Table 1 shows an example of difficult and easy examples according to our definition of difficulty  The underlined text demarcates the (consensus) reference label provided by domain experts  In the difficult examples, crowd workers marked text distinct from these reference annotations; whereas in the easy cases they reproduced them with reasonable fidelity  The difficult sentences usually exhibit complicated structure and feature jargon 
An abstract may contain some `easy' and some `difficult' sentences  We thus perform our analysis at the sentence level  We split abstracts into sentences using spaCy  We excluded sentences that comprise fewer than two tokens, as these are likely an artifact of errors in sentence splitting  In total, this resulted in 57,505 and 2,428 sentences in the train and test set abstracts, respectively 
Quantifying Task Difficulty
The test set includes annotations from both crowd workers and domain experts  We treat the latter as ground truth and then define the difficulty of sentences in terms of the observed agreement between expert and lay annotators  Formally, for annotation task $t$ and instance $i$ :
$$\text{Difficulty}_{ti} = \frac{\sum _{j=1}^n{f(\text{label}_{ij}, y_i})}{n}$$   (Eq  3)
where $f$ is a scoring function that measures the quality of the label from worker $j$ for sentence $i$ , as compared to a ground truth annotation, $y_i$   The difficulty score of sentence $i$ is taken as an average over the scores for all $n$ layworkers  We use Spearmans' correlation coefficient as a scoring function  Specifically, for each sentence we create two vectors comprising counts of how many times each token was annotated by crowd and expert workers, respectively, and calculate the correlation between these  Sentences with no labels are treated as maximally easy; those with only either crowd worker or expert label(s) are assumed maximally difficult 
The training set contains only crowdsourced annotations  To label the training data, we use a 10-fold validation like setting  We iteratively retrain the LSTM-CRF-Pattern sequence tagger of Patel et al  patel2018syntactic on 9 folds of the training data and use that trained model to predict labels for the 10th  In this way we obtain predictions on the full training set  We then use predicted spans as proxy `ground truth' annotations to calculate the difficulty score of sentences as described above; we normalize these to the [ $0, 1$ ] interval  We validate this approximation by comparing the proxy scores against reference scores over the test set, the Pearson's correlation coefficients are 0 57 for Population, 0 71 for Intervention and 0 68 for Outcome 
There exist many sentences that contain neither manual nor predicted annotations  We treat these as maximally easy sentences (with difficulty scores of 0)  Such sentences comprise 51%, 42% and 36% for Population, Interventions and Outcomes data respectively, indicating that it is easier to identify sentences that have no Population spans, but harder to identify sentences that have no Interventions or Outcomes spans  This is intuitive as descriptions of the latter two tend to be more technical and dense with medical jargon 
We show the distribution of the automatically labeled scores for sentences that do contain spans in Figure 1   The mean of the Population (p) sentence scores is significantly lower than that for other types of sentences (i and o), again indicating that they are easier on average to annotate  This aligns with a previous finding that annotating Interventions and Outcomes is more difficult than annotating Participants BIBREF5  
Many sentences contain spans tagged by the LSTM-CRF-Pattern model, but missed by all crowd workers, resulting in a maximally difficult score (1)  Inspection of such sentences revealed that some are truly difficult examples, but others are tagging model errors  In either case, such sentences have confused workers and/or the model, and so we retain them all as `difficult' sentences 
Content describing the p, i and o, respectively, is quite different  As such, one sentence usually contains (at most) only one of these three content types  We thus treat difficulty prediction for the respective label types as separate tasks 
Difficulty is not Worker Agreement
Our definition of difficulty is derived from agreement between expert and crowd annotations for the test data, and agreement between a predictive model and crowd annotations in the training data  It is reasonable to ask if these measures are related to inter-annotator agreement, a metric often used in language technology research to identify ambiguous or difficult items  Here we explicitly verify that our definition of difficulty only weakly correlates with inter-annotator agreement 
We calculate inter-worker agreement between crowd and expert annotators using Spearman's correlation coefficient  As shown in Table 2 , average agreement between domain experts are considerably higher than agreements between crowd workers for all three label types  This is a clear indication that the crowd annotations are noisier 
Furthermore, we compare the correlation between inter-annotator agreement and difficulty scores in the training data  Given that the majority of sentences do not contain a PICO span, we only include in these calculations those that contain a reference label  Pearson's r are 0 34, 0 30 and 0 31 for p, i and o, respectively, confirming that inter-worker agreement and our proposed difficulty score are quite distinct 
Predicting Annotation Difficulty
We treat difficulty prediction as a regression problem, and propose and evaluate neural model variants for the task  We first train RNN BIBREF19 and CNN BIBREF20 models 
We also use the universal sentence encoder (USE) BIBREF6 to induce sentence representations, and train a model using these as features  Following BIBREF6 , we then experiment with an ensemble model that combines the `universal' and task-specific representations to predict annotation difficulty  We expect these universal embeddings to capture general, high-level semantics, and the task specific representations to capture more granular information  Figure 2 depicts the model architecture  Sentences are fed into both the universal sentence encoder and, separately, a task specific neural encoder, yielding two representations  We concatenate these and pass the combined vector to the regression layer 
Experimental Setup and Results
We trained models for each label type separately  Word embeddings were initialized to 300d GloVe vectors BIBREF21 trained on common crawl data; these are fine-tuned during training  We used the Adam optimizer BIBREF22 with learning rate and decay set to 0 001 and 0 99, respectively  We used batch sizes of 16 
We used the large version of the universal sentence encoder with a transformer BIBREF23   We did not update the pretrained sentence encoder parameters during training  All hyperparamaters for all models (including hidden layers, hidden sizes, and dropout) were tuned using Vizier BIBREF24 via 10-fold cross validation on the training set maximizing for F1 
As a baseline, we also trained a linear Support-Vector Regression BIBREF25 model on $n$ -gram features ( $n$ ranges from 1 to 3) 
Table 3 reports Pearson correlation coefficients between the predictions with each of the neural models and the ground truth difficulty scores  Rows 1-4 correspond to individual models, and row 5 reports the ensemble performance  Columns correspond to label type  Results from all models outperform the baseline SVR model: Pearson's correlation coefficients range from 0 550 to 0 622  The regression correlations are the lowest 
The RNN model realizes the strongest performance among the stand-alone (non-ensemble) models, outperforming variants that exploit CNN and USE representations  Combining the RNN and USE further improves results  We hypothesize that this is due to complementary sentence information encoded in universal representations 
For all models, correlations for Intervention and Outcomes are higher than for Population, which is expected given the difficulty distributions in Figure 1   In these, the sentences are more uniformly distributed, with a fair number of difficult and easier sentences  By contrast, in Population there are a greater number of easy sentences and considerably fewer difficult sentences, which makes the difficulty ranking task particularly challenging 
Better IE with Difficulty Prediction
We next present experiments in which we attempt to use the predicted difficulty during training to improve models for information extraction of descriptions of Population, Interventions and Outcomes from medical article abstracts  We investigate two uses: (1) simply removing the most difficult sentences from the training set, and, (2) re-weighting the most difficult sentences 
We again use LSTM-CRF-Pattern as the base model and experimenting on the EBM-NLP corpus BIBREF5   This is trained on either (1) the training set with difficult sentences removed, or (2) the full training set but with instances re-weighted in proportion to their predicted difficulty score  Following BIBREF5 , we use the Adam optimizer with learning rate of 0 001, decay 0 9, batch size 20 and dropout 0 5  We use pretrained 200d GloVe vectors BIBREF21 to initialize word embeddings, and use 100d hidden char representations  Each word is thus represented with 300 dimensions in total  The hidden size is 100 for the LSTM in the character representation component, and 200 for the LSTM in the information extraction component  We train for 15 epochs, saving parameters that achieve the best F1 score on a nested development set 
Removing Difficult Examples
We first evaluate changes in performance induced by training the sequence labeling model using less data by removing difficult sentences prior to training  The hypothesis here is that these difficult instances are likely to introduce more noise than signal  We used a cross-fold approach to predict sentence difficulties, training on 9/10ths of the data and scoring the remaining 1/10th at a time  We then sorted sentences by predicted difficulty scores, and experimented with removing increasing numbers of these (in order of difficulty) prior to training the LSTM-CRF-Pattern model 
Figure 3 shows the results achieved by the LSTM-CRF-Pattern model after discarding increasing amounts of the training data: the $x$ and $y$ axes correspond to the the percentage of data removed and F1 scores, respectively  We contrast removing sentences predicted to be difficult with removing them (a) randomly (i i d ), and, (b) in inverse order of predicted inter-annotator agreement  The agreement prediction model is trained exactly the same like difficult prediction model, with simply changing the difficult score to annotation agreement  F1 scores actually improve (marginally) when we remove the most difficult sentences, up until we drop 4% of the data for Population and Interventions, and 6% for Outcomes  Removing training points at i i d  random degrades performance, as expected  Removing sentences in order of disagreement seems to have similar effect as removing them by difficulty score when removing small amount of the data, but the F1 scores drop much faster when removing more data  These findings indicate that sentences predicted to be difficult are indeed noisy, to the extent that they do not seem to provide the model useful signal 
Re-weighting by Difficulty
We showed above that removing a small number of the most difficult sentences does not harm, and in fact modestly improves, medical IE model performance  However, using the available data we are unable to test if this will be useful in practice, as we would need additional data to determine how many difficult sentences should be dropped 
We instead explore an alternative, practical means of exploiting difficulty predictions: we re-weight sentences during training inversely to their predicted difficulty  Formally, we weight sentence $i$ with difficulty scores above $\tau $ according to: $1-a\cdot (d_i-\tau )/(1-\tau )$ , where $d_i$ is the difficulty score for sentence $i$ , and $a$ is a parameter codifying the minimum weight value  We set $\tau $ to 0 8 so as to only re-weight sentences with difficulty in the top 20th percentile, and we set $a$ to 0 5  The re-weighting is equivalent to down-sampling the difficult sentences  LSTM-CRF-Pattern is our base model 
Table 4 reports the precision, recall and F1 achieved both with and without sentence re-weighting  Re-weighting improves all metrics modestly but consistently  All F1 differences are statistically significant under a sign test ( $p<0 01$ )  The model with best precision is different for Patient, Intervention and Outcome labels  However re-weighting by difficulty does consistently yield the best recall for all three extraction types, with the most notable improvement for i and o, where recall improved by 10 percentage points  This performance increase translated to improvements in F1 across all types, as compared to the base model and to re-weighting by agreement 
Involving Expert Annotators
The preceding experiments demonstrate that re-weighting difficult sentences annotated by the crowd generally improves the extraction models  Presumably the performance is influenced by the annotation quality 
We now examine the possibility that the higher quality and more consistent annotations of domain experts on the difficult instances will benefit the extraction model  This simulates an annotation strategy in which we route difficult instances to domain experts and easier ones to crowd annotators  We also contrast the value of difficult data to that of an i i d  random sample of the same size, both annotated by experts 
Expert annotations of Random and Difficult Instances
We re-annotate by experts a subset of most difficult instances and the same number of random instances  As collecting annotations from experts is slow and expensive, we only re-annotate the difficult instances for the interventions extraction task  We re-annotate the abstracts which cover the sentences with predicted difficulty scores in the top 5 percentile  We rank the abstracts from the training set by the count of difficult sentences, and re-annotate the abstracts that contain the most difficult sentences  Constrained by time and budget, we select only 2000 abstracts for re-annotation; 1000 of these are top-ranked, and 1000 are randomly sampled  This re-annotation cost $3,000  We have released the new annotation data at: https://github com/bepnye/EBM-NLP 
Following BIBREF5 , we recruited five medical experts via Up-work with advanced medical training and strong technical reading/writing skills  The expert annotator were asked to read the entire abstract and highlight, using the BRAT toolkit BIBREF26 , all spans describing medical Interventions  Each abstract is only annotated by one expert  We examined 30 re-annotated abstracts to ensure the annotation quality before hiring the annotator 
Table 5 presents the results of LSTM-CRF-Pattern model trained on the reannotated difficult subset and the random subset  The first two rows show the results for models trained with expert annotations  The model trained on random data has a slightly better F1 than that trained on the same amount of difficult data  The model trained on random data has higher precision but lower recall 
Rows 3 and 4 list the results for models trained on the same data but with crowd annotation  Models trained with expert-annotated data are clearly superior to those trained with crowd labels with respect to F1, indicating that the experts produced higher quality annotations  For crowdsourced annotations, training the model with data sampled at i i d  random achieves 2% higher F1 than when difficult instances are used  When expert annotations are used, this difference is less than 1%  This trend in performance may be explained by differences in annotation quality: the randomly sampled set was more consistently annotated by both experts and crowd because the difficult set is harder  However, in both cases expert annotations are better, with a bigger difference between the expert and crowd models on the difficult set 
The last row is the model trained on all 5k abstracts with crowd annotations  Its F1 score is lower than either expert model trained on only 20% of data, suggesting that expert annotations should be collected whenever possible  Again the crowd model on complete data has higher precision than expert models but its recall is much lower 
Routing To Experts or Crowd
So far a system was trained on one type of data, either labeled by crowd or experts  We now examine the performance of a system trained on data that was routed to either experts or crowd annotators depending on their predicted difficult  Given the results presented so far mixing annotators may be beneficial given their respective trade-offs of precision and recall  We use the annotations from experts for an abstract if it exists otherwise use crowd annotations  The results are presented in Table 6  
Rows 1 and 2 repeat the performance of the models trained on difficult subset and random subset with expert annotations only respectively  The third row is the model trained by combining difficult and random subsets with expert annotations  There are around 250 abstracts in the overlap of these two sets, so there are total 1 75k abstracts used for training the D+R model  Rows 4 to 6 are the models trained on all 5k abstracts with mixed annotations, where Other means the rest of the abstracts with crowd annotation only 
The results show adding more training data with crowd annotation still improves at least 1 point F1 score in all three extraction tasks  The improvement when the difficult subset with expert annotations is mixed with the remaining crowd annotation is 3 5 F1 score, much larger than when a random set of expert annotations are added  The model trained with re-annotating the difficult subset (D+Other) also outperforms the model with re-annotating the random subset (R+Other) by 2 points in F1  The model trained with re-annotating both of difficult and random subsets (D+R+Other), however, achieves only marginally higher F1 than the model trained with the re-annotated difficult subset (D+Other)  In sum, the results clearly indicate that mixing expert and crowd annotations leads to better models than using solely crowd data, and better than using expert data alone  More importantly, there is greater gain in performance when instances are routed according to difficulty, as compared to randomly selecting the data for expert annotators  These findings align with our motivating hypothesis that annotation quality for difficult instances is important for final model performance  They also indicate that mixing annotations from expert and crowd could be an effective way to achieve acceptable model performance given a limited budget 
How Many Expert Annotations?
We established that crowd annotation are still useful in supplementing expert annotations for medical IE  Obtaining expert annotations for the one thousand most difficult instances greatly improved the model performance  However the choice of how many difficult instances to annotate was an uninformed choice  Here we check if less expert data would have yielded similar gains  Future work will need to address how best to choose this parameter for a routing system 
We simulate a routing scenario in which we send consecutive batches of the most difficult examples to the experts for annotation  We track changes in performance as we increase the number of most-difficult-articles sent to domain experts  As shown in Figure 4 , adding expert annotations for difficult articles consistently increases F1 scores  The performance gain is mostly from increased recall; the precision changes only a bit with higher quality annotation  This observation implies that crowd workers often fail to mark target tokens, but do not tend to produce large numbers of false positives  We suspect such failures to identify relevant spans/tokens are due to insufficient domain knowledge possessed by crowd workers 
The F1 score achieved after re-annotating the 600 most-difficult articles reaches 68 1%, which is close to the performance when re-annotating 1000 random articles  This demonstrates the effectiveness of recognizing difficult instances  The trend when we use up all expert data is still upward, so adding even more expert data is likely to further improve performance  Unfortunately we exhausted our budget and were not able to obtain additional expert annotations  It is likely that as the size of the expert annotations increases, the value of crowd annotations will diminish  This investigation is left for future work 
Conclusions
We have introduced the task of predicting annotation difficulty for biomedical information extraction (IE)  We trained neural models using different learned representations to score texts in terms of their difficulty  Results from all models were strong with Pearson’s correlation coefficients higher than 0 45 in almost all evaluations, indicating the feasibility of this task  An ensemble model combining universal and task specific feature sentence vectors yielded the best results 
Experiments on biomedical IE tasks show that removing up to $\sim $ 10% of the sentences predicted to be most difficult did not decrease model performance, and that re-weighting sentences inversely to their difficulty score during training improves predictive performance  Simulations in which difficult examples are routed to experts and other instances to crowd annotators yields the best results, outperforming the strategy of randomly selecting data for expert annotation, and substantially improving upon the approach of relying exclusively on crowd annotations  In future work, routing strategies based on instance difficulty could be further investigated for budget-quality trade-off 
Acknowledgements
This work has been partially supported by NSF1748771 grant  Wallace was support in part by NIH/NLM R01LM012086 
Context 2: Introduction
Pretrained Language Models (PTLMs) such as BERT BIBREF1 have spearheaded advances on many NLP tasks  Usually, PTLMs are pretrained on unlabeled general-domain and/or mixed-domain text, such as Wikipedia, digital books or the Common Crawl corpus 
When applying PTLMs to specific domains, it can be useful to domain-adapt them  Domain adaptation of PTLMs has typically been achieved by pretraining on target-domain text  One such model is BioBERT BIBREF2, which was initialized from general-domain BERT and then pretrained on biomedical scientific publications  The domain adaptation is shown to be helpful for target-domain tasks such as biomedical Named Entity Recognition (NER) or Question Answering (QA)  On the downside, the computational cost of pretraining can be considerable: BioBERTv1 0 was adapted for ten days on eight large GPUs (see Table TABREF1), which is expensive, environmentally unfriendly, prohibitive for small research labs and students, and may delay prototyping on emerging domains 
We therefore propose a fast, CPU-only domain-adaptation method for PTLMs: We train Word2Vec BIBREF3 on target-domain text and align the resulting word vectors with the wordpiece vectors of an existing general-domain PTLM  The PTLM thus gains domain-specific lexical knowledge in the form of additional word vectors, but its deeper layers remain unchanged  Since Word2Vec and the vector space alignment are efficient models, the process requires a fraction of the resources associated with pretraining the PTLM itself, and it can be done on CPU 
In Section SECREF4, we use the proposed method to domain-adapt BERT on PubMed+PMC (the data used for BioBERTv1 0) and/or CORD-19 (Covid-19 Open Research Dataset)  We improve over general-domain BERT on eight out of eight biomedical NER tasks, using a fraction of the compute cost associated with BioBERT  In Section SECREF5, we show how to quickly adapt an existing Question Answering model to text about the Covid-19 pandemic, without any target-domain Language Model pretraining or finetuning 
Related work ::: The BERT PTLM
For our purpose, a PTLM consists of three parts: A tokenizer $\mathcal {T}_\mathrm {LM} : \mathbb {L}^+ \rightarrow \mathbb {L}_\mathrm {LM}^+$, a wordpiece embedding function $\mathcal {E}_\mathrm {LM}: \mathbb {L}_\mathrm {LM} \rightarrow \mathbb {R}^{d_\mathrm {LM}}$ and an encoder function $\mathcal {F}_\mathrm {LM}$  $\mathbb {L}_\mathrm {LM}$ is a limited vocabulary of wordpieces  All words that are not in $\mathbb {L}_\mathrm {LM}$ are tokenized into sequences of shorter wordpieces, e g , tachycardia becomes ta ##chy ##card ##ia  Given a sentence $S = [w_1, \ldots , w_T]$, tokenized as $\mathcal {T}_\mathrm {LM}(S) = [\mathcal {T}_\mathrm {LM}(w_1); \ldots ; \mathcal {T}_\mathrm {LM}(w_T)]$, $\mathcal {E}_\mathrm {LM}$ embeds every wordpiece in $\mathcal {T}_\mathrm {LM}(S)$ into a real-valued, trainable wordpiece vector  The wordpiece vectors of the entire sequence are stacked and fed into $\mathcal {F}_\mathrm {LM}$  Note that we consider position and segment embeddings to be a part of $\mathcal {F}_\mathrm {LM}$ rather than $\mathcal {E}_\mathrm {LM}$ 
In the case of BERT, $\mathcal {F}_\mathrm {LM}$ is a Transformer BIBREF4, followed by a final Feed-Forward Net  During pretraining, the Feed-Forward Net predicts the identity of masked wordpieces  When finetuning on a supervised task, it is usually replaced with a randomly initialized task-specific layer 
Related work ::: Domain-adapted PTLMs
Domain adaptation of PTLMs is typically achieved by pretraining on unlabeled target-domain text  Some examples of such models are BioBERT BIBREF2, which was pretrained on the PubMed and/or PubMed Central (PMC) corpora, SciBERT BIBREF5, which was pretrained on papers from SemanticScholar, ClinicalBERT BIBREF6, BIBREF7 and ClinicalXLNet BIBREF8, which were pretrained on clinical patient notes, and AdaptaBERT BIBREF9, which was pretrained on Early Modern English text  In most cases, a domain-adapted PTLM is initialized from a general-domain PTLM (e g , standard BERT), though BIBREF5 report better results with a model that was pretrained from scratch with a custom wordpiece vocabulary  In this paper, we focus on BioBERT, as its domain adaptation corpora are publicly available 
Related work ::: Word vectors
Word vectors are distributed representations of words that are trained on unlabeled text  Contrary to PTLMs, word vectors are non-contextual, i e , a word type is always assigned the same vector, regardless of context  In this paper, we use Word2Vec BIBREF3 to train word vectors  We will denote the Word2Vec lookup function as $\mathcal {E}_\mathrm {W2V} : \mathbb {L}_\mathrm {W2V} \rightarrow \mathbb {R}^{d_\mathrm {W2V}}$ 
Related work ::: Word vector space alignment
Word vector space alignment has most frequently been explored in the context of cross-lingual word embeddings  For instance, BIBREF10 align English and Spanish Word2Vec spaces by a simple linear transformation  BIBREF11 use a related method to align cross-lingual word vectors and multilingual BERT wordpiece vectors 
Method
In the following, we assume access to a general-domain PTLM, as described in Section SECREF2, and a corpus of unlabeled target-domain text 
Method ::: Creating new input vectors
In a first step, we train Word2Vec on the target-domain corpus  In a second step, we take the intersection of $\mathbb {L}_\mathrm {LM}$ and $\mathbb {L}_\mathrm {W2V}$  In practice, the intersection mostly contains wordpieces from $\mathbb {L}_\mathrm {LM}$ that correspond to standalone words  It also contains single characters and other noise, however, we found that filtering them does not improve alignment quality  In a third step, we use the intersection to fit an unconstrained linear transformation $\mathbf {W} \in \mathbb {R}^{d_\mathrm {LM} \times d_\mathrm {W2V}}$ via least squares:
Intuitively, $\mathbf {W}$ makes Word2Vec vectors “look like” the PTLM's native wordpiece vectors, just like cross-lingual alignment makes word vectors from one language “look like” word vectors from another language  In Table TABREF7 (top), we show examples of within-space and cross-space nearest neighbors after alignment 
Method ::: Updating the wordpiece embedding layer
Next, we redefine the wordpiece embedding layer of the PTLM  The most radical strategy would be to replace the entire layer with the aligned Word2Vec vectors:
In initial experiments, this strategy led to a drop in performance, presumably because function words are not well represented by Word2Vec, and replacing them disrupts BERT's syntactic abilities  To prevent this problem, we leave existing wordpiece vectors intact and only add new ones:
Method ::: Updating the tokenizer
In a final step, we update the tokenizer to account for the added words  Let $\mathcal {T}_\mathrm {LM}$ be the standard BERT tokenizer, and let $\hat{\mathcal {T}}_\mathrm {LM}$ be the tokenizer that treats all words in $\mathbb {L}_\mathrm {LM} \cup \mathbb {L}_\mathrm {W2V}$ as one-wordpiece tokens, while tokenizing any other words as usual 
In practice, a given word may or may not benefit from being tokenized by $\hat{\mathcal {T}}_\mathrm {LM}$ instead of $\mathcal {T}_\mathrm {LM}$  To give a concrete example, 82% of the words in the BC5CDR NER dataset that end in the suffix -ia are inside a disease entity (e g , tachycardia)  $\mathcal {T}_\mathrm {LM}$ tokenizes this word as ta ##chy ##card ##ia, thereby exposing the orthographic cue to the model  As a result, $\mathcal {T}_\mathrm {LM}$ leads to higher recall on -ia diseases  But there are many cases where wordpiece tokenization is meaningless or misleading  For instance euthymia (not a disease) is tokenized by $\mathcal {T}_\mathrm {LM}$ as e ##uth ##ym ##ia, making it likely to be classified as a disease  By contrast, $\hat{\mathcal {T}}_\mathrm {LM}$ gives euthymia a one-wordpiece representation that depends only on distributional semantics  We find that using $\hat{\mathcal {T}}_\mathrm {LM}$ improves precision on -ia diseases 
To combine these complementary strengths, we use a 50/50 mixture of $\mathcal {T}_\mathrm {LM}$-tokenization and $\hat{\mathcal {T}}_\mathrm {LM}$-tokenization when finetuning the PTLM on a task  At test time, we use both tokenizers and mean-pool the outputs  Let $o(\mathcal {T}(S))$ be some output of interest (e g , a logit), given sentence $S$ tokenized by $\mathcal {T}$  We predict:
Experiment 1: Biomedical NER
In this section, we use the proposed method to create GreenBioBERT, an inexpensive and environmentally friendly alternative to BioBERT  Recall that BioBERTv1 0 (biobert_v1 0_pubmed_pmc) was initialized from general-domain BERT (bert-base-cased) and pretrained on PubMed+PMC 
Experiment 1: Biomedical NER ::: Domain adaptation
We train Word2Vec with vector size $d_\mathrm {W2V} = d_\mathrm {LM} = 768$ on PubMed+PMC (see Appendix for details)  Then, we follow the procedure described in Section SECREF3 to update the wordpiece embedding layer and tokenizer of general-domain BERT 
Experiment 1: Biomedical NER ::: Finetuning
We finetune GreenBioBERT on the eight publicly available NER tasks used in BIBREF2  We also do reproduction experiments with general-domain BERT and BioBERTv1 0, using the same setup as our model  We average results over eight random seeds  See Appendix for details on preprocessing, training and hyperparameters 
Experiment 1: Biomedical NER ::: Results and discussion
Table TABREF7 (bottom) shows entity-level precision, recall and F1  For ease of visualization, Figure FIGREF13 shows what portion of the BioBERT – BERT F1 delta is covered  We improve over general-domain BERT on all tasks with varying effect sizes
Context 3: Introduction
Stance detection (also called stance identification or stance classification) is one of the considerably recent research topics in natural language processing (NLP)  It is usually defined as a classification problem where for a text and target pair, the stance of the author of the text for that target is expected as a classification output from the set: {Favor, Against, Neither} BIBREF0  
Stance detection is usually considered as a subtask of sentiment analysis (opinion mining) BIBREF1 topic in NLP  Both are mostly performed on social media texts, particularly on tweets, hence both are important components of social media analysis  Nevertheless, in sentiment analysis, the sentiment of the author of a piece of text usually as Positive, Negative, and Neutral is explored while in stance detection, the stance of the author of the text for a particular target (an entity, event, etc ) either explicitly or implicitly referred to in the text is considered  Like sentiment analysis, stance detection systems can be valuable components of information retrieval and other text analysis systems BIBREF0  
Previous work on stance detection include BIBREF2 where a stance classifier based on sentiment and arguing features is proposed in addition to an arguing lexicon automatically compiled  The ultimate approach performs better than distribution-based and uni-gram-based baseline systems BIBREF2   In BIBREF3 , the authors show that the use of dialogue structure improves stance detection in on-line debates  In BIBREF4 , Hasan and Ng carry out stance detection experiments using different machine learning algorithms, training data sets, features, and inter-post constraints in on-line debates, and draw insightful conclusions based on these experiments  For instance, they find that sequence models like HMMs perform better at stance detection when compared with non-sequence models like Naive Bayes (NB) BIBREF4   In another related study BIBREF5 , the authors conclude that topic-independent features can be exploited for disagreement detection in on-line dialogues  The employed features include agreement, cue words, denial, hedges, duration, polarity, and punctuation BIBREF5   Stance detection on a corpus of student essays is considered in BIBREF6   After using linguistically-motivated feature sets together with multivalued NB and SVM as the learning models, the authors conclude that they outperform two baseline approaches BIBREF6   In BIBREF7 , the author claims that Wikipedia can be used to determine stances about controversial topics based on their previous work regarding controversy extraction on the Web 
Among more recent related work, in BIBREF8 stance detection for unseen targets is studied and bidirectional conditional encoding is employed  The authors state that their approach achieves state-of-the art performance rates BIBREF8 on SemEval 2016 Twitter Stance Detection corpus BIBREF0   In BIBREF9 , a stance-community detection approach called SCIFNET is proposed  SCIFNET creates networks of people who are stance targets, automatically from the related document collections BIBREF9 using stance expansion and refinement techniques to arrive at stance-coherent networks  A tweet data set annotated with stance information regarding six predefined targets is proposed in BIBREF10 where this data set is annotated through crowdsourcing  The authors indicate that the data set is also annotated with sentiment information in addition to stance, so it can help reveal associations between stance and sentiment BIBREF10   Lastly, in BIBREF0 , SemEval 2016's aforementioned shared task on Twitter Stance Detection is described  Also provided are the results of the evaluations of 19 systems participating in two subtasks (one with training data set provided and the other without an annotated data set) of the shared task BIBREF0  
In this paper, we present a tweet data set in Turkish annotated with stance information, where the corresponding annotations are made publicly available  The domain of the tweets comprises two popular football clubs which constitute the targets of the tweets included  We also provide the evaluation results of SVM classifiers (for each target) on this data set using unigram, bigram, and hashtag features 
To the best of our knowledge, the current study is the first one to target at stance detection in Turkish tweets  Together with the provided annotated data set and the corresponding evaluations with the aforementioned SVM classifiers which can be used as baseline systems, our study will hopefully help increase social media analysis studies on Turkish content 
The rest of the paper is organized as follows: In Section SECREF2 , we describe our tweet data set annotated with the target and stance information  Section SECREF3 includes the details of our SVM-based stance classifiers and their evaluation results with discussions  Section SECREF4 includes future research topics based on the current study, and finally Section SECREF5 concludes the paper with a summary 
A Stance Detection Data Set
We have decided to consider tweets about popular sports clubs as our domain for stance detection  Considerable amounts of tweets are being published for sports-related events at every instant  Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey  As is the case for the sentiment analysis tools, the outputs of the stance detection systems on a stream of tweets about these clubs can facilitate the use of the opinions of the football followers by these clubs 
In a previous study on the identification of public health-related tweets, two tweet data sets in Turkish (each set containing 1 million random tweets) have been compiled where these sets belong to two different periods of 20 consecutive days BIBREF11   We have decided to use one of these sets (corresponding to the period between August 18 and September 6, 2015) and firstly filtered the tweets using the possible names used to refer to the target clubs  Then, we have annotated the stance information in the tweets for these targets as Favor or Against  Within the course of this study, we have not considered those tweets in which the target is not explicitly mentioned, as our initial filtering process reveals 
For the purposes of the current study, we have not annotated any tweets with the Neither class  This stance class and even finer-grained classes can be considered in further annotation studies  We should also note that in a few tweets, the target of the stance was the management of the club while in some others a particular footballer of the club is praised or criticised  Still, we have considered the club as the target of the stance in all of the cases and carried out our annotations accordingly 
At the end of the annotation process, we have annotated 700 tweets, where 175 tweets are in favor of and 175 tweets are against Target-1, and similarly 175 tweets are in favor of and 175 are against Target-2  Hence, our data set is a balanced one although it is currently limited in size  The corresponding stance annotations are made publicly available at http://ceng metu edu tr/ INLINEFORM0 e120329/ Turkish_Stance_Detection_Tweet_Dataset csv in Comma Separated Values (CSV) format  The file contains three columns with the corresponding headers  The first column is the tweet id of the corresponding tweet, the second column contains the name of the stance target, and the last column includes the stance of the tweet for the target as Favor or Against 
To the best of our knowledge, this is the first publicly-available stance-annotated data set for Turkish  Hence, it is a significant resource as there is a scarcity of annotated data sets, linguistic resources, and NLP tools available for Turkish  Additionally, to the best of our knowledge, it is also significant for being the first stance-annotated data set including sports-related tweets, as previous stance detection data sets mostly include on-line texts on political/ethical issues 
Stance Detection Experiments Using SVM Classifiers
It is emphasized in the related literature that unigram-based methods are reliable for the stance detection task BIBREF2 and similarly unigram-based models have been used as baseline models in studies such as BIBREF0   In order to be used as a baseline and reference system for further studies on stance detection in Turkish tweets, we have trained two SVM classifiers (one for each target) using unigrams as features  Before the extraction of unigrams, we have employed automated preprocessing to filter out the stopwords in our annotated data set of 700 tweets  The stopword list used is the list presented in BIBREF12 which, in turn, is the slightly extended version of the stopword list provided in BIBREF13  
We have used the SVM implementation available in the Weka data mining application BIBREF14 where this particular implementation employs the SMO algorithm BIBREF15 to train a classifier with a linear kernel  The 10-fold cross-validation results of the two classifiers are provided in Table TABREF1 using the metrics of precision, recall, and F-Measure 
The evaluation results are quite favorable for both targets and particularly higher for Target-1, considering the fact that they are the initial experiments on the data set  The performance of the classifiers is better for the Favor class for both targets when compared with the performance results for the Against class  This outcome may be due to the common use of some terms when expressing positive stance towards sports clubs in Turkish tweets  The same percentage of common terms may not have been observed in tweets during the expression of negative stances towards the targets  Yet, completely the opposite pattern is observed in stance detection results of baseline systems given in BIBREF0 , i e , better F-Measure rates have been obtained for the Against class when compared with the Favor class BIBREF0   Some of the baseline systems reported in BIBREF0 are SVM-based systems using unigrams and ngrams as features similar to our study, but their data sets include all three stance classes of Favor, Against, and Neither, while our data set comprises only tweets classified as belonging to Favor or Against classes  Another difference is that the data sets in BIBREF0 have been divided into training and test sets, while in our study we provide 10-fold cross-validation results on the whole data set  On the other hand, we should also note that SVM-based sentiment analysis systems (such as those given in BIBREF16 ) have been reported to achieve better F-Measure rates for the Positive sentiment class when compared with the results obtained for the Negative class  Therefore, our evaluation results for each stance class seem to be in line with such sentiment analysis systems  Yet, further experiments on the extended versions of our data set should be conducted and the results should again be compared with the stance detection results given in the literature 
We have also evaluated SVM classifiers which use only bigrams as features, as ngram-based classifiers have been reported to perform better for the stance detection problem BIBREF0   However, we have observed that using bigrams as the sole features of the SVM classifiers leads to quite poor results  This observation may be due to the relatively limited size of the tweet data set employed  Still, we can conclude that unigram-based features lead to superior results compared to the results obtained using bigrams as features, based on our experiments on our data set  Yet, ngram-based features may be employed on the extended versions of the data set to verify this conclusion within the course of future work 
With an intention to exploit the contribution of hashtag use to stance detection, we have also used the existence of hashtags in tweets as an additional feature to unigrams  The corresponding evaluation results of the SVM classifiers using unigrams together the existence of hashtags as features are provided in Table TABREF2  
When the results given in Table TABREF2 are compared with the results in Table TABREF1 , a slight decrease in F-Measure (0 5%) for Target-1 is observed, while the overall F-Measure value for Target-2 has increased by 1 8%  Although we could not derive sound conclusions mainly due to the relatively small size of our data set, the increase in the performance of the SVM classifier Target-2 is an encouraging evidence for the exploitation of hashtags in a stance detection system  We leave other ways of exploiting hashtags for stance detection as a future work 
To sum up, our evaluation results are significant as reference results to be used for comparison purposes and provides evidence for the utility of unigram-based and hashtag-related features in SVM classifiers for the stance detection problem in Turkish tweets 
Future Prospects
Future work based on the current study includes the following:
Conclusion
Stance detection is a considerably new research area in natural language processing and is considered within the scope of the well-studied topic of sentiment analysis  It is the detection of stance within text towards a target which may be explicitly specified in the text or not  In this study, we present a stance-annotated tweet data set in Turkish where the targets of the annotated stances are two popular sports clubs in Turkey  The corresponding annotations are made publicly-available for research purposes  To the best of our knowledge, this is the first stance detection data set for the Turkish language and also the first sports-related stance-annotated data set  Also presented in this study are SVM classifiers (one for each target) utilizing unigram and bigram features in addition to using the existence of hashtags as another feature  10-fold cross validation results of these classifiers are presented which can be used as reference results by prospective systems  Both the annotated data set and the classifiers with evaluations are significant since they are the initial contributions to stance detection problem in Turkish tweets 

ground_truth: improvement when the difficult subset with expert annotations is mixed with the remaining crowd annotation is 3.5 F1 score, much larger than when a random set of expert annotations are added

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which eight NER tasks did they evaluate on?
Context 2:  Depending on the points of reference, we cover an average 52% to 60% of the BioBERT – BERT F1 delta (54% for BioBERTv1 0, 60% for BioBERTv1 1 and 52% for our reproduction experiments)  Table TABREF17 (top) shows the importance of vector space alignment: If we replace the aligned Word2Vec vectors with their non-aligned counterparts (by setting $\mathbf {W} = \mathbf {1}$) or with randomly initialized vectors, F1 drops on all tasks 
Experiment 2: Covid-19 QA
In this section, we use the proposed method to quickly adapt an existing general-domain QA model to an emerging target domain: Covid-19  Our baseline model is SQuADBERT (bert-large-uncased-whole-word-masking-finetuned-squad), a version of BERT that was finetuned on general-domain SQuAD BIBREF19  We evaluate on Deepset-AI Covid-QA, a SQuAD-style dataset with 1380 questions (see Appendix for details on data and preprocessing)  We assume that there is no target-domain finetuning data, which is a realistic setup for a new domain 
Experiment 2: Covid-19 QA ::: Domain adaptation
We train Word2Vec with vector size $d_\mathrm {W2V} = d_\mathrm {LM} = 1024$ on CORD-19 (Covid-19 Open Research Dataset) and/or PubMed+PMC  The process takes less than an hour on CORD-19 and about one day on the combined corpus, again without the need for a GPU  Then, we update SQuADBERT's wordpiece embedding layer and tokenizer, as described in Section SECREF3  We refer to the resulting model as GreenCovidSQuADBERT 
Experiment 2: Covid-19 QA ::: Results and discussion
Table TABREF17 (bottom) shows that GreenCovidSQuADBERT outperforms general-domain SQuADBERT in all metrics  Most of the improvement can be achieved with just the small CORD-19 corpus, which is more specific to the target domain (compare “Cord-19 only” and “Cord-19+PubMed+PMC”) 
Conclusion
As a reaction to the trend towards high-resource models, we have proposed an inexpensive, CPU-only method for domain-adapting Pretrained Language Models: We train Word2Vec vectors on target-domain data and align them with the wordpiece vector space of a general-domain PTLM 
On eight biomedical NER tasks, we cover over 50% of the BioBERT – BERT F1 delta, at 5% of BioBERT's domain adaptation CO$_2$ footprint and 2% of its cloud compute cost  We have also shown how to rapidly adapt an existing BERT QA model to an emerging domain – the Covid-19 pandemic – without the need for target-domain Language Model pretraining or finetuning 
We hope that our approach will benefit practitioners with limited time or resources, and that it will encourage environmentally friendlier NLP 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Word2Vec training
We downloaded the PubMed, PMC and CORD-19 corpora from:
https://ftp ncbi nlm nih gov/pub/pmc/oa_bulk/ [20 January 2020, 68GB raw text]
https://ftp ncbi nlm nih gov/pubmed/baseline/ [20 January 2020, 24GB raw text]
https://pages semanticscholar org/coronavirus-research [17 April 2020, 2GB raw text]
We extract all abstracts and text bodies and apply the BERT basic tokenizer (a word tokenizer that standard BERT uses before wordpiece tokenization)  Then, we train CBOW Word2Vec with negative sampling  We use default parameters except for the vector size (which we set to $d_\mathrm {W2V} = d_\mathrm {LM}$) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Pretrained models
General-domain BERT and BioBERTv1 0 were downloaded from:
https://storage googleapis com/bert_models/2018_10_18/cased_L-12_H-768_A-12 zip
https://github com/naver/biobert-pretrained
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Data
We downloaded the NER datasets by following instructions on https://github com/dmis-lab/biobert#Datasets  For detailed dataset statistics, see BIBREF2 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Preprocessing
We cut all sentences into chunks of 30 or fewer whitespace-tokenized words (without splitting inside labeled spans)  Then, we tokenize every chunk $S$ with $\mathcal {T} = \mathcal {T}_\mathrm {LM}$ or $\mathcal {T} = \hat{\mathcal {T}}_\mathrm {LM}$ and add special tokens:
Word-initial wordpieces in $\mathcal {T}(S)$ are labeled as B(egin), I(nside) or O(utside), while non-word-initial wordpieces are labeled as X(ignore) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Modeling, training and inference
We follow BIBREF2's implementation (https://github com/dmis-lab/biobert): We add a randomly initialized softmax classifier on top of the last BERT layer to predict the labels  We finetune the entire model to minimize negative log likelihood, with the standard Adam optimizer BIBREF20 and a linear learning rate scheduler (10% warmup)  Like BIBREF2, we finetune on the concatenation of the training and development set  All finetuning runs were done on a GeForce Titan X GPU (12GB) 
Since we do not have the resources for an extensive hyperparameter search, we use defaults and recommendations from the BioBERT repository: Batch size of 32, peak learning rate of $1 \cdot 10^{-5}$, and 100 epochs 
At inference time, we gather the output logits of word-initial wordpieces only  Since the number of word-initial wordpieces is the same for $\mathcal {T}_\mathrm {LM}(S)$ and $\hat{\mathcal {T}}_\mathrm {LM}(S)$, this makes mean-pooling the logits straightforward 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Note on our reproduction experiments
We found it easier to reproduce or exceed BIBREF2's results for general-domain BERT, compared to their results for BioBERTv1 0 (see Figure FIGREF13, main paper)  While this may be due to hyperparameters, it suggests that BioBERTv1 0 was more strongly tuned than BERT in the original BioBERT paper  This observation does not affect our conclusions, as GreenBioBERT performs better than reproduced BERT as well 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Pretrained model
We downloaded the SQuADBERT baseline from:
https://huggingface co/bert-large-uncased-whole-word-masking-finetuned-squad
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Data
We downloaded the Deepset-AI Covid-QA dataset from:
https://github com/deepset-ai/COVID-QA/blob/master/data/question-answering/200423_covidQA json [24 April 2020]
At the time of writing, the dataset contains 1380 questions and gold answer spans  Every question is associated with one of 98 research papers (contexts)  We treat the entire dataset as a test set 
Note that there are some important differences between the dataset and SQuAD, which make the task challenging:
The contexts are full documents rather than single paragraphs  Thus, the correct answer may appear several times, often with slightly different wordings  Only a single one of the occurrences is annotated as correct, e g :
What was the prevalence of Coronavirus OC43 in community samples in Ilorin, Nigeria?
13 3% (95% CI 6 9-23 6%) # from main text
(13 3%, 10/75)  # from abstract
SQuAD gold answers are defined as the “shortest span in the paragraph that answered the question” BIBREF19, but many Covid-QA gold answers are longer and contain non-essential context, e g :
When was the Middle East Respiratory Syndrome Coronavirus isolated first?
(MERS-CoV) was first isolated in 2012, in a 60-year-old man who died in Jeddah, KSA due to severe acute pneumonia and multiple organ failure
2012,
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Preprocessing
We tokenize every question-context pair $(Q, C)$ with $\mathcal {T} = \mathcal {T}_\mathrm {LM}$ or $\mathcal {T} = \hat{\mathcal {T}}_\mathrm {LM}$, which yields $(\mathcal {T}(Q), \mathcal {T}(C))$  Since $\mathcal {T}(C)$ is usually too long to be digested in a single forward pass, we define a sliding window with width and stride $N = \mathrm {floor}(\frac{509 - |\mathcal {T}(Q)|}{2})$  At step $n$, the “active” window is between $a^{(l)}_n = nN$ and $a^{(r)}_n = \mathrm {min}(|C|, nN+N)$  The input is defined as:
$p^{(l)}_n$ and $p^{(r)}_n$ are chosen such that $|X^{(n)}| = 512$, and such that the active window is in the center of the input (if possible) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Modeling and inference
Feeding $X^{(n)}$ into the pretrained QA model yields start logits $\mathbf {h^{\prime }}^{(\mathrm {start}, n)} \in \mathbb {R}^{|X^{(n)}|}$ and end logits $\mathbf {h^{\prime }}^{(\mathrm {end},n)} \in \mathbb {R}^{|X^{(n)}|}$  We extract and concatenate the slices that correspond to the active windows of all steps:
Next, we map the logits from the wordpiece level to the word level  This allows us to mean-pool the outputs of $\mathcal {T}_\mathrm {LM}$ and $\hat{\mathcal {T}}_\mathrm {LM}$ even when $|\mathcal {T}_\mathrm {LM}(C)| \ne |\hat{\mathcal {T}}_\mathrm {LM}(C)|$ 
Let $c_i$ be a whitespace-delimited word in $C$  Let $\mathcal {T}(C)_{j:j+|\mathcal {T}(c_i)|}$ be the corresponding wordpieces  The start and end logits of $c_i$ are derived as:
Finally, we return the answer span $C_{k:k^{\prime }}$ that maximizes $o^{(\mathrm {start})}_k + o^{(\mathrm {end})}_{k^{\prime }}$, subject to the constraints that $k^{\prime }$ does not precede $k$ and the answer span is not longer than 500 characters 

ground_truth: BC5CDR-disease, NCBI-disease, BC5CDR-chem, BC4CHEMD, BC2GM, JNLPBA, LINNAEUS, Species-800

BC5CDR-disease, NCBI-disease, BC5CDR-chem, BC4CHEMD, BC2GM, JNLPBA, LINNAEUS, Species-800

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which sports clubs are the targets?
Context 4:  This could be due to implementation differences, which unfortunately makes our EN results incomparable  For DE, the arguments are identified using the LTH system BIBREF16 , which gives an F1 score of 86 5% on the CoNLL 2009 DE data  The word alignments for the EN-DE parallel Europarl corpus are computed using GIZA++ BIBREF17   For high-precision, only the intersecting alignments in the two directions are kept  We define two semantic arguments as aligned if their head-words are aligned  In total we get 9 3M arguments for EN (240k CoNLL + 9 1M Europarl) and 4 43M for DE (32k CoNLL + 4 4M Europarl)  Out of these, 0 76M arguments are aligned 
Main Results
Since the CoNLL annotations have 21 semantic roles in total, we use 21 roles in our model as well as the baseline  Following garg2012unsupervised, we set the number of PRs to 2 (excluding INLINEFORM0 , INLINEFORM1 and INLINEFORM2 ), and SRs to 21-2=19  Table TABREF27 shows the results 
In the first setting (Line 1), we train and test the monolingual model on the CoNLL data  We observe significant improvements in F1 score over the Baseline (Line 0) in both languages  Using the CoNLL 2009 dataset alone, titovcrosslingual report an F1 score of 80 9% (PU=86 8%, CO=75 7%) for German  Thus, our monolingual model outperforms their monolingual model in German  For English, they report an F1 score of 83 6% (PU=87 5%, CO=80 1%), but note that our English results are not directly comparable to theirs due to differences argument identification, as discussed in section SECREF25   As their argument identification score is lower, perhaps their system is discarding “difficult” arguments which leads to a higher clustering score 
In the second setting (Line 2), we use the additional monolingual Europarl (EP) data for training  We get equivalent results in English and a significant improvement in German compared to our previous setting (Line 1)  The German dataset in CoNLL is quite small and benefits from the additional EP training data  In contrast, the English model is already quite good due to a relatively big dataset from CoNLL, and good accuracy syntactic parsers  Unfortunately, titovcrosslingual do not report results with this setting 
The third setting (Line 3) gives the results of our multilingual model, which adds the word alignments in the EP data  Comparing with Line 2, we get non-significant improvements in both languages  titovcrosslingual obtain an F1 score of 82 7% (PU=85 0%, CO=80 6%) for German, and 83 7% (PU=86 8%, CO=80 7%) for English  Thus, for German, our multilingual Bayesian model is able to capture the cross-lingual patterns at least as well as the external penalty term in BIBREF6   We cannot compare the English results unfortunately due to differences in argument identification 
We also compared monolingual and bilingual training data using a setting that emulates the standard supervised setup of separate training and test data sets  We train only on the EP dataset and test on the CoNLL dataset  Lines 4 and 5 of Table TABREF27 give the results  The multilingual model obtains small improvements in both languages, which confirms the results from the standard unsupervised setup, comparing lines 2 to 3 
These results indicate that little information can be learned about semantic roles from this parallel data setup  One possible explanation for this result is that the setup itself is inadequate  Given the definition of aligned arguments, only 8% of English arguments and 17% of German arguments are aligned  This plus our experiments suggest that improving the alignment model is a necessary step to making effective use of parallel data in multilingual SRI, for example by joint modeling with SRI  We leave this exploration to future work 
Multilingual Training with Labeled Data for One Language
Another motivation for jointly modeling SRL in multiple languages is the transfer of information from a resource rich language to a resource poor language  We evaluated our model in a very general annotation transfer scenario, where we have a small labeled dataset for one language (source), and a large parallel unlabeled dataset for the source and another (target) language  We investigate whether this setting improves the parameter estimates for the target language  To this end, we clamp the role annotations of the source language in the CoNLL dataset using a predefined mapping, and do not sample them during training  This data gives us good parameters for the source language, which are used to sample the roles of the source language in the unlabeled Europarl data  The CLVs aim to capture this improvement and thereby improve sampling and parameter estimates for the target language  Table TABREF28 shows the results of this experiment  We obtain small improvements in the target languages  As in the unsupervised setting, the small percentage of aligned roles probably limits the impact of the cross-lingual information 
Labeled Data in Monolingual Model
We explored the improvement in the monolingual model in a semi-supervised setting  To this end, we randomly selected INLINEFORM0 of the sentences in the CoNLL dataset as “supervised sentences” and the rest INLINEFORM1 were kept unsupervised  Next, we clamped the role labels of the supervised sentences using the predefined mapping from Section SECREF29   Sampling was done on the unsupervised sentences as usual  We then measured the clustering performance using the trained parameters 
To access the contribution of partial supervision better, we constructed a “supervised baseline” as follows  For predicates seen in the supervised sentences, a MAP estimate of the parameters was calculated using the predefined mapping  For the unseen predicates, the standard baseline was used 
Figures FIGREF33 and FIGREF33 show the performance variation with INLINEFORM0   We make the following observations:
[leftmargin=*]
In both languages, at around INLINEFORM0 , the supervised baseline starts outperforming the semi-supervised model, which suggests that manually labeling about 10% of the sentences is a good enough alternative to our training procedure  Note that 10% amounts to about 3 6k sentences in German and 4k in English  We noticed that the proportion of seen predicates increases dramatically as we increase the proportion of supervised sentences  At 10% supervised sentences, the model has already seen 63% of predicates in German and 44% in English  This explains to some extent why only 10% labeled sentences are enough 
For German, it takes about 3 5% or 1260 supervised sentences to have the same performance increase as 1 5M unlabeled sentences (Line 1 to Line 2 in Table TABREF27 )  Adding about 180 more supervised sentences also covers the benefit obtained by alignments in the multilingual model (Line 2 to Line 3 in Table TABREF27 )  There is no noticeable performance difference in English 
We also evaluated the performance variation on a completely unseen CoNLL test set  Since the test set is very small compared to the training set, the clustering evaluation is not as reliable  Nonetheless, we broadly obtained the same pattern 
Related Work
As discussed in section SECREF24 , our work is closely related to the crosslingual unsupervised SRL work of titovcrosslingual  The idea of using superlingual latent variables to capture cross-lingual information was proposed for POS tagging by naseem2009multilingual, which we use here for SRL  In a semi-supervised setting, pado2009cross used a graph based approach to transfer semantic role annotations from English to German  furstenau2009graph used a graph alignment method to measure the semantic and syntactic similarity between dependency tree arguments of known and unknown verbs 
For monolingual unsupervised SRL, swier2004unsupervised presented the first work on a domain-general corpus, the British National Corpus, using 54 verbs taken from VerbNet  garg2012unsupervised proposed a Bayesian model for this problem that we use here  titov2012bayesian also proposed a closely related Bayesian model  grenager2006unsupervised proposed a generative model but their parameter space consisted of all possible linkings of syntactic constituents and semantic roles, which made unsupervised learning difficult and a separate language-specific rule based method had to be used to constrain this space  Other proposed models include an iterative split-merge algorithm BIBREF18 and a graph-partitioning based approach BIBREF1   marquez2008semantic provide a good overview of the supervised SRL systems 
Conclusions
We propose a Bayesian model of semantic role induction (SRI) that uses crosslingual latent variables to capture role alignments in parallel corpora  The crosslingual latent variables capture correlations between roles in different languages, and regularize the parameter estimates of the monolingual models  Because this is a joint Bayesian model of multilingual SRI, we can apply the same model to a variety of training scenarios just by changing the inference procedure appropriately  We evaluate monolingual SRI with a large unlabeled dataset, bilingual SRI with a parallel corpus, bilingual SRI with annotations available for the source language, and monolingual SRI with a small labeled dataset  Increasing the amount of monolingual unlabeled data significantly improves SRI in German but not in English  Adding word alignments in parallel sentences results in small, non significant improvements, even if there is some labeled data available in the source language  This difficulty in showing the usefulness of parallel corpora for SRI may be due to the current assumptions about role alignments, which mean that only a small percentage of roles are aligned  Further analyses reveals that annotating small amounts of data can easily outperform the performance gains obtained by adding large unlabeled dataset as well as adding parallel corpora 
Future work includes training on different language pairs, on more than two languages, and with more inclusive models of role alignment 
Acknowledgments
This work was funded by the Swiss NSF grant 200021_125137 and EC FP7 grant PARLANCE 

ground_truth: Galatasaray, Fenerbahçe

Galatasaray , Fenerbahçe