prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are two datasets model is applied to?
Context 1: Introduction
“Ché saetta previsa vien più lenta ”
– Dante Alighieri, Divina Commedia, Paradiso
Antisocial behavior is a persistent problem plaguing online conversation platforms; it is both widespread BIBREF0 and potentially damaging to mental and emotional health BIBREF1, BIBREF2  The strain this phenomenon puts on community maintainers has sparked recent interest in computational approaches for assisting human moderators 
Prior work in this direction has largely focused on post-hoc identification of various kinds of antisocial behavior, including hate speech BIBREF3, BIBREF4, harassment BIBREF5, personal attacks BIBREF6, and general toxicity BIBREF7  The fact that these approaches only identify antisocial content after the fact limits their practicality as tools for assisting pre-emptive moderation in conversational domains 
Addressing this limitation requires forecasting the future derailment of a conversation based on early warning signs, giving the moderators time to potentially intervene before any harm is done (BIBREF8 BIBREF8, BIBREF9 BIBREF9, see BIBREF10 BIBREF10 for a discussion)  Such a goal recognizes derailment as emerging from the development of the conversation, and belongs to the broader area of conversational forecasting, which includes future-prediction tasks such as predicting the eventual length of a conversation BIBREF11, whether a persuasion attempt will eventually succeed BIBREF12, BIBREF13, BIBREF14, whether team discussions will eventually lead to an increase in performance BIBREF15, or whether ongoing counseling conversations will eventually be perceived as helpful BIBREF16 
Approaching such conversational forecasting problems, however, requires overcoming several inherent modeling challenges  First, conversations are dynamic and their outcome might depend on how subsequent comments interact with each other  Consider the example in Figure FIGREF2: while no individual comment is outright offensive, a human reader can sense a tension emerging from their succession (e g , dismissive answers to repeated questioning)  Thus a forecasting model needs to capture not only the content of each individual comment, but also the relations between comments  Previous work has largely relied on hand-crafted features to capture such relations—e g , similarity between comments BIBREF16, BIBREF12 or conversation structure BIBREF17, BIBREF18—, though neural attention architectures have also recently shown promise BIBREF19 
The second modeling challenge stems from the fact that conversations have an unknown horizon: they can be of varying lengths, and the to-be-forecasted event can occur at any time  So when is it a good time to make a forecast? Prior work has largely proposed two solutions, both resulting in important practical limitations  One solution is to assume (unrealistic) prior knowledge of when the to-be-forecasted event takes place and extract features up to that point BIBREF20, BIBREF8  Another compromising solution is to extract features from a fixed-length window, often at the start of the conversation BIBREF21, BIBREF15, BIBREF16, BIBREF9  Choosing a catch-all window-size is however impractical: short windows will miss information in comments they do not encompass (e g , a window of only two comments would miss the chain of repeated questioning in comments 3 through 6 of Figure FIGREF2), while longer windows risk missing the to-be-forecasted event altogether if it occurs before the end of the window, which would prevent early detection 
In this work we introduce a model for forecasting conversational events that overcomes both these inherent challenges by processing comments, and their relations, as they happen (i e , in an online fashion)  Our main insight is that models with these properties already exist, albeit geared toward generation rather than prediction: recent work in context-aware dialog generation (or “chatbots”) has proposed sequential neural models that make effective use of the intra-conversational dynamics BIBREF22, BIBREF23, BIBREF24, while concomitantly being able to process the conversation as it develops (see BIBREF25 for a survey) 
In order for these systems to perform well in the generative domain they need to be trained on massive amounts of (unlabeled) conversational data  The main difficulty in directly adapting these models to the supervised domain of conversational forecasting is the relative scarcity of labeled data: for most forecasting tasks, at most a few thousands labeled examples are available, insufficient for the notoriously data-hungry sequential neural models 
To overcome this difficulty, we propose to decouple the objective of learning a neural representation of conversational dynamics from the objective of predicting future events  The former can be pre-trained on large amounts of unsupervised data, similarly to how chatbots are trained  The latter can piggy-back on the resulting representation after fine-tuning it for classification using relatively small labeled data  While similar pre-train-then-fine-tune approaches have recently achieved state-of-the-art performance in a number of NLP tasks—including natural language inference, question answering, and commonsense reasoning (discussed in Section SECREF2)—to the best of our knowledge this is the first attempt at applying this paradigm to conversational forecasting 
To test the effectiveness of this new architecture in forecasting derailment of online conversations, we develop and distribute two new datasets  The first triples in size the highly curated `Conversations Gone Awry' dataset BIBREF9, where civil-starting Wikipedia Talk Page conversations are crowd-labeled according to whether they eventually lead to personal attacks; the second relies on in-the-wild moderation of the popular subreddit ChangeMyView, where the aim is to forecast whether a discussion will later be subject to moderator action due to “rude or hostile” behavior  In both datasets, our model outperforms existing fixed-window approaches, as well as simpler sequential baselines that cannot account for inter-comment relations  Furthermore, by virtue of its online processing of the conversation, our system can provide substantial prior notice of upcoming derailment, triggering on average 3 comments (or 3 hours) before an overtly toxic comment is posted 
To summarize, in this work we:
introduce the first model for forecasting conversational events that can capture the dynamics of a conversation as it develops;
build two diverse datasets (one entirely new, one extending prior work) for the task of forecasting derailment of online conversations;
compare the performance of our model against the current state-of-the-art, and evaluate its ability to provide early warning signs 
Our work is motivated by the goal of assisting human moderators of online communities by preemptively signaling at-risk conversations that might deserve their attention  However, we caution that any automated systems might encode or even amplify the biases existing in the training data BIBREF26, BIBREF27, BIBREF28, so a public-facing implementation would need to be exhaustively scrutinized for such biases BIBREF29 
Further Related Work
Antisocial behavior  Antisocial behavior online comes in many forms, including harassment BIBREF30, cyberbullying BIBREF31, and general aggression BIBREF32  Prior work has sought to understand different aspects of such behavior, including its effect on the communities where it happens BIBREF33, BIBREF34, the actors involved BIBREF35, BIBREF36, BIBREF37, BIBREF38 and connections to the outside world BIBREF39 
Post-hoc classification of conversations  There is a rich body of prior work on classifying the outcome of a conversation after it has concluded, or classifying conversational events after they happened  Many examples exist, but some more closely related to our present work include identifying the winner of a debate BIBREF40, BIBREF41, BIBREF42, identifying successful negotiations BIBREF21, BIBREF43, as well as detecting whether deception BIBREF44, BIBREF45, BIBREF46 or disagreement BIBREF47, BIBREF48, BIBREF49, BIBREF50, BIBREF51 has occurred 
Our goal is different because we wish to forecast conversational events before they happen and while the conversation is still ongoing (potentially allowing for interventions)  Note that some post-hoc tasks can also be re-framed as forecasting tasks (assuming the existence of necessary labels); for instance, predicting whether an ongoing conversation will eventually spark disagreement BIBREF18, rather than detecting already-existing disagreement 
Conversational forecasting  As described in Section SECREF1, prior work on forecasting conversational outcomes and events has largely relied on hand-crafted features to capture aspects of conversational dynamics  Example feature sets include statistical measures based on similarity between utterances BIBREF16, sentiment imbalance BIBREF20, flow of ideas BIBREF20, increase in hostility BIBREF8, reply rate BIBREF11 and graph representations of conversations BIBREF52, BIBREF17  By contrast, we aim to automatically learn neural representations of conversational dynamics through pre-training 
Such hand-crafted features are typically extracted from fixed-length windows of the conversation, leaving unaddressed the problem of unknown horizon  While some work has trained multiple models for different window-lengths BIBREF8, BIBREF18, they consider these models to be independent and, as such, do not address the issue of aggregating them into a single forecast (i e , deciding at what point to make a prediction)  We implement a simple sliding windows solution as a baseline (Section SECREF5) 
Pre-training for NLP  The use of pre-training for natural language tasks has been growing in popularity after recent breakthroughs demonstrating improved performance on a wide array of benchmark tasks BIBREF53, BIBREF54  Existing work has generally used a language modeling objective as the pre-training objective; examples include next-word prediction BIBREF55, sentence autoencoding, BIBREF56, and machine translation BIBREF57  BERT BIBREF58 introduces a variation on this in which the goal is to predict the next sentence in a document given the current sentence  Our pre-training objective is similar in spirit, but operates at a conversation level, rather than a document level  We hence view our objective as conversational modeling rather than (only) language modeling  Furthermore, while BERT's sentence prediction objective is framed as a multiple-choice task, our objective is framed as a generative task 
Derailment Datasets
We consider two datasets, representing related but slightly different forecasting tasks  The first dataset is an expanded version of the annotated Wikipedia conversations dataset from BIBREF9  This dataset uses carefully-controlled crowdsourced labels, strictly filtered to ensure the conversations are civil up to the moment of a personal attack  This is a useful property for the purposes of model analysis, and hence we focus on this as our primary dataset  However, we are conscious of the possibility that these strict labels may not fully capture the kind of behavior that moderators care about in practice  We therefore introduce a secondary dataset, constructed from the subreddit ChangeMyView (CMV) that does not use post-hoc annotations  Instead, the prediction task is to forecast whether the conversation will be subject to moderator action in the future 
Wikipedia data  BIBREF9's `Conversations Gone Awry' dataset consists of 1,270 conversations that took place between Wikipedia editors on publicly accessible talk pages  The conversations are sourced from the WikiConv dataset BIBREF59 and labeled by crowdworkers as either containing a personal attack from within (i e , hostile behavior by one user in the conversation directed towards another) or remaining civil throughout 
A series of controls are implemented to prevent models from picking up on trivial correlations  To prevent models from capturing topic-specific information (e g , political conversations are more likely to derail), each attack-containing conversation is paired with a clean conversation from the same talk page, where the talk page serves as a proxy for topic  To force models to actually capture conversational dynamics rather than detecting already-existing toxicity, human annotations are used to ensure that all comments preceding a personal attack are civil 
To the ends of more effective model training, we elected to expand the `Conversations Gone Awry' dataset, using the original annotation procedure  Since we found that the original data skewed towards shorter conversations, we focused this crowdsourcing run on longer conversations: ones with 4 or more comments preceding the attack  Through this additional crowdsourcing, we expand the dataset to 4,188 conversations, which we are publicly releasing as part of the Cornell Conversational Analysis Toolkit (ConvoKit) 
We perform an 80-20-20 train/dev/test split, ensuring that paired conversations end up in the same split in order to preserve the topic control  Finally, we randomly sample another 1 million conversations from WikiConv to use for the unsupervised pre-training of the generative component 
Reddit CMV data  The CMV dataset is constructed from conversations collected via the Reddit API  In contrast to the Wikipedia-based dataset, we explicitly avoid the use of post-hoc annotation  Instead, we use as our label whether a conversation eventually had a comment removed by a moderator for violation of Rule 2: “Don't be rude or hostile to other users” 
Though the lack of post-hoc annotation limits the degree to which we can impose controls on the data (e g , some conversations may contain toxic comments not flagged by the moderators) we do reproduce as many of the Wikipedia data's controls as we can  Namely, we replicate the topic control pairing by choosing pairs of positive and negative examples that belong to the same top-level post, following BIBREF12; and enforce that the removed comment was made by a user who was previously involved in the conversation  This process results in 6,842 conversations, to which we again apply a pair-preserving 80-20-20 split  Finally, we gather over 600,000 conversations that do not include any removed comment, for unsupervised pre-training 
Conversational Forecasting Model
We now describe our general model for forecasting future conversational events  Our model integrates two components: (a) a generative dialog model that learns to represent conversational dynamics in an unsupervised fashion; and (b) a supervised component that fine-tunes this representation to forecast future events  Figure FIGREF13 provides an overview of the proposed architecture, henceforth CRAFT (Conversational Recurrent Architecture for ForecasTing) 
Terminology  For modeling purposes, we treat a conversation as a sequence of $N$ comments $C = \lbrace c_1,\dots ,c_N\rbrace $  Each comment, in turn, is a sequence of tokens, where the number of tokens may vary from comment to comment  For the $n$-th comment ($1 \le n \le N)$, we let $M_n$ denote the number of tokens  Then, a comment $c_n$ can be represented as a sequence of $M_n$ tokens: $c_n = \lbrace w_1,\dots ,w_{M_n}\rbrace $ 
Generative component  For the generative component of our model, we use a hierarchical recurrent encoder-decoder (HRED) architecture BIBREF60, a modified version of the popular sequence-to-sequence (seq2seq) architecture BIBREF61 designed to account for dependencies between consecutive inputs  BIBREF23 showed that HRED can successfully model conversational context by encoding the temporal structure of previously seen comments, making it an ideal fit for our use case  Here, we provide a high-level summary of the HRED architecture, deferring deeper technical discussion to BIBREF60 and BIBREF23 
An HRED dialog model consists of three components: an utterance encoder, a context encoder, and a decoder  The utterance encoder is responsible for generating semantic vector representations of comments  It consists of a recurrent neural network (RNN) that reads a comment token-by-token, and on each token $w_m$ updates a hidden state $h^{\text{enc}}$ based on the current token and the previous hidden state:
where $f^{\text{RNN}}$ is a nonlinear gating function (our implementation uses GRU BIBREF62)  The final hidden state $h^{\text{enc}}_M$ can be viewed as a vector encoding of the entire comment 
Running the encoder on each comment $c_n$ results in a sequence of $N$ vector encodings  A second encoder, the context encoder, is then run over this sequence:
Each hidden state $h^{\text{con}}_n$ can then be viewed as an encoding of the full conversational context up to and including the $n$-th comment  To generate a response to comment $n$, the context encoding $h^{\text{con}}_n$ is used to initialize the hidden state $h^{\text{dec}}_{0}$ of a decoder RNN  The decoder produces a response token by token using the following recurrence:
where $f^{\text{out}}$ is some function that outputs a probability distribution over words; we implement this using a simple feedforward layer  In our implementation, we further augment the decoder with attention BIBREF63, BIBREF64 over context encoder states to help capture long-term inter-comment dependencies  This generative component can be pre-trained using unlabeled conversational data 
Prediction component  Given a pre-trained HRED dialog model, we aim to extend the model to predict from the conversational context whether the to-be-forecasted event will occur  Our predictor consists of a multilayer perceptron (MLP) with 3 fully-connected layers, leaky ReLU activations between layers, and sigmoid activation for output  For each comment $c_n$, the predictor takes as input the context encoding $h^{\text{con}}_n$ and forwards it through the MLP layers, resulting in an output score that is interpreted as a probability $p_{\text{event}}(c_{n+1})$ that the to-be-forecasted event will happen (e g , that the conversation will derail) 
Training the predictive component starts by initializing the weights of the encoders to the values learned in pre-training  The main training loop then works as follows: for each positive sample—i e , a conversation containing an instance of the to-be-forecasted event (e g , derailment) at comment $c_e$—we feed the context $c_1,\dots ,c_{e-1}$ through the encoder and classifier, and compute cross-entropy loss between the classifier output and expected output of 1  Similarly, for each negative sample—i e , a conversation where none of the comments exhibit the to-be-forecasted event and that ends with $c_N$—we feed the context $c_1,\dots ,c_{N-1}$ through the model and compute loss against an expected output of 0 
Note that the parameters of the generative component are not held fixed during this process; instead, backpropagation is allowed to go all the way through the encoder layers  This process, known as fine-tuning, reshapes the representation learned during pre-training to be more directly useful to prediction BIBREF55 
We implement the model and training code using PyTorch, and we are publicly releasing our implementation and the trained models together with the data as part of ConvoKit 
Forecasting Derailment
We evaluate the performance of CRAFT in the task of forecasting conversational derailment in both the Wikipedia and CMV scenarios  To this end, for each of these datasets we pre-train the generative component on the unlabeled portion of the data and fine-tune it on the labeled training split (data size detailed in Section SECREF3) 
In order to evaluate our sequential system against conversational-level ground truth, we need to aggregate comment level predictions  If any comment in the conversation triggers a positive prediction—i e , $p_{\text{event}}(c_{n+1})$ is greater than a threshold learned on the development split—then the respective conversation is predicted to derail  If this forecast is triggered in a conversation that actually derails, but before the derailment actually happens, then the conversation is counted as a true positive; otherwise it is a false positive  If no positive predictions are triggered for a conversation, but it actually derails then it counts as a false negative; if it does not derail then it is a true negative 
Fixed-length window baselines  We first seek to compare CRAFT to existing, fixed-length window approaches to forecasting  To this end, we implement two such baselines: Awry, which is the state-of-the-art method proposed in BIBREF9 based on pragmatic features in the first comment-reply pair, and BoW, a simple bag-of-words baseline that makes a prediction using TF-IDF weighted bag-of-words features extracted from the first comment-reply pair 
Online forecasting baselines  Next, we consider simpler approaches for making forecasts as the conversations happen (i e , in an online fashion)  First, we propose Cumulative BoW, a model that recomputes bag-of-words features on all comments seen thus far every time a new comment arrives  While this approach does exhibit the desired behavior of producing updated predictions for each new comment, it fails to account for relationships between comments 
This simple cumulative approach cannot be directly extended to models whose features are strictly based on a fixed number of comments, like Awry  An alternative is to use a sliding window: for a feature set based on a window of $W$ comments, upon each new comment we can extract features from a window containing that comment and the $W-1$ comments preceding it  We apply this to the Awry method and call this model Sliding Awry  For both these baselines, we aggregate comment-level predictions in the same way as in our main model 
CRAFT ablations  Finally, we consider two modified versions of the CRAFT model in order to evaluate the impact of two of its key components: (1) the pre-training step, and (2) its ability to capture inter-comment dependencies through its hierarchical memory 
To evaluate the impact of pre-training, we train the prediction component of CRAFT on only the labeled training data, without first pre-training the encoder layers with the unlabeled data  We find that given the relatively small size of labeled data, this baseline fails to successfully learn, and ends up performing at the level of random guessing  This result underscores the need for the pre-training step that can make use of unlabeled data 
To evaluate the impact of the hierarchical memory, we implement a simplified version of CRAFT where the memory size of the context encoder is zero (CRAFT $-$ CE), thus effectively acting as if the pre-training component is a vanilla seq2seq model  In other words, this model cannot capture inter-comment dependencies, and instead at each step makes a prediction based only on the utterance encoding of the latest comment 
Results  Table TABREF17 compares CRAFT to the baselines on the test splits (random baseline is 50%) and illustrates several key findings  First, we find that unsurprisingly, accounting for full conversational context is indeed helpful, with even the simple online baselines outperforming the fixed-window baselines  On both datasets, CRAFT outperforms all baselines (including the other online models) in terms of accuracy and F1  Furthermore, although it loses on precision (to CRAFT $-$ CE) and recall (to Cumulative BoW) individually on the Wikipedia data, CRAFT has the superior balance between the two, having both a visibly higher precision-recall curve and larger area under the curve (AUPR) than the baselines (Figure FIGREF20)  This latter property is particularly useful in a practical setting, as it allows moderators to tune model performance to some desired precision without having to sacrifice as much in the way of recall (or vice versa) compared to the baselines and pre-existing solutions 
Analysis
We now examine the behavior of CRAFT in greater detail, to better understand its benefits and limitations  We specifically address the following questions: (1) How much early warning does the the model provide? (2) Does the model actually learn an order-sensitive representation of conversational context?
Early warning, but how early? The recent interest in forecasting antisocial behavior has been driven by a desire to provide pre-emptive, actionable warning to moderators  But does our model trigger early enough for any such practical goals?
For each personal attack correctly forecasted by our model, we count the number of comments elapsed between the time the model is first triggered and the attack  Figure FIGREF22 shows the distribution of these counts: on average, the model warns of an attack 3 comments before it actually happens (4 comments for CMV)  To further evaluate how much time this early warning would give to the moderator, we also consider the difference in timestamps between the comment where the model first triggers and the comment containing the actual attack  Over 50% of conversations get at least 3 hours of advance warning (2 hours for CMV)  Moreover, 39% of conversations get at least 12 hours of early warning before they derail 
Does order matter? One motivation behind the design of our model was the intuition that comments in a conversation are not independent events; rather, the order in which they appear matters (e g , a blunt comment followed by a polite one feels intuitively different from a polite comment followed by a blunt one)  By design, CRAFT has the capacity to learn an order-sensitive representation of conversational context, but how can we know that this capacity is actually used? It is conceivable that the model is simply computing an order-insensitive “bag-of-features”  Neural network models are notorious for their lack of transparency, precluding an analysis of how exactly CRAFT models conversational context  Nevertheless, through two simple exploratory experiments, we seek to show that it does not completely ignore comment order 
The first experiment for testing whether the model accounts for comment order is a prefix-shuffling experiment, visualized in Figure FIGREF23  For each conversation that the model predicts will derail, let $t$ denote the index of the triggering comment, i e , the index where the model first made a derailment forecast  We then construct synthetic conversations by taking the first $t-1$ comments (henceforth referred to as the prefix) and randomizing their order  Finally, we count how often the model no longer predicts derailment at index $t$ in the synthetic conversations  If the model were ignoring comment order, its prediction should remain unchanged (as it remains for the Cumulative BoW baseline), since the actual content of the first $t$ comments has not changed (and CRAFT inference is deterministic)  We instead find that in roughly one fifth of cases (12% for CMV) the model changes its prediction on the synthetic conversations  This suggests that CRAFT learns an order-sensitive representation of context, not a mere “bag-of-features” 
To more concretely quantify how much this order-sensitive context modeling helps with prediction, we can actively prevent the model from learning and exploiting any order-related dynamics  We achieve this through another type of shuffling experiment, where we go back even further and shuffle the comment order in the conversations used for pre-training, fine-tuning and testing  This procedure preserves the model's ability to capture signals present within the individual comments processed so far, as the utterance encoder is unaffected, but inhibits it from capturing any meaningful order-sensitive dynamics  We find that this hurts the model's performance (65% accuracy for Wikipedia, 59 5% for CMV), lowering it to a level similar to that of the version where we completely disable the context encoder 
Taken together, these experiments provide evidence that CRAFT uses its capacity to model conversational context in an order-sensitive fashion, and that it makes effective use of the dynamics within  An important avenue for future work would be developing more transparent models that can shed light on exactly what kinds of order-related features are being extracted and how they are used in prediction 
Conclusions and Future Work
In this work, we introduced a model for forecasting conversational events that processes comments as they happen and takes the full conversational context into account to make an updated prediction at each step  This model fills a void in the existing literature on conversational forecasting, simultaneously addressing the dual challenges of capturing inter-comment dynamics and dealing with an unknown horizon  We find that our model achieves state-of-the-art performance on the task of forecasting derailment in two different datasets that we release publicly  We further show that the resulting system can provide substantial prior notice of derailment, opening up the potential for preemptive interventions by human moderators BIBREF65 
While we have focused specifically on the task of forecasting derailment, we view this work as a step towards a more general model for real-time forecasting of other types of emergent properties of conversations  Follow-up work could adapt the CRAFT architecture to address other forecasting tasks mentioned in Section SECREF2—including those for which the outcome is extraneous to the conversation  We expect different tasks to be informed by different types of inter-comment dynamics, and further architecture extensions could add additional supervised fine-tuning in order to direct it to focus on specific dynamics that might be relevant to the task (e g , exchange of ideas between interlocutors or stonewalling) 
With respect to forecasting derailment, there remain open questions regarding what human moderators actually desire from an early-warning system, which would affect the design of a practical system based on this work  For instance, how early does a warning need to be in order for moderators to find it useful? What is the optimal balance between precision, recall, and false positive rate at which such a system is truly improving moderator productivity rather than wasting their time through false positives? What are the ethical implications of such a system? Follow-up work could run a user study of a prototype system with actual moderators to address these questions 
A practical limitation of the current analysis is that it relies on balanced datasets, while derailment is a relatively rare event for which a more restrictive trigger threshold would be appropriate  While our analysis of the precision-recall curve suggests the system is robust across multiple thresholds ($AUPR=0 7$), additional work is needed to establish whether the recall tradeoff would be acceptable in practice 
Finally, one major limitation of the present work is that it assigns a single label to each conversation: does it derail or not? In reality, derailment need not spell the end of a conversation; it is possible that a conversation could get back on track, suffer a repeat occurrence of antisocial behavior, or any number of other trajectories  It would be exciting to consider finer-grained forecasting of conversational trajectories, accounting for the natural—and sometimes chaotic—ebb-and-flow of human interactions 
Acknowledgements  We thank Caleb Chiam, Liye Fu, Lillian Lee, Alexandru Niculescu-Mizil, Andrew Wang and Justine Zhang for insightful conversations (with unknown horizon), Aditya Jha for his great help with implementing and running the crowd-sourcing tasks, Thomas Davidson and Claire Liang for exploratory data annotation, as well as the anonymous reviewers for their helpful comments  This work is supported in part by the NSF CAREER award IIS-1750615 and by the NSF Grant SES-1741441 
Context 2: Introduction
Neural networks have been successfully used to describe images with text using sequence-to-sequence models BIBREF0  However, the results are simple and dry captions which are one or two phrases long  Humans looking at a painting see more than just objects  Paintings stimulate sentiments, metaphors and stories as well  Therefore, our goal is to have a neural network describe the painting artistically in a style of choice  As a proof of concept, we present a model which generates Shakespearean prose for a given painting as shown in Figure FIGREF1  Accomplishing this task is difficult with traditional sequence to sequence models since there does not exist a large collection of Shakespearean prose which describes paintings: Shakespeare's works describes a single painting shown in Figure FIGREF3  Fortunately we have a dataset of modern English poems which describe images BIBREF1 and line-by-line modern paraphrases of Shakespeare's plays BIBREF2  Our solution is therefore to combine two separately trained models to synthesize Shakespearean prose for a given painting 
Introduction ::: Related work
A general end-to-end approach to sequence learning BIBREF3 makes minimal assumptions on the sequence structure  This model is widely used in tasks such as machine translation, text summarization, conversational modeling, and image captioning  A generative model using a deep recurrent architecture BIBREF0 has also beeen used for generating phrases describing an image  The task of synthesizing multiple lines of poetry for a given image BIBREF1 is accomplished by extracting poetic clues from images  Given the context image, the network associates image attributes with poetic descriptions using a convolutional neural net  The poem is generated using a recurrent neural net which is trained using multi-adversarial training via policy gradient 
Transforming text from modern English to Shakespearean English using text "style transfer" is challenging  An end to end approach using a sequence-to-sequence model over a parallel text corpus BIBREF2 has been proposed based on machine translation  In the absence of a parallel text corpus, generative adversarial networks (GANs) have been used, which simultaneously train two models: a generative model which captures the data distribution, and a discriminative model which evaluates the performance of the generator  Using a target domain language model as a discriminator has also been employed BIBREF4, providing richer and more stable token-level feedback during the learning process  A key challenge in both image and text style transfer is separating content from style BIBREF5, BIBREF6, BIBREF7  Cross-aligned auto-encoder models have focused on style transfer using non-parallel text BIBREF7  Recently, a fine grained model for text style transfer has been proposed BIBREF8 which controls several factors of variation in textual data by using back-translation  This allows control over multiple attributes, such as gender and sentiment, and fine-grained control over the trade-off between content and style 
Methods
We use a total three datasets: two datasets for generating an English poem from an image, and Shakespeare plays and their English translations for text style transfer 
We train a model for generating poems from images based on two datasets BIBREF1  The first dataset consists of image and poem pairs, namely a multi-modal poem dataset (MultiM-Poem), and the second dataset is a large poem corpus, namely a uni-modal poem dataset (UniM-Poem)  The image and poem pairs are extended by adding the nearest three neighbor poems from the poem corpus without redundancy, and an extended image and poem pair dataset is constructed and denoted as MultiM-Poem(Ex)BIBREF1 
We use a collection of line-by-line modern paraphrases for 16 of Shakespeare’s plays BIBREF2, for training a style transfer network from English poems to Shakespearean prose  We use 18,395 sentences from the training data split  We keep 1,218 sentences in the validation data set and 1,462 sentences in our test set 
Methods ::: Image To Poem Actor-Critic Model
For generating a poem from images we use an existing actor-critic architecture BIBREF1  This involves 3 parallel CNNs: an object CNN, sentiment CNN, and scene CNN, for feature extraction  These features are combined with a skip-thought model which provides poetic clues, which are then fed into a sequence-to-sequence model trained by policy gradient with 2 discriminator networks for rewards  This as a whole forms a pipeline that takes in an image and outputs a poem as shown on the top left of Figure FIGREF4  A CNN-RNN generative model acts as an agent  The parameters of this agent define a policy whose execution determines which word is selected as an action  When the agent selects all words in a poem, it receives a reward  Two discriminative networks, shown on the top right of Figure FIGREF4, are defined to serve as rewards concerning whether the generated poem properly describes the input image and whether the generated poem is poetic  The goal of the poem generation model is to generate a sequence of words as a poem for an image to maximize the expected return 
Methods ::: Shakespearizing Poetic Captions
For Shakespearizing modern English texts, we experimented with various types of sequence to sequence models  Since the size of the parallel translation data available is small, we leverage a dictionary providing a mapping between Shakespearean words and modern English words to retrofit pre-trained word embeddings
Context 3: Introduction
Assembling training corpora of annotated natural language examples in specialized domains such as biomedicine poses considerable challenges  Experts with the requisite domain knowledge to perform high-quality annotation tend to be expensive, while lay annotators may not have the necessary knowledge to provide high-quality annotations  A practical approach for collecting a sufficiently large corpus would be to use crowdsourcing platforms like Amazon Mechanical Turk (MTurk)  However, crowd workers in general are likely to provide noisy annotations BIBREF0 , BIBREF1 , BIBREF2 , an issue exacerbated by the technical nature of specialized content  Some of this noise may reflect worker quality and can be modeled BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4 , but for some instances lay people may simply lack the domain knowledge to provide useful annotation 
In this paper we report experiments on the EBM-NLP corpus comprising crowdsourced annotations of medical literature BIBREF5   We operationalize the concept of annotation difficulty and show how it can be exploited during training to improve information extraction models  We then obtain expert annotations for the abstracts predicted to be most difficult, as well as for a similar number of randomly selected abstracts  The annotation of highly specialized data and the use of lay and expert annotators allow us to examine the following key questions related to lay and expert annotations in specialized domains:
Can we predict item difficulty? We define a training instance as difficult if a lay annotator or an automated model disagree on its labeling  We show that difficulty can be predicted, and that it is distinct from inter-annotator agreement  Further, such predictions can be used during training to improve information extraction models 
Are there systematic differences between expert and lay annotations? We observe decidedly lower agreement between lay workers as compared to domain experts  Lay annotations have high precision but low recall with respect to expert annotations in the new data that we collected  More generally, we expect lay annotations to be lower quality, which may translate to lower precision, recall, or both, compared to expert annotations  Can one rely solely on lay annotations? Reasonable models can be trained using lay annotations alone, but similar performance can be achieved using markedly less expert data  This suggests that the optimal ratio of expert to crowd annotations for specialized tasks will depend on the cost and availability of domain experts  Expert annotations are preferable whenever its collection is practical  But in real-world settings, a combination of expert and lay annotations is better than using lay data alone 
Does it matter what data is annotated by experts? We demonstrate that a system trained on combined data achieves better predictive performance when experts annotate difficult examples rather than instances selected at i i d  random 
Our contributions in this work are summarized as follows  We define a task difficulty prediction task and show how this is related to, but distinct from, inter-worker agreement  We introduce a new model for difficulty prediction combining learned representations induced via a pre-trained `universal' sentence encoder BIBREF6 , and a sentence encoder learned from scratch for this task  We show that predicting annotation difficulty can be used to improve the task routing and model performance for a biomedical information extraction task  Our results open up a new direction for ensuring corpus quality  We believe that item difficulty prediction will likely be useful in other, non-specialized tasks as well, and that the most effective data collection in specialized domains requires research addressing the fundamental questions we examine here 
Related Work
Crowdsourcing annotation is now a well-studied problem BIBREF7 , BIBREF0 , BIBREF1 , BIBREF2   Due to the noise inherent in such annotations, there have also been considerable efforts to develop aggregation models that minimize noise BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4  
There are also several surveys of crowdsourcing in biomedicine specifically BIBREF8 , BIBREF9 , BIBREF10   Some work in this space has contrasted model performance achieved using expert vs  crowd annotated training data BIBREF11 , BIBREF12 , BIBREF13   Dumitrache et al  Dumitrache:2018:CGT:3232718 3152889 concluded that performance is similar under these supervision types, finding no clear advantage from using expert annotators  This differs from our findings, perhaps owing to differences in design  The experts we used already hold advanced medical degrees, for instance, while those in prior work were medical students  Furthermore, the task considered here would appear to be of greater difficulty: even a system trained on $\sim $ 5k instances performs reasonably, but far from perfect  By contrast, in some of the prior work where experts and crowd annotations were deemed equivalent, a classifier trained on 300 examples can achieve very high accuracy BIBREF12  
More relevant to this paper, prior work has investigated methods for `task routing' in active learning scenarios in which supervision is provided by heterogeneous labelers with varying levels of expertise BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF14   The related question of whether effort is better spent collecting additional annotations for already labeled (but potentially noisily so) examples or novel instances has also been addressed BIBREF18   What distinguishes the work here is our focus on providing an operational definition of instance difficulty, showing that this can be predicted, and then using this to inform task routing 
Application Domain
Our specific application concerns annotating abstracts of articles that describe the conduct and results of randomized controlled trials (RCTs)  Experimentation in this domain has become easy with the recent release of the EBM-NLP BIBREF5 corpus, which includes a reasonably large training dataset annotated via crowdsourcing, and a modest test set labeled by individuals with advanced medical training  More specifically, the training set comprises 4,741 medical article abstracts with crowdsourced annotations indicating snippets (sequences) that describe the Participants (p), Interventions (i), and Outcome (o) elements of the respective RCT, and the test set is composed of 191 abstracts with p, i, o sequence annotations from three medical experts 
Table 1 shows an example of difficult and easy examples according to our definition of difficulty  The underlined text demarcates the (consensus) reference label provided by domain experts  In the difficult examples, crowd workers marked text distinct from these reference annotations; whereas in the easy cases they reproduced them with reasonable fidelity  The difficult sentences usually exhibit complicated structure and feature jargon 
An abstract may contain some `easy' and some `difficult' sentences  We thus perform our analysis at the sentence level  We split abstracts into sentences using spaCy  We excluded sentences that comprise fewer than two tokens, as these are likely an artifact of errors in sentence splitting  In total, this resulted in 57,505 and 2,428 sentences in the train and test set abstracts, respectively 
Quantifying Task Difficulty
The test set includes annotations from both crowd workers and domain experts  We treat the latter as ground truth and then define the difficulty of sentences in terms of the observed agreement between expert and lay annotators  Formally, for annotation task $t$ and instance $i$ :
$$\text{Difficulty}_{ti} = \frac{\sum _{j=1}^n{f(\text{label}_{ij}, y_i})}{n}$$   (Eq  3)
where $f$ is a scoring function that measures the quality of the label from worker $j$ for sentence $i$ , as compared to a ground truth annotation, $y_i$   The difficulty score of sentence $i$ is taken as an average over the scores for all $n$ layworkers  We use Spearmans' correlation coefficient as a scoring function  Specifically, for each sentence we create two vectors comprising counts of how many times each token was annotated by crowd and expert workers, respectively, and calculate the correlation between these  Sentences with no labels are treated as maximally easy; those with only either crowd worker or expert label(s) are assumed maximally difficult 
The training set contains only crowdsourced annotations  To label the training data, we use a 10-fold validation like setting  We iteratively retrain the LSTM-CRF-Pattern sequence tagger of Patel et al  patel2018syntactic on 9 folds of the training data and use that trained model to predict labels for the 10th  In this way we obtain predictions on the full training set  We then use predicted spans as proxy `ground truth' annotations to calculate the difficulty score of sentences as described above; we normalize these to the [ $0, 1$ ] interval  We validate this approximation by comparing the proxy scores against reference scores over the test set, the Pearson's correlation coefficients are 0 57 for Population, 0 71 for Intervention and 0 68 for Outcome 
There exist many sentences that contain neither manual nor predicted annotations  We treat these as maximally easy sentences (with difficulty scores of 0)  Such sentences comprise 51%, 42% and 36% for Population, Interventions and Outcomes data respectively, indicating that it is easier to identify sentences that have no Population spans, but harder to identify sentences that have no Interventions or Outcomes spans  This is intuitive as descriptions of the latter two tend to be more technical and dense with medical jargon 
We show the distribution of the automatically labeled scores for sentences that do contain spans in Figure 1   The mean of the Population (p) sentence scores is significantly lower than that for other types of sentences (i and o), again indicating that they are easier on average to annotate  This aligns with a previous finding that annotating Interventions and Outcomes is more difficult than annotating Participants BIBREF5  
Many sentences contain spans tagged by the LSTM-CRF-Pattern model, but missed by all crowd workers, resulting in a maximally difficult score (1)  Inspection of such sentences revealed that some are truly difficult examples, but others are tagging model errors  In either case, such sentences have confused workers and/or the model, and so we retain them all as `difficult' sentences 
Content describing the p, i and o, respectively, is quite different  As such, one sentence usually contains (at most) only one of these three content types  We thus treat difficulty prediction for the respective label types as separate tasks

ground_truth:  `Conversations Gone Awry' dataset, subreddit ChangeMyView

An expanded version of the existing 'Conversations Gone Awry' dataset and the ChangeMyView dataset, a subreddit whose only annotation is whether the conversation required action by the Reddit moderators. 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What models are used for painting embedding and what for language style transfer?
Context 2:  Incorporating this extra information improves the translation task  The large number of shared word types between the source and target sentences indicates that sharing the representation between them is beneficial 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with Attention
We use a sequence-to-sequence model which consists of a single layer unidrectional LSTM encoder and a single layer LSTM decoder and pre-trained retrofitted word embeddings shared between source and target sentences  We experimented with two different types of attention: global attention BIBREF9, in which the model makes use of the output from the encoder and decoder for the current time step only, and Bahdanau attention BIBREF10, where computing attention requires the output of the decoder from the prior time step  We found that global attention performs better in practice for our task of text style transfer 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with a Pointer Network
Since a pair of corresponding Shakespeare and modern English sentences have significant vocabulary overlap we extend the sequence-to-sequence model mentioned above using pointer networks BIBREF11 that provide location based attention and have been used to enable copying of tokens directly from the input  Moreover, there are lot of proper nouns and rare words which might not be predicted by a vanilla sequence to sequence model 
Methods ::: Shakespearizing Poetic Captions ::: Prediction
For both seq2seq models, we use the attention matrices returned at each decoder time step during inference, to compute the next word in the translated sequence if the decoder output at the current time step is the UNK token  We replace the UNKs in the target output with the highest aligned, maximum attention, source word  The seq2seq model with global attention gives the best results with an average target BLEU score of 29 65 on the style transfer dataset, compared with an average target BLEU score of 26 97 using the seq2seq model with pointer networks 
Results
We perform a qualitative analysis of the Shakespearean prose generated for the input paintings  We conducted a survey, in which we presented famous paintings including those shown in Figures FIGREF1 and FIGREF10 and the corresponding Shakespearean prose generated by the model, and asked 32 students to rate them on the basis of content, creativity and similarity to Shakespearean style on a Likert scale of 1-5  Figure FIGREF12 shows the result of our human evaluation 
The average content score across the paintings is 3 7 which demonstrates that the prose generated is relevant to the painting  The average creativity score is 3 9 which demonstrates that the model captures more than basic objects in the painting successfully using poetic clues in the scene  The average style score is 3 9 which demonstrates that the prose generated is perceived to be in the style of Shakespeare 
We also perform a quantitative analysis of style transfer by generating BLEU scores for the model output using the style transfer dataset  The variation of the BLEU scores with the source sentence lengths is shown in Figure FIGREF11  As expected, the BLEU scores decrease with increase in source sentence lengths 
Results ::: Implementation
All models were trained on Google Colab with a single GPU using Python 3 6 and Tensorflow 2 0  The number of hidden units for the encoder and decoder is 1,576 and 256 for seq2seq with global attention and seq2seq with pointer networks respectively  Adam optimizer was used with the default learning rate of 0 001  The model was trained for 25 epochs  We use pre-trained retrofitted word embeddings of dimension 192 
Results ::: Limitations
Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score  This happens when the style transfer dataset does not have similar words in the training set of sentences  A solution would be to expand the style transfer dataset, for a better representation of the poem data 
Results ::: Conclusions and Future Work
In conclusion, combining two pipelines with an intermediate representation works well in practice  We observe that a CNN-RNN based image-to-poem net combined with a seq2seq model with parallel text corpus for text style transfer synthesizes Shakespeare-style prose for a given painting  For the seq2seq model used, we observe that it performs better in practice using global attention as compared with local attention  We make our models and code publicly available BIBREF12  In future work we would like to experiment with GANs in the absence of non-parallel datasets, so that we can use varied styles for text style transfer  We would also like to experiment with cross aligned auto-encoders, which form a latent content representation, to efficiently separate style and content 
Context 3: 
Difficulty is not Worker Agreement
Our definition of difficulty is derived from agreement between expert and crowd annotations for the test data, and agreement between a predictive model and crowd annotations in the training data  It is reasonable to ask if these measures are related to inter-annotator agreement, a metric often used in language technology research to identify ambiguous or difficult items  Here we explicitly verify that our definition of difficulty only weakly correlates with inter-annotator agreement 
We calculate inter-worker agreement between crowd and expert annotators using Spearman's correlation coefficient  As shown in Table 2 , average agreement between domain experts are considerably higher than agreements between crowd workers for all three label types  This is a clear indication that the crowd annotations are noisier 
Furthermore, we compare the correlation between inter-annotator agreement and difficulty scores in the training data  Given that the majority of sentences do not contain a PICO span, we only include in these calculations those that contain a reference label  Pearson's r are 0 34, 0 30 and 0 31 for p, i and o, respectively, confirming that inter-worker agreement and our proposed difficulty score are quite distinct 
Predicting Annotation Difficulty
We treat difficulty prediction as a regression problem, and propose and evaluate neural model variants for the task  We first train RNN BIBREF19 and CNN BIBREF20 models 
We also use the universal sentence encoder (USE) BIBREF6 to induce sentence representations, and train a model using these as features  Following BIBREF6 , we then experiment with an ensemble model that combines the `universal' and task-specific representations to predict annotation difficulty  We expect these universal embeddings to capture general, high-level semantics, and the task specific representations to capture more granular information  Figure 2 depicts the model architecture  Sentences are fed into both the universal sentence encoder and, separately, a task specific neural encoder, yielding two representations  We concatenate these and pass the combined vector to the regression layer 
Experimental Setup and Results
We trained models for each label type separately  Word embeddings were initialized to 300d GloVe vectors BIBREF21 trained on common crawl data; these are fine-tuned during training  We used the Adam optimizer BIBREF22 with learning rate and decay set to 0 001 and 0 99, respectively  We used batch sizes of 16 
We used the large version of the universal sentence encoder with a transformer BIBREF23   We did not update the pretrained sentence encoder parameters during training  All hyperparamaters for all models (including hidden layers, hidden sizes, and dropout) were tuned using Vizier BIBREF24 via 10-fold cross validation on the training set maximizing for F1 
As a baseline, we also trained a linear Support-Vector Regression BIBREF25 model on $n$ -gram features ( $n$ ranges from 1 to 3) 
Table 3 reports Pearson correlation coefficients between the predictions with each of the neural models and the ground truth difficulty scores  Rows 1-4 correspond to individual models, and row 5 reports the ensemble performance  Columns correspond to label type  Results from all models outperform the baseline SVR model: Pearson's correlation coefficients range from 0 550 to 0 622  The regression correlations are the lowest 
The RNN model realizes the strongest performance among the stand-alone (non-ensemble) models, outperforming variants that exploit CNN and USE representations  Combining the RNN and USE further improves results  We hypothesize that this is due to complementary sentence information encoded in universal representations 
For all models, correlations for Intervention and Outcomes are higher than for Population, which is expected given the difficulty distributions in Figure 1   In these, the sentences are more uniformly distributed, with a fair number of difficult and easier sentences  By contrast, in Population there are a greater number of easy sentences and considerably fewer difficult sentences, which makes the difficulty ranking task particularly challenging 
Better IE with Difficulty Prediction
We next present experiments in which we attempt to use the predicted difficulty during training to improve models for information extraction of descriptions of Population, Interventions and Outcomes from medical article abstracts  We investigate two uses: (1) simply removing the most difficult sentences from the training set, and, (2) re-weighting the most difficult sentences 
We again use LSTM-CRF-Pattern as the base model and experimenting on the EBM-NLP corpus BIBREF5   This is trained on either (1) the training set with difficult sentences removed, or (2) the full training set but with instances re-weighted in proportion to their predicted difficulty score  Following BIBREF5 , we use the Adam optimizer with learning rate of 0 001, decay 0 9, batch size 20 and dropout 0 5  We use pretrained 200d GloVe vectors BIBREF21 to initialize word embeddings, and use 100d hidden char representations  Each word is thus represented with 300 dimensions in total  The hidden size is 100 for the LSTM in the character representation component, and 200 for the LSTM in the information extraction component  We train for 15 epochs, saving parameters that achieve the best F1 score on a nested development set 
Removing Difficult Examples
We first evaluate changes in performance induced by training the sequence labeling model using less data by removing difficult sentences prior to training  The hypothesis here is that these difficult instances are likely to introduce more noise than signal  We used a cross-fold approach to predict sentence difficulties, training on 9/10ths of the data and scoring the remaining 1/10th at a time  We then sorted sentences by predicted difficulty scores, and experimented with removing increasing numbers of these (in order of difficulty) prior to training the LSTM-CRF-Pattern model 
Figure 3 shows the results achieved by the LSTM-CRF-Pattern model after discarding increasing amounts of the training data: the $x$ and $y$ axes correspond to the the percentage of data removed and F1 scores, respectively  We contrast removing sentences predicted to be difficult with removing them (a) randomly (i i d ), and, (b) in inverse order of predicted inter-annotator agreement  The agreement prediction model is trained exactly the same like difficult prediction model, with simply changing the difficult score to annotation agreement  F1 scores actually improve (marginally) when we remove the most difficult sentences, up until we drop 4% of the data for Population and Interventions, and 6% for Outcomes  Removing training points at i i d  random degrades performance, as expected  Removing sentences in order of disagreement seems to have similar effect as removing them by difficulty score when removing small amount of the data, but the F1 scores drop much faster when removing more data  These findings indicate that sentences predicted to be difficult are indeed noisy, to the extent that they do not seem to provide the model useful signal 
Re-weighting by Difficulty
We showed above that removing a small number of the most difficult sentences does not harm, and in fact modestly improves, medical IE model performance  However, using the available data we are unable to test if this will be useful in practice, as we would need additional data to determine how many difficult sentences should be dropped 
We instead explore an alternative, practical means of exploiting difficulty predictions: we re-weight sentences during training inversely to their predicted difficulty  Formally, we weight sentence $i$ with difficulty scores above $\tau $ according to: $1-a\cdot (d_i-\tau )/(1-\tau )$ , where $d_i$ is the difficulty score for sentence $i$ , and $a$ is a parameter codifying the minimum weight value  We set $\tau $ to 0 8 so as to only re-weight sentences with difficulty in the top 20th percentile, and we set $a$ to 0 5  The re-weighting is equivalent to down-sampling the difficult sentences  LSTM-CRF-Pattern is our base model 
Table 4 reports the precision, recall and F1 achieved both with and without sentence re-weighting  Re-weighting improves all metrics modestly but consistently  All F1 differences are statistically significant under a sign test ( $p<0 01$ )  The model with best precision is different for Patient, Intervention and Outcome labels

ground_truth: generating a poem from images we use an existing actor-critic architecture, various types of sequence to sequence models

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How much data is needed to train the task-specific encoder?
Context 3:  However re-weighting by difficulty does consistently yield the best recall for all three extraction types, with the most notable improvement for i and o, where recall improved by 10 percentage points  This performance increase translated to improvements in F1 across all types, as compared to the base model and to re-weighting by agreement 
Involving Expert Annotators
The preceding experiments demonstrate that re-weighting difficult sentences annotated by the crowd generally improves the extraction models  Presumably the performance is influenced by the annotation quality 
We now examine the possibility that the higher quality and more consistent annotations of domain experts on the difficult instances will benefit the extraction model  This simulates an annotation strategy in which we route difficult instances to domain experts and easier ones to crowd annotators  We also contrast the value of difficult data to that of an i i d  random sample of the same size, both annotated by experts 
Expert annotations of Random and Difficult Instances
We re-annotate by experts a subset of most difficult instances and the same number of random instances  As collecting annotations from experts is slow and expensive, we only re-annotate the difficult instances for the interventions extraction task  We re-annotate the abstracts which cover the sentences with predicted difficulty scores in the top 5 percentile  We rank the abstracts from the training set by the count of difficult sentences, and re-annotate the abstracts that contain the most difficult sentences  Constrained by time and budget, we select only 2000 abstracts for re-annotation; 1000 of these are top-ranked, and 1000 are randomly sampled  This re-annotation cost $3,000  We have released the new annotation data at: https://github com/bepnye/EBM-NLP 
Following BIBREF5 , we recruited five medical experts via Up-work with advanced medical training and strong technical reading/writing skills  The expert annotator were asked to read the entire abstract and highlight, using the BRAT toolkit BIBREF26 , all spans describing medical Interventions  Each abstract is only annotated by one expert  We examined 30 re-annotated abstracts to ensure the annotation quality before hiring the annotator 
Table 5 presents the results of LSTM-CRF-Pattern model trained on the reannotated difficult subset and the random subset  The first two rows show the results for models trained with expert annotations  The model trained on random data has a slightly better F1 than that trained on the same amount of difficult data  The model trained on random data has higher precision but lower recall 
Rows 3 and 4 list the results for models trained on the same data but with crowd annotation  Models trained with expert-annotated data are clearly superior to those trained with crowd labels with respect to F1, indicating that the experts produced higher quality annotations  For crowdsourced annotations, training the model with data sampled at i i d  random achieves 2% higher F1 than when difficult instances are used  When expert annotations are used, this difference is less than 1%  This trend in performance may be explained by differences in annotation quality: the randomly sampled set was more consistently annotated by both experts and crowd because the difficult set is harder  However, in both cases expert annotations are better, with a bigger difference between the expert and crowd models on the difficult set 
The last row is the model trained on all 5k abstracts with crowd annotations  Its F1 score is lower than either expert model trained on only 20% of data, suggesting that expert annotations should be collected whenever possible  Again the crowd model on complete data has higher precision than expert models but its recall is much lower 
Routing To Experts or Crowd
So far a system was trained on one type of data, either labeled by crowd or experts  We now examine the performance of a system trained on data that was routed to either experts or crowd annotators depending on their predicted difficult  Given the results presented so far mixing annotators may be beneficial given their respective trade-offs of precision and recall  We use the annotations from experts for an abstract if it exists otherwise use crowd annotations  The results are presented in Table 6  
Rows 1 and 2 repeat the performance of the models trained on difficult subset and random subset with expert annotations only respectively  The third row is the model trained by combining difficult and random subsets with expert annotations  There are around 250 abstracts in the overlap of these two sets, so there are total 1 75k abstracts used for training the D+R model  Rows 4 to 6 are the models trained on all 5k abstracts with mixed annotations, where Other means the rest of the abstracts with crowd annotation only 
The results show adding more training data with crowd annotation still improves at least 1 point F1 score in all three extraction tasks  The improvement when the difficult subset with expert annotations is mixed with the remaining crowd annotation is 3 5 F1 score, much larger than when a random set of expert annotations are added  The model trained with re-annotating the difficult subset (D+Other) also outperforms the model with re-annotating the random subset (R+Other) by 2 points in F1  The model trained with re-annotating both of difficult and random subsets (D+R+Other), however, achieves only marginally higher F1 than the model trained with the re-annotated difficult subset (D+Other)  In sum, the results clearly indicate that mixing expert and crowd annotations leads to better models than using solely crowd data, and better than using expert data alone  More importantly, there is greater gain in performance when instances are routed according to difficulty, as compared to randomly selecting the data for expert annotators  These findings align with our motivating hypothesis that annotation quality for difficult instances is important for final model performance  They also indicate that mixing annotations from expert and crowd could be an effective way to achieve acceptable model performance given a limited budget 
How Many Expert Annotations?
We established that crowd annotation are still useful in supplementing expert annotations for medical IE  Obtaining expert annotations for the one thousand most difficult instances greatly improved the model performance  However the choice of how many difficult instances to annotate was an uninformed choice  Here we check if less expert data would have yielded similar gains  Future work will need to address how best to choose this parameter for a routing system 
We simulate a routing scenario in which we send consecutive batches of the most difficult examples to the experts for annotation  We track changes in performance as we increase the number of most-difficult-articles sent to domain experts  As shown in Figure 4 , adding expert annotations for difficult articles consistently increases F1 scores  The performance gain is mostly from increased recall; the precision changes only a bit with higher quality annotation  This observation implies that crowd workers often fail to mark target tokens, but do not tend to produce large numbers of false positives  We suspect such failures to identify relevant spans/tokens are due to insufficient domain knowledge possessed by crowd workers 
The F1 score achieved after re-annotating the 600 most-difficult articles reaches 68 1%, which is close to the performance when re-annotating 1000 random articles  This demonstrates the effectiveness of recognizing difficult instances  The trend when we use up all expert data is still upward, so adding even more expert data is likely to further improve performance  Unfortunately we exhausted our budget and were not able to obtain additional expert annotations  It is likely that as the size of the expert annotations increases, the value of crowd annotations will diminish  This investigation is left for future work 
Conclusions
We have introduced the task of predicting annotation difficulty for biomedical information extraction (IE)  We trained neural models using different learned representations to score texts in terms of their difficulty  Results from all models were strong with Pearson’s correlation coefficients higher than 0 45 in almost all evaluations, indicating the feasibility of this task  An ensemble model combining universal and task specific feature sentence vectors yielded the best results 
Experiments on biomedical IE tasks show that removing up to $\sim $ 10% of the sentences predicted to be most difficult did not decrease model performance, and that re-weighting sentences inversely to their difficulty score during training improves predictive performance  Simulations in which difficult examples are routed to experts and other instances to crowd annotators yields the best results, outperforming the strategy of randomly selecting data for expert annotation, and substantially improving upon the approach of relying exclusively on crowd annotations  In future work, routing strategies based on instance difficulty could be further investigated for budget-quality trade-off 
Acknowledgements
This work has been partially supported by NSF1748771 grant  Wallace was support in part by NIH/NLM R01LM012086 

ground_truth: 57,505 sentences

57,505 sentences