prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What baselines do they compare with?
Context 1: Introduction
There exists a class of language construction known as pun in natural language texts and utterances, where a certain word or other lexical items are used to exploit two or more separate meanings  It has been shown that understanding of puns is an important research question with various real-world applications, such as human-computer interaction BIBREF0 , BIBREF1 and machine translation BIBREF2   Recently, many researchers show their interests in studying puns, like detecting pun sentences BIBREF3 , locating puns in the text BIBREF4 , interpreting pun sentences BIBREF5 and generating sentences containing puns BIBREF6 , BIBREF7 , BIBREF8   A pun is a wordplay in which a certain word suggests two or more meanings by exploiting polysemy, homonymy, or phonological similarity to another sign, for an intended humorous or rhetorical effect  Puns can be generally categorized into two groups, namely heterographic puns (where the pun and its latent target are phonologically similar) and homographic puns (where the two meanings of the pun reflect its two distinct senses) BIBREF9   Consider the following two examples:
The first punning joke exploits the sound similarity between the word “propane" and the latent target “profane", which can be categorized into the group of heterographic puns  Another categorization of English puns is homographic pun, exemplified by the second instance leveraging distinct senses of the word “gut" 
Pun detection is the task of detecting whether there is a pun residing in the given text  The goal of pun location is to find the exact word appearing in the text that implies more than one meanings  Most previous work addresses such two tasks separately and develop separate systems BIBREF10 , BIBREF5   Typically, a system for pun detection is built to make a binary prediction on whether a sentence contains a pun or not, where all instances (with or without puns) are taken into account during training  For the task of pun location, a separate system is used to make a single prediction as to which word in the given sentence in the text that trigger more than one semantic interpretations of the text, where the training data involves only sentences that contain a pun  Therefore, if one is interested in solving both problems at the same time, a pipeline approach that performs pun detection followed by pun location can be used 
Compared to the pipeline methods, joint learning has been shown effective BIBREF11 , BIBREF12 since it is able to reduce error propagation and allows information exchange between tasks which is potentially beneficial to all the tasks  In this work, we demonstrate that the detection and location of puns can be jointly addressed by a single model  The pun detection and location tasks can be combined as a sequence labeling problem, which allows us to jointly detect and locate a pun in a sentence by assigning each word a tag  Since each context contains a maximum of one pun BIBREF9 , we design a novel tagging scheme to capture this structural constraint  Statistics on the corpora also show that a pun tends to appear in the second half of a context  To capture such a structural property, we also incorporate word position knowledge into our structured prediction model  Experiments on the benchmark datasets show that detection and location tasks can reinforce each other, leading to new state-of-the-art performance on these two tasks  To the best of our knowledge, this is the first work that performs joint detection and location of English puns by using a sequence labeling approach 
Problem Definition
We first design a simple tagging scheme consisting of two tags { INLINEFORM0 }:
INLINEFORM0 tag means the current word is not a pun 
INLINEFORM0 tag means the current word is a pun 
If the tag sequence of a sentence contains a INLINEFORM0 tag, then the text contains a pun and the word corresponding to INLINEFORM1 is the pun 
The contexts have the characteristic that each context contains a maximum of one pun BIBREF9   In other words, there exists only one pun if the given sentence is detected as the one containing a pun  Otherwise, there is no pun residing in the text  To capture this interesting property, we propose a new tagging scheme consisting of three tags, namely { INLINEFORM0 } 
INLINEFORM0 tag indicates that the current word appears before the pun in the given context 
INLINEFORM0 tag highlights the current word is a pun 
INLINEFORM0 tag indicates that the current word appears after the pun 
We empirically show that the INLINEFORM0 scheme can guarantee the context property that there exists a maximum of one pun residing in the text 
Given a context from the training set, we will be able to generate its corresponding gold tag sequence using a deterministic procedure  Under the two schemes, if a sentence does not contain any puns, all words will be tagged with INLINEFORM0 or INLINEFORM1 , respectively  Exemplified by the second sentence “Some diets cause a gut reaction," the pun is given as “gut " Thus, under the INLINEFORM2 scheme, it should be tagged with INLINEFORM3 , while the words before it are assigned with the tag INLINEFORM4 and words after it are with INLINEFORM5 , as illustrated in Figure FIGREF8   Likewise, the INLINEFORM6 scheme tags the word “gut" with INLINEFORM7 , while other words are tagged with INLINEFORM8   Therefore, we can combine the pun detection and location tasks into one problem which can be solved by the sequence labeling approach 
Model
Neural models have shown their effectiveness on sequence labeling tasks BIBREF13 , BIBREF14 , BIBREF15   In this work, we adopt the bidirectional Long Short Term Memory (BiLSTM) BIBREF16 networks on top of the Conditional Random Fields BIBREF17 (CRF) architecture to make labeling decisions, which is one of the classical models for sequence labeling  Our model architecture is illustrated in Figure FIGREF8 with a running example  Given a context/sentence INLINEFORM0 where INLINEFORM1 is the length of the context, we generate the corresponding tag sequence INLINEFORM2 based on our designed tagging schemes and the original annotations for pun detection and location provided by the corpora  Our model is then trained on pairs of INLINEFORM3  
Input  The contexts in the pun corpus hold the property that each pun contains exactly one content word, which can be either a noun, a verb, an adjective, or an adverb  To capture this characteristic, we consider lexical features at the character level  Similar to the work of BIBREF15 , the character embeddings are trained by the character-level LSTM networks on the unannotated input sequences  Nonlinear transformations are then applied to the character embeddings by highway networks BIBREF18 , which map the character-level features into different semantic spaces 
We also observe that a pun tends to appear at the end of a sentence  Specifically, based on the statistics, we found that sentences with a pun that locate at the second half of the text account for around 88% and 92% in homographic and heterographic datasets, respectively  We thus introduce a binary feature that indicates if a word is located at the first or the second half of an input sentence to capture such positional information  A binary indicator can be mapped to a vector representation using a randomly initialized embedding table BIBREF19 , BIBREF20   In this work, we directly adopt the value of the binary indicator as part of the input 
The concatenation of the transformed character embeddings, the pre-trained word embeddings BIBREF21 , and the position indicators are taken as input of our model 
Tagging  The input is then fed into a BiLSTM network, which will be able to capture contextual information  For a training instance INLINEFORM0 , we suppose the output by the word-level BiLSTM is INLINEFORM1   The CRF layer is adopted to capture label dependencies and make final tagging decisions at each position, which has been included in many state-of-the-art sequence labeling models BIBREF14 , BIBREF15   The conditional probability is defined as:
where INLINEFORM0 is a set of all possible label sequences consisting of tags from INLINEFORM1 (or INLINEFORM2 ), INLINEFORM3 and INLINEFORM4 are weight and bias parameters corresponding to the label pair INLINEFORM5   During training, we minimize the negative log-likelihood summed over all training instances:
where INLINEFORM0 refers to the INLINEFORM1 -th instance in the training set  During testing, we aim to find the optimal label sequence for a new input INLINEFORM2 :
This search process can be done efficiently using the Viterbi algorithm 
Datasets and Settings
We evaluate our model on two benchmark datasets BIBREF9   The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun  The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun  We notice there is no standard splitting information provided for both datasets  Thus we apply 10-fold cross validation  To make direct comparisons with prior studies, following BIBREF4 , we accumulated the predictions for all ten folds and calculate the scores in the end 
For each fold, we randomly select 10% of the instances from the training set for development  Word embeddings are initialized with the 100-dimensional Glove BIBREF21   The dimension of character embeddings is 30 and they are randomly initialized, which can be fine tuned during training  The pre-trained word embeddings are not updated during training  The dimensions of hidden vectors for both char-level and word-level LSTM units are set to 300  We adopt stochastic gradient descent (SGD) BIBREF26 with a learning rate of 0 015 
For the pun detection task, if the predicted tag sequence contains at least one INLINEFORM0 tag, we regard the output (i e , the prediction of our pun detection model) for this task as true, otherwise false  For the pun location task, a predicted pun is regarded as correct if and only if it is labeled as the gold pun in the dataset  As to pun location, to make fair comparisons with prior studies, we only consider the instances that are labeled as the ones containing a pun  We report precision, recall and INLINEFORM1 score in Table TABREF11   A list of prior works that did not employ joint learning are also shown in the first block of Table TABREF11  
Results
We also implemented a baseline model based on conditional random fields (CRF), where features like POS tags produced by the Stanford POS tagger BIBREF27 , n-grams, label transitions, word suffixes and relative position to the end of the text are considered  We can see that our model with the INLINEFORM0 tagging scheme yields new state-of-the-art INLINEFORM1 scores on pun detection and competitive results on pun location, compared to baselines that do not adopt joint learning in the first block  For location on heterographic puns, our model's performance is slightly lower than the system of BIBREF25 , which is a rule-based locator  Compared to CRF, we can see that our model, either with the INLINEFORM2 or the INLINEFORM3 scheme, yields significantly higher recall on both detection and location tasks, while the precisions are relatively close  This demonstrates the effectiveness of BiLSTM, which learns the contextual features of given texts – such information appears to be helpful in recalling more puns 
Compared to the INLINEFORM0 scheme, the INLINEFORM1 tagging scheme is able to yield better performance on these two tasks  After studying outputs from these two approaches, we found that one leading source of error for the INLINEFORM2 approach is that there exist more than one words in a single instance that are assigned with the INLINEFORM3 tag  However, according to the description of pun in BIBREF9 , each context contains a maximum of one pun  Thus, such a useful structural constraint is not well captured by the simple approach based on the INLINEFORM4 tagging scheme  On the other hand, by applying the INLINEFORM5 tagging scheme, such a constraint is properly captured in the model  As a result, the results for such a approach are significantly better than the approach based on the INLINEFORM6 tagging scheme, as we can observe from the table  Under the same experimental setup, we also attempted to exclude word position features  Results are given by INLINEFORM7 - INLINEFORM8   It is expected that the performance of pun location drops, since such position features are able to capture the interesting property that a pun tends to appear in the second half of a sentence  While such knowledge is helpful for the location task, interestingly, a model without position knowledge yields improved performance on the pun detection task  One possible reason is that detecting whether a sentence contains a pun is not concerned with such word position information 
Additionally, we conduct experiments over sentences containing a pun only, namely 1,607 and 1,271 instances from homographic and heterographic pun corpora separately  It can be regarded as a “pipeline” method where the classifier for pun detection is regarded as perfect  Following the prior work of BIBREF4 , we apply 10-fold cross validation  Since we are given that all input sentences contain a pun, we only report accumulated results on pun location, denoted as Pipeline in Table TABREF11   Compared with our approaches, the performance of such an approach drops significantly  On the other hand, such a fact demonstrates that the two task, detection and location of puns, can reinforce each other  These figures demonstrate the effectiveness of our sequence labeling method to detect and locate English puns in a joint manner 
Error Analysis
We studied the outputs from our system and make some error analysis  We found the errors can be broadly categorized into several types, and we elaborate them here  1) Low word coverage: since the corpora are relatively small, there exist many unseen words in the test set  Learning the representations of such unseen words is challenging, which affects the model's performance  Such errors contribute around 40% of the total errors made by our system  2) Detection errors: we found many errors are due to the model's inability to make correct pun detection  Such inability harms both pun detection and pun location  Although our approach based on the INLINEFORM0 tagging scheme yields relatively higher scores on the detection task, we still found that 40% of the incorrectly predicted instances fall into this group  3) Short sentences: we found it was challenging for our model to make correct predictions when the given text is short  Consider the example “Superglue! Tom rejoined," here the word rejoined is the corresponding pun  However, it would be challenging to figure out the pun with such limited contextual information 
Related Work
Most existing systems address pun detection and location separately  BIBREF22 applied word sense knowledge to conduct pun detection  BIBREF24 trained a bidirectional RNN classifier for detecting homographic puns  Next, a knowledge-based approach is adopted to find the exact pun  Such a system is not applicable to heterographic puns  BIBREF28 applied Google n-gram and word2vec to make decisions  The phonetic distance via the CMU Pronouncing Dictionary is computed to detect heterographic puns  BIBREF10 used the hidden Markov model and a cyclic dependency network with rich features to detect and locate puns  BIBREF23 used a supervised approach to pun detection and a weakly supervised approach to pun location based on the position within the context and part of speech features  BIBREF25 proposed a rule-based system for pun location that scores candidate words according to eleven simple heuristics  Two systems are developed to conduct detection and location separately in the system known as UWAV BIBREF3   The pun detector combines predictions from three classifiers  The pun locator considers word2vec similarity between every pair of words in the context and position to pinpoint the pun  The state-of-the-art system for homographic pun location is a neural method BIBREF4 , where the word senses are incorporated into a bidirectional LSTM model  This method only supports the pun location task on homographic puns  Another line of research efforts related to this work is sequence labeling, such as POS tagging, chunking, word segmentation and NER  The neural methods have shown their effectiveness in this task, such as BiLSTM-CNN BIBREF13 , GRNN BIBREF29 , LSTM-CRF BIBREF30 , LSTM-CNN-CRF BIBREF14 , LM-LSTM-CRF BIBREF15  
In this work, we combine pun detection and location tasks as a single sequence labeling problem  Inspired by the work of BIBREF15 , we also adopt a LSTM-CRF with character embeddings to make labeling decisions 
Conclusion
In this paper, we propose to perform pun detection and location tasks in a joint manner from a sequence labeling perspective  We observe that each text in our corpora contains a maximum of one pun  Hence, we design a novel tagging scheme to incorporate such a constraint  Such a scheme guarantees that there is a maximum of one word that will be tagged as a pun during the testing phase  We also found the interesting structural property such as the fact that most puns tend to appear at the second half of the sentences can be helpful for such a task, but was not explored in previous works  Furthermore, unlike many previous approaches, our approach, though simple, is generally applicable to both heterographic and homographic puns  Empirical results on the benchmark datasets prove the effectiveness of the proposed approach that the two tasks of pun detection and location can be addressed by a single model from a sequence labeling perspective 
Future research includes the investigations on how to make use of richer semantic and linguistic information for detection and location of puns  Research on puns for other languages such as Chinese is still under-explored, which could also be an interesting direction for our future studies 
Acknowledgments
We would like to thank the three anonymous reviewers for their thoughtful and constructive comments  This work is supported by Singapore Ministry of Education Academic Research Fund (AcRF) Tier 2 Project MOE2017-T2-1-156, and is partially supported by SUTD project PIE-SGP-AI-2018-01 
Context 2: Introduction
The task of speculation detection and scope resolution is critical in distinguishing factual information from speculative information  This has multiple use-cases, like systems that determine the veracity of information, and those that involve requirement analysis  This task is particularly important to the biomedical domain, where patient reports and medical articles often use this feature of natural language  This task is commonly broken down into two subtasks: the first subtask, speculation cue detection, is to identify the uncertainty cue in a sentence, while the second subtask: scope resolution, is to identify the scope of that cue  For instance, consider the example:
It might rain tomorrow 
The speculation cue in the sentence above is ‘might’ and the scope of the cue ‘might’ is ‘rain tomorrow’  Thus, the speculation cue is the word that expresses the speculation, while the words affected by the speculation are in the scope of that cue 
This task was the CoNLL-2010 Shared Task (BIBREF0), which had 3 different subtasks  Task 1B was speculation cue detection on the BioScope Corpus, Task 1W was weasel identification from Wikipedia articles, and Task 2 was speculation scope resolution from the BioScope Corpus  For each task, the participants were provided the train and test set, which is henceforth referred to as Task 1B CoNLL and Task 2 CoNLL throughout this paper 
For our experimentation, we use the sub corpora of the BioScope Corpus (BIBREF1), namely the BioScope Abstracts sub corpora, which is referred to as BA, and the BioScope Full Papers sub corpora, which is referred to as BF  We also use the SFU Review Corpus (BIBREF2), which is referred to as SFU 
This subtask of natural language processing, along with another similar subtask, negation detection and scope resolution, have been the subject of a body of work over the years  The approaches used to solve them have evolved from simple rule-based systems (BIBREF3) based on linguistic information extracted from the sentences, to modern deep-learning based methods  The Machine Learning techniques used varied from Maximum Entropy Classifiers (BIBREF4) to Support Vector Machines (BIBREF5,BIBREF6,BIBREF7,BIBREF8), while the deep learning approaches included Recursive Neural Networks (BIBREF9,BIBREF10), Convolutional Neural Networks (BIBREF11) and most recently transfer learning-based architectures like Bidirectional Encoder Representation from Transformers (BERT) (BIBREF12)  Figures FIGREF1 and FIGREF1 contain a summary of the papers addressing speculation detection and scope resolution (BIBREF13, BIBREF5, BIBREF9, BIBREF3, BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF6, BIBREF11, BIBREF18, BIBREF10, BIBREF19, BIBREF7, BIBREF4, BIBREF8) 
Inspired by the most recent approach of applying BERT to negation detection and scope resolution (BIBREF12), we take this approach one step further by performing a comparative analysis of three popular transformer-based architectures: BERT (BIBREF20), XLNet (BIBREF21) and RoBERTa (BIBREF22), applied to speculation detection and scope resolution  We evaluate the performance of each model across all datasets via the single dataset training approach, and report all scores including inter-dataset scores (i e  train on one dataset, evaluate on another) to test the generalizability of the models  This approach outperforms all existing systems on the task of speculation detection and scope resolution  Further, we jointly train on multiple datasets and obtain improvements over the single dataset training approach on most datasets 
Contrary to results observed on benchmark GLUE tasks, we observe XLNet consistently outperforming RoBERTa  To confirm this observation, we apply these models to the negation detection and scope resolution task, and observe a continuity in this trend, reporting state-of-the-art results on three of four datasets on the negation scope resolution task 
The rest of the paper is organized as follows: In Section 2, we provide a detailed description of our methodology and elaborate on the experimentation details  In Section 3, we present our results and analysis on the speculation detection and scope resolution task, using the single dataset and the multiple dataset training approach  In Section 4, we show the results of applying XLNet and RoBERTa on negation detection and scope resolution and propose a few reasons to explain why XLNet performs better than RoBERTa  Finally, the future scope and conclusion is mentioned in Section 5 
Methodology and Experimental Setup
We use the methodology by Khandelwal and Sawant (BIBREF12), and modify it to support experimentation with multiple models 
For Speculation Cue Detection:
Input Sentence: It might rain tomorrow 
True Labels: Not-A-Cue, Cue, Not-A-Cue, Not-A-Cue 
First, this sentence is preprocessed to get the target labels as per the following annotation schema:
1 – Normal Cue 2 – Multiword Cue 3 – Not a cue 4 – Pad token
Thus, the preprocessed sequence is as follows:
Input Sentence: [It, might, rain, tomorrow]
True Labels: [3,1,3,3]
Then, we preprocess the input using the tokenizer for the model being used (BERT, XLNet or RoBERTa): splitting each word into one or more tokens, and converting each token to it’s corresponding tokenID, and padding it to the maximum input length of the model  Thus,
Input Sentence: [wtt(It), wtt(might), wtt(rain), wtt(tom), wtt(## or), wtt(## row), wtt(〈pad 〉),wtt(〈pad 〉)   ]
True Labels: [3,1,3,3,3,3,4,4,4,4,   ]
The word ‘tomorrow' has been split into 3 tokens, ‘tom', ‘##or' and ‘##row'  The function to convert the word to tokenID is represented by wtt 
For Speculation Scope Resolution:
If a sentence has multiple cues, each cue's scope will be resolved individually 
Input Sentence: It might rain tomorrow 
True Labels: Out-Of-Scope, Out-Of-Scope, In-Scope, In-Scope 
First, this sentence is preprocessed to get the target labels as per the following annotation schema:
0 – Out-Of-Scope 1 – In-Scope
Thus, the preprocessed sequence is as follows:
True Scope Labels: [0,0,1,1]
As for cue detection, we preprocess the input using the tokenizer for the model being used  Additionally, we need to indicate which cue's scope we want to find in the input sentence  We do this by inserting a special token representing the token type (according to the cue detection annotation schema) before the cue word whose scope is being resolved  Here, we want to find the scope of the cue ‘might'  Thus,
Input Sentence: [wtt(It), wtt(〈token[1]〉), wtt(might), wtt(rain), wtt(tom), wtt(##or), wtt(##row), wtt(〈pad〉), wtt(〈pad〉)   ]
True Scope Labels: [0,0,1,1,1,1,0,0,0,0,   ]
Now, the preprocessed input for cue detection and similarly for scope detection is fed as input to our model as follows:
X = Model (Input)
Y = W*X + b
The W matrix is a matrix of size n_hidden x num_classes (n_hidden is the size of the representation of a token within the model)  These logits are fed to the loss function 
We use the following variants of each model:
BERT: bert-base-uncaseds3 amazonaws com/models huggingface co/bert/bert-base-uncased tar gz (The model used by BIBREF12)
RoBERTa: roberta-bases3 amazonaws com/models huggingface co/bert/roberta-base-pytorch_model bin (RoBERTa-base does not have an uncased variant)
XLNet: xlnet-base-caseds3 amazonaws com/models huggingface co/bert/xlnet-base-cased-pytorch_model bin (XLNet-base does not have an uncased variant)
The output of the model is a vector of probabilities per token  The loss is calculated for each token, by using the output vector and the true label for that token  We use class weights for the loss function, setting the weight for label 4 to 0 and all other labels to 1 (for cue detection only) to avoid training on padding token’s output 
We calculate the scores (Precision, Recall, F1) for the model per word of the input sentence, not per token that was fed to the model, as the tokens could be different for different models leading to inaccurate scores  For the above example, we calculate the output label for the word ‘tomorrow', not for each token it was split into (‘tom', ‘##or' and ‘##row')  To find the label for each word from the tokens it was split into, we experiment with 2 methods:
Average: We average the output vectors (softmax probabilities) for each token that the word was split into by the model's tokenizer  In the example above, we average the output of ‘tom', ‘##or' and ‘##row' to get the output for ‘tomorrow'  Then, we take an argmax over the resultant vector  This is then compared with the true label for the original word 
First Token: Here, we only consider the first token's probability vector (among all tokens the word was split into) as the output for that word, and get the label by an argmax over this vector  In the example above, we would consider the output vector corresponding to the token ‘tom' as the output for the word ‘tomorrow' 
For cue detection, the results are reported for the Average method only, while we report the scores for both Average and First Token for Scope Resolution 
For fair comparison, we use the same hyperparameters for the entire architecture for all 3 models  Only the tokenizer and the model are changed for each model  All other hyperparameters are kept same  We finetune the models for 60 epochs, using early stopping with a patience of 6 on the F1 score (word level) on the validation dataset  We use an initial learning rate of 3e-5, with a batch size of 8  We use the Categorical Cross Entropy loss function 
We use the Huggingface’s Pytorch Transformer library (BIBREF23) for the models and train all our models on Google Colaboratory 
Results: Speculation Cue Detection and Scope Resolution
We use a default train-validation-test split of 70-15-15 for each dataset  For the speculation detection and scope resolution subtasks using single-dataset training, we report the results as an average of 5 runs of the model  For training the model on multiple datasets, we perform a 70-15-15 split of each training dataset, after which the train and validation part of the individual datasets are merged while the scores are reported for the test part of the individual datasets, which is not used for training or validation  We report the results as an average of 3 runs of the model  Figure FIGREF8 contains results for speculation cue detection and scope resolution when trained on a single dataset  All models perform the best when trained on the same dataset as they are evaluated on, except for BF, which gets the best results when trained on BA  This is because of the transfer learning capabilities of the models and the fact that BF is a smaller dataset than BA (BF: 2670 sentences, BA: 11871 sentences)  For speculation cue detection, there is lesser generalizability for models trained on BF or BA, while there is more generalizability for models trained on SFU  This could be because of the different nature of the biomedical domain 
Figure FIGREF11 contains the results for speculation detection and scope resolution for models trained jointly on multiple datasets  We observe that training on multiple datasets helps the performance of all models on each dataset, as the quantity of data available to train the model increases  We also observe that XLNet consistently outperforms BERT and RoBERTa  To confirm this observation, we apply the 2 models to the related task of negation detection and scope resolution
Negation Cue Detection and Scope Resolution
We use a default train-validation-test split of 70-15-15 for each dataset, and use all 4 datasets (BF, BA, SFU and Sherlock)  The results for BERT are taken from BIBREF12  The results for XLNet and RoBERTa are averaged across 5 runs for statistical significance  Figure FIGREF14 contains results for negation cue detection and scope resolution  We report state-of-the-art results on negation scope resolution on BF, BA and SFU datasets  Contrary to popular opinion, we observe that XLNet is better than RoBERTa for the cue detection and scope resolution tasks  A few possible reasons for this trend are:
Domain specificity, as both negation and speculation are closely related subtasks  Further experimentation on different tasks is needed to verify this 
Most benchmark tasks are sentence classification tasks, whereas the subtasks we experiment on are sequence labelling tasks  Given the pre-training objective of XLNet (training on permutations of the input), it may be able to capture long-term dependencies better, essential for sequence labelling tasks 
We work with the base variants of the models, while most results are reported with the large variants of the models 
Conclusion and Future Scope
In this paper, we expanded on the work of Khandelwal and Sawant (BIBREF12) by looking at alternative transfer-learning models and experimented with training on multiple datasets  On the speculation detection task, we obtained a gain of 0 42 F1 points on BF, 1 98 F1 points on BA and 0 29 F1 points on SFU, while on the scope resolution task, we obtained a gain of 8 06 F1 points on BF, 4 27 F1 points on BA and 11 87 F1 points on SFU, when trained on a single dataset  While training on multiple datasets, we observed a gain of 10 6 F1 points on BF and 1 94 F1 points on BA on the speculation detection task and 2 16 F1 points on BF and 0 25 F1 points on SFU on the scope resolution task over the single dataset training approach  We thus significantly advance the state-of-the-art for speculation detection and scope resolution  On the negation scope resolution task, we applied the XLNet and RoBERTa and obtained a gain of 3 16 F1 points on BF, 0 06 F1 points on BA and 0 3 F1 points on SFU  Thus, we demonstrated the usefulness of transformer-based architectures in the field of negation and speculation detection and scope resolution  We believe that a larger and more general dataset would go a long way in bolstering future research and would help create better systems that are not domain-specific 
Context 3: Introduction
The explosion of available scientific articles in the Biomedical domain has led to the rise of Biomedical Information Extraction (BioIE)  BioIE systems aim to extract information from a wide spectrum of articles including medical literature, biological literature, electronic health records, etc  that can be used by clinicians and researchers in the field  Often the outputs of BioIE systems are used to assist in the creation of databases, or to suggest new paths for research  For example, a ranked list of interacting proteins that are extracted from biomedical literature, but are not present in existing databases, can allow researchers to make informed decisions about which protein/gene to study further  Interactions between drugs are necessary for clinicians who simultaneously administer multiple drugs to their patients  A database of diseases, treatments and tests is beneficial for doctors consulting in complicated medical cases 
The main problems in BioIE are similar to those in Information Extraction:
This paper discusses, in each section, various methods that have been adopted to solve the listed problems  Each section also highlights the difficulty of Information Extraction tasks in the biomedical domain 
This paper is intended as a primer to Biomedical Information Extraction for current NLP researchers  It aims to highlight the diversity of the various techniques from Information Extraction that have been applied in the Biomedical domain  The state of biomedical text mining is reviewed regularly  For more extensive surveys, consult BIBREF0 , BIBREF1 , BIBREF2  
Named Entity Recognition and Fact Extraction
Named Entity Recognition (NER) in the Biomedical domain usually includes recognition of entities such as proteins, genes, diseases, treatments, drugs, etc  Fact extraction involves extraction of Named Entities from a corpus, usually given a certain ontology  When compared to NER in the domain of general text, the biomedical domain has some characteristic challenges:
Some of the earliest systems were heavily dependent on hand-crafted features  The method proposed in BIBREF4 for recognition of protein names in text does not require any prepared dictionary  The work gives examples of diversity in protein names and lists multiple rules depending on simple word features as well as POS tags 
BIBREF5 adopt a machine learning approach for NER  Their NER system extracts medical problems, tests and treatments from discharge summaries and progress notes  They use a semi-Conditional Random Field (semi-CRF) BIBREF6 to output labels over all tokens in the sentence  They use a variety of token, context and sentence level features  They also use some concept mapping features using existing annotation tools, as well as Brown clustering to form 128 clusters over the unlabelled data  The dataset used is the i2b2 2010 challenge dataset  Their system achieves an F-Score of 0 85  BIBREF7 is an incremental paper on NER taggers  It uses 3 types of word-representation techniques (Brown clustering, distributional clustering, word vectors) to improve performance of the NER Conditional Random Field tagger, and achieves marginal F-Score improvements 
BIBREF8 propose a boostrapping mechanism to bootstrap biomedical ontologies using NELL BIBREF9 , which uses a coupled semi-supervised bootstrapping approach to extract facts from text, given an ontology and a small number of “seed” examples for each category  This interesting approach (called BioNELL) uses an ontology of over 100 categories  In contrast to NELL, BioNELL does not contain any relations in the ontology  BioNELL is motivated by the fact that a lot of scientific literature available online is highly reliable due to peer-review  The authors note that the algorithm used by NELL to bootstrap fails in BioNELL due to ambiguities in biomedical literature, and heavy semantic drift  One of the causes for this is that often common words such as “white”, “dad”, “arm” are used as names of genes- this can easily result in semantic drift in one iteration of the bootstrapping  In order to mitigate this, they use Pointwise Mutual Information scores for corpus level statistics, which attributes a small score to common words  In addition, in contrast to NELL, BioNELL only uses high instances as seeds in the next iteration, but adds low ranking instances to the knowledge base  Since evaluation is not possible using Mechanical Turk or a small number of experts (due to the complexity of the task), they use Freebase BIBREF10 , a knowledge base that has some biomedical concepts as well  The lexicon learned using BioNELL is used to train an NER system  The system shows a very high precision, thereby showing that BioNELL learns very few ambiguous terms 
More recently, deep learning techniques have been developed to further enhance the performance of NER systems  BIBREF11 explore recurrent neural networks for the problem of NER in biomedical text 
Relation Extraction
In Biomedical Information Extraction, Relation Extraction involves finding related entities of many different kinds  Some of these include protein-protein interactions, disease-gene relations and drug-drug interactions  Due to the explosion of available biomedical literature, it is impossible for one person to extract relevant relations from published material  Automatic extraction of relations assists in the process of database creation, by suggesting potentially related entities with links to the source article  For example, a database of drug-drug interactions is important for clinicians who administer multiple drugs simultaneously to their patients- it is imperative to know if one drug will have an adverse effect on the other  A variety of methods have been developed for relation extractions, and are often inspired by Relation Extraction in NLP tasks  These include rule-based approaches, hand-crafted patterns, feature-based and kernel machine learning methods, and more recently deep learning architectures  Relation Extraction systems over Biomedical Corpora are often affected by noisy extraction of entities, due to ambiguities in names of proteins, genes, drugs etc 
BIBREF12 was one of the first large scale Information Extraction efforts to study the feasibility of extraction of protein-protein interactions (such as “protein A activates protein B") from Biomedical text  Using 8 hand-crafted regular expressions over a fixed vocabulary, the authors were able to achieve a recall of 30% for interactions present in The Dictionary of Interacting Proteins (DIP) from abstracts in Medline  The method did not differentiate between the type of relation  The reasons for the low recall were the inconsistency in protein nomenclature, information not present in the abstract, and due to specificity of the hand-crafted patterns  On a small subset of extracted relations, they found that about 60% were true interactions between proteins not present in DIP 
BIBREF13 combine sentence level relation extraction for protein interactions with corpus level statistics  Similar to BIBREF12 , they do not consider the type of interaction between proteins- only whether they interact in the general sense of the word  They also do not differentiate between genes and their protein products (which may share the same name)  They use Pointwise Mutual Information (PMI) for corpus level statistics to determine whether a pair of proteins occur together by chance or because they interact  They combine this with a confidence aggregator that takes the maximum of the confidence of the extractor over all extractions for the same protein-pair  The extraction uses a subsequence kernel based on BIBREF14   The integrated model, that combines PMI with aggregate confidence, gives the best performance  Kernel methods have widely been studied for Relation Extraction in Biomedical Literature  Common kernels used usually exploit linguistic information by utilising kernels based on the dependency tree BIBREF15 , BIBREF16 , BIBREF17  
BIBREF18 look at the extraction of diseases and their relevant genes  They use a dictionary from six public databases to annotate genes and diseases in Medline abstracts  In their work, the authors note that when both genes and diseases are correctly identified, they are related in 94% of the cases  The problem then reduces to filtering incorrect matches using the dictionary, which occurs due to false positives resulting from ambiguities in the names as well as ambiguities in abbreviations  To this end, they train a Max-Ent based NER classifier for the task, and get a 26% gain in precision over the unfiltered baseline, with a slight hit in recall  They use POS tags, expanded forms of abbreviations, indicators for Greek letters as well as suffixes and prefixes commonly used in biomedical terms 
BIBREF19 adopt a supervised feature-based approach for the extraction of drug-drug interaction (DDI) for the DDI-2013 dataset BIBREF20   They partition the data in subsets depending on the syntactic features, and train a different model for each  They use lexical, syntactic and verb based features on top of shallow parse features, in addition to a hand-crafted list of trigger words to define their features  An SVM classifier is then trained on the feature vectors, with a positive label if the drug pair interacts, and negative otherwise  Their method beats other systems on the DDI-2013 dataset  Some other feature-based approaches are described in BIBREF21 , BIBREF22  
Distant supervision methods have also been applied to relation extraction over biomedical corpora  In BIBREF23 , 10,000 neuroscience articles are distantly supervised using information from UMLS Semantic Network to classify brain-gene relations into geneExpression and otherRelation  They use lexical (bag of words, contextual) features as well as syntactic (dependency parse features)  They make the “at-least one” assumption, i e  at least one of the sentences extracted for a given entity-pair contains the relation in database  They model it as a multi-instance learning problem and adopt a graphical model similar to BIBREF24   They test using manually annotated examples

ground_truth: They compare with the following models: by Pedersen (2017), by Pramanick and Das (2017), by Mikhalkova and Karyakin (2017),  by Vadehra (2017), Indurthi and Oota (2017), by Vechtomova (2017), by (Cai et al., 2018), and CRF.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What were the baselines?
Context 3:  They note that the F-score achieved are much lesser than that achieved in the general domain in BIBREF24 , and attribute to generally poorer performance of NER tools in the biomedical domain, as well as less training examples  BIBREF25 explore distant supervision methods for protein-protein interaction extraction 
More recently, deep learning methods have been applied to relation extraction in the biomedical domain  One of the main advantages of such methods over traditional feature or kernel based learning methods is that they require minimal feature engineering  In BIBREF26 , skip-gram vectors BIBREF27 are trained over 5 6Gb of unlabelled text  They use these vectors to extract protein-protein interactions by converting them into features for entities, context and the entire sentence  Using an SVM for classification, their method is able to outperform many kernel and feature based methods over a variety of datasets 
BIBREF28 follow a similar method by using word vectors trained on PubMed articles  They use it for the task of relation extraction from clinical text for entities that include problem, treatment and medical test  For a given sentence, given labelled entities, they predict the type of relation exhibited (or None) by the entity pair  These types include “treatment caused medical problem”, “test conducted to investigate medical problem”, “medical problem indicates medical problems”, etc  They use a Convolutional Neural Network (CNN) followed by feedforward neural network architecture for prediction  In addition to pre-trained word vectors as features, for each token they also add features for POS tags, distance from both the entities in the sentence, as well BIO tags for the entities  Their model performs better than a feature based SVM baseline that they train themselves 
The BioNLP'16 Shared Tasks has also introduced some Relation Extraction tasks, in particular the BB3-event subtask that involves predicting whether a “lives-in” relation holds for a Bacteria in a location  Some of the top performing models for this task are deep learning models  BIBREF29 train word embeddings with six billions words of scientific texts from PubMed  They then consider the shortest dependency path between the two entities (Bacteria and location)  For each token in the path, they use word embedding features, POS type embeddings and dependency type embeddings  They train a unidirectional LSTM BIBREF30 over the dependency path, that achieves an F-Score of 52 1% on the test set 
BIBREF31 improve the performance by making modifications to the above model  Instead of using the shortest dependency path, they modify the parse tree based on some pruning strategies  They also add feature embeddings for each token to represent the distance from the entities in the shortest path  They then train a Bidirectional LSTM on the path, and obtain an F-Score of 57 1% 
The recent success of deep learning models in Biomedical Relation Extraction that require minimal feature engineering is promising  This also suggests new avenues of research in the field  An approach as in BIBREF32 can be used to combine multi-instance learning and distant supervision with a neural architecture 
Event Extraction
Event Extraction in the Biomedical domain is a task that has gained more importance recently  Event Extraction goes beyond Relation Extraction  In Biomedical Event Extraction, events generally refer to a change in the state of biological molecules such as proteins and DNA  Generally, it includes detection of targeted event types such as gene expression, regulation, localisation and transcription  Each event type in addition can have multiple arguments that need to be detected  An additional layer of complexity comes from the fact that events can also be arguments of other events, giving rise to a nested structure  This helps to capture the underlying biology better BIBREF1   Detecting the event type often involves recognising and classifying trigger words  Often, these words are verbs such as “activates”, “inhibits”, “phosphorylation” that may indicate a single, or sometimes multiple event types  In this section, we will discuss some of the successful models for Event Extraction in some detail 
Event Extraction gained a lot of interest with the availability of an annotated corpus with the BioNLP'09 Shared Task on Event Extraction BIBREF34   The task involves prediction of trigger words over nine event types such as expression, transcription, catabolism, binding, etc  given only annotation of named entities (proteins, genes, etc )  For each event, its class, trigger expression and arguments need to be extracted  Since the events can be arguments to other events, the final output in general is a graph representation with events and named entities as nodes, and edges that correspond to event arguments  BIBREF33 present a pipeline based method that is heavily dependent on dependency parsing  Their pipeline approach consists of three steps: trigger detection, argument detection and semantic post-processing  While the first two components are learning based systems, the last component is a rule based system  For the BioNLP'09 corpus, only 5% of the events span multiple sentences  Hence the approach does not get affected severely by considering only single sentences  It is important to note that trigger words cannot simply be reduced to a dictionary lookup  This is because a specific word may belong to multiple classes, or may not always be a trigger word for an event  For example, “activate” is found to not be a trigger word in over 70% of the cases  A multi-class SVM is trained for trigger detection on each token, using a large feature set consisting of semantic and syntactic features  It is interesting to note that the hyperparameters of this classifier are optimised based on the performance of the entire end-to-end system 
For the second component to detect arguments, labels for edges between entities must be predicted  For the BioNLP'09 Shared Task, each directed edge from one event node to another event node, or from an event node to a named entity node are classified as “theme”, “cause”, or None  The second component of the pipeline makes these predictions independently  This is also trained using a multi-class SVM which involves heavy use of syntactic features, including the shortest dependency path between the nodes  The authors note that the precision-recall choice of the first component affects the performance of the second component: since the second component is only trained on Gold examples, any error by the first component will lead to a cascading of errors  The final component, which is a semantic post-processing step, consists of rules and heuristics to correct the output of the second component  Since the edge predictions are made independently, it is possible that some event nodes do not have any edges, or have an improper combination of edges  The rule based component corrects these and applies rules to break directed cycles in the graph, and some specific heuristics for different types of events  The final model gives a cumulative F-Score of 52% on the test set, and was the best model on the task 
BIBREF35 note that previous approaches on the task suffer due to the pipeline nature and the propagation of errors  To counter this, they adopt a joint inference method based on Markov Logic Networks BIBREF36 for the same task on BioNLP'09  The Markov Logic Network jointly predicts whether each token is a trigger word, and if yes, the class it belongs to; for each dependency edge, whether it is an argument path leading to a “theme” or a “cause”  By formulating the Event Extraction problem using an MLN, the approach becomes computationally feasible and only linear in the length of the sentence  They incorporate hard constraints to encode rules such as “an argument path must have an event”, “a cause path must start with a regulation event”, etc  In addition, they also include some domain specific soft constraints as well as some linguistically-motivated context-specific soft constraints  In order to train the MLN, stochastic gradient descent was used  Certain heuristic methods are implemented in order to deal with errors due to syntactic parsing, especially ambiguities in PP-attachment and coordination  Their final system is competitive and comes very close to the system by BIBREF33 with an average F-Score of 50%  To further improve the system, they suggest leveraging additional joint-inference opportunities and integrating the syntactic parser better  Some other more recent models for Biomedical Event Extraction include BIBREF37 , BIBREF38  
Conclusion
We have discussed some of the major problems and challenges in BioIE, and seen some of the diverse approaches adopted to solve them  Some interesting problems such as Pathway Extraction for Biological Systems BIBREF39 , BIBREF40 have not been discussed 
Biomedical Information Extraction is a challenging and exciting field for NLP researchers that demands application of state-of-the-art methods  Traditionally, there has been a dependence on hand-crafted features or heavily feature-engineered methods  However, with the advent of deep learning methods, a lot of BioIE tasks are seeing an improvement by adopting deep learning models such as Convolutional Neural Networks and LSTMs, which require minimal feature engineering  Rapid progress in developing better systems for BioIE will be extremely helpful for clinicians and researchers in the Biomedical domain 

ground_truth: varied from Maximum Entropy Classifiers (BIBREF4) to Support Vector Machines (BIBREF5,BIBREF6,BIBREF7,BIBREF8), Recursive Neural Networks (BIBREF9,BIBREF10), Convolutional Neural Networks (BIBREF11) and most recently transfer learning-based architectures like Bidirectional Encoder Representation from Transformers (BERT) (BIBREF12)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Does the paper explore extraction from electronic health records?
Context 4: 
For example, for the sentence:
a 2002 systemic review of herbal products found that several herbs , including peppermint and caraway , have anti-dyspeptic effects for non-ulcer dyspepsia with “ encouraging safety profiles ”   (from BIBREF1 )
I feed into BERT:
[CLS] a 2002 systemic review of herbal products found that several herbs , including peppermint and caraway , [MASK] anti-dyspeptic effects for non-ulcer dyspepsia with “ encouraging safety profiles ”   and look for the score assigned to the words have and has at the masked position 
Similarly, for the pair
the game that the guard hates is bad  
the game that the guard hates are bad  
(from BIBREF3 ), I feed into BERT:
[CLS] the game that the guard hates [MASK] bad  
and compare the scores predicted for is and are 
This differs from BIBREF1 and BIBREF2 by considering the entire sentence (excluding the verb) and not just its prefix leading to the verb, and differs from BIBREF3 by conditioning the focus verb on bidirectional context 
I use the PyTorch implementation of BERT, with the pre-trained models supplied by Google  I experiment with the bert-large-uncased and bert-base-uncased models 
The bi-directional setup precludes using using the NPI stimuli of BIBREF3 , in which the minimal pair differs in two words position, which I discard from the evaluation  I also discard the agreement cases involving the verbs is or are in BIBREF1 and in BIBREF2 , because some of them are copular construction, in which strong agreement hints can be found also on the object following the verb  This is not an issue in the manually constructed BIBREF3 stimuli due to the patterns they chose 
Finally, I discard stimuli in which the focus verb or its plural/singular inflection does not appear as a single word in the BERT word-piece-based vocabulary (and hence cannot be predicted by the model)  This include discarding BIBREF3 stimuli involving the words swims or admires, resulting in 23,368 discarded pairs (out of 152,300)  I similarly discard 680 sentences from BIBREF1 where the focus verb or its inflection were one of 108 out-of-vocabulary tokens, and 28 sentence-pairs (8 tokens) from BIBREF2  
The BERT results are not directly comparable to the numbers reported in previous work  Beyond the differences due to bidirectionality and the discarded stimuli, the BERT models are also trained on a different and larger corpus (covering both wikipedia and books) 
Code is available at https://github com/yoavg/bert-syntax 
Results
Tables 1 , 2 and 3 show the results  All cases exhibit high scores—in the vast majority of the cases substantially higher than reported in previous work  As discussed above, the results are not directly comparable to previous work: the BERT models are trained on different (and larger) data, are allowed to access the suffix of the sentence in addition to its prefix, and are evaluated on somewhat different data due to discarding OOV items  Still, taken together, the high performance numbers indicate that the purely attention-based BERT models are likely capable of capturing the same kind of syntactic regularities that LSTM-based models are capable of capturing, at least as well as the LSTM models and probably better 
Another noticeable and interesting trend is that larger is not necessarily better: the BERT-Base model outperforms the BERT-Large model on many of the syntactic conditions 
Discussion
The BERT models perform remarkably well on all the syntactic test cases  I expected the attention-based mechanism to fail on these (compared to the LSTM-based models), and am surprised by these results  The BIBREF2 and BIBREF3 conditions rule out the possibility of overly relying on selectional preference cues or memorizing the wikipedia training data, and suggest real syntactic generalization is taking place  Exploring the extent to which deep purely-attention-based architectures such as BERT are capable of capturing hierarchy-sensitive and syntactic dependencies—as well as the mechanisms by which this is achieved—is a fascinating area for future research 

ground_truth: Yes