prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How large is their data set?
Context 1: Introduction
The task of document quality assessment is to automatically assess a document according to some predefined inventory of quality labels  This can take many forms, including essay scoring (quality = language quality, coherence, and relevance to a topic), job application filtering (quality = suitability for role + visual/presentational quality of the application), or answer selection in community question answering (quality = actionability + relevance of the answer to the question)  In the case of this paper, we focus on document quality assessment in two contexts: Wikipedia document quality classification, and whether a paper submitted to a conference was accepted or not 
Automatic quality assessment has obvious benefits in terms of time savings and tractability in contexts where the volume of documents is large  In the case of dynamic documents (possibly with multiple authors), such as in the case of Wikipedia, it is particularly pertinent, as any edit potentially has implications for the quality label of that document (and around 10 English Wikipedia documents are edited per second)  Furthermore, when the quality assessment task is decentralized (as in the case of Wikipedia and academic paper assessment), quality criteria are often applied inconsistently by different people, where an automatic document quality assessment system could potentially reduce inconsistencies and enable immediate author feedback 
Current studies on document quality assessment mainly focus on textual features  For example, BIBREF0 examine features such as the article length and the number of headings to predict the quality class of a Wikipedia article  In contrast to these studies, in this paper, we propose to combine text features with visual features, based on a visual rendering of the document  Figure 1 illustrates our intuition, relative to Wikipedia articles  Without being able to read the text, we can tell that the article in Figure 1 has higher quality than Figure 1 , as it has a detailed infobox, extensive references, and a variety of images  Based on this intuition, we aim to answer the following question: can we achieve better accuracy on document quality assessment by complementing textual features with visual features?
Our visual model is based on fine-tuning an Inception V3 model BIBREF1 over visual renderings of documents, while our textual model is based on a hierarchical biLSTM  We further combine the two into a joint model  We perform experiments on two datasets: a Wikipedia dataset novel to this paper, and an arXiv dataset provided by BIBREF2 split into three sub-parts based on subject category  Experimental results on the visual renderings of documents show that implicit quality indicators, such as images and visual layout, can be captured by an image classifier, at a level comparable to a text classifier  When we combine the two models, we achieve state-of-the-art results over 3/4 of our datasets 
This paper makes the following contributions:
All code and data associated with this research will be released on publication 
Related Work
A variety of approaches have been proposed for document quality assessment across different domains: Wikipedia article quality assessment, academic paper rating, content quality assessment in community question answering (cQA), and essay scoring  Among these approaches, some use hand-crafted features while others use neural networks to learn features from documents  For each domain, we first briefly describe feature-based approaches and then review neural network-based approaches  Wikipedia article quality assessment: Quality assessment of Wikipedia articles is a task that assigns a quality class label to a given Wikipedia article, mirroring the quality assessment process that the Wikipedia community carries out manually  Many approaches have been proposed that use features from the article itself, meta-data features (e g , the editors, and Wikipedia article revision history), or a combination of the two  Article-internal features capture information such as whether an article is properly organized, with supporting evidence, and with appropriate terminology  For example, BIBREF3 use writing styles represented by binarized character trigram features to identify featured articles  BIBREF4 and BIBREF0 explore the number of headings, images, and references in the article  BIBREF5 use nine readability scores, such as the percentage of difficult words in the document, to measure the quality of the article  Meta-data features, which are indirect indicators of article quality, are usually extracted from revision history, and the interaction between editors and articles  For example, one heuristic that has been proposed is that higher-quality articles have more edits BIBREF6 , BIBREF7   BIBREF8 use the percentage of registered editors and the total number of editors of an article  Article–editor dependencies have also been explored  For example, BIBREF9 use the authority of editors to measure the quality of Wikipedia articles, where the authority of editors is determined by the articles they edit  Deep learning approaches to predicting Wikipedia article quality have also been proposed  For example, BIBREF10 use a version of doc2vec BIBREF11 to represent articles, and feed the document embeddings into a four hidden layer neural network  BIBREF12 first obtain sentence representations by averaging words within a sentence, and then apply a biLSTM BIBREF13 to learn a document-level representation, which is combined with hand-crafted features as side information  BIBREF14 exploit two stacked biLSTMs to learn document representations 
Academic paper rating: Academic paper rating is a relatively new task in NLP/AI, with the basic formulation being to automatically predict whether to accept or reject a paper  BIBREF2 explore hand-crafted features, such as the length of the title, whether specific words (such as outperform, state-of-the-art, and novel) appear in the abstract, and an embedded representation of the abstract as input to different downstream learners, such as logistic regression, decision tree, and random forest  BIBREF15 exploit a modularized hierarchical convolutional neural network (CNN), where each paper section is treated as a module  For each paper section, they train an attention-based CNN, and an attentive pooling layer is applied to the concatenated representation of each section, which is then fed into a softmax layer 
Content quality assessment in cQA: Automatic quality assessment in cQA is the task of determining whether an answer is of high quality, selected as the best answer, or ranked higher than other answers  To measure answer content quality in cQA, researchers have exploited various features from different sources, such as the answer content itself, the answerer's profile, interactions among users, and usage of the content  The most common feature used is the answer length BIBREF16 , BIBREF17 , with other features including: syntactic and semantic features, such as readability scores  BIBREF18 ; similarity between the question and the answer at lexical, syntactic, and semantic levels BIBREF18 , BIBREF19 , BIBREF20 ; or user data (e g , a user's status points or the number of answers written by the user)  There have also been approaches using neural networks  For example, BIBREF21 combine CNN-learned representations with hand-crafted features to predict answer quality  BIBREF22 use a 2-dimensional CNN to learn the semantic relevance of an answer to the question, and apply an LSTM to the answer sequence to model thread context  BIBREF23 and BIBREF24 model the problem similarly to machine translation quality estimation, treating answers as competing translation hypotheses and the question as the reference translation, and apply neural machine translation to the problem  Essay scoring: Automated essay scoring is the task of assigning a score to an essay, usually in the context of assessing the language ability of a language learner  The quality of an essay is affected by the following four primary dimensions: topic relevance, organization and coherence, word usage and sentence complexity, and grammar and mechanics  To measure whether an essay is relevant to its “prompt” (the description of the essay topic), lexical and semantic overlap is commonly used BIBREF25 , BIBREF26   BIBREF27 explore word features, such as the number of verb formation errors, average word frequency, and average word length, to measure word usage and lexical complexity  BIBREF28 use sentence structure features to measure sentence variety  The effects of grammatical and mechanic errors on the quality of an essay are measured via word and part-of-speech $n$ -gram features and “mechanics” features BIBREF29 (e g , spelling, capitalization, and punctuation), respectively  BIBREF30 , BIBREF31 , and BIBREF32 use an LSTM to obtain an essay representation, which is used as the basis for classification  Similarly, BIBREF33 utilize a CNN to obtain sentence representation and an LSTM to obtain essay representation, with an attention layer at both the sentence and essay levels 
The Proposed Joint Model
We treat document quality assessment as a classification problem, i e , given a document, we predict its quality class (e g , whether an academic paper should be accepted or rejected)  The proposed model is a joint model that integrates visual features learned through Inception V3 with textual features learned through a biLSTM  In this section, we present the details of the visual and textual embeddings, and finally describe how we combine the two  We return to discuss hyper-parameter settings and the experimental configuration in the Experiments section 
Visual Embedding Learning
A wide range of models have been proposed to tackle the image classification task, such as VGG BIBREF34 , ResNet BIBREF35 , Inception V3 BIBREF1 , and Xception BIBREF36   However, to the best of our knowledge, there is no existing work that has proposed to use visual renderings of documents to assess document quality  In this paper, we use Inception V3 pretrained on ImageNet (“Inception” hereafter) to obtain visual embeddings of documents, noting that any image classifier could be applied to our task  The input to Inception is a visual rendering (screenshot) of a document, and the output is a visual embedding, which we will later integrate with our textual embedding 
Based on the observation that it is difficult to decide what types of convolution to apply to each layer (such as 3 $\times $ 3 or 5 $\times $ 5), the basic Inception model applies multiple convolution filters in parallel and concatenates the resulting features, which are fed into the next layer  This has the benefit of capturing both local features through smaller convolutions and abstracted features through larger convolutions  Inception is a hybrid of multiple Inception models of different architectures  To reduce computational cost, Inception also modifies the basic model by applying a 1 $\times $ 1 convolution to the input and factorizing larger convolutions into smaller ones 
Textual Embedding Learning
We adopt a bi-directional LSTM model to generate textual embeddings for document quality assessment, following the method of BIBREF12 (“biLSTM” hereafter)  The input to biLSTM is a textual document, and the output is a textual embedding, which will later integrate with the visual embedding 
For biLSTM, each word is represented as a word embedding BIBREF37 , and an average-pooling layer is applied to the word embeddings to obtain the sentence embedding, which is fed into a bi-directional LSTM to generate the document embedding from the sentence embeddings  Then a max-pooling layer is applied to select the most salient features from the component sentences 
The Joint Model
The proposed joint model (“Joint” hereafter) combines the visual and textual embeddings (output of Inception and biLSTM) via a simple feed-forward layer and softmax over the document label set, as shown in Figure 2   We optimize our model based on cross-entropy loss 
Experiments
In this section, we first describe the two datasets used in our experiments: (1) Wikipedia, and (2) arXiv  Then, we report the experimental details and results 
Datasets
The Wikipedia dataset consists of articles from English Wikipedia, with quality class labels assigned by the Wikipedia community  Wikipedia articles are labelled with one of six quality classes, in descending order of quality: Featured Article (“FA”), Good Article (“GA”), B-class Article (“B”), C-class Article (“C”), Start Article (“Start”), and Stub Article (“Stub”)  A description of the criteria associated with the different classes can be found in the Wikipedia grading scheme page  The quality class of a Wikipedia article is assigned by Wikipedia reviewers or any registered user, who can discuss through the article's talk page to reach consensus  We constructed the dataset by first crawling all articles from each quality class repository, e g , we get FA articles by crawling pages from the FA repository: https://en wikipedia org/wiki/Category:Featured_articles  This resulted in around 5K FA, 28K GA, 212K B, 533K C, 2 6M Start, and 3 2M Stub articles 
We randomly sampled 5,000 articles from each quality class and removed all redirect pages, resulting in a dataset of 29,794 articles  As the wikitext contained in each document contains markup relating to the document category such as {Featured Article} or {geo-stub}, which reveals the label, we remove such information  We additionally randomly partitioned this dataset into training, development, and test splits based on a ratio of 8:1:1  Details of the dataset are summarized in Table 1  
We generate a visual representation of each document via a 1,000 $\times $ 2,000-pixel screenshot of the article via a PhantomJS script over the rendered version of the article, ensuring that the screenshot and wikitext versions of the article are the same version  Any direct indicators of document quality (such as the FA indicator, which is a bronze star icon in the top right corner of the webpage) are removed from the screenshot 
The arXiv dataset BIBREF2 consists of three subsets of academic articles under the arXiv repository of Computer Science (cs), from the three subject areas of: Artificial Intelligence (cs ai), Computation and Language (cs cl), and Machine Learning (cs lg)  In line with the original dataset formulation BIBREF2 , a paper is considered to have been accepted (i e  is positively labeled) if it matches a paper in the DBLP database or is otherwise accepted by any of the following conferences: ACL, EMNLP, NAACL, EACL, TACL, NIPS, ICML, ICLR, or AAAI  Failing this, it is considered to be rejected (noting that some of the papers may not have been submitted to one of these conferences)  The median numbers of pages for papers in cs ai, cs cl, and cs lg are 11, 10, and 12, respectively  To make sure each page in the PDF file has the same size in the screenshot, we crop the PDF file of a paper to the first 12; we pad the PDF file with blank pages if a PDF file has less than 12 pages, using the PyPDF2 Python package  We then use ImageMagick to convert the 12-page PDF file to a single 1,000 $\times $ 2,000 pixel screenshot  Table 2 details this dataset, where the “Accepted” column denotes the percentage of positive instances (accepted papers) in each subset 
Experimental Setting
As discussed above, our model has two main components — biLSTM and Inception— which generate textual and visual representations, respectively  For the biLSTM component, the documents are preprocessed as described in BIBREF12 , where an article is divided into sentences and tokenized using NLTK BIBREF38   Words appearing more than 20 times are retained when building the vocabulary  All other words are replaced by the special UNK token  We use the pre-trained GloVe BIBREF39 50-dimensional word embeddings to represent words  For words not in GloVe, word embeddings are randomly initialized based on sampling from a uniform distribution $U(-1, 1)$   All word embeddings are updated in the training process  We set the LSTM hidden layer size to 256  The concatenation of the forward and backward LSTMs thus gives us 512 dimensions for the document embedding  A dropout layer is applied at the sentence and document level, respectively, with a probability of 0 5 
For Inception, we adopt data augmentation techniques in the training with a “nearest” filling mode, a zoom range of 0 1, a width shift range of 0 1, and a height shift range of 0 1  As the original screenshots have the size of 1,000 $\times 2$ ,000 pixels, they are resized to 500 $\times $ 500 to feed into Inception, where the input shape is (500, 500, 3)  A dropout layer is applied with a probability of 0 5  Then, a GlobalAveragePooling2D layer is applied, which produces a 2,048 dimensional representation 
For the Joint model, we get a representation of 2,560 dimensions by concatenating the 512 dimensional representation from the biLSTM with the 2,048 dimensional representation from Inception  The dropout layer is applied to the two components with a probability of 0 5  For biLSTM, we use a mini-batch size of 128 and a learning rate of 0 001  For both Inception and joint model, we use a mini-batch size of 16 and a learning rate of 0 0001  All hyper-parameters were set empirically over the development data, and the models were optimized using the Adam optimizer BIBREF40  
In the training phase, the weights in Inception are initialized by parameters pretrained on ImageNet, and the weights in biLSTM are randomly initialized (except for the word embeddings)  We train each model for 50 epochs  However, to prevent overfitting, we adopt early stopping, where we stop training the model if the performance on the development set does not improve for 20 epochs  For evaluation, we use (micro-)accuracy, following previous studies BIBREF5 , BIBREF2  
Baseline Approaches
We compare our models against the following five baselines:
Majority: the model labels all test samples with the majority class of the training data 
Benchmark: a benchmark method from the literature  In the case of Wikipedia, this is BIBREF5 , who use structural features and readability scores as features to build a random forest classifier; for arXiv, this is BIBREF2 , who use hand-crafted features, such as the number of references and TF-IDF weighted bag-of-words in abstract, to build a classifier based on the best of logistic regression, multi-layer perception, and AdaBoost 
Doc2Vec: doc2vec BIBREF11 to learn document embeddings with a dimension of 500, and a 4-layer feed-forward classification model on top of this, with 2000, 1000, 500, and 200 dimensions, respectively 
biLSTM: first derive a sentence representation by averaging across words in a sentence, then feed the sentence representation into a biLSTM and a maxpooling layer over output sequence to learn a document level representation with a dimension of 512, which is used to predict document quality 
Inception $_{\text{fixed}}$ : the frozen Inception model, where only parameters in the last layer are fine-tuned during training 
The hyper-parameters of Benchmark, Doc2Vec, and biLSTM are based on the corresponding papers except that: (1) we fine-tune the feed forward layer of Doc2Vec on the development set and train the model 300 epochs on Wikipedia and 50 epochs on arXiv; (2) we do not use hand-crafted features for biLSTM as we want the baselines to be comparable to our models, and the main focus of this paper is not to explore the effects of hand-crafted features (e g , see BIBREF12 ) 
Experimental Results
Table 3 shows the performance of the different models over our two datasets, in the form of the average accuracy on the test set (along with the standard deviation) over 10 runs, with different random initializations 
On Wikipedia, we observe that the performance of biLSTM, Inception, and Joint is much better than that of all four baselines  Inception achieves 2 9% higher accuracy than biLSTM  The performance of Joint achieves an accuracy of 59 4%, which is 5 3% higher than using textual features alone (biLSTM) and 2 4% higher than using visual features alone (Inception)  Based on a one-tailed Wilcoxon signed-rank test, the performance of Joint is statistically significant ( $p<0 05$ )  This shows that the textual and visual features complement each other, achieving state-of-the-art results in combination 
For arXiv, baseline methods Majority, Benchmark, and Inception $_{\text{fixed}}$ outperform biLSTM over cs ai, in large part because of the class imbalance in this dataset (90% of papers are rejected)  Surprisingly, Inception $_{\text{fixed}}$ is better than Majority and Benchmark over the arXiv cs lg subset, which verifies the usefulness of visual features, even when only the last layer is fine-tuned  Table 3 also shows that Inception and biLSTM achieve similar performance on arXiv, showing that textual and visual representations are equally discriminative: Inception and biLSTM are indistinguishable over cs cl; biLSTM achieves 1 8% higher accuracy over cs lg, while Inception achieves 1 3% higher accuracy over cs ai  Once again, the Joint model achieves the highest accuracy on cs ai and cs cl by combining textual and visual representations (at a level of statistical significance for cs ai)  This, again, confirms that textual and visual features complement each other, and together they achieve state-of-the-art results  On arXiv cs lg, Joint achieves a 0 6% higher accuracy than Inception by combining visual features and textual features, but biLSTM achieves the highest accuracy  One characteristic of cs lg documents is that they tend to contain more equations than the other two arXiv datasets, and preliminary analysis suggests that the biLSTM is picking up on a correlation between the volume/style of mathematical presentation and the quality of the document 
Analysis
In this section, we first analyze the performance of Inception and Joint  We also analyze the performance of different models on different quality classes  The high-level representations learned by different models are also visualized and discussed  As the Wikipedia test set is larger and more balanced than that of arXiv, our analysis will focus on Wikipedia 
Inception
To better understand the performance of Inception, we generated the gradient-based class activation map BIBREF41 , by maximizing the outputs of each class in the penultimate layer, as shown in Figure 3   From Figure 3 and Figure 3 , we can see that Inception identifies the two most important regions (one at the top corresponding to the table of contents, and the other at the bottom, capturing both document length and references) that contribute to the FA class prediction, and a region in the upper half of the image that contributes to the GA class prediction (capturing the length of the article body)  From Figure 3 and Figure 3 , we can see that the most important regions in terms of B and C class prediction capture images (down the left and right of the page, in the case of B and C), and document length/references  From Figure 3 and Figure 3 , we can see that Inception finds that images in the top right corner are the strongest predictor of Start class prediction, and (the lack of) images/the link bar down the left side of the document are the most important for Stub class prediction 
Joint
Table 4 shows the confusion matrix of Joint on Wikipedia  We can see that more than 50% of documents for each quality class are correctly classified, except for the C class where more documents are misclassified into B  Analysis shows that when misclassified, documents are usually misclassified into adjacent quality classes, which can be explained by the Wikipedia grading scheme, where the criteria for adjacent quality classes are more similar 
We also provide a breakdown of precision (“ $\mathcal {P}$ ”), recall (“ $\mathcal {R}$ ”), and F1 score (“ $\mathcal {F}_{\beta =1}$ ”) for biLSTM, Inception, and Joint across the quality classes in Table 5   We can see that Joint achieves the highest accuracy in 11 out of 18 cases  It is also worth noting that all models achieve higher scores for FA, GA, and Stub articles than B, C and Start articles  This can be explained in part by the fact that FA and GA articles must pass an official review based on structured criteria, and in part by the fact that Stub articles are usually very short, which is discriminative for Inception, and Joint  All models perform worst on the B and C quality classes  It is difficult to differentiate B articles from C articles even for Wikipedia contributors  As evidence of this, when we crawled a new dataset including talk pages with quality class votes from Wikipedia contributors, we found that among articles with three or more quality labels, over 20% percent of B and C articles have inconsistent votes from Wikipedia contributors, whereas for FA and GA articles the number is only 0 7% 
We further visualize the learned document representations of biLSTM, Inception, and Joint in the form of a t-SNE plot BIBREF42 in Figure 4   The degree of separation between Start and Stub achieved by Inception is much greater than for biLSTM, with the separation between Start and Stub achieved by Joint being the clearest among the three models  Inception and Joint are better than biLSTM at separating Start and C  Joint achieves slightly better performance than Inception in separating GA and FA  We can also see that it is difficult for all models to separate B and C, which is consistent with the findings of Tables 4 and 5  
Conclusions
We proposed to use visual renderings of documents to capture implicit document quality indicators, such as font choices, images, and visual layout, which are not captured in textual content  We applied neural network models to capture visual features given visual renderings of documents  Experimental results show that we achieve a 2 9% higher accuracy than state-of-the-art approaches based on textual features over Wikipedia, and performance competitive with or surpassing state-of-the-art approaches over arXiv  We further proposed a joint model, combining textual and visual representations, to predict the quality of a document  Experimental results show that our joint model outperforms the visual-only model in all cases, and the text-only model on Wikipedia and two subsets of arXiv  These results underline the feasibility of assessing document quality via visual features, and the complementarity of visual and textual document representations for quality assessment 
Context 2: Introduction
With the availability of rich data on users' locations, profiles and search history, personalization has become the leading trend in large-scale information retrieval  However, efficiency through personalization is not yet the most suitable model when tackling domain-specific searches  This is due to several factors, such as the lexical and semantic challenges of domain-specific data that often include advanced argumentation and complex contextual information, the higher sparseness of relevant information sources, and the more pronounced lack of similarities between users' searches 
A recent study on expert search strategies among healthcare information professionals BIBREF0 showed that, for a given search task, they spend an average of 60 minutes per collection or database, 3 minutes to examine the relevance of each document, and 4 hours of total search time  When written in steps, their search strategy spans over 15 lines and can reach up to 105 lines 
With the abundance of information sources in the medical domain, consumers are more and more faced with a similar challenge, one that needs dedicated solutions that can adapt to the heterogeneity and specifics of health-related information 
Dedicated Question Answering (QA) systems are one of the viable solutions to this problem as they are designed to understand natural language questions without relying on external information on the users 
In the context of QA, the goal of Recognizing Question Entailment (RQE) is to retrieve answers to a premise question ( INLINEFORM0 ) by retrieving inferred or entailed questions, called hypothesis questions ( INLINEFORM1 ) that already have associated answers  Therefore, we define the entailment relation between two questions as: a question INLINEFORM2 entails a question INLINEFORM3 if every answer to INLINEFORM4 is also a correct answer to INLINEFORM5 BIBREF1  
RQE is particularly relevant due to the increasing numbers of similar questions posted online BIBREF2 and its ability to solve differently the challenging issues of question understanding and answer extraction  In addition to being used to find relevant answers, these resources can also be used in training models able to recognize inference relations and similarity between questions 
Question similarity has recently attracted international challenges BIBREF3 , BIBREF4 and several research efforts proposing a wide range of approaches, including Logistic Regression, Recurrent Neural Networks (RNNs), Long Short Term Memory cells (LSTMs), and Convolutional Neural Networks (CNNs) BIBREF5 , BIBREF6 , BIBREF1 , BIBREF7  
In this paper, we study question entailment in the medical domain and the effectiveness of the end-to-end RQE-based QA approach by evaluating the relevance of the retrieved answers  Although entailment was attempted in QA before BIBREF8 , BIBREF9 , BIBREF10 , as far as we know, we are the first to introduce and evaluate a full medical question answering approach based on question entailment for free-text questions  Our contributions are:
The next section is dedicated to related work on question answering, question similarity and entailment  In Section SECREF3 , we present two machine learning (ML) and deep learning (DL) methods for RQE and compare their performance using open-domain and clinical datasets  Section SECREF4 describes the new collection of medical question-answer pairs  In Section SECREF5 , we describe our RQE-based approach for QA  Section SECREF6 presents our evaluation of the retrieved answers and the results obtained on TREC 2017 LiveQA medical questions 
Background
In this section we define the RQE task and describe related work at the intersection of question answering, question similarity and textual inference 
Task Definition
The definition of Recognizing Question Entailment (RQE) can have a significant impact on QA results  In related work, the meaning associated with Natural Language Inference (NLI) varies among different tasks and events  For instance, Recognizing Textual Entailment (RTE) was addressed by the PASCAL challenge BIBREF12 , where the entailment relation has been assessed manually by human judges who selected relevant sentences "entailing" a set of hypotheses from a list of documents returned by different Information Retrieval (IR) methods  In another definition, the Stanford Natural Language Inference corpus SNLI BIBREF13 , used three classification labels for the relations between two sentences: entailment, neutral and contradiction  For the entailment label, the annotators who built the corpus were presented with an image and asked to write a caption “that is a definitely true description of the photo”  For the neutral label, they were asked to provide a caption “that might be a true description of the label”  They were asked for a caption that “is definitely a false description of the photo” for the contradiction label 
More recently, the multiNLI corpus BIBREF14 was shared in the scope of the RepEval 2017 shared task BIBREF15   To build the corpus, annotators were presented with a premise text and asked to write three sentences  One novel sentence, which is “necessarily true or appropriate in the same situations as the premise,” for the entailment label, a sentence, which is “necessarily false or inappropriate whenever the premise is true,” for the contradiction label, and a last sentence, “where neither condition applies,” for the neutral label 
Whereas these NLI definitions might be suitable for the broad topic of text understanding, their relation to practical information retrieval or question answering systems is not straightforward 
In contrast, RQE has to be tailored to the question answering task  For instance, if the premise question is "looking for cold medications for a 30 yo woman", a RQE approach should be able to consider the more general (less restricted) question "looking for cold medications" as relevant, since its answers are relevant for the initial question, whereas "looking for medications for a 30 yo woman" is a useless contextualization  The entailment relation we are seeking in the QA context should include relevant and meaningful relaxations of contextual and semantic constraints (cf  Section SECREF13 ) 
Related Work on Question Answering
Classical QA systems face two main challenges related to question analysis and answer extraction  Several QA approaches were proposed in the literature for the open domain BIBREF16 , BIBREF17 and the medical domain BIBREF18 , BIBREF19 , BIBREF20   A variety of methods were developed for question analysis, focus (topic) recognition and question type identification BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24   Similarly, many different approaches tackled document or passage retrieval and answer selection and (re)ranking BIBREF25 , BIBREF26 , BIBREF27  
An alternative approach consists in finding similar questions or FAQs that are already answered BIBREF28 , BIBREF29   One of the earliest question answering systems based on finding similar questions and re-using the existing answers was FAQ FINDER BIBREF30   Another system that complements the existing Q&A services of NetWellness is SimQ BIBREF2 , which allows retrieval of similar web-based consumer health questions  SimQ uses syntactic and semantic features to compute similarity between questions, and UMLS BIBREF31 as a standardized semantic knowledge source  The system achieves 72 2% precision, 78 0% recall and 75 0% F-score on NetWellness questions  However, the method was evaluated only on one question similarity dataset, and the retrieved answers were not evaluated 
The aim of the medical task at TREC 2017 LiveQA was to develop techniques for answering complex questions such as consumer health questions, as well as to identify relevant answer sources that can comply with the sensitivity of medical information retrieval 
The CMU-OAQA system BIBREF32 achieved the best performance of 0 637 average score on the medical task by using an attentional encoder-decoder model for paraphrase identification and answer ranking  The Quora question-similarity dataset was used for training  The PRNA system BIBREF33 achieved the second best performance in the medical task with 0 49 average score using Wikipedia as the first answer source and Yahoo and Google searches as secondary answer sources  Each medical question was decomposed into several subquestions  To extract the answer from the selected text passage, a bi-directional attention model trained on the SQUAD dataset was used 
Deep neural network models have been pushing the limits of performance achieved in QA related tasks using large training datasets  The results obtained by CMU-OAQA and PRNA showed that large open-domain datasets were beneficial for the medical domain  However, the best system (CMU-OAQA) relying on the same training data obtained a score of 1 139 on the LiveQA open-domain task 
While this gap in performance can be explained in part by the discrepancies between the medical test questions and the open-domain questions, it also highlights the need for larger medical datasets to support deep learning approaches in dealing with the linguistic complexity of consumer health questions and the challenge of finding correct and complete answers 
Another technique was used by ECNU-ICA team BIBREF34 based on learning question similarity via two long short-term memory (LSTM) networks applied to obtain the semantic representations of the questions  To construct a collection of similar question pairs, they searched community question answering sites such as Yahoo! and Answers com  In contrast, the ECNU-ICA system achieved the best performance of 1 895 in the open-domain task but an average score of only 0 402 in the medical task  As the ECNU-ICA approach also relied on a neural network for question matching, this result shows that training attention-based decoder-encoder networks on the Quora dataset generalized better to the medical domain than training LSTMs on similar questions from Yahoo! and Answers com 
The CMU-LiveMedQA team BIBREF20 designed a specific system for the medical task  Using only the provided training datasets and the assumption that each question contains only one focus, the CMU-LiveMedQA system obtained an average score of 0 353  They used a convolutional neural network (CNN) model to classify a question into a restricted set of 10 question types and crawled "relevant" online web pages to find the answers  However, the results were lower than those achieved by the systems relying on finding similar answered questions  These results support the relevance of similar question matching for the end-to-end QA task as a new way of approaching QA instead of the classical QA approaches based on Question Analysis and Answer Retrieval 
Related Work on Question Similarity and Entailment
Several efforts focused on recognizing similar questions  Jeon et al  BIBREF35 showed that a retrieval model based on translation probabilities learned from a question and answer archive can recognize semantically similar questions  Duan et al  BIBREF36 proposed a dedicated language modeling approach for question search, using question topic (user's interest) and question focus (certain aspect of the topic) 
Lately, these efforts were supported by a task on Question-Question similarity introduced in the community QA challenge at SemEval (task 3B) BIBREF3   Given a new question, the task focused on reranking all similar questions retrieved by a search engine, assuming that the answers to the similar questions will be correct answers for the new question  Different machine learning and deep learning approaches were tested in the scope of SemEval 2016 BIBREF3 and 2017 BIBREF4 task 3B  The best performing system in 2017 achieved a MAP of 47 22% using supervised Logistic Regression that combined different unsupervised similarity measures such as Cosine and Soft-Cosine BIBREF37   The second best system achieved 46 93% MAP with a learning-to-rank method using Logistic Regression and a rich set of features including lexical and semantic features as well as embeddings generated by different neural networks (siamese, Bi-LSTM, GRU and CNNs) BIBREF38   In the scope of this challenge, a dataset was collected from Qatar Living forum for training  We refer to this dataset as SemEval-cQA 
In another effort, an answer-based definition of RQE was proposed and tested BIBREF1   The authors introduced a dataset of clinical questions and used a feature-based method that provided an Accuracy of 75% on consumer health questions  We will call this dataset Clinical-QE  Dos Santos et al  BIBREF5 proposed a new approach to retrieve semantically equivalent questions combining a bag-of-words representation with a distributed vector representation created by a CNN and user data collected from two Stack Exchange communities  Lei et al  BIBREF7 proposed a recurrent and convolutional model (gated convolution) to map questions to their semantic representations  The models were pre-trained within an encoder-decoder framework 
RQE Approaches and Experiments
The choice of two methods for our empirical study is motivated by the best performance achieved by Logistic Regression in question-question similarity at SemEval 2017 (best system BIBREF37 and second best system BIBREF38 ), and the high performance achieved by neural networks on larger datasets such as SNLI BIBREF13 , BIBREF39 , BIBREF40 , BIBREF41   We first define the RQE task, then present the two approaches, and evaluate their performance on five different datasets 
Definition
In the context of QA, the goal of RQE is to retrieve answers to a new question by retrieving entailed questions with associated answers  We therefore define question entailment as:
a question INLINEFORM0 entails a question INLINEFORM1 if every answer to INLINEFORM2 is also a complete or partial answer to INLINEFORM3  
We present below two examples of consumer health questions INLINEFORM0 and entailed questions INLINEFORM1 :
Example 1 (each answer to the entailed question B1 is a complete answer to A1):
A1: What is the latest news on tennitis, or ringing in the ear, I am 75 years old and have had ringing in the ear since my mid 5os  Thank you 
B1: What is the latest research on Tinnitus?
Example 2 (each answer to the entailed question B2 is a partial answer to A2):
A2: My mother has been diagnosed with Alzheimer's, my father is not of the greatest health either and is the main caregiver for my mother  My question is where do we start with attempting to help our parents w/ the care giving and what sort of financial options are there out there for people on fixed incomes 
B2: What resources are available for Alzheimer's caregivers?
The inclusion of partial answers in the definition of question entailment also allows efficient relaxation of the contextual constraints of the original question INLINEFORM0 to retrieve relevant answers from entailed, but less restricted, questions 
Deep Learning Model
To recognize entailment between two questions INLINEFORM0 (premise) and INLINEFORM1 (hypothesis), we adapted the neural network proposed by Bowman et al  BIBREF13   Our DL model, presented in Figure FIGREF20 , consists of three 600d ReLU layers, with a bottom layer taking the concatenated sentence representations as input and a top layer feeding a softmax classifier  The sentence embedding model sums the Recurrent neural network (RNN) embeddings of its words  The word embeddings are first initialized with pretrained GloVe vectors  This adaptation provided the best performance in previous experiments with RQE data 
GloVe is an unsupervised learning algorithm to generate vector representations for words BIBREF42   Training is performed on aggregated word co-occurrence statistics from a large corpus, and the resulting representations show interesting linear substructures of the word vector space  We use the pretrained common crawl version with 840B tokens and 300d vectors, which are not updated during training 
Logistic Regression Classifier
In this feature-based approach, we use Logistic Regression to classify question pairs into entailment or no-entailment  Logistic Regression achieved good results on this specific task and outperformed other statistical learning algorithms such as SVM and Naive Bayes  In a preprocessing step, we remove stop words and perform word stemming using the Porter algorithm BIBREF43 for all ( INLINEFORM0 , INLINEFORM1 ) pairs 
We use a list of nine features, selected after several experiments on RTE datasets BIBREF12   We compute five similarity measures between the pre-processed questions and use their values as features  We use Word Overlap, the Dice coefficient based on the number of common bigrams, Cosine, Levenshtein, and the Jaccard similarities  Our feature list also includes the maximum and average values obtained with these measures and the question length ratio (length( INLINEFORM0 )/length( INLINEFORM1 ))  We compute a morphosyntactic feature indicating the number of common nouns and verbs between INLINEFORM2 and INLINEFORM3   TreeTagger BIBREF44 was used for POS tagging 
For RQE, we add an additional feature specific to the question type  We use a dictionary lookup to map triggers to the question type (e g  Treatment, Prognosis, Inheritance)  Triggers are identified for each question type based on a manual annotation of a set of medical questions (cf  Section SECREF36 )  This feature has three possible values: 2 (Perfect match between INLINEFORM0 type(s) and INLINEFORM1 type(s)), 1 (Overlap between INLINEFORM2 type(s) and INLINEFORM3 type(s)) and 0 (No common types) 
Datasets Used for the RQE Study
We evaluate the RQE methods (i e  deep learning model and logistic regression classifier) using two datasets of sentence pairs (SNLI and multiNLI), and three datasets of question pairs (Quora, Clinical-QE, and SemEval-cQA) 
The Stanford Natural Language Inference corpus (SNLI) BIBREF13 contains 569,037 sentence pairs written by humans based on image captioning  The training set of the MultiNLI corpus BIBREF14 consists of 393,000 pairs of sentences from five genres of written and spoken English (e g  Travel, Government)  Two other "matched" and "mismatched" sets are also available for development (20,000 pairs)  Both SNLI and multiNLI consider three types of relationships between sentences: entailment, neutral and contradiction  We converted the contradiction and neutral labels to the same non-entailment class 
The QUORA dataset of similar questions was recently published with 404,279 question pairs  We randomly selected three distinct subsets (80%/10%/10%) for training (323,423 pairs), development (40,428 pairs) and test (40,428 pairs) 
The clinical-QE dataset BIBREF1 contains 8,588 question pairs and was constructed using 4,655 clinical questions asked by family doctors BIBREF45   We randomly selected three distinct subsets (80%/10%/10%) for training (6,870 pairs), development (859 pairs) and test (859 pairs) 
The question similarity dataset of SemEval 2016 Task 3B (SemEval-cQA) BIBREF3 contains 3,869 question pairs and aims to re-rank a list of related questions according to their similarity to the original question  The same dataset was used for SemEval 2017 Task 3 BIBREF4  
To construct our test dataset, we used a publicly shared set of Consumer Health Questions (CHQs) received by the U S  National Library of Medicine (NLM), and annotated with named entities, question types, and focus BIBREF46 , BIBREF47   The CHQ dataset consists of 1,721 consumer information requests manually annotated with subquestions, each identified by a question type and a focus 
First, we selected automatically harvested FAQs, from U S  National Institutes of Health (NIH) websites, that share both the same focus and the same question type with the CHQs  As FAQs are most often very short, we first assume that the CHQ entails the FAQ  Two sets of pairs were constructed: (i) positive pairs of CHQs and FAQs sharing at least one common question type and the question focus, and (ii) negative pairs corresponding to a focus mismatch or type mismatch  For each category of negative examples, we randomly selected the same number of pairs for a balanced dataset  Then, we manually validated the constructed pairs and corrected the positive and negative labels when needed  The final RQE dataset contains 850 CHQ-FAQ pairs with 405 positive and 445 negative pairs  Table TABREF26 presents examples from the five training datasets (SNLI, MultiNLI, SemEval-cQA, Clinical-QE and Quora) and the new test dataset of medical CHQ-FAQ pairs 
Results of RQE Approaches
In the first experiment, we evaluated the DL and ML methods on SNLI, multi-NLI, Quora, and Clinical-QE  For the datasets that did not have a development and test sets, we randomly selected two sets, each amounting to 10% of the data, for test and development, and used the remaining 80% for training  For MultiNLI, we used the dev1-matched set for validation and the dev2-mismatched set for testing 
Table TABREF28 presents the results of the first experiment  The DL model with GloVe word embeddings achieved better results on three datasets, with 82 80% Accuracy on SNLI, 78 52% Accuracy on MultiNLI, and 83 62% Accuracy on Quora  Logistic Regression achieved the best Accuracy of 98 60% on Clinical-RQE  We also performed a 10-fold cross-validation on the full Clinical-QE data of 8,588 question pairs, which gave 98 61% Accuracy 
In the second experiment, we used these datasets for training only and compared their performance on our test set of 850 consumer health questions  Table TABREF29 presents the results of this experiment  Logistic Regression trained on the clinical-RQE data outperformed DL models trained on all datasets, with 73 18% Accuracy 
To validate further the performance of the LR method, we evaluated it on question similarity detection  A typical approach to this task is to use an IR method to find similar question candidates, then a more sophisticated method to select and re-rank the similar questions  We followed a similar approach for this evaluation by combining the LR method with the IR baseline provided in the context of SemEval-cQA  The hybrid method combines the score provided by the Logistic Regression model and the reciprocal rank from the IR baseline using a weight-based combination:
INLINEFORM0
The weight INLINEFORM0 was set empirically through several tests on the cQA-2016 development set ( INLINEFORM1 )  Table TABREF30 presents the results on the cQA-2016 and cQA-2017 test datasets  The hybrid method (LR+IR) provided the best results on both datasets  On the 2016 test data, the LR+IR method outperformed the best system in all measures, with 80 57% Accuracy and 77 47% MAP (official system ranking measure in SemEval-cQA)  On the cQA-2017 test data, the LR+IR method obtained 44 66% MAP and outperformed the cQA-2017 best system in Accuracy with 67 27% 
Discussion of RQE Results
When trained and tested on the same corpus, the DL model with GloVe embeddings gave the best results on three datasets (SNLI, MultiNLI and Quora)  Logistic Regression gave the best Accuracy on the Clinical-RQE dataset with 98 60%  When tested on our test set (850 medical CHQs-FAQs pairs), Logistic Regression trained on Clinical-QE gave the best performance with 73 18% Accuracy 
The SNLI and multi-NLI models did not perform well when tested on medical RQE data  We performed additional evaluations using the RTE-1, RTE-2 and RTE-3 open-domain datasets provided by the PASCAL challenge and the results were similar  We have also tested the SemEval-cQA-2016 model and had a similar drop in performance on RQE data  This could be explained by the different types of data leading to wrong internal conceptualizations of medical terms and questions in the deep neural layers

ground_truth: a sample of  29,794 wikipedia articles and 2,794 arXiv papers 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What machine learning and deep learning methods are used for RQE?
Context 2:  This performance drop could also be caused by the complexity of the test consumer health questions that are often composed of several subquestions, contain contextual information, and may contain misspellings and ungrammatical sentences, which makes them more difficult to process BIBREF48   Another aspect is the semantics of the task as discussed in Section SECREF6   The definition of textual entailment in open-domain may not quite apply to question entailment due to the strict semantics  Also the general textual entailment definitions refer only to the premise and hypothesis, while the definition of RQE for question answering relies on the relationship between the sets of answers of the compared questions 
Building a Medical QA Collection from Trusted Resources
A RQE-based QA system requires a collection of question-answer pairs to map new user questions to the existing questions with an RQE approach, rank the retrieved questions, and present their answers to the user 
Method
To construct trusted medical question-answer pairs, we crawled websites from the National Institutes of Health (cf  Section SECREF56 )  Each web page describes a specific topic (e g  name of a disease or a drug), and often includes synonyms of the main topic that we extracted during the crawl 
We constructed hand-crafted patterns for each website to automatically generate the question-answer pairs based on the document structure and the section titles  We also annotated each question with the associated focus (topic of the web page) as well as the question type identified with the designed patterns (cf  Section SECREF36 ) 
To provide additional information about the questions that could be used for diverse IR and NLP tasks, we automatically annotated the questions with the focus, its UMLS Concept Unique Identifier (CUI) and Semantic Type  We combined two methods to recognize named entities from the titles of the crawled articles and their associated UMLS CUIs: (i) exact string matching to the UMLS Metathesaurus, and (ii) MetaMap Lite BIBREF49   We then used the UMLS Semantic Network to retrieve the associated semantic types and groups 
Question Types
The question types were derived after the manual evaluation of 1,721 consumer health questions  Our taxonomy includes 16 types about Diseases, 20 types about Drugs and one type (Information) for the other named entities such as Procedures, Medical exams and Treatments  We describe below the considered question types and examples of associated question patterns 
Question Types about Diseases (16): Information, Research (or Clinical Trial), Causes, Treatment, Prevention, Diagnosis (Exams and Tests), Prognosis, Complications, Symptoms, Inheritance, Susceptibility, Genetic changes, Frequency, Considerations, Contact a medical professional, Support Groups 
Examples:
What research (or clinical trial) is being done for DISEASE?
What is the outlook for DISEASE?
How many people are affected by DISEASE?
When to contact a medical professional about DISEASE?
Who is at risk for DISEASE?
Where to find support for people with DISEASE?
Question Types About Drugs (20): Information, Interaction with medications, Interaction with food, Interaction with herbs and supplements, Important warning, Special instructions, Brand names, How does it work, How effective is it, Indication, Contraindication, Learn more, Side effects, Emergency or overdose, Severe reaction, Forget a dose, Dietary, Why get vaccinated, Storage and disposal, Usage, Dose 
Examples:
Are there interactions between DRUG and herbs and supplements?
What important warning or information should I know about DRUG?
Are there safety concerns or special precautions about DRUG?
What is the action of DRUG and how does it work?
Who should get DRUG and why is it prescribed?
What to do in case of a severe reaction to DRUG?
Question Type for other medical entities (e g  Procedure, Exam, Treatment): Information 
What is Coronary Artery Bypass Surgery?
What are Liver Function Tests?
Medical Resources
We used 12 trusted websites to construct a collection of question-answer pairs  For each website, we extracted the free text of each article as well as the synonyms of the article focus (topic)  These resources and their brief descriptions are provided below:
National Cancer Institute (NCI) : We extracted free text from 116 articles on various cancer types (729 QA pairs)  We manually restructured the content of the articles to generate complete answers (e g  a full answer about the treatment of all stages of a specific type of cancer)  Figure FIGREF54 presents examples of QA pairs generated from a NCI article 
Genetic and Rare Diseases Information Center (GARD): This resource contains information about various aspects of genetic/rare diseases  We extracted all disease question/answer pairs from 4,278 topics (5,394 QA pairs) 
Genetics Home Reference (GHR): This NLM resource contains consumer-oriented information about the effects of genetic variation on human health  We extracted 1,099 articles about diseases from this resource (5,430 QA pairs) 
MedlinePlus Health Topics: This portion of MedlinePlus contains information on symptoms, causes, treatment and prevention for diseases, health conditions and wellness issues  We extracted the free texts in summary sections of 981 articles (981 QA pairs) 
National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) : We extracted text from 174 health information pages on diseases studied by this institute (1,192 QA pairs) 
National Institute of Neurological Disorders and Stroke (NINDS): We extracted free text from 277 information pages on neurological and stroke-related diseases from this resource (1,104 QA pairs) 
NIHSeniorHealth : This website contains health and wellness information for older adults  We extracted 71 articles from this resource (769 QA pairs) 
National Heart, Lung, and Blood Institute (NHLBI) : We extracted text from 135 articles on diseases, tests, procedures, and other relevant topics on disorders of heart, lung, blood, and sleep (559 QA pairs) 
Centers for Disease Control and Prevention (CDC) : We extracted text from 152 articles on diseases and conditions (270 QA pairs) 
MedlinePlus A D A M  Medical Encyclopedia: This resource contains 4,366 articles about conditions, tests, and procedures  17,348 QA pairs were extracted from this resource  Figure FIGREF55 presents examples of QA pairs generated from A D A M encyclopedia 
MedlinePlus Drugs: We extracted free text from 1,316 articles about Drugs and generated 12,889 QA pairs 
MedlinePlus Herbs and Supplements: We extracted free text from 99 articles and generated 792 QA pairs 
The final collection contains 47,457 annotated question-answer pairs about Diseases, Drugs and other named entities (e g  Tests) extracted from these 12 trusted resources 
The Proposed Entailment-based QA System
Our goal is to generate a ranked list of answers for a given Premise Question INLINEFORM0 by ranking the recognized Hypothesis Questions INLINEFORM1   Based on the RQE experiments above (Section SECREF27 ), we selected Logistic Regression trained on the clinical-RQE dataset to recognize entailed questions and rank them with their classification scores 
RQE-based QA Approach
Classifying the full QA collection for each test question is not feasible for real-time applications  Therefore, we first filter the questions with an IR method to retrieve candidate questions, then classify them as entailed (or not) by the user/test question  Based on the positive results of the combination method tested on SemEval-cQA data (Section SECREF27 ), we adopted a combination method to merge the results obtained by the search engine and the RQE scores  The answers are then combined from both methods and ranked using an aggregate score  Figure FIGREF82 presents the overall architecture of the proposed QA system  We describe each module in more details next 
Finding Similar Question Candidates
For each premise question INLINEFORM0 , we use the Terrier search engine to retrieve INLINEFORM1 relevant question candidates INLINEFORM2 and then apply the RQE classifier to predict the labels for the pairs ( INLINEFORM3 , INLINEFORM4 ) 
We indexed the questions of our QA collection without the associated answers  In order to improve the indexing and the performance of question retrieval, we also indexed the synonyms of the question focus and the triggers of the question type with each question  This choice allowed us to avoid the shortcomings of query expansion, including incorrect or irrelevant synonyms and the increased execution time  The synonyms of the question focus (topic) were extracted automatically from the QA collection  The triggers of each question type were defined manually in the question types taxonomy  Below are two examples of indexed questions from our QA collection, with the automatically added focus synonyms and question type triggers:
What are the treatments for Torticollis?
Focus: Torticollis  Question type: Treatment 
Added focus synonyms: "Spasmodic torticollis, Wry neck, Loxia, Cervical dystonia"  Added question type triggers: "relieve, manage, cure, remedy, therapy" 
What is the outlook for Legionnaire disease?
Focus: Legionnaire disease  Question Type: Prognosis 
Added focus synonyms: "Legionella pneumonia, Pontiac fever, Legionellosis"  Added question type triggers: "prognosis, life expectancy" 
The IR task consists of retrieving hypothesis questions INLINEFORM0 relevant to the submitted question INLINEFORM1   As fusion of IR result has shown good performance in different tracks in TREC, we merge the results of the TF-IDF weighting function and the In-expB2 DFR model BIBREF50  
Let INLINEFORM0 = INLINEFORM1 , INLINEFORM2 ,    , INLINEFORM3 be the set of INLINEFORM4 questions retrieved by the first IR model INLINEFORM5 and INLINEFORM6 = INLINEFORM7 , INLINEFORM8 ,    , INLINEFORM9 be the set of INLINEFORM10 questions retrieved by the second IR model INLINEFORM11   We merge both sets by summing the scores of each retrieved question INLINEFORM12 in both INLINEFORM13 and INLINEFORM14 lists, then we re-rank the hypothesis questions INLINEFORM15  
Combining IR and RQE Methods
The IR models and the RQE Logistic Regression model bring different perspectives to the search for relevant candidate questions  In particular, question entailment allows understanding the relations between the important terms, whereas the traditional IR methods identify the important terms, but will not notice if the relations are opposite  Moreover, some of the question types that the RQE classifier learns will not be deemed important terms by traditional IR and the most relevant questions will not be ranked at the top of the list 
Therefore, in our approach, when a question is submitted to the system, candidate questions are fetched using the IR models, then the RQE classifier is applied to filter out the non-entailed questions and re-rank the remaining candidates 
Specifically, we denote INLINEFORM0 the list of question candidates INLINEFORM1 returned by the IR system  The premise question INLINEFORM2 is then used to construct N question pairs INLINEFORM3   The RQE classifier is then applied to filter out the question pairs that are not entailed and re-rank the remaining pairs 
More precisely, let INLINEFORM0 = INLINEFORM1 in INLINEFORM2 be the list of selected candidate questions that have a positive entailment relation with a given premise question INLINEFORM3   We rank INLINEFORM4 by computing a hybrid score INLINEFORM5 for each candidate question INLINEFORM6 taking into account the score of the IR system INLINEFORM7 and the score of the RQE system INLINEFORM8  
For each system INLINEFORM0 INLINEFORM1 , we normalize the associated score by dividing it by the maximum score among the INLINEFORM2 candidate questions retrieved by INLINEFORM3 for INLINEFORM4 :
INLINEFORM0
INLINEFORM0 INLINEFORM1
In our experiments, we fixed the value of INLINEFORM0 to 100  This threshold value was selected as a safe value for this task for the following reasons:
Our collection of 47,457 question-answer pairs was collected from only 12 NIH institutes and is unlikely to contain more than 100 occurrences of the same focus-type pair 
Each question was indexed with additional annotations for the question focus, its synonyms and the question type synonyms 
Evaluating RQE for Medical Question Answering
The objective of this evaluation is to study the effectiveness of RQE for Medical Question Answering, by comparing the answers retrieved by the hybrid entailment-based approach, the IR method and the other QA systems participating to the medical task at TREC 2017 LiveQA challenge (LiveQA-Med) 
Evaluation Method
We developed an interface to perform the manual evaluation of the retrieved answers  Figure 5 presents the evaluation interface showing, for each test question, the top-10 answers of the evaluated QA method and the reference answer(s) used by LiveQA assessors to help judging the retrieved answers by the participating systems 
We used the test questions of the medical task at TREC-2017 LiveQA BIBREF11   These questions are randomly selected from the consumer health questions that the NLM receives daily from all over the world  The test questions cover different medical entities and have a wide list of question types such as Comparison, Diagnosis, Ingredient, Side effects and Tapering 
For a relevant comparison, we used the same judgment scores as the LiveQA Track:
Correct and Complete Answer (4)
Correct but Incomplete (3)
Incorrect but Related (2)
Incorrect (1)
We evaluated the answers returned by the IR-based method and the hybrid QA method (IR+RQE) according to the same reference answers used in LiveQA-Med  The answers were anonymized (the method names were blinded) and presented to 3 assessors: a medical doctor (Assessor A), a medical librarian (B) and a researcher in medical informatics (C)  None of the assessors participated in the development of the QA methods  Assessors B and C evaluated 1,000 answers retrieved by each of the methods (IR and IR+RQE)  Assessor A evaluated 2,000 answers from both methods 
Table TABREF103 presents the inter-annotator agreement (IAA) through F1 score computed by considering one of the assessors as reference  In the first evaluation, we computed the True Positives (TP) and False Positives (FP) over all ratings and the Precision and F1 score  As there are no negative labels (only true or false positives for each category), Recall is 100%  We also computed a partial IAA by grouping the "Correct and Complete Answer" and "Correct but Incomplete" ratings (as Correct), and the "Incorrect but Related" and "Incorrect" ratings (as Incorrect)  The average agreement on distinguishing the Correct and Incorrect answers is 94 33% F1 score  Therefore, we used the evaluations performed by assessor A for both methods  The official results of the TREC LiveQA track relied on one assessor per question as well 
Evaluation of the first retrieved answer
We computed the measures used by TREC LiveQA challenges BIBREF51 , BIBREF11 to evaluate the first retrieved answer for each test question:
avgScore(0-3): the average score over all questions, transferring 1-4 level grades to 0-3 scores  This is the main score used to rank LiveQA runs 
succ@i+: the number of questions with score i or above (i INLINEFORM0 {2  4}) divided by the total number of questions 
prec@i+: the number of questions with score i or above (i INLINEFORM0 {2  4}) divided by number of questions answered by the system 
Table TABREF108 presents the average scores, success and precision results  The hybrid IR+RQE QA system achieved better results than the IR-based system with 0 827 average score  It also achieved a higher score than the best results achieved in the medical challenge at LiveQA'17  Evaluating the RQE system alone is not relevant, as applying RQE on the full collection for each user question is not feasible for a real-time system because of the extended execution time 
Evaluation of the top ten answers
In this evaluation, we used Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) which are commonly used in QA to evaluate the top-10 answers for each question  We consider answers rated as “Correct and Complete Answer” or “Correct but Incomplete” as correct answers, as the test questions contain multiple subquestions while each answer in our QA collection can cover only one subquestion 
MAP is the mean of the Average Precision (AvgP) scores over all questions 
(1) INLINEFORM0
Q is the number of questions  INLINEFORM0 is the AvgP of the INLINEFORM1 question 
INLINEFORM0
K is the number of correct answers  INLINEFORM0 is the rank of INLINEFORM1 correct answer 
MRR is the average of the reciprocal ranks for each question  The reciprocal rank of a question is the multiplicative inverse of the rank of the first correct answer 
(2) INLINEFORM0
Q is the number of questions  INLINEFORM0 is the rank of the first correct answer for the INLINEFORM1 question 
Table TABREF113 presents the MAP@10 and MRR@10 of our QA methods  The IR+RQE system outperforms the IR-based QA system with 0 311 MAP@10 and 0 333 MRR@10 
Discussion of entailment-based QA for the medical domain
In our evaluation, we followed the same LiveQA guidelines with the highest possible rigor  In particular, we consulted with NIST assessors who provided us with the paraphrases of the test questions that they used to judge the answers  Our IAA on the answers rating was also high compared to related tasks, with an 88 5% F1 agreement with the exact four categories and a 94 3% agreement when reducing the categories to two: “Correct” and “Incorrect” answers  Our results show that RQE improves the overall performance and exceeds the best results in the medical LiveQA'17 challenge by a factor of 29 8%  This performance improvement is particularly interesting as:
Our answer source has only 47K question-answer pairs when LiveQA participating systems relied on much larger collections, including the World Wide Web 
Our system answered one subquestion at most when many LiveQA test questions had several subquestions 
The latter observation, (b), makes the hybrid IR+RQE approach even more promising as it gives it a large potential for the improvement of answer completeness 
The former observation, (a), provides another interesting insight: restricting the answer source to only reliable collections can actually improve the QA performance without losing coverage (i e , our QA approach provided at least one answer to each test question and obtained the best relevance score) 
In another observation, the assessors reported that many of the returned answers had a correct question type but a wrong focus, which indicates that including a focus recognition module to filter such wrong answers can improve further the QA performance in terms of precision  Another aspect that was reported is the repetition of the same (or similar) answer from different websites, which could be addressed by improving answer selection with inter-answer comparisons and removal of near-duplicates  Also, half of the LiveQA test questions are about Drugs, when only two of our resources are specialized in Drugs, among 12 sub-collections overall  Accordingly, the assessors noticed that the performance of the QA systems was better on questions about diseases than on questions about drugs, which suggests a need for extending our medical QA collection with more information about drugs and associated question types 
We also looked closely at the private websites used by the LiveQA-Med annotators to provide some of the reference answers for the test questions  For instance, the ConsumerLab website was useful to answer a question about the ingredients of a Drug (COENZYME Q10)  Similarly, the eHealthMe website was used to answer a test question asking about interactions between two drugs (Phentermine and Dicyclomine) when no information was found in DailyMed  eHealthMe provides healthcare big data analysis and private research and studies including self-reported adverse drug effects by patients 
But the question remains on the extent to which such big data and other private websites could be used to automatically answer medical questions if information is otherwise unavailable  Unlike medical professionals, patients do not necessarily have the knowledge and tools to validate such information  An alternative approach could be to put limitations on medical QA systems in terms of the questions that can be answered (e g  "What is my diagnosis for such symptoms") and build classifiers to detect such questions and warn the users about the dangers of looking for their answers online 
More generally, medical QA systems should follow some strict guidelines regarding the goal and background knowledge and resources of each system in order to protect the consumers from misleading or harmful information  Such guidelines could be based (i) on the source of the information such as health and medical information websites sponsored by the U S  government, not-for-profit health or medical organizations, and medical university centers, or (ii) on conventions such as the code of conduct of the HON Foundation (HONcode) that addresses the reliability and usefulness of medical information on the Internet 
Our experiments show that limiting the number of answer sources with such guidelines is not only feasible, but it could also enhance the performance of the QA system from an information retrieval perspective 
Conclusion
In this paper, we carried out an empirical study of machine learning and deep learning methods for Recognizing Question Entailment in the medical domain using several datasets  We developed a RQE-based QA system to answer new medical questions using existing question-answer pairs  We built and shared a collection of 47K medical question-answer pairs  Our QA approach outperformed the best results on TREC-2017 LiveQA medical test questions  The proposed approach can be applied and adapted to open-domain as well as specific-domain QA  Deep learning models achieved interesting results on open-domain and clinical datasets, but obtained a lower performance on consumer health questions  We will continue investigating other network architectures including transfer learning, as well as creation of a large collection of consumer health questions for training to improve the performance of DL models  Future work also includes exploring integration of a Question Focus Recognition module to enhance candidate question retrieval, and expanding our question-answer collection 
Acknowledgements
We thank Halil Kilicoglu (NLM/NIH) for his help with the crawling and the manual evaluation and Sonya E  Shooshan (NLM/NIH) for her help with the judgment of the retrieved answers  We also thank Ellen Voorhees (NIST) for her valuable support with the TREC LiveQA evaluation 
We consider the case of the question number 36 in the TREC-2017 LiveQA medical test dataset:
36  congenital diaphragmatic hernia  what are the causes of congenital diaphragmatic hernia? Can cousin marriage cause this? What kind of lung disease the baby might experience life long?
This question was answered by 5 participating runs (vs  8 runs for other questions), and all submitted answers were wrong (scores of 1 or 2)  However, our IR-based QA system retrieved one excellent answer (score 4) and our hybrid IR+RQE system provided 3 excellent answers 
A) TREC 2017 LiveQA-Med Participants' Results:
B) Our IR-based QA System:
C) Our IR+RQE QA System:

ground_truth: Logistic Regression, neural networks

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Does the paper explore extraction from electronic health records?
Context 3: Introduction
The explosion of available scientific articles in the Biomedical domain has led to the rise of Biomedical Information Extraction (BioIE)  BioIE systems aim to extract information from a wide spectrum of articles including medical literature, biological literature, electronic health records, etc  that can be used by clinicians and researchers in the field  Often the outputs of BioIE systems are used to assist in the creation of databases, or to suggest new paths for research  For example, a ranked list of interacting proteins that are extracted from biomedical literature, but are not present in existing databases, can allow researchers to make informed decisions about which protein/gene to study further  Interactions between drugs are necessary for clinicians who simultaneously administer multiple drugs to their patients  A database of diseases, treatments and tests is beneficial for doctors consulting in complicated medical cases 
The main problems in BioIE are similar to those in Information Extraction:
This paper discusses, in each section, various methods that have been adopted to solve the listed problems  Each section also highlights the difficulty of Information Extraction tasks in the biomedical domain 
This paper is intended as a primer to Biomedical Information Extraction for current NLP researchers  It aims to highlight the diversity of the various techniques from Information Extraction that have been applied in the Biomedical domain  The state of biomedical text mining is reviewed regularly  For more extensive surveys, consult BIBREF0 , BIBREF1 , BIBREF2  
Named Entity Recognition and Fact Extraction
Named Entity Recognition (NER) in the Biomedical domain usually includes recognition of entities such as proteins, genes, diseases, treatments, drugs, etc  Fact extraction involves extraction of Named Entities from a corpus, usually given a certain ontology  When compared to NER in the domain of general text, the biomedical domain has some characteristic challenges:
Some of the earliest systems were heavily dependent on hand-crafted features  The method proposed in BIBREF4 for recognition of protein names in text does not require any prepared dictionary  The work gives examples of diversity in protein names and lists multiple rules depending on simple word features as well as POS tags 
BIBREF5 adopt a machine learning approach for NER  Their NER system extracts medical problems, tests and treatments from discharge summaries and progress notes  They use a semi-Conditional Random Field (semi-CRF) BIBREF6 to output labels over all tokens in the sentence  They use a variety of token, context and sentence level features  They also use some concept mapping features using existing annotation tools, as well as Brown clustering to form 128 clusters over the unlabelled data  The dataset used is the i2b2 2010 challenge dataset  Their system achieves an F-Score of 0 85  BIBREF7 is an incremental paper on NER taggers  It uses 3 types of word-representation techniques (Brown clustering, distributional clustering, word vectors) to improve performance of the NER Conditional Random Field tagger, and achieves marginal F-Score improvements 
BIBREF8 propose a boostrapping mechanism to bootstrap biomedical ontologies using NELL BIBREF9 , which uses a coupled semi-supervised bootstrapping approach to extract facts from text, given an ontology and a small number of “seed” examples for each category  This interesting approach (called BioNELL) uses an ontology of over 100 categories  In contrast to NELL, BioNELL does not contain any relations in the ontology  BioNELL is motivated by the fact that a lot of scientific literature available online is highly reliable due to peer-review  The authors note that the algorithm used by NELL to bootstrap fails in BioNELL due to ambiguities in biomedical literature, and heavy semantic drift  One of the causes for this is that often common words such as “white”, “dad”, “arm” are used as names of genes- this can easily result in semantic drift in one iteration of the bootstrapping  In order to mitigate this, they use Pointwise Mutual Information scores for corpus level statistics, which attributes a small score to common words  In addition, in contrast to NELL, BioNELL only uses high instances as seeds in the next iteration, but adds low ranking instances to the knowledge base  Since evaluation is not possible using Mechanical Turk or a small number of experts (due to the complexity of the task), they use Freebase BIBREF10 , a knowledge base that has some biomedical concepts as well  The lexicon learned using BioNELL is used to train an NER system  The system shows a very high precision, thereby showing that BioNELL learns very few ambiguous terms 
More recently, deep learning techniques have been developed to further enhance the performance of NER systems  BIBREF11 explore recurrent neural networks for the problem of NER in biomedical text 
Relation Extraction
In Biomedical Information Extraction, Relation Extraction involves finding related entities of many different kinds  Some of these include protein-protein interactions, disease-gene relations and drug-drug interactions  Due to the explosion of available biomedical literature, it is impossible for one person to extract relevant relations from published material  Automatic extraction of relations assists in the process of database creation, by suggesting potentially related entities with links to the source article  For example, a database of drug-drug interactions is important for clinicians who administer multiple drugs simultaneously to their patients- it is imperative to know if one drug will have an adverse effect on the other  A variety of methods have been developed for relation extractions, and are often inspired by Relation Extraction in NLP tasks  These include rule-based approaches, hand-crafted patterns, feature-based and kernel machine learning methods, and more recently deep learning architectures  Relation Extraction systems over Biomedical Corpora are often affected by noisy extraction of entities, due to ambiguities in names of proteins, genes, drugs etc 
BIBREF12 was one of the first large scale Information Extraction efforts to study the feasibility of extraction of protein-protein interactions (such as “protein A activates protein B") from Biomedical text  Using 8 hand-crafted regular expressions over a fixed vocabulary, the authors were able to achieve a recall of 30% for interactions present in The Dictionary of Interacting Proteins (DIP) from abstracts in Medline  The method did not differentiate between the type of relation  The reasons for the low recall were the inconsistency in protein nomenclature, information not present in the abstract, and due to specificity of the hand-crafted patterns  On a small subset of extracted relations, they found that about 60% were true interactions between proteins not present in DIP 
BIBREF13 combine sentence level relation extraction for protein interactions with corpus level statistics  Similar to BIBREF12 , they do not consider the type of interaction between proteins- only whether they interact in the general sense of the word  They also do not differentiate between genes and their protein products (which may share the same name)  They use Pointwise Mutual Information (PMI) for corpus level statistics to determine whether a pair of proteins occur together by chance or because they interact  They combine this with a confidence aggregator that takes the maximum of the confidence of the extractor over all extractions for the same protein-pair  The extraction uses a subsequence kernel based on BIBREF14   The integrated model, that combines PMI with aggregate confidence, gives the best performance  Kernel methods have widely been studied for Relation Extraction in Biomedical Literature  Common kernels used usually exploit linguistic information by utilising kernels based on the dependency tree BIBREF15 , BIBREF16 , BIBREF17  
BIBREF18 look at the extraction of diseases and their relevant genes  They use a dictionary from six public databases to annotate genes and diseases in Medline abstracts  In their work, the authors note that when both genes and diseases are correctly identified, they are related in 94% of the cases  The problem then reduces to filtering incorrect matches using the dictionary, which occurs due to false positives resulting from ambiguities in the names as well as ambiguities in abbreviations  To this end, they train a Max-Ent based NER classifier for the task, and get a 26% gain in precision over the unfiltered baseline, with a slight hit in recall  They use POS tags, expanded forms of abbreviations, indicators for Greek letters as well as suffixes and prefixes commonly used in biomedical terms 
BIBREF19 adopt a supervised feature-based approach for the extraction of drug-drug interaction (DDI) for the DDI-2013 dataset BIBREF20   They partition the data in subsets depending on the syntactic features, and train a different model for each  They use lexical, syntactic and verb based features on top of shallow parse features, in addition to a hand-crafted list of trigger words to define their features  An SVM classifier is then trained on the feature vectors, with a positive label if the drug pair interacts, and negative otherwise  Their method beats other systems on the DDI-2013 dataset  Some other feature-based approaches are described in BIBREF21 , BIBREF22  
Distant supervision methods have also been applied to relation extraction over biomedical corpora  In BIBREF23 , 10,000 neuroscience articles are distantly supervised using information from UMLS Semantic Network to classify brain-gene relations into geneExpression and otherRelation  They use lexical (bag of words, contextual) features as well as syntactic (dependency parse features)  They make the “at-least one” assumption, i e  at least one of the sentences extracted for a given entity-pair contains the relation in database  They model it as a multi-instance learning problem and adopt a graphical model similar to BIBREF24   They test using manually annotated examples  They note that the F-score achieved are much lesser than that achieved in the general domain in BIBREF24 , and attribute to generally poorer performance of NER tools in the biomedical domain, as well as less training examples  BIBREF25 explore distant supervision methods for protein-protein interaction extraction 
More recently, deep learning methods have been applied to relation extraction in the biomedical domain  One of the main advantages of such methods over traditional feature or kernel based learning methods is that they require minimal feature engineering  In BIBREF26 , skip-gram vectors BIBREF27 are trained over 5 6Gb of unlabelled text  They use these vectors to extract protein-protein interactions by converting them into features for entities, context and the entire sentence  Using an SVM for classification, their method is able to outperform many kernel and feature based methods over a variety of datasets 
BIBREF28 follow a similar method by using word vectors trained on PubMed articles  They use it for the task of relation extraction from clinical text for entities that include problem, treatment and medical test  For a given sentence, given labelled entities, they predict the type of relation exhibited (or None) by the entity pair  These types include “treatment caused medical problem”, “test conducted to investigate medical problem”, “medical problem indicates medical problems”, etc  They use a Convolutional Neural Network (CNN) followed by feedforward neural network architecture for prediction  In addition to pre-trained word vectors as features, for each token they also add features for POS tags, distance from both the entities in the sentence, as well BIO tags for the entities  Their model performs better than a feature based SVM baseline that they train themselves 
The BioNLP'16 Shared Tasks has also introduced some Relation Extraction tasks, in particular the BB3-event subtask that involves predicting whether a “lives-in” relation holds for a Bacteria in a location  Some of the top performing models for this task are deep learning models  BIBREF29 train word embeddings with six billions words of scientific texts from PubMed  They then consider the shortest dependency path between the two entities (Bacteria and location)  For each token in the path, they use word embedding features, POS type embeddings and dependency type embeddings  They train a unidirectional LSTM BIBREF30 over the dependency path, that achieves an F-Score of 52 1% on the test set 
BIBREF31 improve the performance by making modifications to the above model  Instead of using the shortest dependency path, they modify the parse tree based on some pruning strategies  They also add feature embeddings for each token to represent the distance from the entities in the shortest path  They then train a Bidirectional LSTM on the path, and obtain an F-Score of 57 1% 
The recent success of deep learning models in Biomedical Relation Extraction that require minimal feature engineering is promising  This also suggests new avenues of research in the field  An approach as in BIBREF32 can be used to combine multi-instance learning and distant supervision with a neural architecture 
Event Extraction
Event Extraction in the Biomedical domain is a task that has gained more importance recently  Event Extraction goes beyond Relation Extraction  In Biomedical Event Extraction, events generally refer to a change in the state of biological molecules such as proteins and DNA  Generally, it includes detection of targeted event types such as gene expression, regulation, localisation and transcription  Each event type in addition can have multiple arguments that need to be detected  An additional layer of complexity comes from the fact that events can also be arguments of other events, giving rise to a nested structure  This helps to capture the underlying biology better BIBREF1   Detecting the event type often involves recognising and classifying trigger words  Often, these words are verbs such as “activates”, “inhibits”, “phosphorylation” that may indicate a single, or sometimes multiple event types  In this section, we will discuss some of the successful models for Event Extraction in some detail 
Event Extraction gained a lot of interest with the availability of an annotated corpus with the BioNLP'09 Shared Task on Event Extraction BIBREF34   The task involves prediction of trigger words over nine event types such as expression, transcription, catabolism, binding, etc  given only annotation of named entities (proteins, genes, etc )  For each event, its class, trigger expression and arguments need to be extracted  Since the events can be arguments to other events, the final output in general is a graph representation with events and named entities as nodes, and edges that correspond to event arguments  BIBREF33 present a pipeline based method that is heavily dependent on dependency parsing  Their pipeline approach consists of three steps: trigger detection, argument detection and semantic post-processing  While the first two components are learning based systems, the last component is a rule based system  For the BioNLP'09 corpus, only 5% of the events span multiple sentences  Hence the approach does not get affected severely by considering only single sentences  It is important to note that trigger words cannot simply be reduced to a dictionary lookup  This is because a specific word may belong to multiple classes, or may not always be a trigger word for an event  For example, “activate” is found to not be a trigger word in over 70% of the cases  A multi-class SVM is trained for trigger detection on each token, using a large feature set consisting of semantic and syntactic features  It is interesting to note that the hyperparameters of this classifier are optimised based on the performance of the entire end-to-end system 
For the second component to detect arguments, labels for edges between entities must be predicted  For the BioNLP'09 Shared Task, each directed edge from one event node to another event node, or from an event node to a named entity node are classified as “theme”, “cause”, or None  The second component of the pipeline makes these predictions independently  This is also trained using a multi-class SVM which involves heavy use of syntactic features, including the shortest dependency path between the nodes  The authors note that the precision-recall choice of the first component affects the performance of the second component: since the second component is only trained on Gold examples, any error by the first component will lead to a cascading of errors  The final component, which is a semantic post-processing step, consists of rules and heuristics to correct the output of the second component  Since the edge predictions are made independently, it is possible that some event nodes do not have any edges, or have an improper combination of edges  The rule based component corrects these and applies rules to break directed cycles in the graph, and some specific heuristics for different types of events  The final model gives a cumulative F-Score of 52% on the test set, and was the best model on the task 
BIBREF35 note that previous approaches on the task suffer due to the pipeline nature and the propagation of errors  To counter this, they adopt a joint inference method based on Markov Logic Networks BIBREF36 for the same task on BioNLP'09  The Markov Logic Network jointly predicts whether each token is a trigger word, and if yes, the class it belongs to; for each dependency edge, whether it is an argument path leading to a “theme” or a “cause”  By formulating the Event Extraction problem using an MLN, the approach becomes computationally feasible and only linear in the length of the sentence  They incorporate hard constraints to encode rules such as “an argument path must have an event”, “a cause path must start with a regulation event”, etc  In addition, they also include some domain specific soft constraints as well as some linguistically-motivated context-specific soft constraints  In order to train the MLN, stochastic gradient descent was used  Certain heuristic methods are implemented in order to deal with errors due to syntactic parsing, especially ambiguities in PP-attachment and coordination  Their final system is competitive and comes very close to the system by BIBREF33 with an average F-Score of 50%  To further improve the system, they suggest leveraging additional joint-inference opportunities and integrating the syntactic parser better  Some other more recent models for Biomedical Event Extraction include BIBREF37 , BIBREF38  
Conclusion
We have discussed some of the major problems and challenges in BioIE, and seen some of the diverse approaches adopted to solve them  Some interesting problems such as Pathway Extraction for Biological Systems BIBREF39 , BIBREF40 have not been discussed 
Biomedical Information Extraction is a challenging and exciting field for NLP researchers that demands application of state-of-the-art methods  Traditionally, there has been a dependence on hand-crafted features or heavily feature-engineered methods  However, with the advent of deep learning methods, a lot of BioIE tasks are seeing an improvement by adopting deep learning models such as Convolutional Neural Networks and LSTMs, which require minimal feature engineering  Rapid progress in developing better systems for BioIE will be extremely helpful for clinicians and researchers in the Biomedical domain 

ground_truth: Yes