prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the tagging scheme employed?
Context 1: Introduction
There exists a class of language construction known as pun in natural language texts and utterances, where a certain word or other lexical items are used to exploit two or more separate meanings  It has been shown that understanding of puns is an important research question with various real-world applications, such as human-computer interaction BIBREF0 , BIBREF1 and machine translation BIBREF2   Recently, many researchers show their interests in studying puns, like detecting pun sentences BIBREF3 , locating puns in the text BIBREF4 , interpreting pun sentences BIBREF5 and generating sentences containing puns BIBREF6 , BIBREF7 , BIBREF8   A pun is a wordplay in which a certain word suggests two or more meanings by exploiting polysemy, homonymy, or phonological similarity to another sign, for an intended humorous or rhetorical effect  Puns can be generally categorized into two groups, namely heterographic puns (where the pun and its latent target are phonologically similar) and homographic puns (where the two meanings of the pun reflect its two distinct senses) BIBREF9   Consider the following two examples:
The first punning joke exploits the sound similarity between the word “propane" and the latent target “profane", which can be categorized into the group of heterographic puns  Another categorization of English puns is homographic pun, exemplified by the second instance leveraging distinct senses of the word “gut" 
Pun detection is the task of detecting whether there is a pun residing in the given text  The goal of pun location is to find the exact word appearing in the text that implies more than one meanings  Most previous work addresses such two tasks separately and develop separate systems BIBREF10 , BIBREF5   Typically, a system for pun detection is built to make a binary prediction on whether a sentence contains a pun or not, where all instances (with or without puns) are taken into account during training  For the task of pun location, a separate system is used to make a single prediction as to which word in the given sentence in the text that trigger more than one semantic interpretations of the text, where the training data involves only sentences that contain a pun  Therefore, if one is interested in solving both problems at the same time, a pipeline approach that performs pun detection followed by pun location can be used 
Compared to the pipeline methods, joint learning has been shown effective BIBREF11 , BIBREF12 since it is able to reduce error propagation and allows information exchange between tasks which is potentially beneficial to all the tasks  In this work, we demonstrate that the detection and location of puns can be jointly addressed by a single model  The pun detection and location tasks can be combined as a sequence labeling problem, which allows us to jointly detect and locate a pun in a sentence by assigning each word a tag  Since each context contains a maximum of one pun BIBREF9 , we design a novel tagging scheme to capture this structural constraint  Statistics on the corpora also show that a pun tends to appear in the second half of a context  To capture such a structural property, we also incorporate word position knowledge into our structured prediction model  Experiments on the benchmark datasets show that detection and location tasks can reinforce each other, leading to new state-of-the-art performance on these two tasks  To the best of our knowledge, this is the first work that performs joint detection and location of English puns by using a sequence labeling approach 
Problem Definition
We first design a simple tagging scheme consisting of two tags { INLINEFORM0 }:
INLINEFORM0 tag means the current word is not a pun 
INLINEFORM0 tag means the current word is a pun 
If the tag sequence of a sentence contains a INLINEFORM0 tag, then the text contains a pun and the word corresponding to INLINEFORM1 is the pun 
The contexts have the characteristic that each context contains a maximum of one pun BIBREF9   In other words, there exists only one pun if the given sentence is detected as the one containing a pun  Otherwise, there is no pun residing in the text  To capture this interesting property, we propose a new tagging scheme consisting of three tags, namely { INLINEFORM0 } 
INLINEFORM0 tag indicates that the current word appears before the pun in the given context 
INLINEFORM0 tag highlights the current word is a pun 
INLINEFORM0 tag indicates that the current word appears after the pun 
We empirically show that the INLINEFORM0 scheme can guarantee the context property that there exists a maximum of one pun residing in the text 
Given a context from the training set, we will be able to generate its corresponding gold tag sequence using a deterministic procedure  Under the two schemes, if a sentence does not contain any puns, all words will be tagged with INLINEFORM0 or INLINEFORM1 , respectively  Exemplified by the second sentence “Some diets cause a gut reaction," the pun is given as “gut " Thus, under the INLINEFORM2 scheme, it should be tagged with INLINEFORM3 , while the words before it are assigned with the tag INLINEFORM4 and words after it are with INLINEFORM5 , as illustrated in Figure FIGREF8   Likewise, the INLINEFORM6 scheme tags the word “gut" with INLINEFORM7 , while other words are tagged with INLINEFORM8   Therefore, we can combine the pun detection and location tasks into one problem which can be solved by the sequence labeling approach 
Model
Neural models have shown their effectiveness on sequence labeling tasks BIBREF13 , BIBREF14 , BIBREF15   In this work, we adopt the bidirectional Long Short Term Memory (BiLSTM) BIBREF16 networks on top of the Conditional Random Fields BIBREF17 (CRF) architecture to make labeling decisions, which is one of the classical models for sequence labeling  Our model architecture is illustrated in Figure FIGREF8 with a running example  Given a context/sentence INLINEFORM0 where INLINEFORM1 is the length of the context, we generate the corresponding tag sequence INLINEFORM2 based on our designed tagging schemes and the original annotations for pun detection and location provided by the corpora  Our model is then trained on pairs of INLINEFORM3  
Input  The contexts in the pun corpus hold the property that each pun contains exactly one content word, which can be either a noun, a verb, an adjective, or an adverb  To capture this characteristic, we consider lexical features at the character level  Similar to the work of BIBREF15 , the character embeddings are trained by the character-level LSTM networks on the unannotated input sequences  Nonlinear transformations are then applied to the character embeddings by highway networks BIBREF18 , which map the character-level features into different semantic spaces 
We also observe that a pun tends to appear at the end of a sentence  Specifically, based on the statistics, we found that sentences with a pun that locate at the second half of the text account for around 88% and 92% in homographic and heterographic datasets, respectively  We thus introduce a binary feature that indicates if a word is located at the first or the second half of an input sentence to capture such positional information  A binary indicator can be mapped to a vector representation using a randomly initialized embedding table BIBREF19 , BIBREF20   In this work, we directly adopt the value of the binary indicator as part of the input 
The concatenation of the transformed character embeddings, the pre-trained word embeddings BIBREF21 , and the position indicators are taken as input of our model 
Tagging  The input is then fed into a BiLSTM network, which will be able to capture contextual information  For a training instance INLINEFORM0 , we suppose the output by the word-level BiLSTM is INLINEFORM1   The CRF layer is adopted to capture label dependencies and make final tagging decisions at each position, which has been included in many state-of-the-art sequence labeling models BIBREF14 , BIBREF15   The conditional probability is defined as:
where INLINEFORM0 is a set of all possible label sequences consisting of tags from INLINEFORM1 (or INLINEFORM2 ), INLINEFORM3 and INLINEFORM4 are weight and bias parameters corresponding to the label pair INLINEFORM5   During training, we minimize the negative log-likelihood summed over all training instances:
where INLINEFORM0 refers to the INLINEFORM1 -th instance in the training set  During testing, we aim to find the optimal label sequence for a new input INLINEFORM2 :
This search process can be done efficiently using the Viterbi algorithm 
Datasets and Settings
We evaluate our model on two benchmark datasets BIBREF9   The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun  The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun  We notice there is no standard splitting information provided for both datasets  Thus we apply 10-fold cross validation  To make direct comparisons with prior studies, following BIBREF4 , we accumulated the predictions for all ten folds and calculate the scores in the end 
For each fold, we randomly select 10% of the instances from the training set for development  Word embeddings are initialized with the 100-dimensional Glove BIBREF21   The dimension of character embeddings is 30 and they are randomly initialized, which can be fine tuned during training  The pre-trained word embeddings are not updated during training  The dimensions of hidden vectors for both char-level and word-level LSTM units are set to 300  We adopt stochastic gradient descent (SGD) BIBREF26 with a learning rate of 0 015 
For the pun detection task, if the predicted tag sequence contains at least one INLINEFORM0 tag, we regard the output (i e , the prediction of our pun detection model) for this task as true, otherwise false  For the pun location task, a predicted pun is regarded as correct if and only if it is labeled as the gold pun in the dataset  As to pun location, to make fair comparisons with prior studies, we only consider the instances that are labeled as the ones containing a pun  We report precision, recall and INLINEFORM1 score in Table TABREF11   A list of prior works that did not employ joint learning are also shown in the first block of Table TABREF11  
Results
We also implemented a baseline model based on conditional random fields (CRF), where features like POS tags produced by the Stanford POS tagger BIBREF27 , n-grams, label transitions, word suffixes and relative position to the end of the text are considered  We can see that our model with the INLINEFORM0 tagging scheme yields new state-of-the-art INLINEFORM1 scores on pun detection and competitive results on pun location, compared to baselines that do not adopt joint learning in the first block  For location on heterographic puns, our model's performance is slightly lower than the system of BIBREF25 , which is a rule-based locator  Compared to CRF, we can see that our model, either with the INLINEFORM2 or the INLINEFORM3 scheme, yields significantly higher recall on both detection and location tasks, while the precisions are relatively close  This demonstrates the effectiveness of BiLSTM, which learns the contextual features of given texts – such information appears to be helpful in recalling more puns 
Compared to the INLINEFORM0 scheme, the INLINEFORM1 tagging scheme is able to yield better performance on these two tasks  After studying outputs from these two approaches, we found that one leading source of error for the INLINEFORM2 approach is that there exist more than one words in a single instance that are assigned with the INLINEFORM3 tag  However, according to the description of pun in BIBREF9 , each context contains a maximum of one pun  Thus, such a useful structural constraint is not well captured by the simple approach based on the INLINEFORM4 tagging scheme  On the other hand, by applying the INLINEFORM5 tagging scheme, such a constraint is properly captured in the model  As a result, the results for such a approach are significantly better than the approach based on the INLINEFORM6 tagging scheme, as we can observe from the table  Under the same experimental setup, we also attempted to exclude word position features  Results are given by INLINEFORM7 - INLINEFORM8   It is expected that the performance of pun location drops, since such position features are able to capture the interesting property that a pun tends to appear in the second half of a sentence  While such knowledge is helpful for the location task, interestingly, a model without position knowledge yields improved performance on the pun detection task  One possible reason is that detecting whether a sentence contains a pun is not concerned with such word position information 
Additionally, we conduct experiments over sentences containing a pun only, namely 1,607 and 1,271 instances from homographic and heterographic pun corpora separately  It can be regarded as a “pipeline” method where the classifier for pun detection is regarded as perfect  Following the prior work of BIBREF4 , we apply 10-fold cross validation  Since we are given that all input sentences contain a pun, we only report accumulated results on pun location, denoted as Pipeline in Table TABREF11   Compared with our approaches, the performance of such an approach drops significantly  On the other hand, such a fact demonstrates that the two task, detection and location of puns, can reinforce each other  These figures demonstrate the effectiveness of our sequence labeling method to detect and locate English puns in a joint manner 
Error Analysis
We studied the outputs from our system and make some error analysis  We found the errors can be broadly categorized into several types, and we elaborate them here  1) Low word coverage: since the corpora are relatively small, there exist many unseen words in the test set  Learning the representations of such unseen words is challenging, which affects the model's performance  Such errors contribute around 40% of the total errors made by our system  2) Detection errors: we found many errors are due to the model's inability to make correct pun detection  Such inability harms both pun detection and pun location  Although our approach based on the INLINEFORM0 tagging scheme yields relatively higher scores on the detection task, we still found that 40% of the incorrectly predicted instances fall into this group  3) Short sentences: we found it was challenging for our model to make correct predictions when the given text is short  Consider the example “Superglue! Tom rejoined," here the word rejoined is the corresponding pun  However, it would be challenging to figure out the pun with such limited contextual information 
Related Work
Most existing systems address pun detection and location separately  BIBREF22 applied word sense knowledge to conduct pun detection  BIBREF24 trained a bidirectional RNN classifier for detecting homographic puns  Next, a knowledge-based approach is adopted to find the exact pun  Such a system is not applicable to heterographic puns  BIBREF28 applied Google n-gram and word2vec to make decisions  The phonetic distance via the CMU Pronouncing Dictionary is computed to detect heterographic puns  BIBREF10 used the hidden Markov model and a cyclic dependency network with rich features to detect and locate puns  BIBREF23 used a supervised approach to pun detection and a weakly supervised approach to pun location based on the position within the context and part of speech features  BIBREF25 proposed a rule-based system for pun location that scores candidate words according to eleven simple heuristics  Two systems are developed to conduct detection and location separately in the system known as UWAV BIBREF3   The pun detector combines predictions from three classifiers  The pun locator considers word2vec similarity between every pair of words in the context and position to pinpoint the pun  The state-of-the-art system for homographic pun location is a neural method BIBREF4 , where the word senses are incorporated into a bidirectional LSTM model  This method only supports the pun location task on homographic puns  Another line of research efforts related to this work is sequence labeling, such as POS tagging, chunking, word segmentation and NER  The neural methods have shown their effectiveness in this task, such as BiLSTM-CNN BIBREF13 , GRNN BIBREF29 , LSTM-CRF BIBREF30 , LSTM-CNN-CRF BIBREF14 , LM-LSTM-CRF BIBREF15  
In this work, we combine pun detection and location tasks as a single sequence labeling problem  Inspired by the work of BIBREF15 , we also adopt a LSTM-CRF with character embeddings to make labeling decisions 
Conclusion
In this paper, we propose to perform pun detection and location tasks in a joint manner from a sequence labeling perspective  We observe that each text in our corpora contains a maximum of one pun  Hence, we design a novel tagging scheme to incorporate such a constraint  Such a scheme guarantees that there is a maximum of one word that will be tagged as a pun during the testing phase  We also found the interesting structural property such as the fact that most puns tend to appear at the second half of the sentences can be helpful for such a task, but was not explored in previous works  Furthermore, unlike many previous approaches, our approach, though simple, is generally applicable to both heterographic and homographic puns  Empirical results on the benchmark datasets prove the effectiveness of the proposed approach that the two tasks of pun detection and location can be addressed by a single model from a sequence labeling perspective 
Future research includes the investigations on how to make use of richer semantic and linguistic information for detection and location of puns  Research on puns for other languages such as Chinese is still under-explored, which could also be an interesting direction for our future studies 
Acknowledgments
We would like to thank the three anonymous reviewers for their thoughtful and constructive comments  This work is supported by Singapore Ministry of Education Academic Research Fund (AcRF) Tier 2 Project MOE2017-T2-1-156, and is partially supported by SUTD project PIE-SGP-AI-2018-01 
Context 2: Introduction
With the availability of rich data on users' locations, profiles and search history, personalization has become the leading trend in large-scale information retrieval  However, efficiency through personalization is not yet the most suitable model when tackling domain-specific searches  This is due to several factors, such as the lexical and semantic challenges of domain-specific data that often include advanced argumentation and complex contextual information, the higher sparseness of relevant information sources, and the more pronounced lack of similarities between users' searches 
A recent study on expert search strategies among healthcare information professionals BIBREF0 showed that, for a given search task, they spend an average of 60 minutes per collection or database, 3 minutes to examine the relevance of each document, and 4 hours of total search time  When written in steps, their search strategy spans over 15 lines and can reach up to 105 lines 
With the abundance of information sources in the medical domain, consumers are more and more faced with a similar challenge, one that needs dedicated solutions that can adapt to the heterogeneity and specifics of health-related information 
Dedicated Question Answering (QA) systems are one of the viable solutions to this problem as they are designed to understand natural language questions without relying on external information on the users 
In the context of QA, the goal of Recognizing Question Entailment (RQE) is to retrieve answers to a premise question ( INLINEFORM0 ) by retrieving inferred or entailed questions, called hypothesis questions ( INLINEFORM1 ) that already have associated answers  Therefore, we define the entailment relation between two questions as: a question INLINEFORM2 entails a question INLINEFORM3 if every answer to INLINEFORM4 is also a correct answer to INLINEFORM5 BIBREF1  
RQE is particularly relevant due to the increasing numbers of similar questions posted online BIBREF2 and its ability to solve differently the challenging issues of question understanding and answer extraction  In addition to being used to find relevant answers, these resources can also be used in training models able to recognize inference relations and similarity between questions 
Question similarity has recently attracted international challenges BIBREF3 , BIBREF4 and several research efforts proposing a wide range of approaches, including Logistic Regression, Recurrent Neural Networks (RNNs), Long Short Term Memory cells (LSTMs), and Convolutional Neural Networks (CNNs) BIBREF5 , BIBREF6 , BIBREF1 , BIBREF7  
In this paper, we study question entailment in the medical domain and the effectiveness of the end-to-end RQE-based QA approach by evaluating the relevance of the retrieved answers  Although entailment was attempted in QA before BIBREF8 , BIBREF9 , BIBREF10 , as far as we know, we are the first to introduce and evaluate a full medical question answering approach based on question entailment for free-text questions  Our contributions are:
The next section is dedicated to related work on question answering, question similarity and entailment  In Section SECREF3 , we present two machine learning (ML) and deep learning (DL) methods for RQE and compare their performance using open-domain and clinical datasets  Section SECREF4 describes the new collection of medical question-answer pairs  In Section SECREF5 , we describe our RQE-based approach for QA  Section SECREF6 presents our evaluation of the retrieved answers and the results obtained on TREC 2017 LiveQA medical questions 
Background
In this section we define the RQE task and describe related work at the intersection of question answering, question similarity and textual inference 
Task Definition
The definition of Recognizing Question Entailment (RQE) can have a significant impact on QA results  In related work, the meaning associated with Natural Language Inference (NLI) varies among different tasks and events  For instance, Recognizing Textual Entailment (RTE) was addressed by the PASCAL challenge BIBREF12 , where the entailment relation has been assessed manually by human judges who selected relevant sentences "entailing" a set of hypotheses from a list of documents returned by different Information Retrieval (IR) methods  In another definition, the Stanford Natural Language Inference corpus SNLI BIBREF13 , used three classification labels for the relations between two sentences: entailment, neutral and contradiction  For the entailment label, the annotators who built the corpus were presented with an image and asked to write a caption “that is a definitely true description of the photo”  For the neutral label, they were asked to provide a caption “that might be a true description of the label”  They were asked for a caption that “is definitely a false description of the photo” for the contradiction label 
More recently, the multiNLI corpus BIBREF14 was shared in the scope of the RepEval 2017 shared task BIBREF15   To build the corpus, annotators were presented with a premise text and asked to write three sentences  One novel sentence, which is “necessarily true or appropriate in the same situations as the premise,” for the entailment label, a sentence, which is “necessarily false or inappropriate whenever the premise is true,” for the contradiction label, and a last sentence, “where neither condition applies,” for the neutral label 
Whereas these NLI definitions might be suitable for the broad topic of text understanding, their relation to practical information retrieval or question answering systems is not straightforward 
In contrast, RQE has to be tailored to the question answering task  For instance, if the premise question is "looking for cold medications for a 30 yo woman", a RQE approach should be able to consider the more general (less restricted) question "looking for cold medications" as relevant, since its answers are relevant for the initial question, whereas "looking for medications for a 30 yo woman" is a useless contextualization  The entailment relation we are seeking in the QA context should include relevant and meaningful relaxations of contextual and semantic constraints (cf  Section SECREF13 ) 
Related Work on Question Answering
Classical QA systems face two main challenges related to question analysis and answer extraction  Several QA approaches were proposed in the literature for the open domain BIBREF16 , BIBREF17 and the medical domain BIBREF18 , BIBREF19 , BIBREF20   A variety of methods were developed for question analysis, focus (topic) recognition and question type identification BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24   Similarly, many different approaches tackled document or passage retrieval and answer selection and (re)ranking BIBREF25 , BIBREF26 , BIBREF27  
An alternative approach consists in finding similar questions or FAQs that are already answered BIBREF28 , BIBREF29   One of the earliest question answering systems based on finding similar questions and re-using the existing answers was FAQ FINDER BIBREF30   Another system that complements the existing Q&A services of NetWellness is SimQ BIBREF2 , which allows retrieval of similar web-based consumer health questions  SimQ uses syntactic and semantic features to compute similarity between questions, and UMLS BIBREF31 as a standardized semantic knowledge source  The system achieves 72 2% precision, 78 0% recall and 75 0% F-score on NetWellness questions  However, the method was evaluated only on one question similarity dataset, and the retrieved answers were not evaluated 
The aim of the medical task at TREC 2017 LiveQA was to develop techniques for answering complex questions such as consumer health questions, as well as to identify relevant answer sources that can comply with the sensitivity of medical information retrieval 
The CMU-OAQA system BIBREF32 achieved the best performance of 0 637 average score on the medical task by using an attentional encoder-decoder model for paraphrase identification and answer ranking  The Quora question-similarity dataset was used for training  The PRNA system BIBREF33 achieved the second best performance in the medical task with 0 49 average score using Wikipedia as the first answer source and Yahoo and Google searches as secondary answer sources  Each medical question was decomposed into several subquestions  To extract the answer from the selected text passage, a bi-directional attention model trained on the SQUAD dataset was used 
Deep neural network models have been pushing the limits of performance achieved in QA related tasks using large training datasets  The results obtained by CMU-OAQA and PRNA showed that large open-domain datasets were beneficial for the medical domain  However, the best system (CMU-OAQA) relying on the same training data obtained a score of 1 139 on the LiveQA open-domain task 
While this gap in performance can be explained in part by the discrepancies between the medical test questions and the open-domain questions, it also highlights the need for larger medical datasets to support deep learning approaches in dealing with the linguistic complexity of consumer health questions and the challenge of finding correct and complete answers 
Another technique was used by ECNU-ICA team BIBREF34 based on learning question similarity via two long short-term memory (LSTM) networks applied to obtain the semantic representations of the questions  To construct a collection of similar question pairs, they searched community question answering sites such as Yahoo! and Answers com  In contrast, the ECNU-ICA system achieved the best performance of 1 895 in the open-domain task but an average score of only 0 402 in the medical task  As the ECNU-ICA approach also relied on a neural network for question matching, this result shows that training attention-based decoder-encoder networks on the Quora dataset generalized better to the medical domain than training LSTMs on similar questions from Yahoo! and Answers com 
The CMU-LiveMedQA team BIBREF20 designed a specific system for the medical task  Using only the provided training datasets and the assumption that each question contains only one focus, the CMU-LiveMedQA system obtained an average score of 0 353  They used a convolutional neural network (CNN) model to classify a question into a restricted set of 10 question types and crawled "relevant" online web pages to find the answers  However, the results were lower than those achieved by the systems relying on finding similar answered questions  These results support the relevance of similar question matching for the end-to-end QA task as a new way of approaching QA instead of the classical QA approaches based on Question Analysis and Answer Retrieval 
Related Work on Question Similarity and Entailment
Several efforts focused on recognizing similar questions  Jeon et al  BIBREF35 showed that a retrieval model based on translation probabilities learned from a question and answer archive can recognize semantically similar questions  Duan et al  BIBREF36 proposed a dedicated language modeling approach for question search, using question topic (user's interest) and question focus (certain aspect of the topic) 
Lately, these efforts were supported by a task on Question-Question similarity introduced in the community QA challenge at SemEval (task 3B) BIBREF3   Given a new question, the task focused on reranking all similar questions retrieved by a search engine, assuming that the answers to the similar questions will be correct answers for the new question  Different machine learning and deep learning approaches were tested in the scope of SemEval 2016 BIBREF3 and 2017 BIBREF4 task 3B  The best performing system in 2017 achieved a MAP of 47 22% using supervised Logistic Regression that combined different unsupervised similarity measures such as Cosine and Soft-Cosine BIBREF37   The second best system achieved 46 93% MAP with a learning-to-rank method using Logistic Regression and a rich set of features including lexical and semantic features as well as embeddings generated by different neural networks (siamese, Bi-LSTM, GRU and CNNs) BIBREF38   In the scope of this challenge, a dataset was collected from Qatar Living forum for training  We refer to this dataset as SemEval-cQA 
In another effort, an answer-based definition of RQE was proposed and tested BIBREF1   The authors introduced a dataset of clinical questions and used a feature-based method that provided an Accuracy of 75% on consumer health questions  We will call this dataset Clinical-QE  Dos Santos et al  BIBREF5 proposed a new approach to retrieve semantically equivalent questions combining a bag-of-words representation with a distributed vector representation created by a CNN and user data collected from two Stack Exchange communities  Lei et al  BIBREF7 proposed a recurrent and convolutional model (gated convolution) to map questions to their semantic representations  The models were pre-trained within an encoder-decoder framework 
RQE Approaches and Experiments
The choice of two methods for our empirical study is motivated by the best performance achieved by Logistic Regression in question-question similarity at SemEval 2017 (best system BIBREF37 and second best system BIBREF38 ), and the high performance achieved by neural networks on larger datasets such as SNLI BIBREF13 , BIBREF39 , BIBREF40 , BIBREF41   We first define the RQE task, then present the two approaches, and evaluate their performance on five different datasets 
Definition
In the context of QA, the goal of RQE is to retrieve answers to a new question by retrieving entailed questions with associated answers  We therefore define question entailment as:
a question INLINEFORM0 entails a question INLINEFORM1 if every answer to INLINEFORM2 is also a complete or partial answer to INLINEFORM3  
We present below two examples of consumer health questions INLINEFORM0 and entailed questions INLINEFORM1 :
Example 1 (each answer to the entailed question B1 is a complete answer to A1):
A1: What is the latest news on tennitis, or ringing in the ear, I am 75 years old and have had ringing in the ear since my mid 5os  Thank you 
B1: What is the latest research on Tinnitus?
Example 2 (each answer to the entailed question B2 is a partial answer to A2):
A2: My mother has been diagnosed with Alzheimer's, my father is not of the greatest health either and is the main caregiver for my mother  My question is where do we start with attempting to help our parents w/ the care giving and what sort of financial options are there out there for people on fixed incomes 
B2: What resources are available for Alzheimer's caregivers?
The inclusion of partial answers in the definition of question entailment also allows efficient relaxation of the contextual constraints of the original question INLINEFORM0 to retrieve relevant answers from entailed, but less restricted, questions 
Deep Learning Model
To recognize entailment between two questions INLINEFORM0 (premise) and INLINEFORM1 (hypothesis), we adapted the neural network proposed by Bowman et al  BIBREF13   Our DL model, presented in Figure FIGREF20 , consists of three 600d ReLU layers, with a bottom layer taking the concatenated sentence representations as input and a top layer feeding a softmax classifier  The sentence embedding model sums the Recurrent neural network (RNN) embeddings of its words  The word embeddings are first initialized with pretrained GloVe vectors  This adaptation provided the best performance in previous experiments with RQE data 
GloVe is an unsupervised learning algorithm to generate vector representations for words BIBREF42   Training is performed on aggregated word co-occurrence statistics from a large corpus, and the resulting representations show interesting linear substructures of the word vector space  We use the pretrained common crawl version with 840B tokens and 300d vectors, which are not updated during training 
Logistic Regression Classifier
In this feature-based approach, we use Logistic Regression to classify question pairs into entailment or no-entailment  Logistic Regression achieved good results on this specific task and outperformed other statistical learning algorithms such as SVM and Naive Bayes  In a preprocessing step, we remove stop words and perform word stemming using the Porter algorithm BIBREF43 for all ( INLINEFORM0 , INLINEFORM1 ) pairs 
We use a list of nine features, selected after several experiments on RTE datasets BIBREF12   We compute five similarity measures between the pre-processed questions and use their values as features  We use Word Overlap, the Dice coefficient based on the number of common bigrams, Cosine, Levenshtein, and the Jaccard similarities  Our feature list also includes the maximum and average values obtained with these measures and the question length ratio (length( INLINEFORM0 )/length( INLINEFORM1 ))  We compute a morphosyntactic feature indicating the number of common nouns and verbs between INLINEFORM2 and INLINEFORM3   TreeTagger BIBREF44 was used for POS tagging 
For RQE, we add an additional feature specific to the question type  We use a dictionary lookup to map triggers to the question type (e g  Treatment, Prognosis, Inheritance)  Triggers are identified for each question type based on a manual annotation of a set of medical questions (cf  Section SECREF36 )  This feature has three possible values: 2 (Perfect match between INLINEFORM0 type(s) and INLINEFORM1 type(s)), 1 (Overlap between INLINEFORM2 type(s) and INLINEFORM3 type(s)) and 0 (No common types) 
Datasets Used for the RQE Study
We evaluate the RQE methods (i e  deep learning model and logistic regression classifier) using two datasets of sentence pairs (SNLI and multiNLI), and three datasets of question pairs (Quora, Clinical-QE, and SemEval-cQA) 
The Stanford Natural Language Inference corpus (SNLI) BIBREF13 contains 569,037 sentence pairs written by humans based on image captioning  The training set of the MultiNLI corpus BIBREF14 consists of 393,000 pairs of sentences from five genres of written and spoken English (e g  Travel, Government)  Two other "matched" and "mismatched" sets are also available for development (20,000 pairs)  Both SNLI and multiNLI consider three types of relationships between sentences: entailment, neutral and contradiction  We converted the contradiction and neutral labels to the same non-entailment class 
The QUORA dataset of similar questions was recently published with 404,279 question pairs  We randomly selected three distinct subsets (80%/10%/10%) for training (323,423 pairs), development (40,428 pairs) and test (40,428 pairs) 
The clinical-QE dataset BIBREF1 contains 8,588 question pairs and was constructed using 4,655 clinical questions asked by family doctors BIBREF45   We randomly selected three distinct subsets (80%/10%/10%) for training (6,870 pairs), development (859 pairs) and test (859 pairs) 
The question similarity dataset of SemEval 2016 Task 3B (SemEval-cQA) BIBREF3 contains 3,869 question pairs and aims to re-rank a list of related questions according to their similarity to the original question  The same dataset was used for SemEval 2017 Task 3 BIBREF4  
To construct our test dataset, we used a publicly shared set of Consumer Health Questions (CHQs) received by the U S  National Library of Medicine (NLM), and annotated with named entities, question types, and focus BIBREF46 , BIBREF47   The CHQ dataset consists of 1,721 consumer information requests manually annotated with subquestions, each identified by a question type and a focus 
First, we selected automatically harvested FAQs, from U S  National Institutes of Health (NIH) websites, that share both the same focus and the same question type with the CHQs  As FAQs are most often very short, we first assume that the CHQ entails the FAQ  Two sets of pairs were constructed: (i) positive pairs of CHQs and FAQs sharing at least one common question type and the question focus, and (ii) negative pairs corresponding to a focus mismatch or type mismatch  For each category of negative examples, we randomly selected the same number of pairs for a balanced dataset  Then, we manually validated the constructed pairs and corrected the positive and negative labels when needed  The final RQE dataset contains 850 CHQ-FAQ pairs with 405 positive and 445 negative pairs  Table TABREF26 presents examples from the five training datasets (SNLI, MultiNLI, SemEval-cQA, Clinical-QE and Quora) and the new test dataset of medical CHQ-FAQ pairs 
Results of RQE Approaches
In the first experiment, we evaluated the DL and ML methods on SNLI, multi-NLI, Quora, and Clinical-QE  For the datasets that did not have a development and test sets, we randomly selected two sets, each amounting to 10% of the data, for test and development, and used the remaining 80% for training  For MultiNLI, we used the dev1-matched set for validation and the dev2-mismatched set for testing 
Table TABREF28 presents the results of the first experiment  The DL model with GloVe word embeddings achieved better results on three datasets, with 82 80% Accuracy on SNLI, 78 52% Accuracy on MultiNLI, and 83 62% Accuracy on Quora  Logistic Regression achieved the best Accuracy of 98 60% on Clinical-RQE  We also performed a 10-fold cross-validation on the full Clinical-QE data of 8,588 question pairs, which gave 98 61% Accuracy 
In the second experiment, we used these datasets for training only and compared their performance on our test set of 850 consumer health questions  Table TABREF29 presents the results of this experiment  Logistic Regression trained on the clinical-RQE data outperformed DL models trained on all datasets, with 73 18% Accuracy 
To validate further the performance of the LR method, we evaluated it on question similarity detection  A typical approach to this task is to use an IR method to find similar question candidates, then a more sophisticated method to select and re-rank the similar questions  We followed a similar approach for this evaluation by combining the LR method with the IR baseline provided in the context of SemEval-cQA  The hybrid method combines the score provided by the Logistic Regression model and the reciprocal rank from the IR baseline using a weight-based combination:
INLINEFORM0
The weight INLINEFORM0 was set empirically through several tests on the cQA-2016 development set ( INLINEFORM1 )  Table TABREF30 presents the results on the cQA-2016 and cQA-2017 test datasets  The hybrid method (LR+IR) provided the best results on both datasets  On the 2016 test data, the LR+IR method outperformed the best system in all measures, with 80 57% Accuracy and 77 47% MAP (official system ranking measure in SemEval-cQA)  On the cQA-2017 test data, the LR+IR method obtained 44 66% MAP and outperformed the cQA-2017 best system in Accuracy with 67 27% 
Discussion of RQE Results
When trained and tested on the same corpus, the DL model with GloVe embeddings gave the best results on three datasets (SNLI, MultiNLI and Quora)  Logistic Regression gave the best Accuracy on the Clinical-RQE dataset with 98 60%  When tested on our test set (850 medical CHQs-FAQs pairs), Logistic Regression trained on Clinical-QE gave the best performance with 73 18% Accuracy 
The SNLI and multi-NLI models did not perform well when tested on medical RQE data  We performed additional evaluations using the RTE-1, RTE-2 and RTE-3 open-domain datasets provided by the PASCAL challenge and the results were similar  We have also tested the SemEval-cQA-2016 model and had a similar drop in performance on RQE data  This could be explained by the different types of data leading to wrong internal conceptualizations of medical terms and questions in the deep neural layers
Context 3: Introduction
Accurate language identification (LID) is the first step in many natural language processing and machine comprehension pipelines  If the language of a piece of text is known then the appropriate downstream models like parts of speech taggers and language models can be applied as required 
LID is further also an important step in harvesting scarce language resources  Harvested data can be used to bootstrap more accurate LID models and in doing so continually improve the quality of the harvested data  Availability of data is still one of the big roadblocks for applying data driven approaches like supervised machine learning in developing countries 
Having 11 official languages of South Africa has lead to initiatives (discussed in the next section) that have had positive effect on the availability of language resources for research  However, many of the South African languages are still under resourced from the point of view of building data driven models for machine comprehension and process automation 
Table TABREF2 shows the percentages of first language speakers for each of the official languages of South Africa  These are four conjunctively written Nguni languages (zul, xho, nbl, ssw), Afrikaans (afr) and English (eng), three disjunctively written Sotho languages (nso, sot, tsn), as well as tshiVenda (ven) and Xitsonga (tso)  The Nguni languages are similar to each other and harder to distinguish  The same is true of the Sotho languages 
This paper presents a hierarchical naive Bayesian and lexicon based classifier for LID of short pieces of text of 15-20 characters long  The algorithm is evaluated against recent approaches using existing test sets from previous works on South African languages as well as the Discriminating between Similar Languages (DSL) 2015 and 2017 shared tasks 
Section SECREF2 reviews existing works on the topic and summarises the remaining research problems  Section SECREF3 of the paper discusses the proposed algorithm and Section SECREF4 presents comparative results 
Related Works
The focus of this section is on recently published datasets and LID research applicable to the South African context  An in depth survey of algorithms, features, datasets, shared tasks and evaluation methods may be found in BIBREF0 
The datasets for the DSL 2015 & DSL 2017 shared tasks BIBREF1 are often used in LID benchmarks and also available on Kaggle   The DSL datasets, like other LID datasets, consists of text sentences labelled by language  The 2017 dataset, for example, contains 14 languages over 6 language groups with 18000 training samples and 1000 testing samples per language 
The recently published JW300 parallel corpus BIBREF2 covers over 300 languages with around 100 thousand parallel sentences per language pair on average  In South Africa, a multilingual corpus of academic texts produced by university students with different mother tongues is being developed BIBREF3  The WiLI-2018 benchmark dataset BIBREF4 for monolingual written natural language identification includes around 1000 paragraphs of 235 languages  A possibly useful link can also be made BIBREF5 between Native Language Identification (NLI) (determining the native language of the author of a text) and Language Variety Identification (LVI) (classification of different varieties of a single language) which opens up more datasets  The Leipzig Corpora Collection BIBREF6, the Universal Declaration of Human Rights and Tatoeba are also often used sources of data 
The NCHLT text corpora BIBREF7 is likely a good starting point for a shared LID task dataset for the South African languages BIBREF8  The NCHLT text corpora contains enough data to have 3500 training samples and 600 testing samples of 300+ character sentences per language  Researchers have recently started applying existing algorithms for tasks like neural machine translation in earnest to such South African language datasets BIBREF9 
Existing NLP datasets, models and services BIBREF10 are available for South African languages  These include an LID algorithm BIBREF11 that uses a character level n-gram language model  Multiple papers have shown that 'shallow' naive Bayes classifiers BIBREF12, BIBREF8, BIBREF13, BIBREF14, SVMs BIBREF15 and similar models work very well for doing LID  The DSL 2017 paper BIBREF1, for example, gives an overview of the solutions of all of the teams that competed on the shared task and the winning approach BIBREF16 used an SVM with character n-gram, parts of speech tag features and some other engineered features  The winning approach for DSL 2015 used an ensemble naive Bayes classifier  The fasttext classifier BIBREF17 is perhaps one of the best known efficient 'shallow' text classifiers that have been used for LID 

ground_truth: A new tagging scheme that tags the words before and after the pun as well as the pun words.

a new tagging scheme consisting of three tags, namely { INLINEFORM0 }

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What machine learning and deep learning methods are used for RQE?
Context 2:  This performance drop could also be caused by the complexity of the test consumer health questions that are often composed of several subquestions, contain contextual information, and may contain misspellings and ungrammatical sentences, which makes them more difficult to process BIBREF48   Another aspect is the semantics of the task as discussed in Section SECREF6   The definition of textual entailment in open-domain may not quite apply to question entailment due to the strict semantics  Also the general textual entailment definitions refer only to the premise and hypothesis, while the definition of RQE for question answering relies on the relationship between the sets of answers of the compared questions 
Building a Medical QA Collection from Trusted Resources
A RQE-based QA system requires a collection of question-answer pairs to map new user questions to the existing questions with an RQE approach, rank the retrieved questions, and present their answers to the user 
Method
To construct trusted medical question-answer pairs, we crawled websites from the National Institutes of Health (cf  Section SECREF56 )  Each web page describes a specific topic (e g  name of a disease or a drug), and often includes synonyms of the main topic that we extracted during the crawl 
We constructed hand-crafted patterns for each website to automatically generate the question-answer pairs based on the document structure and the section titles  We also annotated each question with the associated focus (topic of the web page) as well as the question type identified with the designed patterns (cf  Section SECREF36 ) 
To provide additional information about the questions that could be used for diverse IR and NLP tasks, we automatically annotated the questions with the focus, its UMLS Concept Unique Identifier (CUI) and Semantic Type  We combined two methods to recognize named entities from the titles of the crawled articles and their associated UMLS CUIs: (i) exact string matching to the UMLS Metathesaurus, and (ii) MetaMap Lite BIBREF49   We then used the UMLS Semantic Network to retrieve the associated semantic types and groups 
Question Types
The question types were derived after the manual evaluation of 1,721 consumer health questions  Our taxonomy includes 16 types about Diseases, 20 types about Drugs and one type (Information) for the other named entities such as Procedures, Medical exams and Treatments  We describe below the considered question types and examples of associated question patterns 
Question Types about Diseases (16): Information, Research (or Clinical Trial), Causes, Treatment, Prevention, Diagnosis (Exams and Tests), Prognosis, Complications, Symptoms, Inheritance, Susceptibility, Genetic changes, Frequency, Considerations, Contact a medical professional, Support Groups 
Examples:
What research (or clinical trial) is being done for DISEASE?
What is the outlook for DISEASE?
How many people are affected by DISEASE?
When to contact a medical professional about DISEASE?
Who is at risk for DISEASE?
Where to find support for people with DISEASE?
Question Types About Drugs (20): Information, Interaction with medications, Interaction with food, Interaction with herbs and supplements, Important warning, Special instructions, Brand names, How does it work, How effective is it, Indication, Contraindication, Learn more, Side effects, Emergency or overdose, Severe reaction, Forget a dose, Dietary, Why get vaccinated, Storage and disposal, Usage, Dose 
Examples:
Are there interactions between DRUG and herbs and supplements?
What important warning or information should I know about DRUG?
Are there safety concerns or special precautions about DRUG?
What is the action of DRUG and how does it work?
Who should get DRUG and why is it prescribed?
What to do in case of a severe reaction to DRUG?
Question Type for other medical entities (e g  Procedure, Exam, Treatment): Information 
What is Coronary Artery Bypass Surgery?
What are Liver Function Tests?
Medical Resources
We used 12 trusted websites to construct a collection of question-answer pairs  For each website, we extracted the free text of each article as well as the synonyms of the article focus (topic)  These resources and their brief descriptions are provided below:
National Cancer Institute (NCI) : We extracted free text from 116 articles on various cancer types (729 QA pairs)  We manually restructured the content of the articles to generate complete answers (e g  a full answer about the treatment of all stages of a specific type of cancer)  Figure FIGREF54 presents examples of QA pairs generated from a NCI article 
Genetic and Rare Diseases Information Center (GARD): This resource contains information about various aspects of genetic/rare diseases  We extracted all disease question/answer pairs from 4,278 topics (5,394 QA pairs) 
Genetics Home Reference (GHR): This NLM resource contains consumer-oriented information about the effects of genetic variation on human health  We extracted 1,099 articles about diseases from this resource (5,430 QA pairs) 
MedlinePlus Health Topics: This portion of MedlinePlus contains information on symptoms, causes, treatment and prevention for diseases, health conditions and wellness issues  We extracted the free texts in summary sections of 981 articles (981 QA pairs) 
National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) : We extracted text from 174 health information pages on diseases studied by this institute (1,192 QA pairs) 
National Institute of Neurological Disorders and Stroke (NINDS): We extracted free text from 277 information pages on neurological and stroke-related diseases from this resource (1,104 QA pairs) 
NIHSeniorHealth : This website contains health and wellness information for older adults  We extracted 71 articles from this resource (769 QA pairs) 
National Heart, Lung, and Blood Institute (NHLBI) : We extracted text from 135 articles on diseases, tests, procedures, and other relevant topics on disorders of heart, lung, blood, and sleep (559 QA pairs) 
Centers for Disease Control and Prevention (CDC) : We extracted text from 152 articles on diseases and conditions (270 QA pairs) 
MedlinePlus A D A M  Medical Encyclopedia: This resource contains 4,366 articles about conditions, tests, and procedures  17,348 QA pairs were extracted from this resource  Figure FIGREF55 presents examples of QA pairs generated from A D A M encyclopedia 
MedlinePlus Drugs: We extracted free text from 1,316 articles about Drugs and generated 12,889 QA pairs 
MedlinePlus Herbs and Supplements: We extracted free text from 99 articles and generated 792 QA pairs 
The final collection contains 47,457 annotated question-answer pairs about Diseases, Drugs and other named entities (e g  Tests) extracted from these 12 trusted resources 
The Proposed Entailment-based QA System
Our goal is to generate a ranked list of answers for a given Premise Question INLINEFORM0 by ranking the recognized Hypothesis Questions INLINEFORM1   Based on the RQE experiments above (Section SECREF27 ), we selected Logistic Regression trained on the clinical-RQE dataset to recognize entailed questions and rank them with their classification scores 
RQE-based QA Approach
Classifying the full QA collection for each test question is not feasible for real-time applications  Therefore, we first filter the questions with an IR method to retrieve candidate questions, then classify them as entailed (or not) by the user/test question  Based on the positive results of the combination method tested on SemEval-cQA data (Section SECREF27 ), we adopted a combination method to merge the results obtained by the search engine and the RQE scores  The answers are then combined from both methods and ranked using an aggregate score  Figure FIGREF82 presents the overall architecture of the proposed QA system  We describe each module in more details next 
Finding Similar Question Candidates
For each premise question INLINEFORM0 , we use the Terrier search engine to retrieve INLINEFORM1 relevant question candidates INLINEFORM2 and then apply the RQE classifier to predict the labels for the pairs ( INLINEFORM3 , INLINEFORM4 ) 
We indexed the questions of our QA collection without the associated answers  In order to improve the indexing and the performance of question retrieval, we also indexed the synonyms of the question focus and the triggers of the question type with each question  This choice allowed us to avoid the shortcomings of query expansion, including incorrect or irrelevant synonyms and the increased execution time  The synonyms of the question focus (topic) were extracted automatically from the QA collection  The triggers of each question type were defined manually in the question types taxonomy  Below are two examples of indexed questions from our QA collection, with the automatically added focus synonyms and question type triggers:
What are the treatments for Torticollis?
Focus: Torticollis  Question type: Treatment 
Added focus synonyms: "Spasmodic torticollis, Wry neck, Loxia, Cervical dystonia"  Added question type triggers: "relieve, manage, cure, remedy, therapy" 
What is the outlook for Legionnaire disease?
Focus: Legionnaire disease  Question Type: Prognosis 
Added focus synonyms: "Legionella pneumonia, Pontiac fever, Legionellosis"  Added question type triggers: "prognosis, life expectancy" 
The IR task consists of retrieving hypothesis questions INLINEFORM0 relevant to the submitted question INLINEFORM1   As fusion of IR result has shown good performance in different tracks in TREC, we merge the results of the TF-IDF weighting function and the In-expB2 DFR model BIBREF50  
Let INLINEFORM0 = INLINEFORM1 , INLINEFORM2 ,    , INLINEFORM3 be the set of INLINEFORM4 questions retrieved by the first IR model INLINEFORM5 and INLINEFORM6 = INLINEFORM7 , INLINEFORM8 ,    , INLINEFORM9 be the set of INLINEFORM10 questions retrieved by the second IR model INLINEFORM11   We merge both sets by summing the scores of each retrieved question INLINEFORM12 in both INLINEFORM13 and INLINEFORM14 lists, then we re-rank the hypothesis questions INLINEFORM15  
Combining IR and RQE Methods
The IR models and the RQE Logistic Regression model bring different perspectives to the search for relevant candidate questions  In particular, question entailment allows understanding the relations between the important terms, whereas the traditional IR methods identify the important terms, but will not notice if the relations are opposite  Moreover, some of the question types that the RQE classifier learns will not be deemed important terms by traditional IR and the most relevant questions will not be ranked at the top of the list 
Therefore, in our approach, when a question is submitted to the system, candidate questions are fetched using the IR models, then the RQE classifier is applied to filter out the non-entailed questions and re-rank the remaining candidates 
Specifically, we denote INLINEFORM0 the list of question candidates INLINEFORM1 returned by the IR system  The premise question INLINEFORM2 is then used to construct N question pairs INLINEFORM3   The RQE classifier is then applied to filter out the question pairs that are not entailed and re-rank the remaining pairs 
More precisely, let INLINEFORM0 = INLINEFORM1 in INLINEFORM2 be the list of selected candidate questions that have a positive entailment relation with a given premise question INLINEFORM3   We rank INLINEFORM4 by computing a hybrid score INLINEFORM5 for each candidate question INLINEFORM6 taking into account the score of the IR system INLINEFORM7 and the score of the RQE system INLINEFORM8  
For each system INLINEFORM0 INLINEFORM1 , we normalize the associated score by dividing it by the maximum score among the INLINEFORM2 candidate questions retrieved by INLINEFORM3 for INLINEFORM4 :
INLINEFORM0
INLINEFORM0 INLINEFORM1
In our experiments, we fixed the value of INLINEFORM0 to 100  This threshold value was selected as a safe value for this task for the following reasons:
Our collection of 47,457 question-answer pairs was collected from only 12 NIH institutes and is unlikely to contain more than 100 occurrences of the same focus-type pair 
Each question was indexed with additional annotations for the question focus, its synonyms and the question type synonyms 
Evaluating RQE for Medical Question Answering
The objective of this evaluation is to study the effectiveness of RQE for Medical Question Answering, by comparing the answers retrieved by the hybrid entailment-based approach, the IR method and the other QA systems participating to the medical task at TREC 2017 LiveQA challenge (LiveQA-Med) 
Evaluation Method
We developed an interface to perform the manual evaluation of the retrieved answers  Figure 5 presents the evaluation interface showing, for each test question, the top-10 answers of the evaluated QA method and the reference answer(s) used by LiveQA assessors to help judging the retrieved answers by the participating systems 
We used the test questions of the medical task at TREC-2017 LiveQA BIBREF11   These questions are randomly selected from the consumer health questions that the NLM receives daily from all over the world  The test questions cover different medical entities and have a wide list of question types such as Comparison, Diagnosis, Ingredient, Side effects and Tapering 
For a relevant comparison, we used the same judgment scores as the LiveQA Track:
Correct and Complete Answer (4)
Correct but Incomplete (3)
Incorrect but Related (2)
Incorrect (1)
We evaluated the answers returned by the IR-based method and the hybrid QA method (IR+RQE) according to the same reference answers used in LiveQA-Med  The answers were anonymized (the method names were blinded) and presented to 3 assessors: a medical doctor (Assessor A), a medical librarian (B) and a researcher in medical informatics (C)  None of the assessors participated in the development of the QA methods  Assessors B and C evaluated 1,000 answers retrieved by each of the methods (IR and IR+RQE)  Assessor A evaluated 2,000 answers from both methods 
Table TABREF103 presents the inter-annotator agreement (IAA) through F1 score computed by considering one of the assessors as reference  In the first evaluation, we computed the True Positives (TP) and False Positives (FP) over all ratings and the Precision and F1 score  As there are no negative labels (only true or false positives for each category), Recall is 100%  We also computed a partial IAA by grouping the "Correct and Complete Answer" and "Correct but Incomplete" ratings (as Correct), and the "Incorrect but Related" and "Incorrect" ratings (as Incorrect)  The average agreement on distinguishing the Correct and Incorrect answers is 94 33% F1 score  Therefore, we used the evaluations performed by assessor A for both methods  The official results of the TREC LiveQA track relied on one assessor per question as well 
Evaluation of the first retrieved answer
We computed the measures used by TREC LiveQA challenges BIBREF51 , BIBREF11 to evaluate the first retrieved answer for each test question:
avgScore(0-3): the average score over all questions, transferring 1-4 level grades to 0-3 scores  This is the main score used to rank LiveQA runs 
succ@i+: the number of questions with score i or above (i INLINEFORM0 {2  4}) divided by the total number of questions 
prec@i+: the number of questions with score i or above (i INLINEFORM0 {2  4}) divided by number of questions answered by the system 
Table TABREF108 presents the average scores, success and precision results  The hybrid IR+RQE QA system achieved better results than the IR-based system with 0 827 average score  It also achieved a higher score than the best results achieved in the medical challenge at LiveQA'17  Evaluating the RQE system alone is not relevant, as applying RQE on the full collection for each user question is not feasible for a real-time system because of the extended execution time 
Evaluation of the top ten answers
In this evaluation, we used Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) which are commonly used in QA to evaluate the top-10 answers for each question  We consider answers rated as “Correct and Complete Answer” or “Correct but Incomplete” as correct answers, as the test questions contain multiple subquestions while each answer in our QA collection can cover only one subquestion 
MAP is the mean of the Average Precision (AvgP) scores over all questions 
(1) INLINEFORM0
Q is the number of questions  INLINEFORM0 is the AvgP of the INLINEFORM1 question 
INLINEFORM0
K is the number of correct answers  INLINEFORM0 is the rank of INLINEFORM1 correct answer 
MRR is the average of the reciprocal ranks for each question  The reciprocal rank of a question is the multiplicative inverse of the rank of the first correct answer 
(2) INLINEFORM0
Q is the number of questions  INLINEFORM0 is the rank of the first correct answer for the INLINEFORM1 question 
Table TABREF113 presents the MAP@10 and MRR@10 of our QA methods  The IR+RQE system outperforms the IR-based QA system with 0 311 MAP@10 and 0 333 MRR@10 
Discussion of entailment-based QA for the medical domain
In our evaluation, we followed the same LiveQA guidelines with the highest possible rigor  In particular, we consulted with NIST assessors who provided us with the paraphrases of the test questions that they used to judge the answers  Our IAA on the answers rating was also high compared to related tasks, with an 88 5% F1 agreement with the exact four categories and a 94 3% agreement when reducing the categories to two: “Correct” and “Incorrect” answers  Our results show that RQE improves the overall performance and exceeds the best results in the medical LiveQA'17 challenge by a factor of 29 8%  This performance improvement is particularly interesting as:
Our answer source has only 47K question-answer pairs when LiveQA participating systems relied on much larger collections, including the World Wide Web 
Our system answered one subquestion at most when many LiveQA test questions had several subquestions 
The latter observation, (b), makes the hybrid IR+RQE approach even more promising as it gives it a large potential for the improvement of answer completeness 
The former observation, (a), provides another interesting insight: restricting the answer source to only reliable collections can actually improve the QA performance without losing coverage (i e , our QA approach provided at least one answer to each test question and obtained the best relevance score) 
In another observation, the assessors reported that many of the returned answers had a correct question type but a wrong focus, which indicates that including a focus recognition module to filter such wrong answers can improve further the QA performance in terms of precision  Another aspect that was reported is the repetition of the same (or similar) answer from different websites, which could be addressed by improving answer selection with inter-answer comparisons and removal of near-duplicates  Also, half of the LiveQA test questions are about Drugs, when only two of our resources are specialized in Drugs, among 12 sub-collections overall  Accordingly, the assessors noticed that the performance of the QA systems was better on questions about diseases than on questions about drugs, which suggests a need for extending our medical QA collection with more information about drugs and associated question types 
We also looked closely at the private websites used by the LiveQA-Med annotators to provide some of the reference answers for the test questions  For instance, the ConsumerLab website was useful to answer a question about the ingredients of a Drug (COENZYME Q10)  Similarly, the eHealthMe website was used to answer a test question asking about interactions between two drugs (Phentermine and Dicyclomine) when no information was found in DailyMed  eHealthMe provides healthcare big data analysis and private research and studies including self-reported adverse drug effects by patients 
But the question remains on the extent to which such big data and other private websites could be used to automatically answer medical questions if information is otherwise unavailable  Unlike medical professionals, patients do not necessarily have the knowledge and tools to validate such information  An alternative approach could be to put limitations on medical QA systems in terms of the questions that can be answered (e g  "What is my diagnosis for such symptoms") and build classifiers to detect such questions and warn the users about the dangers of looking for their answers online 
More generally, medical QA systems should follow some strict guidelines regarding the goal and background knowledge and resources of each system in order to protect the consumers from misleading or harmful information  Such guidelines could be based (i) on the source of the information such as health and medical information websites sponsored by the U S  government, not-for-profit health or medical organizations, and medical university centers, or (ii) on conventions such as the code of conduct of the HON Foundation (HONcode) that addresses the reliability and usefulness of medical information on the Internet 
Our experiments show that limiting the number of answer sources with such guidelines is not only feasible, but it could also enhance the performance of the QA system from an information retrieval perspective 
Conclusion
In this paper, we carried out an empirical study of machine learning and deep learning methods for Recognizing Question Entailment in the medical domain using several datasets  We developed a RQE-based QA system to answer new medical questions using existing question-answer pairs  We built and shared a collection of 47K medical question-answer pairs  Our QA approach outperformed the best results on TREC-2017 LiveQA medical test questions  The proposed approach can be applied and adapted to open-domain as well as specific-domain QA  Deep learning models achieved interesting results on open-domain and clinical datasets, but obtained a lower performance on consumer health questions  We will continue investigating other network architectures including transfer learning, as well as creation of a large collection of consumer health questions for training to improve the performance of DL models  Future work also includes exploring integration of a Question Focus Recognition module to enhance candidate question retrieval, and expanding our question-answer collection 
Acknowledgements
We thank Halil Kilicoglu (NLM/NIH) for his help with the crawling and the manual evaluation and Sonya E  Shooshan (NLM/NIH) for her help with the judgment of the retrieved answers  We also thank Ellen Voorhees (NIST) for her valuable support with the TREC LiveQA evaluation 
We consider the case of the question number 36 in the TREC-2017 LiveQA medical test dataset:
36  congenital diaphragmatic hernia  what are the causes of congenital diaphragmatic hernia? Can cousin marriage cause this? What kind of lung disease the baby might experience life long?
This question was answered by 5 participating runs (vs  8 runs for other questions), and all submitted answers were wrong (scores of 1 or 2)  However, our IR-based QA system retrieved one excellent answer (score 4) and our hybrid IR+RQE system provided 3 excellent answers 
A) TREC 2017 LiveQA-Med Participants' Results:
B) Our IR-based QA System:
C) Our IR+RQE QA System:
Context 3: 
Multiple papers have proposed hierarchical stacked classifiers (including lexicons) that would for example first classify a piece of text by language group and then by exact language BIBREF18, BIBREF19, BIBREF8, BIBREF0  Some work has also been done on classifying surnames between Tshivenda, Xitsonga and Sepedi BIBREF20  Additionally, data augmentation BIBREF21 and adversarial training BIBREF22 approaches are potentially very useful to reduce the requirement for data 
Researchers have investigated deeper LID models like bidirectional recurrent neural networks BIBREF23 or ensembles of recurrent neural networks BIBREF24  The latter is reported to achieve 95 12% in the DSL 2015 shared task  In these models text features can include character and word n-grams as well as informative character and word-level features learnt BIBREF25 from the training data  The neural methods seem to work well in tasks where more training data is available 
In summary, LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched  Increased confusion can in general be expected between shorter pieces of text and languages that are more closely related  Shallow methods still seem to work well compared to deeper models for LID  Other remaining research opportunities seem to be data harvesting, building standardised datasets and creating shared tasks for South Africa and Africa  Support for language codes that include more languages seems to be growing and discoverability of research is improving with more survey papers coming out  Paywalls also seem to no longer be a problem; the references used in this paper was either openly published or available as preprint papers 
Methodology
The proposed LID algorithm builds on the work in BIBREF8 and BIBREF26  We apply a naive Bayesian classifier with character (2, 4 & 6)-grams, word unigram and word bigram features with a hierarchical lexicon based classifier 
The naive Bayesian classifier is trained to predict the specific language label of a piece of text, but used to first classify text as belonging to either the Nguni family, the Sotho family, English, Afrikaans, Xitsonga or Tshivenda  The scikit-learn multinomial naive Bayes classifier is used for the implementation with an alpha smoothing value of 0 01 and hashed text features 
The lexicon based classifier is then used to predict the specific language within a language group  For the South African languages this is done for the Nguni and Sotho groups  If the lexicon prediction of the specific language has high confidence then its result is used as the final label else the naive Bayesian classifier's specific language prediction is used as the final result  The lexicon is built over all the data and therefore includes the vocabulary from both the training and testing sets 
The lexicon based classifier is designed to trade higher precision for lower recall  The proposed implementation is considered confident if the number of words from the winning language is at least one more than the number of words considered to be from the language scored in second place 
The stacked classifier is tested against three public LID implementations BIBREF17, BIBREF23, BIBREF8  The LID implementation described in BIBREF17 is available on GitHub and is trained and tested according to a post on the fasttext blog  Character (5-6)-gram features with 16 dimensional vectors worked the best  The implementation discussed in BIBREF23 is available from https://github com/tomkocmi/LanideNN  Following the instructions for an OSX pip install of an old r0 8 release of TensorFlow, the LanideNN code could be executed in Python 3 7 4

ground_truth: Logistic Regression, neural networks

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which languages are similar to each other?
Context 3:  Settings were left at their defaults and a learning rate of 0 001 was used followed by a refinement with learning rate of 0 0001  Only one code modification was applied to return the results from a method that previously just printed to screen  The LID algorithm described in BIBREF8 is also available on GitHub 
The stacked classifier is also tested against the results reported for four other algorithms BIBREF16, BIBREF26, BIBREF24, BIBREF15  All the comparisons are done using the NCHLT BIBREF7, DSL 2015 BIBREF19 and DSL 2017 BIBREF1 datasets discussed in Section SECREF2 
Results and Analysis
The average classification accuracy results are summarised in Table TABREF9  The accuracies reported are for classifying a piece of text by its specific language label  Classifying text only by language group or family is a much easier task as reported in BIBREF8 
Different variations of the proposed classifier were evaluated  A single NB classifier (NB), a stack of two NB classifiers (NB+NB), a stack of a NB classifier and lexicon (NB+Lex) and a lexicon (Lex) by itself  A lexicon with a 50% training token dropout is also listed to show the impact of the lexicon support on the accuracy 
From the results it seems that the DSL 2017 task might be harder than the DSL 2015 and NCHLT tasks  Also, the results for the implementation discussed in BIBREF23 might seem low, but the results reported in that paper is generated on longer pieces of text so lower scores on the shorter pieces of text derived from the NCHLT corpora is expected 
The accuracy of the proposed algorithm seems to be dependent on the support of the lexicon  Without a good lexicon a non-stacked naive Bayesian classifier might even perform better 
The execution performance of some of the LID implementations are shown in Table TABREF10  Results were generated on an early 2015 13-inch Retina MacBook Pro with a 2 9 GHz CPU (Turbo Boosted to 3 4 GHz) and 8GB RAM  The C++ implementation in BIBREF17 is the fastest  The implementation in BIBREF8 makes use of un-hashed feature representations which causes it to be slower than the proposed sklearn implementation  The execution performance of BIBREF23 might improve by a factor of five to ten when executed on a GPU 
Conclusion
LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched  The proposed algorithm was evaluated on three existing datasets and compared to the implementations of three public LID implementations as well as to reported results of four other algorithms  It performed well relative to the other methods beating their results  However, the performance is dependent on the support of the lexicon 
We would like to investigate the value of a lexicon in a production system and how to possibly maintain it using self-supervised learning  We are investigating the application of deeper language models some of which have been used in more recent DSL shared tasks  We would also like to investigate data augmentation strategies to reduce the amount of training data that is required 
Further research opportunities include data harvesting, building standardised datasets and shared tasks for South Africa as well as the rest of Africa  In general, the support for language codes that include more languages seems to be growing, discoverability of research is improving and paywalls seem to no longer be a big problem in getting access to published research 

ground_truth: Nguni languages (zul, xho, nbl, ssw), Sotho languages (nso, sot, tsn)

The Nguni languages are similar to each other, The same is true of the Sotho languages