prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What machine learning and deep learning methods are used for RQE?
Context 1: Introduction
With the availability of rich data on users' locations, profiles and search history, personalization has become the leading trend in large-scale information retrieval  However, efficiency through personalization is not yet the most suitable model when tackling domain-specific searches  This is due to several factors, such as the lexical and semantic challenges of domain-specific data that often include advanced argumentation and complex contextual information, the higher sparseness of relevant information sources, and the more pronounced lack of similarities between users' searches 
A recent study on expert search strategies among healthcare information professionals BIBREF0 showed that, for a given search task, they spend an average of 60 minutes per collection or database, 3 minutes to examine the relevance of each document, and 4 hours of total search time  When written in steps, their search strategy spans over 15 lines and can reach up to 105 lines 
With the abundance of information sources in the medical domain, consumers are more and more faced with a similar challenge, one that needs dedicated solutions that can adapt to the heterogeneity and specifics of health-related information 
Dedicated Question Answering (QA) systems are one of the viable solutions to this problem as they are designed to understand natural language questions without relying on external information on the users 
In the context of QA, the goal of Recognizing Question Entailment (RQE) is to retrieve answers to a premise question ( INLINEFORM0 ) by retrieving inferred or entailed questions, called hypothesis questions ( INLINEFORM1 ) that already have associated answers  Therefore, we define the entailment relation between two questions as: a question INLINEFORM2 entails a question INLINEFORM3 if every answer to INLINEFORM4 is also a correct answer to INLINEFORM5 BIBREF1  
RQE is particularly relevant due to the increasing numbers of similar questions posted online BIBREF2 and its ability to solve differently the challenging issues of question understanding and answer extraction  In addition to being used to find relevant answers, these resources can also be used in training models able to recognize inference relations and similarity between questions 
Question similarity has recently attracted international challenges BIBREF3 , BIBREF4 and several research efforts proposing a wide range of approaches, including Logistic Regression, Recurrent Neural Networks (RNNs), Long Short Term Memory cells (LSTMs), and Convolutional Neural Networks (CNNs) BIBREF5 , BIBREF6 , BIBREF1 , BIBREF7  
In this paper, we study question entailment in the medical domain and the effectiveness of the end-to-end RQE-based QA approach by evaluating the relevance of the retrieved answers  Although entailment was attempted in QA before BIBREF8 , BIBREF9 , BIBREF10 , as far as we know, we are the first to introduce and evaluate a full medical question answering approach based on question entailment for free-text questions  Our contributions are:
The next section is dedicated to related work on question answering, question similarity and entailment  In Section SECREF3 , we present two machine learning (ML) and deep learning (DL) methods for RQE and compare their performance using open-domain and clinical datasets  Section SECREF4 describes the new collection of medical question-answer pairs  In Section SECREF5 , we describe our RQE-based approach for QA  Section SECREF6 presents our evaluation of the retrieved answers and the results obtained on TREC 2017 LiveQA medical questions 
Background
In this section we define the RQE task and describe related work at the intersection of question answering, question similarity and textual inference 
Task Definition
The definition of Recognizing Question Entailment (RQE) can have a significant impact on QA results  In related work, the meaning associated with Natural Language Inference (NLI) varies among different tasks and events  For instance, Recognizing Textual Entailment (RTE) was addressed by the PASCAL challenge BIBREF12 , where the entailment relation has been assessed manually by human judges who selected relevant sentences "entailing" a set of hypotheses from a list of documents returned by different Information Retrieval (IR) methods  In another definition, the Stanford Natural Language Inference corpus SNLI BIBREF13 , used three classification labels for the relations between two sentences: entailment, neutral and contradiction  For the entailment label, the annotators who built the corpus were presented with an image and asked to write a caption “that is a definitely true description of the photo”  For the neutral label, they were asked to provide a caption “that might be a true description of the label”  They were asked for a caption that “is definitely a false description of the photo” for the contradiction label 
More recently, the multiNLI corpus BIBREF14 was shared in the scope of the RepEval 2017 shared task BIBREF15   To build the corpus, annotators were presented with a premise text and asked to write three sentences  One novel sentence, which is “necessarily true or appropriate in the same situations as the premise,” for the entailment label, a sentence, which is “necessarily false or inappropriate whenever the premise is true,” for the contradiction label, and a last sentence, “where neither condition applies,” for the neutral label 
Whereas these NLI definitions might be suitable for the broad topic of text understanding, their relation to practical information retrieval or question answering systems is not straightforward 
In contrast, RQE has to be tailored to the question answering task  For instance, if the premise question is "looking for cold medications for a 30 yo woman", a RQE approach should be able to consider the more general (less restricted) question "looking for cold medications" as relevant, since its answers are relevant for the initial question, whereas "looking for medications for a 30 yo woman" is a useless contextualization  The entailment relation we are seeking in the QA context should include relevant and meaningful relaxations of contextual and semantic constraints (cf  Section SECREF13 ) 
Related Work on Question Answering
Classical QA systems face two main challenges related to question analysis and answer extraction  Several QA approaches were proposed in the literature for the open domain BIBREF16 , BIBREF17 and the medical domain BIBREF18 , BIBREF19 , BIBREF20   A variety of methods were developed for question analysis, focus (topic) recognition and question type identification BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24   Similarly, many different approaches tackled document or passage retrieval and answer selection and (re)ranking BIBREF25 , BIBREF26 , BIBREF27  
An alternative approach consists in finding similar questions or FAQs that are already answered BIBREF28 , BIBREF29   One of the earliest question answering systems based on finding similar questions and re-using the existing answers was FAQ FINDER BIBREF30   Another system that complements the existing Q&A services of NetWellness is SimQ BIBREF2 , which allows retrieval of similar web-based consumer health questions  SimQ uses syntactic and semantic features to compute similarity between questions, and UMLS BIBREF31 as a standardized semantic knowledge source  The system achieves 72 2% precision, 78 0% recall and 75 0% F-score on NetWellness questions  However, the method was evaluated only on one question similarity dataset, and the retrieved answers were not evaluated 
The aim of the medical task at TREC 2017 LiveQA was to develop techniques for answering complex questions such as consumer health questions, as well as to identify relevant answer sources that can comply with the sensitivity of medical information retrieval 
The CMU-OAQA system BIBREF32 achieved the best performance of 0 637 average score on the medical task by using an attentional encoder-decoder model for paraphrase identification and answer ranking  The Quora question-similarity dataset was used for training  The PRNA system BIBREF33 achieved the second best performance in the medical task with 0 49 average score using Wikipedia as the first answer source and Yahoo and Google searches as secondary answer sources  Each medical question was decomposed into several subquestions  To extract the answer from the selected text passage, a bi-directional attention model trained on the SQUAD dataset was used 
Deep neural network models have been pushing the limits of performance achieved in QA related tasks using large training datasets  The results obtained by CMU-OAQA and PRNA showed that large open-domain datasets were beneficial for the medical domain  However, the best system (CMU-OAQA) relying on the same training data obtained a score of 1 139 on the LiveQA open-domain task 
While this gap in performance can be explained in part by the discrepancies between the medical test questions and the open-domain questions, it also highlights the need for larger medical datasets to support deep learning approaches in dealing with the linguistic complexity of consumer health questions and the challenge of finding correct and complete answers 
Another technique was used by ECNU-ICA team BIBREF34 based on learning question similarity via two long short-term memory (LSTM) networks applied to obtain the semantic representations of the questions  To construct a collection of similar question pairs, they searched community question answering sites such as Yahoo! and Answers com  In contrast, the ECNU-ICA system achieved the best performance of 1 895 in the open-domain task but an average score of only 0 402 in the medical task  As the ECNU-ICA approach also relied on a neural network for question matching, this result shows that training attention-based decoder-encoder networks on the Quora dataset generalized better to the medical domain than training LSTMs on similar questions from Yahoo! and Answers com 
The CMU-LiveMedQA team BIBREF20 designed a specific system for the medical task  Using only the provided training datasets and the assumption that each question contains only one focus, the CMU-LiveMedQA system obtained an average score of 0 353  They used a convolutional neural network (CNN) model to classify a question into a restricted set of 10 question types and crawled "relevant" online web pages to find the answers  However, the results were lower than those achieved by the systems relying on finding similar answered questions  These results support the relevance of similar question matching for the end-to-end QA task as a new way of approaching QA instead of the classical QA approaches based on Question Analysis and Answer Retrieval 
Related Work on Question Similarity and Entailment
Several efforts focused on recognizing similar questions  Jeon et al  BIBREF35 showed that a retrieval model based on translation probabilities learned from a question and answer archive can recognize semantically similar questions  Duan et al  BIBREF36 proposed a dedicated language modeling approach for question search, using question topic (user's interest) and question focus (certain aspect of the topic) 
Lately, these efforts were supported by a task on Question-Question similarity introduced in the community QA challenge at SemEval (task 3B) BIBREF3   Given a new question, the task focused on reranking all similar questions retrieved by a search engine, assuming that the answers to the similar questions will be correct answers for the new question  Different machine learning and deep learning approaches were tested in the scope of SemEval 2016 BIBREF3 and 2017 BIBREF4 task 3B  The best performing system in 2017 achieved a MAP of 47 22% using supervised Logistic Regression that combined different unsupervised similarity measures such as Cosine and Soft-Cosine BIBREF37   The second best system achieved 46 93% MAP with a learning-to-rank method using Logistic Regression and a rich set of features including lexical and semantic features as well as embeddings generated by different neural networks (siamese, Bi-LSTM, GRU and CNNs) BIBREF38   In the scope of this challenge, a dataset was collected from Qatar Living forum for training  We refer to this dataset as SemEval-cQA 
In another effort, an answer-based definition of RQE was proposed and tested BIBREF1   The authors introduced a dataset of clinical questions and used a feature-based method that provided an Accuracy of 75% on consumer health questions  We will call this dataset Clinical-QE  Dos Santos et al  BIBREF5 proposed a new approach to retrieve semantically equivalent questions combining a bag-of-words representation with a distributed vector representation created by a CNN and user data collected from two Stack Exchange communities  Lei et al  BIBREF7 proposed a recurrent and convolutional model (gated convolution) to map questions to their semantic representations  The models were pre-trained within an encoder-decoder framework 
RQE Approaches and Experiments
The choice of two methods for our empirical study is motivated by the best performance achieved by Logistic Regression in question-question similarity at SemEval 2017 (best system BIBREF37 and second best system BIBREF38 ), and the high performance achieved by neural networks on larger datasets such as SNLI BIBREF13 , BIBREF39 , BIBREF40 , BIBREF41   We first define the RQE task, then present the two approaches, and evaluate their performance on five different datasets 
Definition
In the context of QA, the goal of RQE is to retrieve answers to a new question by retrieving entailed questions with associated answers  We therefore define question entailment as:
a question INLINEFORM0 entails a question INLINEFORM1 if every answer to INLINEFORM2 is also a complete or partial answer to INLINEFORM3  
We present below two examples of consumer health questions INLINEFORM0 and entailed questions INLINEFORM1 :
Example 1 (each answer to the entailed question B1 is a complete answer to A1):
A1: What is the latest news on tennitis, or ringing in the ear, I am 75 years old and have had ringing in the ear since my mid 5os  Thank you 
B1: What is the latest research on Tinnitus?
Example 2 (each answer to the entailed question B2 is a partial answer to A2):
A2: My mother has been diagnosed with Alzheimer's, my father is not of the greatest health either and is the main caregiver for my mother  My question is where do we start with attempting to help our parents w/ the care giving and what sort of financial options are there out there for people on fixed incomes 
B2: What resources are available for Alzheimer's caregivers?
The inclusion of partial answers in the definition of question entailment also allows efficient relaxation of the contextual constraints of the original question INLINEFORM0 to retrieve relevant answers from entailed, but less restricted, questions 
Deep Learning Model
To recognize entailment between two questions INLINEFORM0 (premise) and INLINEFORM1 (hypothesis), we adapted the neural network proposed by Bowman et al  BIBREF13   Our DL model, presented in Figure FIGREF20 , consists of three 600d ReLU layers, with a bottom layer taking the concatenated sentence representations as input and a top layer feeding a softmax classifier  The sentence embedding model sums the Recurrent neural network (RNN) embeddings of its words  The word embeddings are first initialized with pretrained GloVe vectors  This adaptation provided the best performance in previous experiments with RQE data 
GloVe is an unsupervised learning algorithm to generate vector representations for words BIBREF42   Training is performed on aggregated word co-occurrence statistics from a large corpus, and the resulting representations show interesting linear substructures of the word vector space  We use the pretrained common crawl version with 840B tokens and 300d vectors, which are not updated during training 
Logistic Regression Classifier
In this feature-based approach, we use Logistic Regression to classify question pairs into entailment or no-entailment  Logistic Regression achieved good results on this specific task and outperformed other statistical learning algorithms such as SVM and Naive Bayes  In a preprocessing step, we remove stop words and perform word stemming using the Porter algorithm BIBREF43 for all ( INLINEFORM0 , INLINEFORM1 ) pairs 
We use a list of nine features, selected after several experiments on RTE datasets BIBREF12   We compute five similarity measures between the pre-processed questions and use their values as features  We use Word Overlap, the Dice coefficient based on the number of common bigrams, Cosine, Levenshtein, and the Jaccard similarities  Our feature list also includes the maximum and average values obtained with these measures and the question length ratio (length( INLINEFORM0 )/length( INLINEFORM1 ))  We compute a morphosyntactic feature indicating the number of common nouns and verbs between INLINEFORM2 and INLINEFORM3   TreeTagger BIBREF44 was used for POS tagging 
For RQE, we add an additional feature specific to the question type  We use a dictionary lookup to map triggers to the question type (e g  Treatment, Prognosis, Inheritance)  Triggers are identified for each question type based on a manual annotation of a set of medical questions (cf  Section SECREF36 )  This feature has three possible values: 2 (Perfect match between INLINEFORM0 type(s) and INLINEFORM1 type(s)), 1 (Overlap between INLINEFORM2 type(s) and INLINEFORM3 type(s)) and 0 (No common types) 
Datasets Used for the RQE Study
We evaluate the RQE methods (i e  deep learning model and logistic regression classifier) using two datasets of sentence pairs (SNLI and multiNLI), and three datasets of question pairs (Quora, Clinical-QE, and SemEval-cQA) 
The Stanford Natural Language Inference corpus (SNLI) BIBREF13 contains 569,037 sentence pairs written by humans based on image captioning  The training set of the MultiNLI corpus BIBREF14 consists of 393,000 pairs of sentences from five genres of written and spoken English (e g  Travel, Government)  Two other "matched" and "mismatched" sets are also available for development (20,000 pairs)  Both SNLI and multiNLI consider three types of relationships between sentences: entailment, neutral and contradiction  We converted the contradiction and neutral labels to the same non-entailment class 
The QUORA dataset of similar questions was recently published with 404,279 question pairs  We randomly selected three distinct subsets (80%/10%/10%) for training (323,423 pairs), development (40,428 pairs) and test (40,428 pairs) 
The clinical-QE dataset BIBREF1 contains 8,588 question pairs and was constructed using 4,655 clinical questions asked by family doctors BIBREF45   We randomly selected three distinct subsets (80%/10%/10%) for training (6,870 pairs), development (859 pairs) and test (859 pairs) 
The question similarity dataset of SemEval 2016 Task 3B (SemEval-cQA) BIBREF3 contains 3,869 question pairs and aims to re-rank a list of related questions according to their similarity to the original question  The same dataset was used for SemEval 2017 Task 3 BIBREF4  
To construct our test dataset, we used a publicly shared set of Consumer Health Questions (CHQs) received by the U S  National Library of Medicine (NLM), and annotated with named entities, question types, and focus BIBREF46 , BIBREF47   The CHQ dataset consists of 1,721 consumer information requests manually annotated with subquestions, each identified by a question type and a focus 
First, we selected automatically harvested FAQs, from U S  National Institutes of Health (NIH) websites, that share both the same focus and the same question type with the CHQs  As FAQs are most often very short, we first assume that the CHQ entails the FAQ  Two sets of pairs were constructed: (i) positive pairs of CHQs and FAQs sharing at least one common question type and the question focus, and (ii) negative pairs corresponding to a focus mismatch or type mismatch  For each category of negative examples, we randomly selected the same number of pairs for a balanced dataset  Then, we manually validated the constructed pairs and corrected the positive and negative labels when needed  The final RQE dataset contains 850 CHQ-FAQ pairs with 405 positive and 445 negative pairs  Table TABREF26 presents examples from the five training datasets (SNLI, MultiNLI, SemEval-cQA, Clinical-QE and Quora) and the new test dataset of medical CHQ-FAQ pairs 
Results of RQE Approaches
In the first experiment, we evaluated the DL and ML methods on SNLI, multi-NLI, Quora, and Clinical-QE  For the datasets that did not have a development and test sets, we randomly selected two sets, each amounting to 10% of the data, for test and development, and used the remaining 80% for training  For MultiNLI, we used the dev1-matched set for validation and the dev2-mismatched set for testing 
Table TABREF28 presents the results of the first experiment  The DL model with GloVe word embeddings achieved better results on three datasets, with 82 80% Accuracy on SNLI, 78 52% Accuracy on MultiNLI, and 83 62% Accuracy on Quora  Logistic Regression achieved the best Accuracy of 98 60% on Clinical-RQE  We also performed a 10-fold cross-validation on the full Clinical-QE data of 8,588 question pairs, which gave 98 61% Accuracy 
In the second experiment, we used these datasets for training only and compared their performance on our test set of 850 consumer health questions  Table TABREF29 presents the results of this experiment  Logistic Regression trained on the clinical-RQE data outperformed DL models trained on all datasets, with 73 18% Accuracy 
To validate further the performance of the LR method, we evaluated it on question similarity detection  A typical approach to this task is to use an IR method to find similar question candidates, then a more sophisticated method to select and re-rank the similar questions  We followed a similar approach for this evaluation by combining the LR method with the IR baseline provided in the context of SemEval-cQA  The hybrid method combines the score provided by the Logistic Regression model and the reciprocal rank from the IR baseline using a weight-based combination:
INLINEFORM0
The weight INLINEFORM0 was set empirically through several tests on the cQA-2016 development set ( INLINEFORM1 )  Table TABREF30 presents the results on the cQA-2016 and cQA-2017 test datasets  The hybrid method (LR+IR) provided the best results on both datasets  On the 2016 test data, the LR+IR method outperformed the best system in all measures, with 80 57% Accuracy and 77 47% MAP (official system ranking measure in SemEval-cQA)  On the cQA-2017 test data, the LR+IR method obtained 44 66% MAP and outperformed the cQA-2017 best system in Accuracy with 67 27% 
Discussion of RQE Results
When trained and tested on the same corpus, the DL model with GloVe embeddings gave the best results on three datasets (SNLI, MultiNLI and Quora)  Logistic Regression gave the best Accuracy on the Clinical-RQE dataset with 98 60%  When tested on our test set (850 medical CHQs-FAQs pairs), Logistic Regression trained on Clinical-QE gave the best performance with 73 18% Accuracy 
The SNLI and multi-NLI models did not perform well when tested on medical RQE data  We performed additional evaluations using the RTE-1, RTE-2 and RTE-3 open-domain datasets provided by the PASCAL challenge and the results were similar  We have also tested the SemEval-cQA-2016 model and had a similar drop in performance on RQE data  This could be explained by the different types of data leading to wrong internal conceptualizations of medical terms and questions in the deep neural layers  This performance drop could also be caused by the complexity of the test consumer health questions that are often composed of several subquestions, contain contextual information, and may contain misspellings and ungrammatical sentences, which makes them more difficult to process BIBREF48   Another aspect is the semantics of the task as discussed in Section SECREF6   The definition of textual entailment in open-domain may not quite apply to question entailment due to the strict semantics  Also the general textual entailment definitions refer only to the premise and hypothesis, while the definition of RQE for question answering relies on the relationship between the sets of answers of the compared questions 
Building a Medical QA Collection from Trusted Resources
A RQE-based QA system requires a collection of question-answer pairs to map new user questions to the existing questions with an RQE approach, rank the retrieved questions, and present their answers to the user 
Method
To construct trusted medical question-answer pairs, we crawled websites from the National Institutes of Health (cf  Section SECREF56 )  Each web page describes a specific topic (e g  name of a disease or a drug), and often includes synonyms of the main topic that we extracted during the crawl 
We constructed hand-crafted patterns for each website to automatically generate the question-answer pairs based on the document structure and the section titles  We also annotated each question with the associated focus (topic of the web page) as well as the question type identified with the designed patterns (cf  Section SECREF36 ) 
To provide additional information about the questions that could be used for diverse IR and NLP tasks, we automatically annotated the questions with the focus, its UMLS Concept Unique Identifier (CUI) and Semantic Type  We combined two methods to recognize named entities from the titles of the crawled articles and their associated UMLS CUIs: (i) exact string matching to the UMLS Metathesaurus, and (ii) MetaMap Lite BIBREF49   We then used the UMLS Semantic Network to retrieve the associated semantic types and groups 
Question Types
The question types were derived after the manual evaluation of 1,721 consumer health questions  Our taxonomy includes 16 types about Diseases, 20 types about Drugs and one type (Information) for the other named entities such as Procedures, Medical exams and Treatments  We describe below the considered question types and examples of associated question patterns 
Question Types about Diseases (16): Information, Research (or Clinical Trial), Causes, Treatment, Prevention, Diagnosis (Exams and Tests), Prognosis, Complications, Symptoms, Inheritance, Susceptibility, Genetic changes, Frequency, Considerations, Contact a medical professional, Support Groups 
Examples:
What research (or clinical trial) is being done for DISEASE?
What is the outlook for DISEASE?
How many people are affected by DISEASE?
When to contact a medical professional about DISEASE?
Who is at risk for DISEASE?
Where to find support for people with DISEASE?
Question Types About Drugs (20): Information, Interaction with medications, Interaction with food, Interaction with herbs and supplements, Important warning, Special instructions, Brand names, How does it work, How effective is it, Indication, Contraindication, Learn more, Side effects, Emergency or overdose, Severe reaction, Forget a dose, Dietary, Why get vaccinated, Storage and disposal, Usage, Dose 
Examples:
Are there interactions between DRUG and herbs and supplements?
What important warning or information should I know about DRUG?
Are there safety concerns or special precautions about DRUG?
What is the action of DRUG and how does it work?
Who should get DRUG and why is it prescribed?
What to do in case of a severe reaction to DRUG?
Question Type for other medical entities (e g  Procedure, Exam, Treatment): Information 
What is Coronary Artery Bypass Surgery?
What are Liver Function Tests?
Medical Resources
We used 12 trusted websites to construct a collection of question-answer pairs  For each website, we extracted the free text of each article as well as the synonyms of the article focus (topic)  These resources and their brief descriptions are provided below:
National Cancer Institute (NCI) : We extracted free text from 116 articles on various cancer types (729 QA pairs)  We manually restructured the content of the articles to generate complete answers (e g  a full answer about the treatment of all stages of a specific type of cancer)  Figure FIGREF54 presents examples of QA pairs generated from a NCI article 
Genetic and Rare Diseases Information Center (GARD): This resource contains information about various aspects of genetic/rare diseases  We extracted all disease question/answer pairs from 4,278 topics (5,394 QA pairs) 
Genetics Home Reference (GHR): This NLM resource contains consumer-oriented information about the effects of genetic variation on human health  We extracted 1,099 articles about diseases from this resource (5,430 QA pairs) 
MedlinePlus Health Topics: This portion of MedlinePlus contains information on symptoms, causes, treatment and prevention for diseases, health conditions and wellness issues  We extracted the free texts in summary sections of 981 articles (981 QA pairs) 
National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) : We extracted text from 174 health information pages on diseases studied by this institute (1,192 QA pairs) 
National Institute of Neurological Disorders and Stroke (NINDS): We extracted free text from 277 information pages on neurological and stroke-related diseases from this resource (1,104 QA pairs) 
NIHSeniorHealth : This website contains health and wellness information for older adults  We extracted 71 articles from this resource (769 QA pairs) 
National Heart, Lung, and Blood Institute (NHLBI) : We extracted text from 135 articles on diseases, tests, procedures, and other relevant topics on disorders of heart, lung, blood, and sleep (559 QA pairs) 
Centers for Disease Control and Prevention (CDC) : We extracted text from 152 articles on diseases and conditions (270 QA pairs) 
MedlinePlus A D A M  Medical Encyclopedia: This resource contains 4,366 articles about conditions, tests, and procedures  17,348 QA pairs were extracted from this resource  Figure FIGREF55 presents examples of QA pairs generated from A D A M encyclopedia 
MedlinePlus Drugs: We extracted free text from 1,316 articles about Drugs and generated 12,889 QA pairs 
MedlinePlus Herbs and Supplements: We extracted free text from 99 articles and generated 792 QA pairs 
The final collection contains 47,457 annotated question-answer pairs about Diseases, Drugs and other named entities (e g  Tests) extracted from these 12 trusted resources 
The Proposed Entailment-based QA System
Our goal is to generate a ranked list of answers for a given Premise Question INLINEFORM0 by ranking the recognized Hypothesis Questions INLINEFORM1   Based on the RQE experiments above (Section SECREF27 ), we selected Logistic Regression trained on the clinical-RQE dataset to recognize entailed questions and rank them with their classification scores 
RQE-based QA Approach
Classifying the full QA collection for each test question is not feasible for real-time applications  Therefore, we first filter the questions with an IR method to retrieve candidate questions, then classify them as entailed (or not) by the user/test question  Based on the positive results of the combination method tested on SemEval-cQA data (Section SECREF27 ), we adopted a combination method to merge the results obtained by the search engine and the RQE scores  The answers are then combined from both methods and ranked using an aggregate score  Figure FIGREF82 presents the overall architecture of the proposed QA system  We describe each module in more details next 
Finding Similar Question Candidates
For each premise question INLINEFORM0 , we use the Terrier search engine to retrieve INLINEFORM1 relevant question candidates INLINEFORM2 and then apply the RQE classifier to predict the labels for the pairs ( INLINEFORM3 , INLINEFORM4 ) 
We indexed the questions of our QA collection without the associated answers  In order to improve the indexing and the performance of question retrieval, we also indexed the synonyms of the question focus and the triggers of the question type with each question  This choice allowed us to avoid the shortcomings of query expansion, including incorrect or irrelevant synonyms and the increased execution time  The synonyms of the question focus (topic) were extracted automatically from the QA collection  The triggers of each question type were defined manually in the question types taxonomy  Below are two examples of indexed questions from our QA collection, with the automatically added focus synonyms and question type triggers:
What are the treatments for Torticollis?
Focus: Torticollis  Question type: Treatment 
Added focus synonyms: "Spasmodic torticollis, Wry neck, Loxia, Cervical dystonia"  Added question type triggers: "relieve, manage, cure, remedy, therapy" 
What is the outlook for Legionnaire disease?
Focus: Legionnaire disease  Question Type: Prognosis 
Added focus synonyms: "Legionella pneumonia, Pontiac fever, Legionellosis"  Added question type triggers: "prognosis, life expectancy" 
The IR task consists of retrieving hypothesis questions INLINEFORM0 relevant to the submitted question INLINEFORM1   As fusion of IR result has shown good performance in different tracks in TREC, we merge the results of the TF-IDF weighting function and the In-expB2 DFR model BIBREF50  
Let INLINEFORM0 = INLINEFORM1 , INLINEFORM2 ,    , INLINEFORM3 be the set of INLINEFORM4 questions retrieved by the first IR model INLINEFORM5 and INLINEFORM6 = INLINEFORM7 , INLINEFORM8 ,    , INLINEFORM9 be the set of INLINEFORM10 questions retrieved by the second IR model INLINEFORM11   We merge both sets by summing the scores of each retrieved question INLINEFORM12 in both INLINEFORM13 and INLINEFORM14 lists, then we re-rank the hypothesis questions INLINEFORM15  
Combining IR and RQE Methods
The IR models and the RQE Logistic Regression model bring different perspectives to the search for relevant candidate questions  In particular, question entailment allows understanding the relations between the important terms, whereas the traditional IR methods identify the important terms, but will not notice if the relations are opposite  Moreover, some of the question types that the RQE classifier learns will not be deemed important terms by traditional IR and the most relevant questions will not be ranked at the top of the list 
Therefore, in our approach, when a question is submitted to the system, candidate questions are fetched using the IR models, then the RQE classifier is applied to filter out the non-entailed questions and re-rank the remaining candidates 
Specifically, we denote INLINEFORM0 the list of question candidates INLINEFORM1 returned by the IR system  The premise question INLINEFORM2 is then used to construct N question pairs INLINEFORM3   The RQE classifier is then applied to filter out the question pairs that are not entailed and re-rank the remaining pairs 
More precisely, let INLINEFORM0 = INLINEFORM1 in INLINEFORM2 be the list of selected candidate questions that have a positive entailment relation with a given premise question INLINEFORM3   We rank INLINEFORM4 by computing a hybrid score INLINEFORM5 for each candidate question INLINEFORM6 taking into account the score of the IR system INLINEFORM7 and the score of the RQE system INLINEFORM8  
For each system INLINEFORM0 INLINEFORM1 , we normalize the associated score by dividing it by the maximum score among the INLINEFORM2 candidate questions retrieved by INLINEFORM3 for INLINEFORM4 :
INLINEFORM0
INLINEFORM0 INLINEFORM1
In our experiments, we fixed the value of INLINEFORM0 to 100  This threshold value was selected as a safe value for this task for the following reasons:
Our collection of 47,457 question-answer pairs was collected from only 12 NIH institutes and is unlikely to contain more than 100 occurrences of the same focus-type pair 
Each question was indexed with additional annotations for the question focus, its synonyms and the question type synonyms 
Evaluating RQE for Medical Question Answering
The objective of this evaluation is to study the effectiveness of RQE for Medical Question Answering, by comparing the answers retrieved by the hybrid entailment-based approach, the IR method and the other QA systems participating to the medical task at TREC 2017 LiveQA challenge (LiveQA-Med) 
Evaluation Method
We developed an interface to perform the manual evaluation of the retrieved answers  Figure 5 presents the evaluation interface showing, for each test question, the top-10 answers of the evaluated QA method and the reference answer(s) used by LiveQA assessors to help judging the retrieved answers by the participating systems 
We used the test questions of the medical task at TREC-2017 LiveQA BIBREF11   These questions are randomly selected from the consumer health questions that the NLM receives daily from all over the world  The test questions cover different medical entities and have a wide list of question types such as Comparison, Diagnosis, Ingredient, Side effects and Tapering 
For a relevant comparison, we used the same judgment scores as the LiveQA Track:
Correct and Complete Answer (4)
Correct but Incomplete (3)
Incorrect but Related (2)
Incorrect (1)
We evaluated the answers returned by the IR-based method and the hybrid QA method (IR+RQE) according to the same reference answers used in LiveQA-Med  The answers were anonymized (the method names were blinded) and presented to 3 assessors: a medical doctor (Assessor A), a medical librarian (B) and a researcher in medical informatics (C)  None of the assessors participated in the development of the QA methods  Assessors B and C evaluated 1,000 answers retrieved by each of the methods (IR and IR+RQE)  Assessor A evaluated 2,000 answers from both methods 
Table TABREF103 presents the inter-annotator agreement (IAA) through F1 score computed by considering one of the assessors as reference  In the first evaluation, we computed the True Positives (TP) and False Positives (FP) over all ratings and the Precision and F1 score  As there are no negative labels (only true or false positives for each category), Recall is 100%  We also computed a partial IAA by grouping the "Correct and Complete Answer" and "Correct but Incomplete" ratings (as Correct), and the "Incorrect but Related" and "Incorrect" ratings (as Incorrect)  The average agreement on distinguishing the Correct and Incorrect answers is 94 33% F1 score  Therefore, we used the evaluations performed by assessor A for both methods  The official results of the TREC LiveQA track relied on one assessor per question as well 
Evaluation of the first retrieved answer
We computed the measures used by TREC LiveQA challenges BIBREF51 , BIBREF11 to evaluate the first retrieved answer for each test question:
avgScore(0-3): the average score over all questions, transferring 1-4 level grades to 0-3 scores  This is the main score used to rank LiveQA runs 
succ@i+: the number of questions with score i or above (i INLINEFORM0 {2  4}) divided by the total number of questions 
prec@i+: the number of questions with score i or above (i INLINEFORM0 {2  4}) divided by number of questions answered by the system 
Table TABREF108 presents the average scores, success and precision results  The hybrid IR+RQE QA system achieved better results than the IR-based system with 0 827 average score  It also achieved a higher score than the best results achieved in the medical challenge at LiveQA'17  Evaluating the RQE system alone is not relevant, as applying RQE on the full collection for each user question is not feasible for a real-time system because of the extended execution time 
Evaluation of the top ten answers
In this evaluation, we used Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) which are commonly used in QA to evaluate the top-10 answers for each question  We consider answers rated as “Correct and Complete Answer” or “Correct but Incomplete” as correct answers, as the test questions contain multiple subquestions while each answer in our QA collection can cover only one subquestion 
MAP is the mean of the Average Precision (AvgP) scores over all questions 
(1) INLINEFORM0
Q is the number of questions  INLINEFORM0 is the AvgP of the INLINEFORM1 question 
INLINEFORM0
K is the number of correct answers  INLINEFORM0 is the rank of INLINEFORM1 correct answer 
MRR is the average of the reciprocal ranks for each question  The reciprocal rank of a question is the multiplicative inverse of the rank of the first correct answer 
(2) INLINEFORM0
Q is the number of questions  INLINEFORM0 is the rank of the first correct answer for the INLINEFORM1 question 
Table TABREF113 presents the MAP@10 and MRR@10 of our QA methods  The IR+RQE system outperforms the IR-based QA system with 0 311 MAP@10 and 0 333 MRR@10 
Discussion of entailment-based QA for the medical domain
In our evaluation, we followed the same LiveQA guidelines with the highest possible rigor  In particular, we consulted with NIST assessors who provided us with the paraphrases of the test questions that they used to judge the answers  Our IAA on the answers rating was also high compared to related tasks, with an 88 5% F1 agreement with the exact four categories and a 94 3% agreement when reducing the categories to two: “Correct” and “Incorrect” answers  Our results show that RQE improves the overall performance and exceeds the best results in the medical LiveQA'17 challenge by a factor of 29 8%  This performance improvement is particularly interesting as:
Our answer source has only 47K question-answer pairs when LiveQA participating systems relied on much larger collections, including the World Wide Web 
Our system answered one subquestion at most when many LiveQA test questions had several subquestions 
The latter observation, (b), makes the hybrid IR+RQE approach even more promising as it gives it a large potential for the improvement of answer completeness 
The former observation, (a), provides another interesting insight: restricting the answer source to only reliable collections can actually improve the QA performance without losing coverage (i e , our QA approach provided at least one answer to each test question and obtained the best relevance score) 
In another observation, the assessors reported that many of the returned answers had a correct question type but a wrong focus, which indicates that including a focus recognition module to filter such wrong answers can improve further the QA performance in terms of precision  Another aspect that was reported is the repetition of the same (or similar) answer from different websites, which could be addressed by improving answer selection with inter-answer comparisons and removal of near-duplicates  Also, half of the LiveQA test questions are about Drugs, when only two of our resources are specialized in Drugs, among 12 sub-collections overall  Accordingly, the assessors noticed that the performance of the QA systems was better on questions about diseases than on questions about drugs, which suggests a need for extending our medical QA collection with more information about drugs and associated question types 
We also looked closely at the private websites used by the LiveQA-Med annotators to provide some of the reference answers for the test questions  For instance, the ConsumerLab website was useful to answer a question about the ingredients of a Drug (COENZYME Q10)  Similarly, the eHealthMe website was used to answer a test question asking about interactions between two drugs (Phentermine and Dicyclomine) when no information was found in DailyMed  eHealthMe provides healthcare big data analysis and private research and studies including self-reported adverse drug effects by patients 
But the question remains on the extent to which such big data and other private websites could be used to automatically answer medical questions if information is otherwise unavailable  Unlike medical professionals, patients do not necessarily have the knowledge and tools to validate such information  An alternative approach could be to put limitations on medical QA systems in terms of the questions that can be answered (e g  "What is my diagnosis for such symptoms") and build classifiers to detect such questions and warn the users about the dangers of looking for their answers online 
More generally, medical QA systems should follow some strict guidelines regarding the goal and background knowledge and resources of each system in order to protect the consumers from misleading or harmful information  Such guidelines could be based (i) on the source of the information such as health and medical information websites sponsored by the U S  government, not-for-profit health or medical organizations, and medical university centers, or (ii) on conventions such as the code of conduct of the HON Foundation (HONcode) that addresses the reliability and usefulness of medical information on the Internet 
Our experiments show that limiting the number of answer sources with such guidelines is not only feasible, but it could also enhance the performance of the QA system from an information retrieval perspective 
Conclusion
In this paper, we carried out an empirical study of machine learning and deep learning methods for Recognizing Question Entailment in the medical domain using several datasets  We developed a RQE-based QA system to answer new medical questions using existing question-answer pairs  We built and shared a collection of 47K medical question-answer pairs  Our QA approach outperformed the best results on TREC-2017 LiveQA medical test questions  The proposed approach can be applied and adapted to open-domain as well as specific-domain QA  Deep learning models achieved interesting results on open-domain and clinical datasets, but obtained a lower performance on consumer health questions  We will continue investigating other network architectures including transfer learning, as well as creation of a large collection of consumer health questions for training to improve the performance of DL models  Future work also includes exploring integration of a Question Focus Recognition module to enhance candidate question retrieval, and expanding our question-answer collection 
Acknowledgements
We thank Halil Kilicoglu (NLM/NIH) for his help with the crawling and the manual evaluation and Sonya E  Shooshan (NLM/NIH) for her help with the judgment of the retrieved answers  We also thank Ellen Voorhees (NIST) for her valuable support with the TREC LiveQA evaluation 
We consider the case of the question number 36 in the TREC-2017 LiveQA medical test dataset:
36  congenital diaphragmatic hernia  what are the causes of congenital diaphragmatic hernia? Can cousin marriage cause this? What kind of lung disease the baby might experience life long?
This question was answered by 5 participating runs (vs  8 runs for other questions), and all submitted answers were wrong (scores of 1 or 2)  However, our IR-based QA system retrieved one excellent answer (score 4) and our hybrid IR+RQE system provided 3 excellent answers 
A) TREC 2017 LiveQA-Med Participants' Results:
B) Our IR-based QA System:
C) Our IR+RQE QA System:
Context 2: Introduction
Deep neural networks (DNNs), in particular convolutional and recurrent neural networks, with huge architectures have been proven successful in wide range of tasks including audio processing such as speech to text [1 - 4], emotion recognition [5 - 8], speech/non-speech (e g , of non-speech include noise, music, etc ,) classification [9 - 12], etc 
Training these deep architectures require large amount of annotated data, as a result, they cannot be used in low data resource scenarios which is common in speech-based applications [13 - 15]  Apart from collecting large data corpus, annotating the data is also very difficult, and requires manual supervision and efforts  Especially, annotation of speech for tasks like emotion recognition also suffer from lack of agreement among the annotators [16]  Hence, there is a need to build reliable systems that can work in low resource scenario 
In this work, we propose a novel approach to address the task of classification in low data resource scenarios  Our approach involves simultaneously considering more than one sample (in this work, two samples are considered) to train the classifier  We call this approach as simultaneous two sample learning (s2sL)  The proposed approach is also applicable to low resource data suffering with data imbalance  The contributions of this paper are:
Proposed approach
The s2sL approach proposed to address low data resource problem is explained in this Section  In this work, we use MLP (modified to handle our data representation) as the base classifier  Here, we explain the s2sL approach by considering two-class classification task 
Data representation
Consider a two-class classification task with INLINEFORM0 denoting the set of class labels, and let INLINEFORM1 and INLINEFORM2 be the number of samples corresponding to INLINEFORM3 and INLINEFORM4 , respectively  In general, to train a classifier, the samples in the train set are provided as input-output pairs as follows  DISPLAYFORM0
where INLINEFORM0 refers to the INLINEFORM1 -dimensional feature vector representing the INLINEFORM2 sample corresponding to INLINEFORM3 class label, and INLINEFORM4 , refers to output label of INLINEFORM5 class  In the proposed data representation format, called simultaneous two sample (s2s) representation, we will simultaneously consider two samples as follows  DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 are the INLINEFORM2 -dimensional feature vectors representing the INLINEFORM3 sample in INLINEFORM4 class and INLINEFORM5 sample in the INLINEFORM6 class, respectively; and INLINEFORM7 , INLINEFORM8 refers to the output labels of INLINEFORM9 and INLINEFORM10 class, respectively 
Hence, in s2s data representation, we will have an input feature vector of length INLINEFORM0 i e , INLINEFORM1 , and output class labels as either INLINEFORM2 or INLINEFORM3   Each sample can be combined with all the samples (i e , with ( INLINEFORM4 ) samples) in the dataset  Therefore, by representing the data in the s2s format, the number of samples in the train set increases to INLINEFORM5 from INLINEFORM6 samples  We hypothesize that the s2s format is expected to help the network not only to learn the characteristics of the two classes separately, but also the difference and similarity in characteristics of the two classes 
Classifier training
MLP, the most commonly used feed forward neural network, is considered as the base classifier to validate our proposed s2s framework  Generally, MLPs are trained using the data format given by eq  INLINEFORM0   But to train the MLP on our s2s based data representation (as in eq  INLINEFORM1 ), the following modifications are made to the MLP architecture (refer to Figure FIGREF4 ) 
We have INLINEFORM0 units (instead of INLINEFORM1 units) in the input layer to accept the two samples i e , INLINEFORM2 and INLINEFORM3 , simultaneously 
The structure of the hidden layer in this approach is similar to that of a regular MLP  The number of hidden layers and hidden units can be varied depending upon the complexity of the problem  The number of units in the hidden layer is selected empirically by varying the number of hidden units from 2 to twice the length of the input layer (i e , 2 to INLINEFORM0 ) and the unit at which the highest performance is obtained are selected  In this paper, we considered only a single hidden layer  Rectified linear units (ReLU) are used for hidden layer 
The output layer will consist of units equal to twice the considered number of classes in the classification task i e, the output layer will have four units for two-class classification task  The sigmoid activation function (not softmax) is used on the output layer units  Unlike regular MLP, we use sigmoid activation units in the output layer, with binary cross-entropy as the cost function, because the output labels in the proposed s2s based data representation will have more than one unit active at a time (not one-hot encoded output) and this condition cannot be handled using softmax function 
As can be seen from Figure FIGREF4 , the output layer in our proposed method has outputs INLINEFORM0 and INLINEFORM1 which corresponds to the outputs associated with the input feature vectors INLINEFORM2 and INLINEFORM3 , respectively  For a two-class classification problem, there will be four units in the output layer and the possible output labels are INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 corresponding to the class labels INLINEFORM8 and INLINEFORM9 , respectively  This architecture is referred to as s2s-MLP  In s2sL, s2s-MLP is trained using the s2s data representation format  Further, the s2s-MLP is trained using adam optimizer 
Classifier testing
Generally, the feature vector corresponding to the test sample is provided as input to the trained MLP in the testing phase and the class label is decided based on the obtained output 
However, in s2sL method, the feature vector corresponding to the test sample should also be converted to the s2s data representation format to test the trained s2s-MLP  We propose a testing approach, where the given test sample is combined with a set of preselected reference samples, whose class label is known a priori, to generate multiple instances of the same test sample as follows  DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 refer to the INLINEFORM2 -dimensional feature vector corresponding to the test sample and the INLINEFORM3 reference sample, respectively  INLINEFORM4 refers to the considered number of reference samples  These reference samples can be selected from any of the two classes 
For testing the s2s-MLP (as shown in Figure FIGREF4 ), each test sample INLINEFORM0 (same as ' INLINEFORM1 ' in (3)) is combined with all the INLINEFORM2 reference samples ( INLINEFORM3 ) to form INLINEFORM4 different instances of the same test sample INLINEFORM5   The corresponding outputs ( INLINEFORM6 ) obtained from s2s-MLP for the INLINEFORM7 generated instances of INLINEFORM8 are combined by voting-based decision approach to obtain the final decision INLINEFORM9   The class label that gets maximum votes is considered as the predicted output label 
Experiments
We validate the performance of the proposed s2sL by providing the preliminary results obtained on two different tasks namely, Speech/Music discrimination and emotion classification  We considered the GTZAN Music-Speech dataset [17], consisting of 120 audio files (60 speech and 60 music), for task of classifying speech and music  Each audio file (of 2 seconds duration) is represented using a 13-dimensional mel-frequency cepstral coefficient (MFCC) vector, where each MFCC vector is the average of all the frame level (frame size of 30 msec and an overlap of 10 msec) MFCC vectors  It is to be noted that our main intention for this task is not better feature selection, but to demonstrate the effectiveness of our approach, in particular for low data scenarios 
The standard Berlin speech emotion database (EMO-DB) [18] consisting of 535 utterances corresponding to 7 different emotions is considered for the task of emotion classification  Each utterance is represented by a 19-dimensional feature vector obtained by using the feature selection algorithm from WEKA toolkit [19] on the 384-dimensional utterance level feature vector obtained using openSMILE toolkit [20]  For two class classification, we considered the two most confusing emotion pairs i e , (Neutral,Sad) and (Anger, Happy)  Data corresponding to Speech/Music classification (60 speech and 60 music samples) and Neutral/Sad classification (79 neutral and 62 sad utterances) is balanced whereas Anger/Happy classification task has data imbalance, with anger forming the majority class (127 samples) and happy forming the minority class (71 samples)  Therefore, we show the performance of s2sL on both, balanced and imbalanced datasets 
All experimental results are validated using 5-fold cross validation (80% of data for training and 20% for testing in each fold)  Further, to analyze the effectiveness of s2sL in low resource scenarios, different proportions of training data, within each fold, are considered to train the system  For this analysis, we considered 4 different proportions i e , INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 of the training data to train the classifier  For instance, INLINEFORM4 means considering only half of the original training data to train the classifier, and INLINEFORM5 means considering the complete training data  5-fold cross validation is considered for all data proportions  Accuracy (in %) is used as a performance measure for balanced data classification tasks (i e , Speech/Music classification and Neutral/Sad emotion classification), whereas the more preferred INLINEFORM6 measure [21] is used as a measure for imbalanced data classification task (i e , Anger/Happy emotion classification) 
Table TABREF14 show the results obtained for proposed s2sL approach in comparison to that of MLP for the tasks of Speech/Music and Neutral/Sad classification, by considering different proportions of training data  The values in Table TABREF14 are mean accuracies (in %) obtained by 5-fold cross validation  It can be observed from Table TABREF14 that for both tasks, s2sL method outperforms MLP, especially at low resource conditions  s2sL shows an absolute improvement in accuracy of INLINEFORM0 % and INLINEFORM1 % over MLP for Speech/Music and Neutral/Sad classification tasks, respectively, when INLINEFORM2 of the original training data is used in experiments 
Table TABREF14 show the results (in terms of INLINEFORM0 values) obtained for proposed s2sL approach in comparison to that of MLP for Anger/Happy classification (data imbalance problem)  Here, state-of-the-art methods i e , Eusboost [22] and MWMOTE [23] are also considered for comparison  It can be observed from Table TABREF14 that the s2sL method outperforms MLP, and also performs better than Eusboost and MWMOTE techniques on imbalanced data (around 3 % absolute improvement in INLINEFORM1 value for s2sL compared to MWMOTE, when INLINEFORM2 of the training data is considered)  In particular, at lower amounts of training data, s2sL outperforms all the other methods, illustrating its effectiveness even for low resourced data imbalance problems  s2sL method shows an absolute improvement of 6% ( INLINEFORM3 ) in INLINEFORM4 value over the second best ( INLINEFORM5 for MWMOTE), when only INLINEFORM6 of the training data is used 
Conclusions
In this paper, we introduced a novel s2s framework to effectively learn the class discriminative characteristics, even from low data resources  In this framework, more than one sample (here, two samples) are simultaneously considered to train the classifier  Further, this framework allows to generate multiple instances of the same test sample, by considering preselected reference samples, to achieve a more profound decision making  We illustrated the significance of our approach by providing the experimental results for two different tasks namely, speech/music discrimination and emotion classification  Further, we showed that the s2s framework can also handle the low resourced data imbalance problem 
References
[1] Hinton, G , Deng, L , Yu, D , Dahl, G  E , Mohamed, A  R , Jaitly, N , Senior, A , Vanhoucke, V , Nguyen, P , Sainath, T  N  & Kingsbury, B  (2012) Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups  IEEE Signal Processing Magazine, pp  82–97 
[2] Vinyals, O , Ravuri, S  V  & Povey, D  (2012) Revisiting recurrent neural networks for robust ASR  In Proc  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp  4085–4088 
[3] Lu, L , Zhang, X , Cho K  & Renals, S  (2015) A study of the recurrent neural network encoder-decoder for large vocabulary speech recognition  In Proc  INTERSPEECH, pp  3249–3253 
[4] Zhang, Y , Pezeshki, M , Brakel, P , Zhang, S , Bengio, C L Y  & Courville, A  (2017) Towards end-to-end speech recognition with deep convolutional neural networks  arXiv preprint, arXiv:1701

ground_truth: Logistic Regression, neural networks

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which models/frameworks do they compare to?
Context 2: 02720 
[5] Han, K , Yu, D  & Tashev, I  (2014) Speech emotion recognition using deep neural network and extreme learning machine  In Proc  Interspeech, pp  223–227 
[6] Trigeorgis, G , Ringeval, F , Brueckner, R , Marchi, E , Nicolaou, M  A , Schuller, B , & Zafeiriou, S  (2016) Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network  In Proc  ICASSP, pp  5200–5204) 
[7] Huang, Che-Wei, & Shrikanth S  Narayanan  (2016) Attention Assisted Discovery of Sub-Utterance Structure in Speech Emotion Recognition  In Proc  INTERSPEECH, pp  1387–1391 
[8] Zhang, Z , Ringeval, F , Han, J , Deng, J , Marchi, E  & Schuller, B  (2016) Facing realism in spontaneous emotion recognition from speech: Feature enhancement by autoencoder with LSTM neural networks  In Proc  INTERSPEECH, pp  3593–3597 
[9] Scheirer, E  D  & Slaney, M  (2003) Multi-feature speech/music discrimination system  U S  Patent 6,570,991 
[10] Pikrakis, A  & Theodoridis S  (2014) Speech-music discrimination: A deep learning perspective  In Proc  European Signal Processing Conference (EUSIPCO), pp  616–620 
[11] Choi, K , Fazekas, G  & Sandler, M  (2016) Automatic tagging using deep convolutional neural networks  arXiv preprint, arXiv:1606 00298 
[12] Zazo Candil, R , Sainath, T N , Simko, G  & Parada, C  (2016) Feature learning with raw-waveform CLDNNs for Voice Activity Detection  In Proc  Interspeech, pp  3668–3672 
[13] Tian, L , Moore, JD  & Lai, C  (2015) Emotion recognition in spontaneous and acted dialogues  In Proc  International Conference on Affective Computing and Intelligent Interaction (ACII), pp  698–704 
[14] Dumpala, S H , Chakraborty, R  & Kopparapu, S K  (2017) k-FFNN: A priori knowledge infused feed-forward neural networks  arXiv preprint, arXiv:1704 07055 
[15] Dumpala, S H , Chakraborty, R  & Kopparapu, S K  (2018) Knowledge-driven feed-forward neural network for audio affective content analysis  To appear in AAAI-2018 workshop on Affective Content Analysis 
[16] Deng, J , Zhang, Z , Marchi, E  & Schuller, B  (2013) Sparse autoencoder-based feature transfer learning for speech emotion recognition  In Proc  Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp  511–516 
[17] George Tzanetakis, Gtzan musicspeech  Availabe on-line at http://marsyas info/download/datasets 
[18] Burkhardt, F , Paeschke, A , Rolfes, M , Sendlmeier, W F  & Weiss, B  (2005) A database of german emotional speech  In Proc  Interspeech, pp  1517–1520 
[19] Hall, M , Frank, E , Holmes, G , Pfahringer, B , Reutemann, P  & Witten, I H  (2009) The WEKA data mining software: an update  ACM SIGKDD explorations newsletter, 11(1), pp  10–18 
[20] Eyben, F , Weninger, F , Gross, F  & Schuller, B  (2013) Recent developments in opensmile, the munich open-source multimedia feature extractor  In Proc  ACM international conference on Multimedia, pp  835–838 
[21] Maratea, A , Petrosino, A  & Manzo, M  (2014) Adjusted f-measure and kernel scaling for imbalanced data learning  Information Sciences 257:331–341 
[22] Galar, M , Fernandez, A , Barrenechea, E  & Herrera, F  (2013) Eusboost: Enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling  Pattern Recognition 46(12):3460–3471 
[23] Barua, S , Islam, M  M , Yao, X  & Murase, K  (2014) Mwmote–majority weighted minority oversampling technique for imbalanced data set learning  IEEE Transactions on Knowledge and Data Engineering 26(2):405–425 
Context 3: Introduction
Over the past two decades, the emergence of social media has enabled the proliferation of traceable human behavior  The content posted by users can reflect who their friends are, what topics they are interested in, or which company they are working for  At the same time, users are listing a number of profile fields to define themselves to others  The utilization of such metadata has proven important in facilitating further developments of applications in advertising BIBREF0 , personalization BIBREF1 , and recommender systems BIBREF2   However, profile information can be limited, depending on the platform, or it is often deliberately omitted BIBREF3   To uncloak this information, a number of studies have utilized social media users' footprints to approximate their profiles 
This paper explores the potential of predicting a user's industry –the aggregate of enterprises in a particular field– by identifying industry indicative text in social media  The accurate prediction of users' industry can have a big impact on targeted advertising by minimizing wasted advertising BIBREF4 and improved personalized user experience  A number of studies in the social sciences have associated language use with social factors such as occupation, social class, education, and income BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8   An additional goal of this paper is to examine such findings, and in particular the link between language and occupational class, through a data-driven approach 
In addition, we explore how meaning changes depending on the occupational context  By leveraging word embeddings, we seek to quantify how, for example, cloud might mean a separate concept (e g , condensed water vapor) in the text written by users that work in environmental jobs while it might be used differently by users in technology occupations (e g , Internet-based computing) 
Specifically, this paper makes four main contributions  First, we build a large, industry-annotated dataset that contains over 20,000 blog users  In addition to their posted text, we also link a number of user metadata including their gender, location, occupation, introduction and interests 
Second, we build content-based classifiers for the industry prediction task and study the effect of incorporating textual features from the users' profile metadata using various meta-classification techniques, significantly improving both the overall accuracy and the average per industry accuracy 
Next, after examining which words are indicative for each industry, we build vector-space representations of word meanings and calculate one deviation for each industry, illustrating how meaning is differentiated based on the users' industries  We qualitatively examine the resulting industry-informed semantic representations of words by listing the words per industry that are most similar to job related and general interest terms 
Finally, we rank the different industries based on the normalized relative frequencies of emotionally charged words (positive and negative) and, in addition, discover that, for both genders, these frequencies do not statistically significantly correlate with an industry's gender dominance ratio 
After discussing related work in Section SECREF2 , we present the dataset used in this study in Section SECREF3   In Section SECREF4 we evaluate two feature selection methods and examine the industry inference problem using the text of the users' postings  We then augment our content-based classifier by building an ensemble that incorporates several metadata classifiers  We list the most industry indicative words and expose how each industrial semantic field varies with respect to a variety of terms in Section SECREF5   We explore how the frequencies of emotionally charged words in each gender correlate with the industries and their respective gender dominance ratio and, finally, conclude in Section SECREF6  
Related Work
Alongside the wide adoption of social media by the public, researchers have been leveraging the newly available data to create and refine models of users' behavior and profiling  There exists a myriad research that analyzes language in order to profile social media users  Some studies sought to characterize users' personality BIBREF9 , BIBREF10 , while others sequenced the expressed emotions BIBREF11 , studied mental disorders BIBREF12 , and the progression of health conditions BIBREF13   At the same time, a number of researchers sought to predict the social media users' age and/or gender BIBREF14 , BIBREF15 , BIBREF16 , while others targeted and analyzed the ethnicity, nationality, and race of the users BIBREF17 , BIBREF18 , BIBREF19   One of the profile fields that has drawn a great deal of attention is the location of a user  Among others, Hecht et al  Hecht11 predicted Twitter users' locations using machine learning on nationwide and state levels  Later, Han et al  Han14 identified location indicative words to predict the location of Twitter users down to the city level 
As a separate line of research, a number of studies have focused on discovering the political orientation of users BIBREF15 , BIBREF20 , BIBREF21   Finally, Li et al  Li14a proposed a way to model major life events such as getting married, moving to a new place, or graduating  In a subsequent study, BIBREF22 described a weakly supervised information extraction method that was used in conjunction with social network information to identify the name of a user's spouse, the college they attended, and the company where they are employed 
The line of work that is most closely related to our research is the one concerned with understanding the relation between people's language and their industry  Previous research from the fields of psychology and economics have explored the potential for predicting one's occupation from their ability to use math and verbal symbols BIBREF23 and the relationship between job-types and demographics BIBREF24   More recently, Huang et al  Huang15 used machine learning to classify Sina Weibo users to twelve different platform-defined occupational classes highlighting the effect of homophily in user interactions  This work examined only users that have been verified by the Sina Weibo platform, introducing a potential bias in the resulting dataset  Finally, Preotiuc-Pietro et al  Preoctiuc15 predicted the occupational class of Twitter users using the Standard Occupational Classification (SOC) system, which groups the different jobs based on skill requirements  In that work, the data collection process was limited to only users that specifically mentioned their occupation in their self-description in a way that could be directly mapped to a SOC occupational class  The mapping between a substring of their self-description and a SOC occupational class was done manually  Because of the manual annotation step, their method was not scalable; moreover, because they identified the occupation class inside a user self-description, only a very small fraction of the Twitter users could be included (in their case, 5,191 users) 
Both of these recent studies are based on micro-blogging platforms, which inherently restrict the number of characters that a post can have, and consequently the way that users can express themselves 
Moreover, both studies used off-the-shelf occupational taxonomies (rather than self-declared occupation categories), resulting in classes that are either too generic (e g , media, welfare and electronic are three of the twelve Sina Weibo categories), or too intermixed (e g , an assistant accountant is in a different class from an accountant in SOC)  To address these limitations, we investigate the industry prediction task in a large blog corpus consisting of over 20K American users, 40K web-blogs, and 560K blog posts 
Dataset
We compile our industry-annotated dataset by identifying blogger profiles located in the U S  on the profile finder on http://www blogger com, and scraping only those users that had the industry profile element completed 
For each of these bloggers, we retrieve all their blogs, and for each of these blogs we download the 21 most recent blog postings  We then clean these blog posts of HTML tags and tokenize them, and drop those bloggers whose cumulative textual content in their posts is less than 600 characters  Following these guidelines, we identified all the U S  bloggers with completed industry information 
Traditionally, standardized industry taxonomies organize economic activities into groups based on similar production processes, products or services, delivery systems or behavior in financial markets  Following such assumptions and regardless of their many similarities, a tomato farmer would be categorized into a distinct industry from a tobacco farmer  As demonstrated in Preotiuc-Pietro et al  Preoctiuc15 such groupings can cause unwarranted misclassifications 
The Blogger platform provides a total of 39 different industry options  Even though a completed industry value is an implicit text annotation, we acknowledge the same problem noted in previous studies: some categories are too broad, while others are very similar  To remedy this and following Guibert et al  Guibert71, who argued that the denominations used in a classification must reflect the purpose of the study, we group the different Blogger industries based on similar educational background and similar technical terminology  To do that, we exclude very general categories and merge conceptually similar ones  Examples of broad categories are the Education and the Student options: a teacher could be teaching in any concentration, while a student could be enrolled in any discipline  Examples of conceptually similar categories are the Investment Banking and the Banking options 
The final set of categories is shown in Table TABREF1 , along with the number of users in each category  The resulting dataset consists of 22,880 users, 41,094 blogs, and 561,003 posts  Table TABREF2 presents additional statistics of our dataset 
Text-based Industry Modeling
After collecting our dataset, we split it into three sets: a train set, a development set, and a test set  The sizes of these sets are 17,880, 2,500, and 2,500 users, respectively, with users randomly assigned to these sets  In all the experiments that follow, we evaluate our classifiers by training them on the train set, configure the parameters and measure performance on the development set, and finally report the prediction accuracy and results on the test set  Note that all the experiments are performed at user level, i e , all the data for one user is compiled into one instance in our data sets 
To measure the performance of our classifiers, we use the prediction accuracy  However, as shown in Table TABREF1 , the available data is skewed across categories, which could lead to somewhat distorted accuracy numbers depending on how well a model learns to predict the most populous classes  Moreover, accuracy alone does not provide a great deal of insight into the individual performance per industry, which is one of the main objectives in this study  Therefore, in our results below, we report: (1) micro-accuracy ( INLINEFORM0 ), calculated as the percentage of correctly classified instances out of all the instances in the development (test) data; and (2) macro-accuracy ( INLINEFORM1 ), calculated as the average of the per-category accuracies, where the per-category accuracy is the percentage of correctly classified instances out of the instances belonging to one category in the development (test) data 
Leveraging Blog Content
In this section, we seek the effectiveness of using solely textual features obtained from the users' postings to predict their industry 
The industry prediction baseline Majority is set by discovering the most frequently featured class in our training set and picking that class in all predictions in the respective development or testing set 
After excluding all the words that are not used by at least three separate users in our training set, we build our AllWords model by counting the frequencies of all the remaining words and training a multinomial Naive Bayes classifier  As seen in Figure FIGREF3 , we can far exceed the Majority baseline performance by incorporating basic language signals into machine learning algorithms (173% INLINEFORM0 improvement) 
We additionally explore the potential of improving our text classification task by applying a number of feature ranking methods and selecting varying proportions of top ranked features in an attempt to exclude noisy features  We start by ranking the different features, w, according to their Information Gain Ratio score (IGR) with respect to every industry, i, and training our classifier using different proportions of the top features  INLINEFORM0 INLINEFORM1
Even though we find that using the top 95% of all the features already exceeds the performance of the All Words model on the development data, we further experiment with ranking our features with a more aggressive formula that heavily promotes the features that are tightly associated with any industry category  Therefore, for every word in our training set, we define our newly introduced ranking method, the Aggressive Feature Ranking (AFR), as: INLINEFORM0
In Figure FIGREF3 we illustrate the performance of all four methods in our industry prediction task on the development data  Note that for each method, we provide both the accuracy ( INLINEFORM0 ) and the average per-class accuracy ( INLINEFORM1 )  The Majority and All Words methods apply to all the features; therefore, they are represented as a straight line in the figure  The IGR and AFR methods are applied to varying subsets of the features using a 5% step 
Our experiments demonstrate that the word choice that the users make in their posts correlates with their industry  The first observation in Figure FIGREF3 is that the INLINEFORM0 is proportional to INLINEFORM1 ; as INLINEFORM2 increases, so does INLINEFORM3   Secondly, the best result on the development set is achieved by using the top 90% of the features using the AFR method  Lastly, the improvements of the IGR and AFR feature selections are not substantially better in comparison to All Words (at most 5% improvement between All Words and AFR), which suggest that only a few noisy features exist and most of the words play some role in shaping the “language" of an industry 
As a final evaluation, we apply on the test data the classifier found to work best on the development data (AFR feature selection, top 90% features), for an INLINEFORM0 of 0 534 and INLINEFORM1 of 0 477 
Leveraging User Metadata
Together with the industry information and the most recent postings of each blogger, we also download a number of accompanying profile elements  Using these additional elements, we explore the potential of incorporating users' metadata in our classifiers 
Table TABREF7 shows the different user metadata we consider together with their coverage percentage (not all users provide a value for all of the profile elements)  With the exception of the gender field, the remaining metadata elements shown in Table TABREF7 are completed by the users as a freely editable text field  This introduces a considerable amount of noise in the set of possible metadata values  Examples of noise in the occupation field include values such as “Retired”, “I work ”, or “momma” which are not necessarily informative for our industry prediction task 
To examine whether the metadata fields can help in the prediction of a user's industry, we build classifiers using the different metadata elements  For each metadata element that has a textual value, we use all the words in the training set for that field as features  The only two exceptions are the state field, which is encoded as one feature that can take one out of 50 different values representing the 50 U S  states; and the gender field, which is encoded as a feature with a distinct value for each user gender option: undefined, male, or female 
As shown in Table TABREF9 , we build four different classifiers using the multinomial NB algorithm: Occu (which uses the words found in the occupation profile element), Intro (introduction), Inter (interests), and Gloc (combined gender, city, state) 
In general, all the metadata classifiers perform better than our majority baseline ( INLINEFORM0 of 18 88%)  For the Gloc classifier, this result is in alignment with previous studies BIBREF24   However, the only metadata classifier that outperforms the content classifier is the Occu classifier, which despite missing and noisy occupation values exceeds the content classifier's performance by an absolute 3 2% 
To investigate the promise of combining the five different classifiers we have built so far, we calculate their inter-prediction agreement using Fleiss's Kappa BIBREF25 , as well as the lower prediction bounds using the double fault measure BIBREF26   The Kappa values, presented in the lower left side of Table TABREF10 , express the classification agreement for categorical items, in this case the users' industry  Lower values, especially values below 30%, mean smaller agreement  Since all five classifiers have better-than-baseline accuracy, this low agreement suggests that their predictions could potentially be combined to achieve a better accumulated result 
Moreover, the double fault measure values, which are presented in the top-right hand side of Table TABREF10 , express the proportion of test cases for which both of the two respective classifiers make false predictions, essentially providing the lowest error bound for the pairwise ensemble classifier performance  The lower those numbers are, the greater the accuracy potential of any meta-classification scheme that combines those classifiers  Once again, the low double fault measure values suggest potential gain from a combination of the base classifiers into an ensemble of models 
After establishing the promise of creating an ensemble of classifiers, we implement two meta-classification approaches  First, we combine our classifiers using features concatenation (or early fusion)  Starting with our content-based classifier (Text), we successively add the features derived from each metadata element  The results, both micro- and macro-accuracy, are presented in Table TABREF12   Even though all these four feature concatenation ensembles outperform the content-based classifier in the development set, they fail to outperform the Occu classifier 
Second, we explore the potential of using stacked generalization (or late fusion) BIBREF27   The base classifiers, referred to as L0 classifiers, are trained on different folds of the training set and used to predict the class of the remaining instances  Those predictions are then used together with the true label of the training instances to train a second classifier, referred to as the L1 classifier: this L1 is used to produce the final prediction on both the development data and the test data  Traditionally, stacking uses different machine learning algorithms on the same training data  However in our case, we use the same algorithm (multinomial NB) on heterogeneous data (i e , different types of data such as content, occupation, introduction, interests, gender, city and state) in order to exploit all available sources of information 
The ensemble learning results on the development set are shown in Table TABREF12   We notice a constant improvement for both metrics when adding more classifiers to our ensemble except for the Gloc classifier, which slightly reduces the performance  The best result is achieved using an ensemble of the Text, Occu, Intro, and Inter L0 classifiers; the respective performance on the test set is an INLINEFORM0 of 0 643 and an INLINEFORM1 of 0 564  Finally, we present in Figure FIGREF11 the prediction accuracy for the final classifier for each of the different industries in our test dataset  Evidently, some industries are easier to predict than others  For example, while the Real Estate and Religion industries achieve accuracy figures above 80%, other industries, such as the Banking industry, are predicted correctly in less than 17% of the time  Anecdotal evidence drawn from the examination of the confusion matrix does not encourage any strong association of the Banking class with any other  The misclassifications are roughly uniform across all other classes, suggesting that the users in the Banking industry use language in a non-distinguishing way 
Qualitative Analysis
In this section, we provide a qualitative analysis of the language of the different industries 
Top-Ranked Words
To conduct a qualitative exploration of which words indicate the industry of a user, Table TABREF14 shows the three top-ranking content words for the different industries using the AFR method 
Not surprisingly, the top ranked words align well with what we would intuitively expect for each industry  Even though most of these words are potentially used by many users regardless of their industry in our dataset, they are still distinguished by the AFR method because of the different frequencies of these words in the text of each industry 
Industry-specific Word Similarities
Next, we examine how the meaning of a word is shaped by the context in which it is uttered  In particular, we qualitatively investigate how the speakers' industry affects meaning by learning vector-space representations of words that take into account such contextual information  To achieve this, we apply the contextualized word embeddings proposed by Bamman et al  Bamman14, which are based on an extension of the “skip-gram" language model BIBREF28  
In addition to learning a global representation for each word, these contextualized embeddings compute one deviation from the common word embedding representation for each contextual variable, in this case, an industry option  These deviations capture the terms' meaning variations (shifts in the INLINEFORM0 -dimensional space of the representations, where INLINEFORM1 in our experiments) in the text of the different industries, however all the embeddings are in the same vector space to allow for comparisons to one another 
Using the word representations learned for each industry, we present in Table TABREF16 the terms in the Technology and the Tourism industries that have the highest cosine similarity with a job-related word, customers  Similarly, Table TABREF17 shows the words in the Environment and the Tourism industries that are closest in meaning to a general interest word, food  More examples are given in the Appendix SECREF8  
The terms that rank highest in each industry are noticeably different  For example, as seen in Table TABREF17 , while food in the Environment industry is similar to nutritionally and locally, in the Tourism industry the same word relates more to terms such as delicious and pastries  These results not only emphasize the existing differences in how people in different industries perceive certain terms, but they also demonstrate that those differences can effectively be captured in the resulting word embeddings 
Emotional Orientation per Industry and Gender
As a final analysis, we explore how words that are emotionally charged relate to different industries  To quantify the emotional orientation of a text, we use the Positive Emotion and Negative Emotion categories in the Linguistic Inquiry and Word Count (LIWC) dictionary BIBREF29   The LIWC dictionary contains lists of words that have been shown to correlate with the psychological states of people that use them; for example, the Positive Emotion category contains words such as “happy,” “pretty,” and “good ”
For the text of all the users in each industry we measure the frequencies of Positive Emotion and Negative Emotion words normalized by the text's length  Table TABREF20 presents the industries' ranking for both categories of words based on their relative frequencies in the text of each industry 
We further perform a breakdown per-gender, where we once again calculate the proportion of emotionally charged words in each industry, but separately for each gender  We find that the industry rankings of the relative frequencies INLINEFORM0 of emotionally charged words for the two genders are statistically significantly correlated, which suggests that regardless of their gender, users use positive (or negative) words with a relative frequency that correlates with their industry  (In other words, even if e g , Fashion has a larger number of women users, both men and women working in Fashion will tend to use more positive words than the corresponding gender in another industry with a larger number of men users such as Automotive )
Finally, motivated by previous findings of correlations between job satisfaction and gender dominance in the workplace BIBREF30 , we explore the relationship between the usage of Positive Emotion and Negative Emotion words and the gender dominance in an industry  Although we find that there are substantial gender imbalances in each industry (Appendix SECREF9 ), we did not find any statistically significant correlation between the gender dominance ratio in the different industries and the usage of positive (or negative) emotional words in either gender in our dataset 
Conclusion
In this paper, we examined the task of predicting a social media user's industry  We introduced an annotated dataset of over 20,000 blog users and applied a content-based classifier in conjunction with two feature selection methods for an overall accuracy of up to 0 534, which represents a large improvement over the majority class baseline of 0 188 
We also demonstrated how the user metadata can be incorporated in our classifiers  Although concatenation of features drawn both from blog content and profile elements did not yield any clear improvements over the best individual classifiers, we found that stacking improves the prediction accuracy to an overall accuracy of 0 643, as measured on our test dataset  A more in-depth analysis showed that not all industries are equally easy to predict: while industries such as Real Estate and Religion are clearly distinguishable with accuracy figures over 0 80, others such as Banking are much harder to predict 
Finally, we presented a qualitative analysis to provide some insights into the language of different industries, which highlighted differences in the top-ranked words in each industry, word semantic similarities, and the relative frequency of emotionally charged words 
Acknowledgments
This material is based in part upon work supported by the National Science Foundation (#1344257) and by the John Templeton Foundation (#48503)  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the John Templeton Foundation 
Additional Examples of Word Similarities

ground_truth: MLP

Eusboost, MWMOTE

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How many users do they look at?
Context 4:  The sentences are separated by empty newline  A sample sentence from the dataset is presented in table FIGREF13 
Dataset Statistics ::: ILPRL dataset
After much time, we received the dataset from Bal Krishna Bal, ILPRL, KU  This dataset follows standard CoNLL-2003 IOB formatBIBREF25 with POS tags  This dataset is prepared by ILPRL Lab, KU and KEIV Technologies  Few corrections like correcting the NER tags had to be made on the dataset  The statistics of both the dataset is presented in table TABREF23 
Table TABREF24 presents the total entities (PER, LOC, ORG and MISC) from both of the dataset used in our experiments  The dataset is divided into three parts with 64%, 16% and 20% of the total dataset into training set, development set and test set respectively 
Experiments
In this section, we present the details about training our neural network  The neural network architecture are implemented using PyTorch framework BIBREF26  The training is performed on a single Nvidia Tesla P100 SXM2  We first run our experiment on BiLSTM, BiLSTM-CNN, BiLSTM-CRF BiLSTM-CNN-CRF using the hyper-parameters mentioned in Table TABREF30  The training and evaluation was done on sentence-level  The RNN variants are initialized randomly from $(-\sqrt{k},\sqrt{k})$ where $k=\frac{1}{hidden\_size}$ 
First we loaded our dataset and built vocabulary using torchtext library  This eased our process of data loading using its SequenceTaggingDataset class  We trained our model with shuffled training set using Adam optimizer with hyper-parameters mentioned in table TABREF30  All our models were trained on single layer of LSTM network  We found out Adam was giving better performance and faster convergence compared to Stochastic Gradient Descent (SGD)  We chose those hyper-parameters after many ablation studies  The dropout of 0 5 is applied after LSTM layer 
For CNN, we used 30 different filters of sizes 3, 4 and 5  The embeddings of each character or grapheme involved in a given word, were passed through the pipeline of Convolution, Rectified Linear Unit and Max-Pooling  The resulting vectors were concatenated and applied dropout of 0 5 before passing into linear layer to obtain the embedding size of 30 for the given word  This resulting embedding is concatenated with word embeddings, which is again concatenated with one-hot POS vector 
Experiments ::: Tagging Scheme
Currently, for our experiments we trained our model on IO (Inside, Outside) format for both the dataset, hence the dataset does not contain any B-type annotation unlike in BIO (Beginning, Inside, Outside) scheme 
Experiments ::: Early Stopping
We used simple early stopping technique where if the validation loss does not decrease after 10 epochs, the training was stopped, else the training will run upto 100 epochs  In our experience, training usually stops around 30-50 epochs 
Experiments ::: Hyper-parameters Tuning
We ran our experiment looking for the best hyper-parameters by changing learning rate from (0,1, 0 01, 0 001, 0 0001), weight decay from [$10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$, $10^{-7}$], batch size from [1, 2, 4, 8, 16, 32, 64, 128], hidden size from [8, 16, 32, 64, 128, 256, 512 1024]  Table TABREF30 shows all other hyper-parameter used in our experiment for both of the dataset 
Experiments ::: Effect of Dropout
Figure FIGREF31 shows how we end up choosing 0 5 as dropout rate  When the dropout layer was not used, the F1 score are at the lowest  As, we slowly increase the dropout rate, the F1 score also gradually increases, however after dropout rate = 0 5, the F1 score starts falling down  Therefore, we have chosen 0 5 as dropout rate for all other experiments performed 
Evaluation
In this section, we present the details regarding evaluation and comparison of our models with other baselines 
Table TABREF25 shows the study of various embeddings and comparison among each other in OurNepali dataset  Here, raw dataset represents such dataset where post-positions are not lemmatized  We can observe that pre-trained embeddings significantly improves the score compared to randomly initialized embedding  We can deduce that Skip Gram models perform better compared CBOW models for word2vec and fasttext  Here, fastText_Pretrained represents the embedding readily available in fastText website, while other embeddings are trained on the Nepali National Corpus as mentioned in sub-section SECREF11  From this table TABREF25, we can clearly observe that model using fastText_Skip Gram embeddings outperforms all other models 
Table TABREF35 shows the model architecture comparison between all the models experimented  The features used for Stanford CRF classifier are words, letter n-grams of upto length 6, previous word and next word  This model is trained till the current function value is less than $1\mathrm {e}{-2}$  The hyper-parameters of neural network experiments are set as shown in table TABREF30  Since, word embedding of character-level and grapheme-level is random, their scores are near 
All models are evaluated using CoNLL-2003 evaluation scriptBIBREF25 to calculate entity-wise precision, recall and f1 score 
Discussion
In this paper we present that we can exploit the power of neural network to train the model to perform downstream NLP tasks like Named Entity Recognition even in Nepali language  We showed that the word vectors learned through fasttext skip gram model performs better than other word embedding because of its capability to represent sub-word and this is particularly important to capture morphological structure of words and sentences in highly inflectional like Nepali  This concept can come handy in other Devanagari languages as well because the written scripts have similar syntactical structure 
We also found out that stemming post-positions can help a lot in improving model performance because of inflectional characteristics of Nepali language  So when we separate out its inflections or morphemes, we can minimize the variations of same word which gives its root word a stronger word vector representations compared to its inflected versions 
We can clearly imply from tables TABREF23, TABREF24, and TABREF35 that we need more data to get better results because OurNepali dataset volume is almost ten times bigger compared to ILPRL dataset in terms of entities 
Conclusion and Future work
In this paper, we proposed a novel NER for Nepali language and achieved relative improvement of upto 10% and studies different factors effecting the performance of the NER for Nepali language 
We also present a neural architecture BiLSTM+CNN(grapheme-level) which turns out to be performing on par with BiLSTM+CNN(character-level) under the same configuration  We believe this will not only help Nepali language but also other languages falling under the umbrellas of Devanagari languages  Our model BiLSTM+CNN(grapheme-level) and BiLSTM+CNN(G)+POS outperforms all other model experimented in OurNepali and ILPRL dataset respectively 
Since this is the first named entity recognition research in Nepal language using neural network, there are many rooms for improvement  We believe initializing the grapheme-level embedding with fasttext embeddings might help boosting the performance, rather than randomly initializing it  In future, we plan to apply other latest techniques like BERT, ELMo and FLAIR to study its effect on low-resource language like Nepali  We also plan to improve the model using cross-lingual or multi-lingual parameter sharing techniques by jointly training with other Devanagari languages like Hindi and Bengali 
Finally, we would like to contribute our dataset to Nepali NLP community to move forward the research going on in language understanding domain  We believe there should be special committee to create and maintain such dataset for Nepali NLP and organize various competitions which would elevate the NLP research in Nepal 
Some of the future works are listed below:
Proper initialization of grapheme level embedding from fasttext embeddings 
Apply robust POS-tagger for Nepali dataset
Lemmatize the OurNepali dataset with robust and efficient lemmatizer
Improve Nepali language score with cross-lingual learning techniques
Create more dataset using Wikipedia/Wikidata framework
Acknowledgments
The authors of this paper would like to express sincere thanks to Bal Krishna Bal, Kathmandu University Professor for providing us the POS-tagged Nepali NER data 

ground_truth: 22,880 users

20,000