prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What machine learning and deep learning methods are used for RQE?
Context 1: Introduction
With the availability of rich data on users' locations, profiles and search history, personalization has become the leading trend in large-scale information retrieval  However, efficiency through personalization is not yet the most suitable model when tackling domain-specific searches  This is due to several factors, such as the lexical and semantic challenges of domain-specific data that often include advanced argumentation and complex contextual information, the higher sparseness of relevant information sources, and the more pronounced lack of similarities between users' searches 
A recent study on expert search strategies among healthcare information professionals BIBREF0 showed that, for a given search task, they spend an average of 60 minutes per collection or database, 3 minutes to examine the relevance of each document, and 4 hours of total search time  When written in steps, their search strategy spans over 15 lines and can reach up to 105 lines 
With the abundance of information sources in the medical domain, consumers are more and more faced with a similar challenge, one that needs dedicated solutions that can adapt to the heterogeneity and specifics of health-related information 
Dedicated Question Answering (QA) systems are one of the viable solutions to this problem as they are designed to understand natural language questions without relying on external information on the users 
In the context of QA, the goal of Recognizing Question Entailment (RQE) is to retrieve answers to a premise question ( INLINEFORM0 ) by retrieving inferred or entailed questions, called hypothesis questions ( INLINEFORM1 ) that already have associated answers  Therefore, we define the entailment relation between two questions as: a question INLINEFORM2 entails a question INLINEFORM3 if every answer to INLINEFORM4 is also a correct answer to INLINEFORM5 BIBREF1  
RQE is particularly relevant due to the increasing numbers of similar questions posted online BIBREF2 and its ability to solve differently the challenging issues of question understanding and answer extraction  In addition to being used to find relevant answers, these resources can also be used in training models able to recognize inference relations and similarity between questions 
Question similarity has recently attracted international challenges BIBREF3 , BIBREF4 and several research efforts proposing a wide range of approaches, including Logistic Regression, Recurrent Neural Networks (RNNs), Long Short Term Memory cells (LSTMs), and Convolutional Neural Networks (CNNs) BIBREF5 , BIBREF6 , BIBREF1 , BIBREF7  
In this paper, we study question entailment in the medical domain and the effectiveness of the end-to-end RQE-based QA approach by evaluating the relevance of the retrieved answers  Although entailment was attempted in QA before BIBREF8 , BIBREF9 , BIBREF10 , as far as we know, we are the first to introduce and evaluate a full medical question answering approach based on question entailment for free-text questions  Our contributions are:
The next section is dedicated to related work on question answering, question similarity and entailment  In Section SECREF3 , we present two machine learning (ML) and deep learning (DL) methods for RQE and compare their performance using open-domain and clinical datasets  Section SECREF4 describes the new collection of medical question-answer pairs  In Section SECREF5 , we describe our RQE-based approach for QA  Section SECREF6 presents our evaluation of the retrieved answers and the results obtained on TREC 2017 LiveQA medical questions 
Background
In this section we define the RQE task and describe related work at the intersection of question answering, question similarity and textual inference 
Task Definition
The definition of Recognizing Question Entailment (RQE) can have a significant impact on QA results  In related work, the meaning associated with Natural Language Inference (NLI) varies among different tasks and events  For instance, Recognizing Textual Entailment (RTE) was addressed by the PASCAL challenge BIBREF12 , where the entailment relation has been assessed manually by human judges who selected relevant sentences "entailing" a set of hypotheses from a list of documents returned by different Information Retrieval (IR) methods  In another definition, the Stanford Natural Language Inference corpus SNLI BIBREF13 , used three classification labels for the relations between two sentences: entailment, neutral and contradiction  For the entailment label, the annotators who built the corpus were presented with an image and asked to write a caption “that is a definitely true description of the photo”  For the neutral label, they were asked to provide a caption “that might be a true description of the label”  They were asked for a caption that “is definitely a false description of the photo” for the contradiction label 
More recently, the multiNLI corpus BIBREF14 was shared in the scope of the RepEval 2017 shared task BIBREF15   To build the corpus, annotators were presented with a premise text and asked to write three sentences  One novel sentence, which is “necessarily true or appropriate in the same situations as the premise,” for the entailment label, a sentence, which is “necessarily false or inappropriate whenever the premise is true,” for the contradiction label, and a last sentence, “where neither condition applies,” for the neutral label 
Whereas these NLI definitions might be suitable for the broad topic of text understanding, their relation to practical information retrieval or question answering systems is not straightforward 
In contrast, RQE has to be tailored to the question answering task  For instance, if the premise question is "looking for cold medications for a 30 yo woman", a RQE approach should be able to consider the more general (less restricted) question "looking for cold medications" as relevant, since its answers are relevant for the initial question, whereas "looking for medications for a 30 yo woman" is a useless contextualization  The entailment relation we are seeking in the QA context should include relevant and meaningful relaxations of contextual and semantic constraints (cf  Section SECREF13 ) 
Related Work on Question Answering
Classical QA systems face two main challenges related to question analysis and answer extraction  Several QA approaches were proposed in the literature for the open domain BIBREF16 , BIBREF17 and the medical domain BIBREF18 , BIBREF19 , BIBREF20   A variety of methods were developed for question analysis, focus (topic) recognition and question type identification BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24   Similarly, many different approaches tackled document or passage retrieval and answer selection and (re)ranking BIBREF25 , BIBREF26 , BIBREF27  
An alternative approach consists in finding similar questions or FAQs that are already answered BIBREF28 , BIBREF29   One of the earliest question answering systems based on finding similar questions and re-using the existing answers was FAQ FINDER BIBREF30   Another system that complements the existing Q&A services of NetWellness is SimQ BIBREF2 , which allows retrieval of similar web-based consumer health questions  SimQ uses syntactic and semantic features to compute similarity between questions, and UMLS BIBREF31 as a standardized semantic knowledge source  The system achieves 72 2% precision, 78 0% recall and 75 0% F-score on NetWellness questions  However, the method was evaluated only on one question similarity dataset, and the retrieved answers were not evaluated 
The aim of the medical task at TREC 2017 LiveQA was to develop techniques for answering complex questions such as consumer health questions, as well as to identify relevant answer sources that can comply with the sensitivity of medical information retrieval 
The CMU-OAQA system BIBREF32 achieved the best performance of 0 637 average score on the medical task by using an attentional encoder-decoder model for paraphrase identification and answer ranking  The Quora question-similarity dataset was used for training  The PRNA system BIBREF33 achieved the second best performance in the medical task with 0 49 average score using Wikipedia as the first answer source and Yahoo and Google searches as secondary answer sources  Each medical question was decomposed into several subquestions  To extract the answer from the selected text passage, a bi-directional attention model trained on the SQUAD dataset was used 
Deep neural network models have been pushing the limits of performance achieved in QA related tasks using large training datasets  The results obtained by CMU-OAQA and PRNA showed that large open-domain datasets were beneficial for the medical domain  However, the best system (CMU-OAQA) relying on the same training data obtained a score of 1 139 on the LiveQA open-domain task 
While this gap in performance can be explained in part by the discrepancies between the medical test questions and the open-domain questions, it also highlights the need for larger medical datasets to support deep learning approaches in dealing with the linguistic complexity of consumer health questions and the challenge of finding correct and complete answers 
Another technique was used by ECNU-ICA team BIBREF34 based on learning question similarity via two long short-term memory (LSTM) networks applied to obtain the semantic representations of the questions  To construct a collection of similar question pairs, they searched community question answering sites such as Yahoo! and Answers com  In contrast, the ECNU-ICA system achieved the best performance of 1 895 in the open-domain task but an average score of only 0 402 in the medical task  As the ECNU-ICA approach also relied on a neural network for question matching, this result shows that training attention-based decoder-encoder networks on the Quora dataset generalized better to the medical domain than training LSTMs on similar questions from Yahoo! and Answers com 
The CMU-LiveMedQA team BIBREF20 designed a specific system for the medical task  Using only the provided training datasets and the assumption that each question contains only one focus, the CMU-LiveMedQA system obtained an average score of 0 353  They used a convolutional neural network (CNN) model to classify a question into a restricted set of 10 question types and crawled "relevant" online web pages to find the answers  However, the results were lower than those achieved by the systems relying on finding similar answered questions  These results support the relevance of similar question matching for the end-to-end QA task as a new way of approaching QA instead of the classical QA approaches based on Question Analysis and Answer Retrieval 
Related Work on Question Similarity and Entailment
Several efforts focused on recognizing similar questions  Jeon et al  BIBREF35 showed that a retrieval model based on translation probabilities learned from a question and answer archive can recognize semantically similar questions  Duan et al  BIBREF36 proposed a dedicated language modeling approach for question search, using question topic (user's interest) and question focus (certain aspect of the topic) 
Lately, these efforts were supported by a task on Question-Question similarity introduced in the community QA challenge at SemEval (task 3B) BIBREF3   Given a new question, the task focused on reranking all similar questions retrieved by a search engine, assuming that the answers to the similar questions will be correct answers for the new question  Different machine learning and deep learning approaches were tested in the scope of SemEval 2016 BIBREF3 and 2017 BIBREF4 task 3B  The best performing system in 2017 achieved a MAP of 47 22% using supervised Logistic Regression that combined different unsupervised similarity measures such as Cosine and Soft-Cosine BIBREF37   The second best system achieved 46 93% MAP with a learning-to-rank method using Logistic Regression and a rich set of features including lexical and semantic features as well as embeddings generated by different neural networks (siamese, Bi-LSTM, GRU and CNNs) BIBREF38   In the scope of this challenge, a dataset was collected from Qatar Living forum for training  We refer to this dataset as SemEval-cQA 
In another effort, an answer-based definition of RQE was proposed and tested BIBREF1   The authors introduced a dataset of clinical questions and used a feature-based method that provided an Accuracy of 75% on consumer health questions  We will call this dataset Clinical-QE  Dos Santos et al  BIBREF5 proposed a new approach to retrieve semantically equivalent questions combining a bag-of-words representation with a distributed vector representation created by a CNN and user data collected from two Stack Exchange communities  Lei et al  BIBREF7 proposed a recurrent and convolutional model (gated convolution) to map questions to their semantic representations  The models were pre-trained within an encoder-decoder framework 
RQE Approaches and Experiments
The choice of two methods for our empirical study is motivated by the best performance achieved by Logistic Regression in question-question similarity at SemEval 2017 (best system BIBREF37 and second best system BIBREF38 ), and the high performance achieved by neural networks on larger datasets such as SNLI BIBREF13 , BIBREF39 , BIBREF40 , BIBREF41   We first define the RQE task, then present the two approaches, and evaluate their performance on five different datasets 
Definition
In the context of QA, the goal of RQE is to retrieve answers to a new question by retrieving entailed questions with associated answers  We therefore define question entailment as:
a question INLINEFORM0 entails a question INLINEFORM1 if every answer to INLINEFORM2 is also a complete or partial answer to INLINEFORM3  
We present below two examples of consumer health questions INLINEFORM0 and entailed questions INLINEFORM1 :
Example 1 (each answer to the entailed question B1 is a complete answer to A1):
A1: What is the latest news on tennitis, or ringing in the ear, I am 75 years old and have had ringing in the ear since my mid 5os  Thank you 
B1: What is the latest research on Tinnitus?
Example 2 (each answer to the entailed question B2 is a partial answer to A2):
A2: My mother has been diagnosed with Alzheimer's, my father is not of the greatest health either and is the main caregiver for my mother  My question is where do we start with attempting to help our parents w/ the care giving and what sort of financial options are there out there for people on fixed incomes 
B2: What resources are available for Alzheimer's caregivers?
The inclusion of partial answers in the definition of question entailment also allows efficient relaxation of the contextual constraints of the original question INLINEFORM0 to retrieve relevant answers from entailed, but less restricted, questions 
Deep Learning Model
To recognize entailment between two questions INLINEFORM0 (premise) and INLINEFORM1 (hypothesis), we adapted the neural network proposed by Bowman et al  BIBREF13   Our DL model, presented in Figure FIGREF20 , consists of three 600d ReLU layers, with a bottom layer taking the concatenated sentence representations as input and a top layer feeding a softmax classifier  The sentence embedding model sums the Recurrent neural network (RNN) embeddings of its words  The word embeddings are first initialized with pretrained GloVe vectors  This adaptation provided the best performance in previous experiments with RQE data 
GloVe is an unsupervised learning algorithm to generate vector representations for words BIBREF42   Training is performed on aggregated word co-occurrence statistics from a large corpus, and the resulting representations show interesting linear substructures of the word vector space  We use the pretrained common crawl version with 840B tokens and 300d vectors, which are not updated during training 
Logistic Regression Classifier
In this feature-based approach, we use Logistic Regression to classify question pairs into entailment or no-entailment  Logistic Regression achieved good results on this specific task and outperformed other statistical learning algorithms such as SVM and Naive Bayes  In a preprocessing step, we remove stop words and perform word stemming using the Porter algorithm BIBREF43 for all ( INLINEFORM0 , INLINEFORM1 ) pairs 
We use a list of nine features, selected after several experiments on RTE datasets BIBREF12   We compute five similarity measures between the pre-processed questions and use their values as features  We use Word Overlap, the Dice coefficient based on the number of common bigrams, Cosine, Levenshtein, and the Jaccard similarities  Our feature list also includes the maximum and average values obtained with these measures and the question length ratio (length( INLINEFORM0 )/length( INLINEFORM1 ))  We compute a morphosyntactic feature indicating the number of common nouns and verbs between INLINEFORM2 and INLINEFORM3   TreeTagger BIBREF44 was used for POS tagging 
For RQE, we add an additional feature specific to the question type  We use a dictionary lookup to map triggers to the question type (e g  Treatment, Prognosis, Inheritance)  Triggers are identified for each question type based on a manual annotation of a set of medical questions (cf  Section SECREF36 )  This feature has three possible values: 2 (Perfect match between INLINEFORM0 type(s) and INLINEFORM1 type(s)), 1 (Overlap between INLINEFORM2 type(s) and INLINEFORM3 type(s)) and 0 (No common types) 
Datasets Used for the RQE Study
We evaluate the RQE methods (i e  deep learning model and logistic regression classifier) using two datasets of sentence pairs (SNLI and multiNLI), and three datasets of question pairs (Quora, Clinical-QE, and SemEval-cQA) 
The Stanford Natural Language Inference corpus (SNLI) BIBREF13 contains 569,037 sentence pairs written by humans based on image captioning  The training set of the MultiNLI corpus BIBREF14 consists of 393,000 pairs of sentences from five genres of written and spoken English (e g  Travel, Government)  Two other "matched" and "mismatched" sets are also available for development (20,000 pairs)  Both SNLI and multiNLI consider three types of relationships between sentences: entailment, neutral and contradiction  We converted the contradiction and neutral labels to the same non-entailment class 
The QUORA dataset of similar questions was recently published with 404,279 question pairs  We randomly selected three distinct subsets (80%/10%/10%) for training (323,423 pairs), development (40,428 pairs) and test (40,428 pairs) 
The clinical-QE dataset BIBREF1 contains 8,588 question pairs and was constructed using 4,655 clinical questions asked by family doctors BIBREF45   We randomly selected three distinct subsets (80%/10%/10%) for training (6,870 pairs), development (859 pairs) and test (859 pairs) 
The question similarity dataset of SemEval 2016 Task 3B (SemEval-cQA) BIBREF3 contains 3,869 question pairs and aims to re-rank a list of related questions according to their similarity to the original question  The same dataset was used for SemEval 2017 Task 3 BIBREF4  
To construct our test dataset, we used a publicly shared set of Consumer Health Questions (CHQs) received by the U S  National Library of Medicine (NLM), and annotated with named entities, question types, and focus BIBREF46 , BIBREF47   The CHQ dataset consists of 1,721 consumer information requests manually annotated with subquestions, each identified by a question type and a focus 
First, we selected automatically harvested FAQs, from U S  National Institutes of Health (NIH) websites, that share both the same focus and the same question type with the CHQs  As FAQs are most often very short, we first assume that the CHQ entails the FAQ  Two sets of pairs were constructed: (i) positive pairs of CHQs and FAQs sharing at least one common question type and the question focus, and (ii) negative pairs corresponding to a focus mismatch or type mismatch  For each category of negative examples, we randomly selected the same number of pairs for a balanced dataset  Then, we manually validated the constructed pairs and corrected the positive and negative labels when needed  The final RQE dataset contains 850 CHQ-FAQ pairs with 405 positive and 445 negative pairs  Table TABREF26 presents examples from the five training datasets (SNLI, MultiNLI, SemEval-cQA, Clinical-QE and Quora) and the new test dataset of medical CHQ-FAQ pairs 
Results of RQE Approaches
In the first experiment, we evaluated the DL and ML methods on SNLI, multi-NLI, Quora, and Clinical-QE  For the datasets that did not have a development and test sets, we randomly selected two sets, each amounting to 10% of the data, for test and development, and used the remaining 80% for training  For MultiNLI, we used the dev1-matched set for validation and the dev2-mismatched set for testing 
Table TABREF28 presents the results of the first experiment  The DL model with GloVe word embeddings achieved better results on three datasets, with 82 80% Accuracy on SNLI, 78 52% Accuracy on MultiNLI, and 83 62% Accuracy on Quora  Logistic Regression achieved the best Accuracy of 98 60% on Clinical-RQE  We also performed a 10-fold cross-validation on the full Clinical-QE data of 8,588 question pairs, which gave 98 61% Accuracy 
In the second experiment, we used these datasets for training only and compared their performance on our test set of 850 consumer health questions  Table TABREF29 presents the results of this experiment  Logistic Regression trained on the clinical-RQE data outperformed DL models trained on all datasets, with 73 18% Accuracy 
To validate further the performance of the LR method, we evaluated it on question similarity detection  A typical approach to this task is to use an IR method to find similar question candidates, then a more sophisticated method to select and re-rank the similar questions  We followed a similar approach for this evaluation by combining the LR method with the IR baseline provided in the context of SemEval-cQA  The hybrid method combines the score provided by the Logistic Regression model and the reciprocal rank from the IR baseline using a weight-based combination:
INLINEFORM0
The weight INLINEFORM0 was set empirically through several tests on the cQA-2016 development set ( INLINEFORM1 )  Table TABREF30 presents the results on the cQA-2016 and cQA-2017 test datasets  The hybrid method (LR+IR) provided the best results on both datasets  On the 2016 test data, the LR+IR method outperformed the best system in all measures, with 80 57% Accuracy and 77 47% MAP (official system ranking measure in SemEval-cQA)  On the cQA-2017 test data, the LR+IR method obtained 44 66% MAP and outperformed the cQA-2017 best system in Accuracy with 67 27% 
Discussion of RQE Results
When trained and tested on the same corpus, the DL model with GloVe embeddings gave the best results on three datasets (SNLI, MultiNLI and Quora)  Logistic Regression gave the best Accuracy on the Clinical-RQE dataset with 98 60%  When tested on our test set (850 medical CHQs-FAQs pairs), Logistic Regression trained on Clinical-QE gave the best performance with 73 18% Accuracy 
The SNLI and multi-NLI models did not perform well when tested on medical RQE data  We performed additional evaluations using the RTE-1, RTE-2 and RTE-3 open-domain datasets provided by the PASCAL challenge and the results were similar  We have also tested the SemEval-cQA-2016 model and had a similar drop in performance on RQE data  This could be explained by the different types of data leading to wrong internal conceptualizations of medical terms and questions in the deep neural layers  This performance drop could also be caused by the complexity of the test consumer health questions that are often composed of several subquestions, contain contextual information, and may contain misspellings and ungrammatical sentences, which makes them more difficult to process BIBREF48   Another aspect is the semantics of the task as discussed in Section SECREF6   The definition of textual entailment in open-domain may not quite apply to question entailment due to the strict semantics  Also the general textual entailment definitions refer only to the premise and hypothesis, while the definition of RQE for question answering relies on the relationship between the sets of answers of the compared questions 
Building a Medical QA Collection from Trusted Resources
A RQE-based QA system requires a collection of question-answer pairs to map new user questions to the existing questions with an RQE approach, rank the retrieved questions, and present their answers to the user 
Method
To construct trusted medical question-answer pairs, we crawled websites from the National Institutes of Health (cf  Section SECREF56 )  Each web page describes a specific topic (e g  name of a disease or a drug), and often includes synonyms of the main topic that we extracted during the crawl 
We constructed hand-crafted patterns for each website to automatically generate the question-answer pairs based on the document structure and the section titles  We also annotated each question with the associated focus (topic of the web page) as well as the question type identified with the designed patterns (cf  Section SECREF36 ) 
To provide additional information about the questions that could be used for diverse IR and NLP tasks, we automatically annotated the questions with the focus, its UMLS Concept Unique Identifier (CUI) and Semantic Type  We combined two methods to recognize named entities from the titles of the crawled articles and their associated UMLS CUIs: (i) exact string matching to the UMLS Metathesaurus, and (ii) MetaMap Lite BIBREF49   We then used the UMLS Semantic Network to retrieve the associated semantic types and groups 
Question Types
The question types were derived after the manual evaluation of 1,721 consumer health questions  Our taxonomy includes 16 types about Diseases, 20 types about Drugs and one type (Information) for the other named entities such as Procedures, Medical exams and Treatments  We describe below the considered question types and examples of associated question patterns 
Question Types about Diseases (16): Information, Research (or Clinical Trial), Causes, Treatment, Prevention, Diagnosis (Exams and Tests), Prognosis, Complications, Symptoms, Inheritance, Susceptibility, Genetic changes, Frequency, Considerations, Contact a medical professional, Support Groups 
Examples:
What research (or clinical trial) is being done for DISEASE?
What is the outlook for DISEASE?
How many people are affected by DISEASE?
When to contact a medical professional about DISEASE?
Who is at risk for DISEASE?
Where to find support for people with DISEASE?
Question Types About Drugs (20): Information, Interaction with medications, Interaction with food, Interaction with herbs and supplements, Important warning, Special instructions, Brand names, How does it work, How effective is it, Indication, Contraindication, Learn more, Side effects, Emergency or overdose, Severe reaction, Forget a dose, Dietary, Why get vaccinated, Storage and disposal, Usage, Dose 
Examples:
Are there interactions between DRUG and herbs and supplements?
What important warning or information should I know about DRUG?
Are there safety concerns or special precautions about DRUG?
What is the action of DRUG and how does it work?
Who should get DRUG and why is it prescribed?
What to do in case of a severe reaction to DRUG?
Question Type for other medical entities (e g  Procedure, Exam, Treatment): Information 
What is Coronary Artery Bypass Surgery?
What are Liver Function Tests?
Medical Resources
We used 12 trusted websites to construct a collection of question-answer pairs  For each website, we extracted the free text of each article as well as the synonyms of the article focus (topic)  These resources and their brief descriptions are provided below:
National Cancer Institute (NCI) : We extracted free text from 116 articles on various cancer types (729 QA pairs)  We manually restructured the content of the articles to generate complete answers (e g  a full answer about the treatment of all stages of a specific type of cancer)  Figure FIGREF54 presents examples of QA pairs generated from a NCI article 
Genetic and Rare Diseases Information Center (GARD): This resource contains information about various aspects of genetic/rare diseases  We extracted all disease question/answer pairs from 4,278 topics (5,394 QA pairs) 
Genetics Home Reference (GHR): This NLM resource contains consumer-oriented information about the effects of genetic variation on human health  We extracted 1,099 articles about diseases from this resource (5,430 QA pairs) 
MedlinePlus Health Topics: This portion of MedlinePlus contains information on symptoms, causes, treatment and prevention for diseases, health conditions and wellness issues  We extracted the free texts in summary sections of 981 articles (981 QA pairs) 
National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) : We extracted text from 174 health information pages on diseases studied by this institute (1,192 QA pairs) 
National Institute of Neurological Disorders and Stroke (NINDS): We extracted free text from 277 information pages on neurological and stroke-related diseases from this resource (1,104 QA pairs) 
NIHSeniorHealth : This website contains health and wellness information for older adults  We extracted 71 articles from this resource (769 QA pairs) 
National Heart, Lung, and Blood Institute (NHLBI) : We extracted text from 135 articles on diseases, tests, procedures, and other relevant topics on disorders of heart, lung, blood, and sleep (559 QA pairs) 
Centers for Disease Control and Prevention (CDC) : We extracted text from 152 articles on diseases and conditions (270 QA pairs) 
MedlinePlus A D A M  Medical Encyclopedia: This resource contains 4,366 articles about conditions, tests, and procedures  17,348 QA pairs were extracted from this resource  Figure FIGREF55 presents examples of QA pairs generated from A D A M encyclopedia 
MedlinePlus Drugs: We extracted free text from 1,316 articles about Drugs and generated 12,889 QA pairs 
MedlinePlus Herbs and Supplements: We extracted free text from 99 articles and generated 792 QA pairs 
The final collection contains 47,457 annotated question-answer pairs about Diseases, Drugs and other named entities (e g  Tests) extracted from these 12 trusted resources 
The Proposed Entailment-based QA System
Our goal is to generate a ranked list of answers for a given Premise Question INLINEFORM0 by ranking the recognized Hypothesis Questions INLINEFORM1   Based on the RQE experiments above (Section SECREF27 ), we selected Logistic Regression trained on the clinical-RQE dataset to recognize entailed questions and rank them with their classification scores 
RQE-based QA Approach
Classifying the full QA collection for each test question is not feasible for real-time applications  Therefore, we first filter the questions with an IR method to retrieve candidate questions, then classify them as entailed (or not) by the user/test question  Based on the positive results of the combination method tested on SemEval-cQA data (Section SECREF27 ), we adopted a combination method to merge the results obtained by the search engine and the RQE scores  The answers are then combined from both methods and ranked using an aggregate score  Figure FIGREF82 presents the overall architecture of the proposed QA system  We describe each module in more details next 
Finding Similar Question Candidates
For each premise question INLINEFORM0 , we use the Terrier search engine to retrieve INLINEFORM1 relevant question candidates INLINEFORM2 and then apply the RQE classifier to predict the labels for the pairs ( INLINEFORM3 , INLINEFORM4 ) 
We indexed the questions of our QA collection without the associated answers  In order to improve the indexing and the performance of question retrieval, we also indexed the synonyms of the question focus and the triggers of the question type with each question  This choice allowed us to avoid the shortcomings of query expansion, including incorrect or irrelevant synonyms and the increased execution time  The synonyms of the question focus (topic) were extracted automatically from the QA collection  The triggers of each question type were defined manually in the question types taxonomy  Below are two examples of indexed questions from our QA collection, with the automatically added focus synonyms and question type triggers:
What are the treatments for Torticollis?
Focus: Torticollis  Question type: Treatment 
Added focus synonyms: "Spasmodic torticollis, Wry neck, Loxia, Cervical dystonia"  Added question type triggers: "relieve, manage, cure, remedy, therapy" 
What is the outlook for Legionnaire disease?
Focus: Legionnaire disease  Question Type: Prognosis 
Added focus synonyms: "Legionella pneumonia, Pontiac fever, Legionellosis"  Added question type triggers: "prognosis, life expectancy" 
The IR task consists of retrieving hypothesis questions INLINEFORM0 relevant to the submitted question INLINEFORM1   As fusion of IR result has shown good performance in different tracks in TREC, we merge the results of the TF-IDF weighting function and the In-expB2 DFR model BIBREF50  
Let INLINEFORM0 = INLINEFORM1 , INLINEFORM2 ,    , INLINEFORM3 be the set of INLINEFORM4 questions retrieved by the first IR model INLINEFORM5 and INLINEFORM6 = INLINEFORM7 , INLINEFORM8 ,    , INLINEFORM9 be the set of INLINEFORM10 questions retrieved by the second IR model INLINEFORM11   We merge both sets by summing the scores of each retrieved question INLINEFORM12 in both INLINEFORM13 and INLINEFORM14 lists, then we re-rank the hypothesis questions INLINEFORM15  
Combining IR and RQE Methods
The IR models and the RQE Logistic Regression model bring different perspectives to the search for relevant candidate questions  In particular, question entailment allows understanding the relations between the important terms, whereas the traditional IR methods identify the important terms, but will not notice if the relations are opposite  Moreover, some of the question types that the RQE classifier learns will not be deemed important terms by traditional IR and the most relevant questions will not be ranked at the top of the list 
Therefore, in our approach, when a question is submitted to the system, candidate questions are fetched using the IR models, then the RQE classifier is applied to filter out the non-entailed questions and re-rank the remaining candidates 
Specifically, we denote INLINEFORM0 the list of question candidates INLINEFORM1 returned by the IR system  The premise question INLINEFORM2 is then used to construct N question pairs INLINEFORM3   The RQE classifier is then applied to filter out the question pairs that are not entailed and re-rank the remaining pairs 
More precisely, let INLINEFORM0 = INLINEFORM1 in INLINEFORM2 be the list of selected candidate questions that have a positive entailment relation with a given premise question INLINEFORM3   We rank INLINEFORM4 by computing a hybrid score INLINEFORM5 for each candidate question INLINEFORM6 taking into account the score of the IR system INLINEFORM7 and the score of the RQE system INLINEFORM8  
For each system INLINEFORM0 INLINEFORM1 , we normalize the associated score by dividing it by the maximum score among the INLINEFORM2 candidate questions retrieved by INLINEFORM3 for INLINEFORM4 :
INLINEFORM0
INLINEFORM0 INLINEFORM1
In our experiments, we fixed the value of INLINEFORM0 to 100  This threshold value was selected as a safe value for this task for the following reasons:
Our collection of 47,457 question-answer pairs was collected from only 12 NIH institutes and is unlikely to contain more than 100 occurrences of the same focus-type pair 
Each question was indexed with additional annotations for the question focus, its synonyms and the question type synonyms 
Evaluating RQE for Medical Question Answering
The objective of this evaluation is to study the effectiveness of RQE for Medical Question Answering, by comparing the answers retrieved by the hybrid entailment-based approach, the IR method and the other QA systems participating to the medical task at TREC 2017 LiveQA challenge (LiveQA-Med) 
Evaluation Method
We developed an interface to perform the manual evaluation of the retrieved answers  Figure 5 presents the evaluation interface showing, for each test question, the top-10 answers of the evaluated QA method and the reference answer(s) used by LiveQA assessors to help judging the retrieved answers by the participating systems 
We used the test questions of the medical task at TREC-2017 LiveQA BIBREF11   These questions are randomly selected from the consumer health questions that the NLM receives daily from all over the world  The test questions cover different medical entities and have a wide list of question types such as Comparison, Diagnosis, Ingredient, Side effects and Tapering 
For a relevant comparison, we used the same judgment scores as the LiveQA Track:
Correct and Complete Answer (4)
Correct but Incomplete (3)
Incorrect but Related (2)
Incorrect (1)
We evaluated the answers returned by the IR-based method and the hybrid QA method (IR+RQE) according to the same reference answers used in LiveQA-Med  The answers were anonymized (the method names were blinded) and presented to 3 assessors: a medical doctor (Assessor A), a medical librarian (B) and a researcher in medical informatics (C)  None of the assessors participated in the development of the QA methods  Assessors B and C evaluated 1,000 answers retrieved by each of the methods (IR and IR+RQE)  Assessor A evaluated 2,000 answers from both methods 
Table TABREF103 presents the inter-annotator agreement (IAA) through F1 score computed by considering one of the assessors as reference  In the first evaluation, we computed the True Positives (TP) and False Positives (FP) over all ratings and the Precision and F1 score  As there are no negative labels (only true or false positives for each category), Recall is 100%  We also computed a partial IAA by grouping the "Correct and Complete Answer" and "Correct but Incomplete" ratings (as Correct), and the "Incorrect but Related" and "Incorrect" ratings (as Incorrect)  The average agreement on distinguishing the Correct and Incorrect answers is 94 33% F1 score  Therefore, we used the evaluations performed by assessor A for both methods  The official results of the TREC LiveQA track relied on one assessor per question as well 
Evaluation of the first retrieved answer
We computed the measures used by TREC LiveQA challenges BIBREF51 , BIBREF11 to evaluate the first retrieved answer for each test question:
avgScore(0-3): the average score over all questions, transferring 1-4 level grades to 0-3 scores  This is the main score used to rank LiveQA runs 
succ@i+: the number of questions with score i or above (i INLINEFORM0 {2  4}) divided by the total number of questions 
prec@i+: the number of questions with score i or above (i INLINEFORM0 {2  4}) divided by number of questions answered by the system 
Table TABREF108 presents the average scores, success and precision results  The hybrid IR+RQE QA system achieved better results than the IR-based system with 0 827 average score  It also achieved a higher score than the best results achieved in the medical challenge at LiveQA'17  Evaluating the RQE system alone is not relevant, as applying RQE on the full collection for each user question is not feasible for a real-time system because of the extended execution time 
Evaluation of the top ten answers
In this evaluation, we used Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) which are commonly used in QA to evaluate the top-10 answers for each question  We consider answers rated as “Correct and Complete Answer” or “Correct but Incomplete” as correct answers, as the test questions contain multiple subquestions while each answer in our QA collection can cover only one subquestion 
MAP is the mean of the Average Precision (AvgP) scores over all questions 
(1) INLINEFORM0
Q is the number of questions  INLINEFORM0 is the AvgP of the INLINEFORM1 question 
INLINEFORM0
K is the number of correct answers  INLINEFORM0 is the rank of INLINEFORM1 correct answer 
MRR is the average of the reciprocal ranks for each question  The reciprocal rank of a question is the multiplicative inverse of the rank of the first correct answer 
(2) INLINEFORM0
Q is the number of questions  INLINEFORM0 is the rank of the first correct answer for the INLINEFORM1 question 
Table TABREF113 presents the MAP@10 and MRR@10 of our QA methods  The IR+RQE system outperforms the IR-based QA system with 0 311 MAP@10 and 0 333 MRR@10 
Discussion of entailment-based QA for the medical domain
In our evaluation, we followed the same LiveQA guidelines with the highest possible rigor  In particular, we consulted with NIST assessors who provided us with the paraphrases of the test questions that they used to judge the answers  Our IAA on the answers rating was also high compared to related tasks, with an 88 5% F1 agreement with the exact four categories and a 94 3% agreement when reducing the categories to two: “Correct” and “Incorrect” answers  Our results show that RQE improves the overall performance and exceeds the best results in the medical LiveQA'17 challenge by a factor of 29 8%  This performance improvement is particularly interesting as:
Our answer source has only 47K question-answer pairs when LiveQA participating systems relied on much larger collections, including the World Wide Web 
Our system answered one subquestion at most when many LiveQA test questions had several subquestions 
The latter observation, (b), makes the hybrid IR+RQE approach even more promising as it gives it a large potential for the improvement of answer completeness 
The former observation, (a), provides another interesting insight: restricting the answer source to only reliable collections can actually improve the QA performance without losing coverage (i e , our QA approach provided at least one answer to each test question and obtained the best relevance score) 
In another observation, the assessors reported that many of the returned answers had a correct question type but a wrong focus, which indicates that including a focus recognition module to filter such wrong answers can improve further the QA performance in terms of precision  Another aspect that was reported is the repetition of the same (or similar) answer from different websites, which could be addressed by improving answer selection with inter-answer comparisons and removal of near-duplicates  Also, half of the LiveQA test questions are about Drugs, when only two of our resources are specialized in Drugs, among 12 sub-collections overall  Accordingly, the assessors noticed that the performance of the QA systems was better on questions about diseases than on questions about drugs, which suggests a need for extending our medical QA collection with more information about drugs and associated question types 
We also looked closely at the private websites used by the LiveQA-Med annotators to provide some of the reference answers for the test questions  For instance, the ConsumerLab website was useful to answer a question about the ingredients of a Drug (COENZYME Q10)  Similarly, the eHealthMe website was used to answer a test question asking about interactions between two drugs (Phentermine and Dicyclomine) when no information was found in DailyMed  eHealthMe provides healthcare big data analysis and private research and studies including self-reported adverse drug effects by patients 
But the question remains on the extent to which such big data and other private websites could be used to automatically answer medical questions if information is otherwise unavailable  Unlike medical professionals, patients do not necessarily have the knowledge and tools to validate such information  An alternative approach could be to put limitations on medical QA systems in terms of the questions that can be answered (e g  "What is my diagnosis for such symptoms") and build classifiers to detect such questions and warn the users about the dangers of looking for their answers online 
More generally, medical QA systems should follow some strict guidelines regarding the goal and background knowledge and resources of each system in order to protect the consumers from misleading or harmful information  Such guidelines could be based (i) on the source of the information such as health and medical information websites sponsored by the U S  government, not-for-profit health or medical organizations, and medical university centers, or (ii) on conventions such as the code of conduct of the HON Foundation (HONcode) that addresses the reliability and usefulness of medical information on the Internet 
Our experiments show that limiting the number of answer sources with such guidelines is not only feasible, but it could also enhance the performance of the QA system from an information retrieval perspective 
Conclusion
In this paper, we carried out an empirical study of machine learning and deep learning methods for Recognizing Question Entailment in the medical domain using several datasets  We developed a RQE-based QA system to answer new medical questions using existing question-answer pairs  We built and shared a collection of 47K medical question-answer pairs  Our QA approach outperformed the best results on TREC-2017 LiveQA medical test questions  The proposed approach can be applied and adapted to open-domain as well as specific-domain QA  Deep learning models achieved interesting results on open-domain and clinical datasets, but obtained a lower performance on consumer health questions  We will continue investigating other network architectures including transfer learning, as well as creation of a large collection of consumer health questions for training to improve the performance of DL models  Future work also includes exploring integration of a Question Focus Recognition module to enhance candidate question retrieval, and expanding our question-answer collection 
Acknowledgements
We thank Halil Kilicoglu (NLM/NIH) for his help with the crawling and the manual evaluation and Sonya E  Shooshan (NLM/NIH) for her help with the judgment of the retrieved answers  We also thank Ellen Voorhees (NIST) for her valuable support with the TREC LiveQA evaluation 
We consider the case of the question number 36 in the TREC-2017 LiveQA medical test dataset:
36  congenital diaphragmatic hernia  what are the causes of congenital diaphragmatic hernia? Can cousin marriage cause this? What kind of lung disease the baby might experience life long?
This question was answered by 5 participating runs (vs  8 runs for other questions), and all submitted answers were wrong (scores of 1 or 2)  However, our IR-based QA system retrieved one excellent answer (score 4) and our hybrid IR+RQE system provided 3 excellent answers 
A) TREC 2017 LiveQA-Med Participants' Results:
B) Our IR-based QA System:
C) Our IR+RQE QA System:
Context 2: Introduction
Long short term memory (LSTM) units BIBREF1 are popular for many sequence modeling tasks and are used extensively in language modeling  A key to their success is their articulated gating structure, which allows for more control over the information passed along the recurrence  However, despite the sophistication of the gating mechanisms employed in LSTMs and similar recurrent units, the input and context vectors are treated with simple linear transformations prior to gating  Non-linear transformations such as convolutions BIBREF2 have been used, but these have not achieved the performance of well regularized LSTMs for language modeling BIBREF3  
A natural way to improve the expressiveness of linear transformations is to increase the number of dimensions of the input and context vectors, but this comes with a significant increase in the number of parameters which may limit generalizability  An example is shown in Figure FIGREF1 , where LSTMs performance decreases with the increase in dimensions of the input and context vectors  Moreover, the semantics of the input and context vectors are different, suggesting that each may benefit from specialized treatment 
Guided by these insights, we introduce a new recurrent unit, the Pyramidal Recurrent Unit (PRU), which is based on the LSTM gating structure  Figure FIGREF2 provides an overview of the PRU  At the heart of the PRU is the pyramidal transformation (PT), which uses subsampling to effect multiple views of the input vector  The subsampled representations are combined in a pyramidal fusion structure, resulting in richer interactions between the individual dimensions of the input vector than is possible with a linear transformation  Context vectors, which have already undergone this transformation in the previous cell, are modified with a grouped linear transformation (GLT) which allows the network to learn latent representations in high dimensional space with fewer parameters and better generalizability (see Figure FIGREF1 ) 
We show that PRUs can better model contextual information and demonstrate performance gains on the task of language modeling  The PRU improves the perplexity of the current state-of-the-art language model BIBREF0 by up to 1 3 points, reaching perplexities of 56 56 and 64 53 on the Penn Treebank and WikiText2 datasets while learning 15-20% fewer parameters  Replacing an LSTM with a PRU results in improvements in perplexity across a variety of experimental settings  We provide detailed ablations which motivate the design of the PRU architecture, as well as detailed analysis of the effect of the PRU on other components of the language model 
Related work
Multiple methods, including a variety of gating structures and transformations, have been proposed to improve the performance of recurrent neural networks (RNNs)  We first describe these approaches and then provide an overview of recent work in language modeling 
Pyramidal Recurrent Units
We introduce Pyramidal Recurrent Units (PRUs), a new RNN architecture which improves modeling of context by allowing for higher dimensional vector representations while learning fewer parameters  Figure FIGREF2 provides an overview of PRU  We first elaborate on the details of the pyramidal transformation and the grouped linear transformation  We then describe our recurrent unit, PRU 
Pyramidal transformation for input
The basic transformation in many recurrent units is a linear transformation INLINEFORM0 defined as: DISPLAYFORM0
where INLINEFORM0 are learned weights that linearly map INLINEFORM1 to INLINEFORM2   To simplify notation, we omit the biases 
Motivated by successful applications of sub-sampling in computer vision (e g , BIBREF22 , BIBREF23 , BIBREF9 , BIBREF24 ), we subsample input vector INLINEFORM0 into INLINEFORM1 pyramidal levels to achieve representation of the input vector at multiple scales  This sub-sampling operation produces INLINEFORM2 vectors, represented as INLINEFORM3 , where INLINEFORM4 is the sampling rate and INLINEFORM5   We learn scale-specific transformations INLINEFORM6 for each INLINEFORM7   The transformed subsamples are concatenated to produce the pyramidal analog to INLINEFORM8 , here denoted as INLINEFORM9 : DISPLAYFORM0
where INLINEFORM0 indicates concatenation  We note that pyramidal transformation with INLINEFORM1 is the same as the linear transformation 
To improve gradient flow inside the recurrent unit, we combine the input and output using an element-wise sum (when dimension matches) to produce residual analog of pyramidal transformation, as shown in Figure FIGREF2 BIBREF25  
We sub-sample the input vector INLINEFORM0 into INLINEFORM1 pyramidal levels using the kernel-based approach BIBREF8 , BIBREF9   Let us assume that we have a kernel INLINEFORM2 with INLINEFORM3 elements  Then, the input vector INLINEFORM4 can be sub-sampled as: DISPLAYFORM0
where INLINEFORM0 represents the stride and INLINEFORM1  
The number of parameters learned by the linear transformation and the pyramidal transformation with INLINEFORM0 pyramidal levels to map INLINEFORM1 to INLINEFORM2 are INLINEFORM3 and INLINEFORM4 respectively  Thus, pyramidal transformation reduces the parameters of a linear transformation by a factor of INLINEFORM5   For example, the pyramidal transformation (with INLINEFORM6 and INLINEFORM7 ) learns INLINEFORM8 fewer parameters than the linear transformation 
Grouped linear transformation for context
Many RNN architectures apply linear transformations to both the input and context vector  However, this may not be ideal due to the differing semantics of each vector  In many NLP applications including language modeling, the input vector is a dense word embedding which is shared across all contexts for a given word in a dataset  In contrast, the context vector is highly contextualized by the current sequence  The differences between the input and context vector motivate their separate treatment in the PRU architecture 
The weights learned using the linear transformation (Eq  EQREF9 ) are reused over multiple time steps, which makes them prone to over-fitting BIBREF26   To combat over-fitting, various methods, such as variational dropout BIBREF26 and weight dropout BIBREF0 , have been proposed to regularize these recurrent connections  To further improve generalization abilities while simultaneously enabling the recurrent unit to learn representations at very high dimensional space, we propose to use grouped linear transformation (GLT) instead of standard linear transformation for recurrent connections BIBREF27   While pyramidal and linear transformations can be applied to transform context vectors, our experimental results in Section SECREF39 suggests that GLTs are more effective 
The linear transformation INLINEFORM0 maps INLINEFORM1 linearly to INLINEFORM2   Grouped linear transformations break the linear interactions by factoring the linear transformation into two steps  First, a GLT splits the input vector INLINEFORM3 into INLINEFORM4 smaller groups such that INLINEFORM5   Second, a linear transformation INLINEFORM6 is applied to map INLINEFORM7 linearly to INLINEFORM8 , for each INLINEFORM9   The INLINEFORM10 resultant output vectors INLINEFORM11 are concatenated to produce the final output vector INLINEFORM12   DISPLAYFORM0
GLTs learn representations at low dimensionality  Therefore, a GLT requires INLINEFORM0 fewer parameters than the linear transformation  We note that GLTs are subset of linear transformations  In a linear transformation, each neuron receives an input from each element in the input vector while in a GLT, each neuron receives an input from a subset of the input vector  Therefore, GLT is the same as a linear transformation when INLINEFORM1  
Pyramidal Recurrent Unit
We extend the basic gating architecture of LSTM with the pyramidal and grouped linear transformations outlined above to produce the Pyramidal Recurrent Unit (PRU), whose improved sequence modeling capacity is evidenced in Section SECREF4  
At time INLINEFORM0 , the PRU combines the input vector INLINEFORM1 and the previous context vector (or previous hidden state vector) INLINEFORM2 using the following transformation function as: DISPLAYFORM0
where INLINEFORM0 indexes the various gates in the LSTM model, and INLINEFORM1 and INLINEFORM2 represents the pyramidal and grouped linear transformations defined in Eqns  EQREF10 and EQREF15 , respectively 
We will now incorporate INLINEFORM0 into LSTM gating architecture to produce PRU  At time INLINEFORM1 , a PRU cell takes INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 as inputs to produce forget INLINEFORM5 , input INLINEFORM6 , output INLINEFORM7 , and content INLINEFORM8 gate signals  The inputs are combined with these gate signals to produce context vector INLINEFORM9 and cell state INLINEFORM10   Mathematically, the PRU with the LSTM gating architecture can be defined as: DISPLAYFORM0
where INLINEFORM0 represents the element-wise multiplication operation, and INLINEFORM1 and INLINEFORM2 are the sigmoid and hyperbolic tangent activation functions  We note that LSTM is a special case of PRU when INLINEFORM3 = INLINEFORM4 =1 
Experiments
To showcase the effectiveness of the PRU, we evaluate the performance on two standard datasets for word-level language modeling and compare with state-of-the-art methods  Additionally, we provide a detailed examination of the PRU and its behavior on the language modeling tasks 
Set-up
Following recent works, we compare on two widely used datasets, the Penn Treebank (PTB) BIBREF28 as prepared by BIBREF29 and WikiText2 (WT-2) BIBREF20   For both datasets, we follow the same training, validation, and test splits as in BIBREF0  
We extend the language model, AWD-LSTM BIBREF0 , by replacing LSTM layers with PRU  Our model uses 3-layers of PRU with an embedding size of 400  The number of parameters learned by state-of-the-art methods vary from 18M to 66M with majority of the methods learning about 22M to 24M parameters on the PTB dataset  For a fair comparison with state-of-the-art methods, we fix the model size to 19M and vary the value of INLINEFORM0 and hidden layer sizes so that total number of learned parameters is similar across different configurations  We use 1000, 1200, and 1400 as hidden layer sizes for values of INLINEFORM1 =1,2, and 4, respectively  We use the same settings for the WT-2 dataset  We set the number of pyramidal levels INLINEFORM2 to two in our experiments and use average pooling for sub-sampling  These values are selected based on our ablation experiments on the validation set (Section SECREF39 )  We measure the performance of our models in terms of word-level perplexity  We follow the same training strategy as in BIBREF0  
To understand the effect of regularization methods on the performance of PRUs, we perform experiments under two different settings: (1) Standard dropout: We use a standard dropout BIBREF12 with probability of 0 5 after embedding layer, the output between LSTM layers, and the output of final LSTM layer  (2) Advanced dropout: We use the same dropout techniques with the same dropout values as in BIBREF0   We call this model as AWD-PRU 
Results
Table TABREF23 compares the performance of the PRU with state-of-the-art methods  We can see that the PRU achieves the best performance with fewer parameters 
PRUs achieve either the same or better performance than LSTMs  In particular, the performance of PRUs improves with the increasing value of INLINEFORM0   At INLINEFORM1 , PRUs outperform LSTMs by about 4 points on the PTB dataset and by about 3 points on the WT-2 dataset  This is explained in part by the regularization effect of the grouped linear transformation (Figure FIGREF1 )  With grouped linear and pyramidal transformations, PRUs learn rich representations at very high dimensional space while learning fewer parameters  On the other hand, LSTMs overfit to the training data at such high dimensions and learn INLINEFORM2 to INLINEFORM3 more parameters than PRUs 
With the advanced dropouts, the performance of PRUs improves by about 4 points on the PTB dataset and 7 points on the WT-2 dataset  This further improves with finetuning on the PTB (about 2 points) and WT-2 (about 1 point) datasets 
For similar number of parameters, the PRU with standard dropout outperforms most of the state-of-the-art methods by large margin on the PTB dataset (e g  RAN BIBREF7 by 16 points with 4M less parameters, QRNN BIBREF33 by 16 points with 1M more parameters, and NAS BIBREF31 by 1 58 points with 6M less parameters)  With advanced dropouts, the PRU delivers the best performance  On both datasets, the PRU improves the perplexity by about 1 point while learning 15-20% fewer parameters 
PRU is a drop-in replacement for LSTM, therefore, it can improve language models with modern inference techniques such as dynamic evaluation BIBREF21   When we evaluate PRU-based language models (only with standard dropout) with dynamic evaluation on the PTB test set, the perplexity of PRU ( INLINEFORM0 ) improves from 62 42 to 55 23 while the perplexity of an LSTM ( INLINEFORM1 ) with similar settings improves from 66 29 to 58 79; suggesting that modern inference techniques are equally applicable to PRU-based language models 
Analysis
It is shown above that the PRU can learn representations at higher dimensionality with more generalization power, resulting in performance gains for language modeling  A closer analysis of the impact of the PRU in a language modeling system reveals several factors that help explain how the PRU achieves these gains 
As exemplified in Table TABREF34 , the PRU tends toward more confident decisions, placing more of the probability mass on the top next-word prediction than the LSTM  To quantify this effect, we calculate the entropy of the next-token distribution for both the PRU and the LSTM using 3687 contexts from the PTB validation set  Figure FIGREF32 shows a histogram of the entropies of the distribution, where bins of size 0 23 are used to effect categories  We see that the PRU more often produces lower entropy distributions corresponding to higher confidences for next-token choices  This is evidenced by the mass of the red PRU curve lying in the lower entropy ranges compared to the blue LSTM's curve  The PRU can produce confident decisions in part because more information is encoded in the higher dimensional context vectors 
The PRU has the ability to model individual words at different resolutions through the pyramidal transform; which provides multiple paths for the gradient to the embedding layer (similar to multi-task learning) and improves the flow of information  When considering the embeddings by part of speech, we find that the pyramid level 1 embeddings exhibit higher variance than the LSTM across all POS categories (Figure FIGREF33 ), and that pyramid level 2 embeddings show extremely low variance  We hypothesize that the LSTM must encode both coarse group similarities and individual word differences into the same vector space, reducing the space between individual words of the same category  The PRU can rely on the subsampled embeddings to account for coarse-grained group similarities, allowing for finer individual word distinctions in the embedding layer  This hypothesis is strengthened by the entropy results described above: a model which can make finer distinctions between individual words can more confidently assign probability mass  A model that cannot make these distinctions, such as the LSTM, must spread its probability mass across a larger class of similar words 
Saliency analysis using gradients help identify relevant words in a test sequence that contribute to the prediction BIBREF34 , BIBREF35 , BIBREF36   These approaches compute the relevance as the squared norm of the gradients obtained through back-propagation  Table TABREF34 visualizes the heatmaps for different sequences  PRUs, in general, give more relevance to contextual words than LSTMs, such as southeast (sample 1), cost (sample 2), face (sample 4), and introduced (sample 5), which help in making more confident decisions  Furthermore, when gradients during back-propagation are visualized BIBREF37 (Table TABREF34 ), we find that PRUs have better gradient coverage than LSTMs, suggesting PRUs use more features than LSTMs that contributes to the decision  This also suggests that PRUs update more parameters at each iteration which results in faster training  Language model in BIBREF0 takes 500 and 750 epochs to converge with PRU and LSTM as a recurrent unit, respectively 
Ablation studies
In this section, we provide a systematic analysis of our design choices  Our training methodology is the same as described in Section SECREF19 with the standard dropouts  For a thorough understanding of our design choices, we use a language model with a single layer of PRU and fix the size of embedding and hidden layers to 600  The word-level perplexities are reported on the validation sets of the PTB and the WT-2 datasets 
The two hyper-parameters that control the trade-off between performance and number of parameters in PRUs are the number of pyramidal levels INLINEFORM0 and groups INLINEFORM1   Figure FIGREF35 provides a trade-off between perplexity and recurrent unit (RU) parameters 
Variable INLINEFORM0 and fixed INLINEFORM1 : When we increase the number of pyramidal levels INLINEFORM2 at a fixed value of INLINEFORM3 , the performance of the PRU drops by about 1 to 4 points while reducing the total number of recurrent unit parameters by up to 15%  We note that the PRU with INLINEFORM4 at INLINEFORM5 delivers similar performance as the LSTM while learning about 15% fewer recurrent unit parameters 
Fixed INLINEFORM0 and variable INLINEFORM1 : When we vary the value of INLINEFORM2 at fixed number of pyramidal levels INLINEFORM3 , the total number of recurrent unit parameters decreases significantly with a minimal impact on the perplexity  For example, PRUs with INLINEFORM4 and INLINEFORM5 learns 77% fewer recurrent unit parameters while its perplexity (lower is better) increases by about 12% in comparison to LSTMs  Moreover, the decrease in number of parameters at higher value of INLINEFORM6 enables PRUs to learn the representations in high dimensional space with better generalizability (Table TABREF23 ) 
Table TABREF43 shows the impact of different transformations of the input vector INLINEFORM0 and the context vector INLINEFORM1   We make following observations: (1) Using the pyramidal transformation for the input vectors improves the perplexity by about 1 point on both the PTB and WT-2 datasets while reducing the number of recurrent unit parameters by about 14% (see R1 and R4)  We note that the performance of the PRU drops by up to 1 point when residual connections are not used (R4 and R6)  (2) Using the grouped linear transformation for context vectors reduces the total number of recurrent unit parameters by about 75% while the performance drops by about 11% (see R3 and R4)  When we use the pyramidal transformation instead of the linear transformation, the performance drops by up to 2% while there is no significant drop in the number of parameters (R4 and R5) 
We set sub-sampling kernel INLINEFORM0 (Eq  EQREF12 ) with stride INLINEFORM1 and size of 3 ( INLINEFORM2 ) in four different ways: (1) Skip: We skip every other element in the input vector  (2) Convolution: We initialize the elements of INLINEFORM3 randomly from normal distribution and learn them during training the model  We limit the output values between -1 and 1 using INLINEFORM4 activation function to make training stable  (3) Avg  pool: We initialize the elements of INLINEFORM5 to INLINEFORM6   (4) Max pool: We select the maximum value in the kernel window INLINEFORM7  
Table TABREF45 compares the performance of the PRU with different sampling methods  Average pooling performs the best while skipping give comparable performance  Both of these methods enable the network to learn richer word representations while representing the input vector in different forms, thus delivering higher performance  Surprisingly, a convolution-based sub-sampling method does not perform as well as the averaging method  The INLINEFORM0 function used after convolution limits the range of output values which are further limited by the LSTM gating structure, thereby impeding in the flow of information inside the cell  Max pooling forces the network to learn representations from high magnitude elements, thus distinguishing features between elements vanishes, resulting in poor performance 
Conclusion
We introduce the Pyramidal Recurrent Unit, which better model contextual information by admitting higher dimensional representations with good generalizability  When applied to the task of language modeling, PRUs improve perplexity across several settings, including recent state-of-the-art systems  Our analysis shows that the PRU improves the flow of gradient and expand the word embedding subspace, resulting in more confident decisions  Here we have shown improvements for language modeling  In future, we plan to study the performance of PRUs on different tasks, including machine translation and question answering  In addition, we will study the performance of the PRU on language modeling with more recent inference techniques, such as dynamic evaluation and mixture of softmax 
Acknowledgments
This research was supported by NSF (IIS 1616112, III 1703166), Allen Distinguished Investigator Award, and gifts from Allen Institute for AI, Google, Amazon, and Bloomberg  We are grateful to Aaron Jaech, Hannah Rashkin, Mandar Joshi, Aniruddha Kembhavi, and anonymous reviewers for their helpful comments 

ground_truth: Logistic Regression, neural networks

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what previous RNN models do they compare with?
Context 3: Introduction
Text classification has become an indispensable task due to the rapid growth in the number of texts in digital form available online  It aims to classify different texts, also called documents, into a fixed number of predefined categories, helping to organize data, and making easier for users to find the desired information  Over the past three decades, many methods based on machine learning and statistical models have been applied to perform this task, such as latent semantic analysis (LSA), support vector machines (SVM), and multinomial naive Bayes (MNB) 
The first step in utilizing such methods to categorize textual data is to convert the texts into a vector representation  One of the most popular text representation models is the bag-of-words model BIBREF0 , which represents each document in a collection as a vector in a vector space  Each dimension of the vectors represents a term (e g , a word, a sequence of words), and its value encodes a weight, which can be how many times the term occurs in the document 
Despite showing positive results in tasks such as language modeling and classification BIBREF1 , BIBREF2 , BIBREF3 , the BOW representation has limitations: first, feature vectors are commonly very high-dimensional, resulting in sparse document representations, which are hard to model due to space and time complexity  Second, BOW does not consider the proximity of words and their position in the text and consequently cannot encode the words semantic meanings 
To solve these problems, neural networks have been employed to learn vector representations of words BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7   In particular, the word2vec representation BIBREF8 has gained attention  Given a training corpus, word2vec can generate a vector for each word in the corpus that encodes its semantic information  These word vectors are distributed in such a way that words from similar contexts are represented by word vectors with high correlation, while words from different contexts are represented by word vectors with low correlation 
One crucial aspect of the word2vec representation is that arithmetic and distance calculation between two word vectors can be performed, giving information about their semantic relationship  However, rather than looking at pairs of word vectors, we are interested in studying the relationship between sets of vectors as a whole and, therefore, it is desirable to have a text representation based on a set of these word vectors 
To tackle this problem, we introduce the novel concept of word subspace  It is mathematically defined as a low dimensional linear subspace in a word vector space with high dimensionality  Given that words from texts of the same class belong to the same context, it is possible to model word vectors of each class as word subspaces and efficiently compare them in terms of similarity by using canonical angles between the word subspaces  Through this representation, most of the variability of the class is retained  Consequently, a word subspace can effectively and compactly represent the context of the corresponding text  We achieve this framework through the mutual subspace method (MSM) BIBREF9  
The word subspace of each text class is modeled by applying PCA without data centering to the set of word vectors of the class  When modeling the word subspaces, we assume only one occurrence of each word inside the class 
However, as seen in the BOW approach, the frequency of words inside a text is an informative feature that should be considered  In order to introduce this feature in the word subspace modeling and enhance its performance, we further extend the concept of word subspace to the term-frequency (TF) weighted word subspace 
In this extension, we consider a set of weights, which encodes the words frequencies, when performing the PCA  Text classification with TF weighted word subspace can also be performed under the framework of MSM  We show the validity of our modeling through experiments on the Reuters database, an established database for natural language processing tasks  We demonstrate the effectiveness of the word subspace formulation and its extension, comparing our methods' performance to various state-of-art methods 
The main contributions of our work are:
The remainder of this paper is organized as follows  In Section "Related Work" , we describe the main works related to text classification  In Section "Word subspace" , we present the formulation of our proposed word subspace  In Section "Conventional text classification methods" , we explain how text classification with word subspaces is performed under the MSM framework  Then, we present the TF weighted word subspace extension in Section "TF weighted word subspace"   Evaluation experiments and their results are described in Section "Experimental Evaluation"   Further discussion is then presented in Section "Discussion" , and our conclusions are described in Section "Conclusions and Future Work"  
Related Work
In this section, we outline relevant work towards text classification  We start by describing how text data is conventionally represented using the bag-of-words model and then follow to describe the conventional methods utilized in text classification 
Text Representation with bag-of-words
The bag-of-words representation comes from the hypothesis that frequencies of words in a document can indicate the relevance of the document to a query BIBREF0 , that is, if documents and a query have similar frequencies for the same words, they might have a similar meaning  This representation is based on the vector space model (VSM), that was developed for the SMART information retrieval system BIBREF10   In the VSM, the main idea is that documents in a collection can be represented as a vector in a vector space, where vectors close to each other represent semantically similar documents 
More formally, a document $d$ can be represented by a vector in $\mathbb {R}^{n}$ , where each dimension represents a different term  A term can be a single word, constituting the conventional bag-of-words, or combinations of $N$ words, constituting the bag-of-N-grams  If a term occurs in the document, its position in the vector will have a non-zero value, also known as term weight  Two documents in the VSM can be compared to each other by taking the cosine distance between them BIBREF1  
There are several ways to compute the term weights  Among them, we can highlight some: Binary weights, term-frequency (TF) weights, and term-frequency inverse document-frequency (TF-IDF) weights 
Consider a corpus with documents $D = \lbrace d_i\rbrace _{i=1}^{|D|}$ and a vocabulary with all terms in the corpus $V = \lbrace w_i\rbrace _{i=1}^{|V|}$   The term weights can be defined as:
Binary weight: If a term occurs in the document, its weight is 1  Otherwise, it is zero 
Term-frequency weight (TF): The weight of a term $w$ is defined by the number of times it occurs in the document $d$  
$$TF(w,d) = n_d^w$$   (Eq  8)
Inverse document-frequency: The weight of a term $w$ , given the corpus $D$ , is defined as the total number of documents $|D|$ divided by the number of documents that have the term $w$ , $|D^w|$  
$$IDF(w | D) = \frac{|D|}{|D^w|}$$   (Eq  10)
Term-frequency inverse document-frequency (TF-IDF): The weight of a term $w$ is defined by the multiplication of its term-frequency and its inverse document-frequency  When considering only the TF weights, all terms have the same importance among the corpus  By using the IDF weight, words that are more common across all documents in $D$ receive a smaller weight, giving more importance to rare terms in the corpus 
$$TFIDF(w,d | D)=TF \times IDF$$   (Eq  12)
In very large corpus, it is common to consider the logarithm of the IDF in order to dampen its effect 
$$TFIDF(w,d | D)=TF \times log_{10}(IDF)$$   (Eq  13)
Conventional text classification methods
Multi-variate Bernoulli (MVB) and multinomial naive Bayes (MNB) are two generative models based on the naive Bayes assumption  In other words, they assume that all attributes (e g , the frequency of each word, the presence or absence of a word) of each text are independent of each other given the context of the class BIBREF11  
In the MVB model, a document is represented by a vector generated by a bag-of-words with binary weights  In this case, a document can be considered an event, and the presence or the absence of the words to be the attributes of the event  On the other hand, the MNB model represents each document as a vector generated by a bag-of-words with TF weights  Here, the individual word occurrences are considered as events and the document is a collection of word events 
Both these models use the Bayes rule to classify a document  Consider that each document should be classified into one of the classes in $C=\lbrace c_j\rbrace _{j=1}^{|C|}$   The probability of each class given the document is defined as:
$$P(c_j|d_i) = \frac{P(d_i|c_j)P(c_j)}{P(d_i)} $$   (Eq  16)
The prior $P(d_i)$ is the same for all classes, so to determine the class to which $d_i$ belongs to, the following equation can be used:
$$prediction(d_i) = argmax_{c_j}P(d_i|c_j)P(c_j)$$   (Eq  17)
The prior $P(c_j)$ can be obtained by the following equation:
$$P(c_j) = \frac{1+|D_j|}{|C|+|D|},$$   (Eq  18)
where $|D_j|$ is the number of documents in class $c_j$  
As for the posterior $P(d_i|c_j)$ , different calculations are performed for each model  For MVB, it is defined as:
$$P(d_i|c_j) = \prod _{k=1}^{|V|}P(w_k|c_j)^{t_i^k}(1-P(w_k|c_j))^{1-t_i^k},$$   (Eq  19)
where $w_k$ is the k-th word in the vocabulary $V$ , and $t_i^k$ is the value (0 or 1) of the k-th element of the vector of document $d_i$  
For the MNB, it is defined as:
$$P(d_i|c_j) = P(|d_i|)|d_i|!\prod _{k=1}^{|V|}\frac{P(w_k|c_j)^{n_i^k}}{n_i^k!},$$   (Eq  20)
where $|d_i|$ is the number of words in document $d_i$ and $n_i^k$ is the k-th element of the vector of document $d_i$ and it represents how many times word $w_k$ occurs in $d_i$  
Finally, the posterior $P(w_k|c_j)$ can be obtained by the following equation:
$$P(w_k|c_j) = \frac{1+|D_j^k|}{|C|+|D|},$$   (Eq  21)
where $|D_j^k|$ is the number of documents in class $c_j$ that contain the word $w_k$  
In general, MVB tends to perform better than MNB at small vocabulary sizes whereas MNB is more efficient on large vocabularies 
Despite being robust tools for text classification, both these models depend directly on the bag-of-words features and do not naturally work with representations such as word2vec 
Latent semantic analysis (LSA), or latent semantic indexing (LSI), was proposed in BIBREF12 , and it extends the vector space model by using singular value decomposition (SVD) to find a set of underlying latent variables which spans the meaning of texts 
It is built from a term-document matrix, in which each row represents a term, and each column represents a document  This matrix can be built by concatenating the vectors of all documents in a corpus, obtained using the bag-of-words model, that is, $ {X} = [ {v}_1, {v}_2,    , {v}_{|D|}]$ , where ${v}_i$ is the vector representation obtained using the bag-of-words model 
In this method, the term-document matrix is decomposed using the singular value decomposition,
$${X} = {U\Sigma V}^\top ,$$   (Eq  23)
where $U$ and $V$ are orthogonal matrices and correspond to the left singular vectors and right singular vectors of $X$ , respectively  $\Sigma $ is a diagonal matrix, and it contains the square roots of the eigenvalues of $X^TX$ and $XX^T$   LSA finds a low-rank approximation of $X$ by selecting only the $k$ largest singular values and its respective singular vectors,
$${X}_k = {U}_k{\Sigma }_k {V}_k^{\top } $$   (Eq  24)
To compare two documents, we project both of them into this lower dimension space and calculate the cosine distance between them  The projection ${\hat{d}}$ of document ${d}$ is obtained by the following equation:
$${\hat{d}} = {\Sigma }_k^{-1} {U}_k^\top {d} $$   (Eq  25)
Despite its extensive application on text classification BIBREF13 , BIBREF14 , BIBREF15 , this method was initially proposed for document indexing and, therefore, does not encode any class information when modeling the low-rank approximation  To perform classification, 1-nearest neighbor is usually performed, placing a query document into the class of the nearest training document 
The support vector machine (SVM) was first presented in BIBREF16 and performs the separation between samples of two different classes by projecting them onto a higher dimensionality space  It was first applied in text classification by BIBREF17 and have since been successfully applied in many tasks related to natural language processing BIBREF18 , BIBREF19  
Consider a training data set $D$ , with $n$ samples
$$D = \lbrace ({x}_i,c_i)|{x}_i\in \mathbb {R}^p, c_i \in \lbrace -1,1\rbrace  \rbrace _{i=1}^{n},$$   (Eq  27)
where $c_i$ represents the class to which ${x}_i$ belongs to  Each ${x}_i$ is a $p$ -dimensional vector  The goal is to find the hyperplane that divides the points from $c_i = 1$ from the points from $c_i = -1$   This hyperplane can be written as a set of points $x$ satisfying:
$${w} \cdot {x} - b = 0,$$   (Eq  28)
where $\cdot $ denotes the dot product  The vector ${w}$ is perpendicular to the hyperplane  The parameter $\frac{b}{\Vert {w}\Vert }$ determines the offset of the hyperplane from the origin along the normal vector ${w}$  
We wish to choose ${w}$ and $b$ , so they maximize the distance between the parallel hyperplanes that are as far apart as possible, while still separating the data 
If the training data is linearly separable, we can select two hyperplanes in a way that there are no points between them and then try to maximize the distance  In other words, minimize $\Vert {w}\Vert $ subject to $c_i({w}\cdot {x}_u-b) \ge 1, i=\lbrace 1,2,   ,n\rbrace $   If the training data is not linearly separable, the kernel trick can be applied, where every dot product is replaced by a non-linear kernel function 
Word subspace
All methods mentioned above utilize the BOW features to represent a document  Although this representation is simple and powerful, its main problem lies on disregarding the word semantics within a document, where the context and meaning could offer many benefits to the model such as identification of synonyms 
In our formulation, words are represented as vectors in a real-valued feature vector space $\mathbb {R}^{p}$ , by using word2vec BIBREF8   Through this representation, it is possible to calculate the distance between two words, where words from similar contexts are represented by vectors close to each other, while words from different contexts are represented as far apart vectors  Also, this representation brings the new concept of arithmetic operations between words, where operations such as addition and subtraction carry meaning (eg , “king”-“man”+“woman”=“queen”) BIBREF20  
Consider a set of documents which belong to the same context $D_c = \lbrace d_i\rbrace _{i=1}^{|D_c|}$   Each document $d_i$ is represented by a set of $N_i$ words, $d_i = \lbrace w_k\rbrace _{k=1}^{N_i}$   By considering that all words from documents of the same context belong to the same distribution, a set of words $W_c = \lbrace w_k\rbrace _{k=1}^{N_c}$ with the words in the context $c$ is obtained

ground_truth: Variational LSTM, CharCNN, Pointer Sentinel-LSTM, RHN, NAS Cell, SRU, QRNN, RAN, 4-layer skip-connection LSTM, AWD-LSTM, Quantized LSTM

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What can word subspace represent?
Context 3: 
We then translate these words into word vectors using word2vec, resulting in a set of word vectors $X_c = \lbrace {x}^k_c\rbrace _{k=1}^{N_c} \in \mathbb {R}^p$   This set of word vectors is modeled into a word subspace, which is a compact, scalable and meaningful representation of the whole set  Such a word subspace is generated by applying PCA to the set of word vectors 
First, we compute an autocorrelation matrix, ${R}_c$ :
$${R}_c = \frac{1}{N_c}\sum _{i=1}^{N_c}{x}^{i}_c{x}_c^{i^{\top }} $$   (Eq  29)
The orthonormal basis vectors of $m_c$ -dimensional subspace ${Y}_c$ are obtained as the eigenvectors with the $m_c$ largest eigenvalues of the matrix ${R}_c$   We represent a subspace ${Y}_c$ by the matrix ${Y}_c \in \mathbb {R}^{p \times m_c}$ , which has the corresponding orthonormal basis vectors as its column vectors 
Text classification based on word subspace
We formulate our problem as a single label classification problem  Given a set of training documents, which we will refer as corpus, $D = \lbrace d_i\rbrace _{i=1}^{|D|}$ , with known classes $C = \lbrace c_j\rbrace _{j=1}^{|C|}$ , we wish to classify a query document $d_q$ into one of the classes in $C$  
Text classification based on word subspace can be performed under the framework of mutual subspace method (MSM)  This task involves two different stages: A learning stage, where the word subspace for each class is modeled, and a classification stage, where the word subspace for a query is modeled and compared to the word subspaces of the classes 
In the learning stage, it is assumed that all documents of the same class belong to the same context, resulting in a set of words $W_c = \lbrace w_c^k\rbrace _{k=1}^{N_c}$   This set assumes that each word appears only once in each class  Each set $\lbrace W_c\rbrace _{c=1}^{|C|}$ is then modeled into a word subspace ${Y}_c$ , as explained in Section "Word subspace"   As the number of words in each class may vary largely, the dimension $m_c$ of each class word subspace is not set to the same value 
In the classification stage, for a query document $d_q$ , it is also assumed that each word occurs only once, generating a subspace ${Y}_q$  
To measure the similarity between a class word subspace ${Y}_c$ and a query word subspace ${Y}_q$ , the canonical angles between the two word subspaces are used BIBREF21   There are several methods for calculating canonical angles BIBREF22 , BIBREF23 , and BIBREF24 , among which the simplest and most practical is the singular value decomposition (SVD)  Consider, for example, two subspaces, one from the training data and another from the query, represented as matrices of bases, ${Y}_{c} = [{\Phi }_{1} \ldots {\Phi }_{m_c}] \in \mathbb {R}^{p \times m_c}$ and ${Y}_{q} = [{\Psi }_{1} \ldots {\Psi }_{m_q}] \in \mathbb {R}^{p \times m_q}$ , where ${\Phi }_{i}$ are the bases for ${Y}_c$ and ${\Psi }_{i}$ are the bases for ${Y}_q$   Let the SVD of ${Y}_c^{\top }{Y}_q \in \mathbb {R}^{m_c \times m_q}$ be ${Y}_c^{\top }{Y}_q = {U \Sigma V}^{\top }$ , where ${Y}_q$0 , ${Y}_q$1 represents the set of singular values  The canonical angles ${Y}_q$2 can be obtained as ${Y}_q$3 ${Y}_q$4   The similarity between the two subspaces is measured by ${Y}_q$5 angles as follows:
$$S_{({Y}_c,{Y}_q)}[t] = \frac{1}{t}\sum _{i = 1}^{t} \cos ^{2} \theta _{i},\; 1 \le t \le m_q, \; m_q \le m_c $$   (Eq  30)
Fig  1 shows the modeling and comparison of sets of words by MSM  This method can compare sets of different sizes, and naturally encodes proximity between sets with related words 
Finally, the class with the highest similarity with $d_q$ is assigned as the class of $d_q$ :
$$prediction(d_q) = argmax_c(S_{({Y}_c,{Y}_q)}) $$   (Eq  32)
TF weighted word subspace
The word subspace formulation presented in Section "Word subspace" is a practical and compact way to represent sets of word vectors, retaining most of the variability of features  However, as seen in the BOW features, the frequency of words is relevant information that can improve the characterization of a text  To incorporate this information into the word subspace modeling, we propose an extension of the word subspace, called the term-frequency (TF) weighted word subspace 
Like the word subspace, the TF weighted word subspace is mathematically defined as a low-dimensional linear subspace in a word vector space with high dimensionality  However, a weighted version of the PCA BIBREF25 , BIBREF26 is utilized to incorporate the information given by the frequencies of words (term-frequencies)  This TF weighted word subspace is equivalent to the word subspace if we consider all occurrences of the words 
Consider the set of word vectors $\lbrace {x}_c^k\rbrace _{k=1}^{N_c} \in \mathbb {R}^{p}$ , which represents each word in the context $c$ , and the set of weights $\lbrace \omega _i\rbrace _{i=1}^{N_c}$ , which represent the frequencies of the words in the context $c$  
We incorporate these frequencies into the subspace calculation by weighting the data matrix ${X}$ as follows:
$${\widetilde{X}}={X}{\Omega }^{1/2},$$   (Eq  33)
where ${X} \in \mathbb {R}^{p \times N_c}$ is a matrix containing the word vectors $\lbrace {x}_c^k\rbrace _{k=1}^{N_c}$ and ${\Omega }$ is a diagonal matrix containing the weights $\lbrace \omega _i\rbrace _{i=1}^{N_c}$  
We then perform PCA by solving the SVD of the matrix ${\widetilde{X}}$ :
$${\widetilde{X}}={AMB}^{\top },$$   (Eq  34)
where the columns of the orthogonal matrices ${A}$ and ${B}$ are, respectively, the left-singular vectors and right-singular vectors of the matrix ${\widetilde{X}}$ , and the diagonal matrix ${M}$ contains singular values of ${\widetilde{X}}$  
Finally, the orthonormal basis vectors of the $m_c$ -dimensional TF weighted subspace ${W}$ are the column vectors in ${A}$ corresponding to the $m_c$ largest singular values in ${M}$  
Text classification with TF weighted word subspace can also be performed under the framework of MSM  In this paper, we will refer to MSM with TF weighted word subspace as TF-MSM 
Experimental Evaluation
In this section we describe the experiments performed to demonstrate the validity of our proposed method and its extension  We used the Reuters-8 dataset without stop words from BIBREF27 aiming at single-label classification, which is a preprocessed format of the Reuters-21578  Words in the texts were considered as they appeared, without performing stemming or typo correction  This database has eight different classes with the number of samples varying from 51 to over 3000 documents, as can be seen in Table 1  
To obtain the vector representation of words, we used a freely available word2vec model, trained by BIBREF8 , on approximately 100 billion words, which encodes the vector representation in $\mathbb {R}^{300}$ of over 3 million words from several different languages  Since we decided to focus on English words only, we filtered these vectors to about 800 thousand words, excluding all words with non-roman characters 
To show the validity of our word subspace representation for text classification and the proposed extension, we divided our experiment section into two parts: The first one aims to verify if sets of word vectors are suitable for subspace representation, and the second one puts our methods in practice in a text classification test, comparing our results with the conventional methods described in Section "Related Work"  
Evaluation of the word subspace representation
In this experiment, we modeled the word vectors from each class in the Reuters-8 database into a word subspace  The primary goal is to visualize how much of the text data can be represented by a lower dimensional subspace 
Subspace representations are very efficient in compactly represent data that is close to a normal distribution  This characteristic is due to the application of the PCA, that is optimal to find the direction with the highest variation within the data 
In PCA, the principal components give the directions of maximum variance, while their corresponding eigenvalues give the variance of the data in each of them  Therefore, by observing the distribution of the eigenvalues computed when performing PCA in the modeling of the subspaces, we can suggest if the data is suitable or not for subspace representation 
For each class, we normalized the eigenvalues by the largest one of the class  Fig  2 shows the mean of the eigenvalues and the standard deviation among classes  It is possible to see that the first largest eigenvalues retain larger variance than the smallest ones  In fact, looking at the first 150 largest eigenvalues, we can see that they retain, on average, 86 37% of the data variance  Also, by observing the standard deviation, we can understand that the eigenvalues distribution among classes follows the same pattern, that is, most of the variance is in the first dimensions  This plot indicates that text data represented by vectors generated with word2vec is suitable for subspace representation 
Text classification experiment
In this experiment, we performed text classification among the classes in the Reuters-8 database  We compared the classification using the word subspace, and its weighted extension, based on MSM (to which we will refer as MSM and TF-MSM, respectively) with the baselines presented in Section "Related Work" : MVB, MNB, LSA, and SVM  Since none of the baseline methods work with vector set classification, we also compared to a simple baseline for comparing sets of vectors, defined as the average of similarities between all vector pair combinations of two given sets  For two matrices ${A}$ and ${B}$ , containing the sets of vectors $\lbrace  {x}^{i}_a \rbrace _{i = 1}^{N_A}$ and $\lbrace  {x}^{i}_b \rbrace _{i = 1}^{N_B}$ , respectively, where $N_A$ and $N_B$ are the number of main words in each set, the similarity is defined as:
$$Sim_{(A,B)} = \frac{1}{N_A N_B}\sum _{i}^{N_A}\sum _{j}^{N_B}{{x}_a^i}^{\top }{x}_b^j $$   (Eq  41)
We refer to this baseline as similarity average (SA)  For this method, we only considered one occurrence of each word in each set 
Different features were used, depending on the method  Classification with SA, MSM, and TF-MSM was performed using word2vec features, to which we refer as w2v  For MVB, due to its nature, only bag-of-words features with binary weights were used (binBOW)  For the same reason, we only used bag-of-words features with term-frequency weights (tfBOW) with MNB  Classification with LSA and SVM is usually performed using bag-of-words features and, therefore, we tested with binBOW, tfBOW, and with the term-frequency inverse document-frequency weight, tfidfBOW  We also tested them using word2vec vectors  In this case, we considered each word vector from all documents in each class to be a single sample 
To determine the dimensions of the class subspaces and query subspace of MSM and TF-MSM, and the dimension of the approximation performed by LSA, we performed a 10-fold cross validation, wherein each fold, the data were randomly divided into train (60%), validation (20%) and test set (20%) 
The results can be seen in Table 2   The simplest baseline, SA with w2v, achieved an accuracy rate of 78 73%  This result is important because it shows the validity of the word2vec representation, performing better than more elaborate methods based on BOW, such as MVB with binBOW 
LSA with BOW features was almost 10% more accurate than SA, where the best results with binary weights were achieved with an approximation with 130 dimensions, with TF weights were achieved with 50 dimensions, and with TF-IDF weights were achieved with 30 dimensions  SVM with BOW features was about 3% more accurate than LSA, with binary weights leading to a higher accuracy rate 
It is interesting to note that despite the reasonably high accuracy rates achieved using LSA and SVM with BOW features, they poorly performed when using w2v features 
Among the baselines, the best method was MNB with tfBOW features, with an accuracy of 91 47%, being the only conventional method to outperform MSM  MSM with w2v had an accuracy rate of 90 62%, with the best results achieved with word subspace dimensions for the training classes ranging from 150 to 181, and for the query ranging from 3 to 217  Incorporating the frequency information in the subspace modeling resulted in higher accuracy, with TF-MSM achieving 92 01%, with dimensions of word subspaces for training classes ranging from 150 to 172, and for the query, ranging from 2 to 109  To confirm that TF-MSM is significantly more accurate than MNB, we performed a t-test to compare their results  It resulted in a p-value of 0 031, which shows that at a 95% significance level, TF-MSM has produced better results 
Discussion
Given the observation of the eigenvalues distribution of word vectors, we could see that word vectors that belong to the same context, i e , same class, are suitable for subspace representation  Our analysis showed that half of the word vector space dimensions suffice to represent most of the variability of the data in each class of the Reuters-8 database 
The results from the text classification experiment showed that subspace-based methods performed better than the text classification methods discussed in this work  Ultimately, our proposed TF weighted word subspace with MSM surpassed all the other methods  word2vec features are reliable tools to represent the semantic meaning of the words and when treated as sets of word vectors, they are capable of representing the content of texts  However, despite the fact that word vectors can be treated separately, conventional methods such as SVM and LSA may not be suitable for text classification using word vectors 
Among the conventional methods, LSA and SVM achieved about 86% and 89%, respectively, when using bag-of-words features  Interestingly, both methods had better performance when using binary weights  For LSA, we can see that despite the slight differences in the performance, tfidfBOW required approximations with smaller dimensions  SVM had the lowest accuracy rate when using the tfidfBOW features  One possible explanation for this is that TF-IDF weights are useful when rare words and very frequent words exist in the corpus, giving higher weights for rare words and lower weights for common words  Since we removed the stop words, the most frequent words among the training documents were not considered and, therefore, using TF-IDF weights did not improve the results 
Only MNB with tfBOW performed better than MSM  This result may be because tfBOW features encode the word frequencies, while MSM only considers a single occurrence of words  When incorporating the word frequencies with our TF weighted word subspace, we achieved a higher accuracy of 92 01%, performing better than MNB at a significance level of 95% 
Conclusions and Future Work
In this paper, we proposed a new method for text classification, based on the novel concept of word subspace under the MSM framework  We also proposed the term-frequency weighted word subspace which can incorporate the frequency of words directly in the modeling of the subspace by using a weighted version of PCA 
Most of the conventional text classification methods are based on the bag-of-words features, which are very simple to compute and had been proved to produce positive results  However, bag-of-words are commonly high dimensional models, with a sparse representation, which is computationally heavy to model  Also, bag-of-words fail to convey the semantic meaning of words inside a text  Due to these problems, neural networks started to be applied to generate a vector representation of words  Despite the fact that these representations can encode the semantic meaning of words, conventional methods do not work well when considering word vectors separately 
In our work, we focused on the word2vec representation, which can embed the semantic structure of words, rendering vector angles as a useful metric to show meaningful similarities between words  Our experiments showed that our word subspace modeling along with the MSM outperforms most of the conventional methods  Ultimately, our TF weighted subspace formulation resulted in significantly higher accuracy when compared to all conventional text classification methods discussed in this work  It is important to note that our method does not consider the order of the words in a text, resulting in a loss of context information  As a future work, we wish to extend our word subspace concept further in mainly two directions  First, we seek to encode word order, which may enrich the representation of context information  Second, we wish to model dynamic context change, enabling analysis of large documents, by having a long-short memory to interpret information using cues from different parts of a text 
Acknowledgment
This work is supported by JSPS KAKENHI Grant Number JP16H02842 and the Japanese Ministry of Education, Culture, Sports, Science, and Technology (MEXT) scholarship 

ground_truth: Word vectors, usually in the context of others within the same class