prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the languages they use in their experiment?
Context 1: Introduction
While most NLP resources are English-specific, there have been several recent efforts to build multilingual benchmarks  One possibility is to collect and annotate data in multiple languages separately BIBREF0, but most existing datasets have been created through translation BIBREF1, BIBREF2  This approach has two desirable properties: it relies on existing professional translation services rather than requiring expertise in multiple languages, and it results in parallel evaluation sets that offer a meaningful measure of the cross-lingual transfer gap of different models  The resulting multilingual datasets are generally used for evaluation only, relying on existing English datasets for training 
Closely related to that, cross-lingual transfer learning aims to leverage large datasets available in one language—typically English—to build multilingual models that can generalize to other languages  Previous work has explored 3 main approaches to that end: machine translating the test set into English and using a monolingual English model (Translate-Test), machine translating the training set into each target language and training the models on their respective languages (Translate-Train), or using English data to fine-tune a multilingual model that is then transferred to the rest of languages (Zero-Shot) 
The dataset creation and transfer procedures described above result in a mixture of original, human translated and machine translated data when dealing with cross-lingual models  In fact, the type of text a system is trained on does not typically match the type of text it is exposed to at test time: Translate-Test systems are trained on original data and evaluated on machine translated test sets, Zero-Shot systems are trained on original data and evaluated on human translated test sets, and Translate-Train systems are trained on machine translated data and evaluated on human translated test sets 
Despite overlooked to date, we show that such mismatch has a notable impact in the performance of existing cross-lingual models  By using back-translation BIBREF3 to paraphrase each training instance, we obtain another English version of the training set that better resembles the test set, obtaining substantial improvements for the Translate-Test and Zero-Shot approaches in cross-lingual Natural Language Inference (NLI)  While improvements brought by machine translation have previously been attributed to data augmentation BIBREF4, we reject this hypothesis and show that the phenomenon is only present in translated test sets, but not in original ones  Instead, our analysis reveals that this behavior is caused by subtle artifacts arising from the translation process itself  In particular, we show that translating different parts of each instance separately (e g  the premise and the hypothesis in NLI) can alter superficial patterns in the data (e g  the degree of lexical overlap between them), which severely affects the generalization ability of current models  Based on the gained insights, we improve the state-of-the-art in XNLI, and show that some previous findings need to be reconsidered in the light of this phenomenon 
Related work ::: Cross-lingual transfer learning 
Current cross-lingual models work by pre-training multilingual representations using some form of language modeling, which are then fine-tuned on the relevant task and transferred to different languages  Some authors leverage parallel data to that end BIBREF5, BIBREF6, but training a model akin to BERT BIBREF7 on the combination of monolingual corpora in multiple languages is also effective BIBREF8  Closely related to our work, BIBREF4 showed that replacing segments of the training data with their translation during fine-tuning is helpful  However, they attribute this behavior to a data augmentation effect, which we believe should be reconsidered given the new evidence we provide 
Related work ::: Multilingual benchmarks 
Most benchmarks covering a wide set of languages have been created through translation, as it is the case of XNLI BIBREF1 for NLI, PAWS-X BIBREF9 for adversarial paraphrase identification, and XQuAD BIBREF2 and MLQA BIBREF10 for Question Answering (QA)  A notable exception is TyDi QA BIBREF0, a contemporaneous QA dataset that was separately annotated in 11 languages  Other cross-lingual datasets leverage existing multilingual resources, as it is the case of MLDoc BIBREF11 for document classification and Wikiann BIBREF12 for named entity recognition  Concurrent to our work, BIBREF13 combine some of these datasets into a single multilingual benchmark, and evaluate some well-known methods on it 
Related work ::: Annotation artifacts 
Several studies have shown that NLI datasets like SNLI BIBREF14 and MultiNLI BIBREF15 contain spurious patterns that can be exploited to obtain strong results without making real inferential decisions  For instance, BIBREF16 and BIBREF17 showed that a hypothesis-only baseline performs better than chance due to cues on their lexical choice and sentence length  Similarly, BIBREF18 showed that NLI models tend to predict entailment for sentence pairs with a high lexical overlap  Several authors have worked on adversarial datasets to diagnose these issues and provide a more challenging benchmark BIBREF19, BIBREF20, BIBREF21  Besides NLI, other tasks like QA have also been found to be susceptible to annotation artifacts BIBREF22, BIBREF23  While previous work has focused on the monolingual scenario, we show that translation can interfere with these artifacts in multilingual settings 
Related work ::: Translationese 
Translated texts are known to have unique features like simplification, explicitation, normalization and interference, which are refer to as translationese BIBREF24  This phenomenon has been reported to have a notable impact in machine translation evaluation BIBREF25, BIBREF26  For instance, back-translation brings large BLEU gains for reversed test sets (i e  when translationese is on the source side and original text is used as reference), but its effect diminishes in the natural direction BIBREF27  While connected, the phenomenon we analyze is different in that it arises from translation inconsistencies due to the lack of context, and affects cross-lingual transfer learning rather than machine translation 
Experimental design
Our goal is to analyze the effect of both human and machine translation in cross-lingual models  For that purpose, the core idea of our work is to (i) use machine translation to either translate the training set into other languages, or generate English paraphrases of it through back-translation, and (ii) evaluate the resulting systems on original, human translated and machine translated test sets in comparison with systems trained on original data  We next describe the models used in our experiments (§SECREF6), the specific training variants explored (§SECREF8), and the evaluation procedure followed (§SECREF10) 
Experimental design ::: Models and transfer methods
We experiment with two models that are representative of the state-of-the-art in monolingual and cross-lingual pre-training: (i) Roberta BIBREF28, which is an improved version of BERT that uses masked language modeling to pre-train an English Transformer model, and (ii) XLM-R BIBREF8, which is a multilingual extension of the former pre-trained on 100 languages  In both cases, we use the large models released by the authors under the fairseq repository  As discussed next, we explore different variants of the training set to fine-tune each model on different tasks  At test time, we try both machine translating the test set into English (Translate-Test) and, in the case of XLM-R, using the actual test set in the target language (Zero-Shot) 
Experimental design ::: Training variants
We try 3 variants of each training set to fine-tune our models: (i) the original one in English (Orig), (ii) an English paraphrase of it generated through back-translation using Spanish or Finnish as pivot (BT-ES and BT-FI), and (iii) a machine translated version in Spanish or Finnish (MT-ES and MT-FI)  For sentences occurring multiple times in the training set (e g  premises repeated for multiple hypotheses), we use the exact same translation for all occurrences, as our goal is to understand the inherent effect of translation rather than its potential application as a data augmentation method 
In order to train the machine translation systems for MT-XX and BT-XX, we use the big Transformer model BIBREF29 with the same settings as BIBREF30 and SentencePiece tokenization BIBREF31 with a joint vocabulary of 32k subwords  For English-Spanish, we train for 10 epochs on all parallel data from WMT 2013 BIBREF32 and ParaCrawl v5 0 BIBREF33  For English-Finnish, we train for 40 epochs on Europarl and Wiki Titles from WMT 2019 BIBREF34, ParaCrawl v5 0, and DGT, EUbookshop and TildeMODEL from OPUS BIBREF35  In both cases, we remove sentences longer than 250 tokens, with a source/target ratio exceeding 1 5, or for which langid py BIBREF36 predicts a different language, resulting in a final corpus size of 48M and 7M sentence pairs, respectively  We use sampling decoding with a temperature of 0 5 for inference, which produces more diverse translations than beam search BIBREF37 and performed better in our preliminary experiments 
Experimental design ::: Tasks and evaluation procedure
We use the following tasks for our experiments:
Experimental design ::: Tasks and evaluation procedure ::: Natural Language Inference (NLI) 
Given a premise and a hypothesis, the task is to determine whether there is an entailment, neutral or contradiction relation between them  We fine-tune our models on MultiNLI BIBREF15 for 10 epochs using the same settings as BIBREF28  In most of our experiments, we evaluate on XNLI BIBREF1, which comprises 2490 development and 5010 test instances in 15 languages  These were originally annotated in English, and the resulting premises and hypotheses were independently translated into the rest of the languages by professional translators  For the Translate-Test approach, we use the machine translated versions from the authors  Following BIBREF8, we select the best epoch checkpoint according to the average accuracy in the development set 
Experimental design ::: Tasks and evaluation procedure ::: Question Answering (QA) 
Given a context paragraph and a question, the task is to identify the span answering the question in the context  We fine-tune our models on SQuAD v1 1 BIBREF38 for 2 epochs using the same settings as BIBREF28, and report test results for the last epoch  We use two datasets for evaluation: XQuAD BIBREF2, a subset of the SQuAD development set translated into 10 other languages, and MLQA BIBREF10 a dataset consisting of parallel context paragraphs plus the corresponding questions annotated in English and translated into 6 other languages  In both cases, the translation was done by professional translators at the document level (i e  when translating a question, the text answering it was also shown)  For our BT-XX and MT-XX variants, we translate the context paragraph and the questions independently, and map the answer spans using the same procedure as BIBREF39  For the Translate-Test approach, we use the official machine translated versions of MLQA, run inference over them, and map the predicted answer spans back to the target language 
Both for NLI and QA, we run each system 5 times with different random seeds and report the average results  Space permitting, we also report the standard deviation across the 5 runs 
NLI experiments
We next discuss our main results in the XNLI development set (§SECREF15, §SECREF16), run additional experiments to better understand the behavior of our different variants (§SECREF17, §SECREF22, §SECREF25), and compare our results to previous work in the XNLI test set (§SECREF30) 
NLI experiments ::: Translate-Test results
We start by analyzing XNLI development results for Translate-Test  Recall that, in this approach, the test set is machine translated into English, but training is typically done on original English data  Our BT-ES and BT-FI variants close this gap by training on a machine translated English version of the training set generated through back-translation  As shown in Table TABREF9, this brings substantial gains for both Roberta and XLM-R, with an average improvement of 4 6 points in the best case  Quite remarkably, MT-ES and MT-FI also outperform Orig by a substantial margin, and are only 0 8 points below their BT-ES and BT-FI counterparts  Recall that, for these two systems, training is done in machine translated Spanish or Finnish, while inference is done in machine translated English  This shows that the loss of performance when generalizing from original data to machine translated data is substantially larger than the loss of performance when generalizing from one language to another 
NLI experiments ::: Zero-Shot results
We next analyze the results for the Zero-Shot approach  In this case, inference is done in the test set in each target language which, in the case of XNLI, was human translated from English  As such, different from the Translate-Test approach, neither training on original data (Orig) nor training on machine translated data (BT-XX and MT-XX) makes use of the exact same type of text that the system is exposed to at test time  However, as shown in Table TABREF9, both BT-XX and MT-XX outperform Orig by approximately 2 points, which suggests that our (back-)translated versions of the training set are more similar to the human translated test sets than the original one  This also provides a new perspective on the Translate-Train approach, which was reported to outperform Orig in previous work BIBREF5: while the original motivation was to train the model on the same language that it is tested on, our results show that machine translating the training set is beneficial even when the target language is different 
NLI experiments ::: Original vs  translated test sets
So as to understand whether the improvements observed so far are limited to translated test sets or apply more generally, we conduct additional experiments comparing translated test sets to original ones  However, to the best of our knowledge, all existing non-English NLI benchmarks were created through translation  For that reason, we build a new test set that mimics XNLI, but is annotated in Spanish rather than English  We first collect the premises from a filtered version of CommonCrawl BIBREF42, taking a subset of 5 websites that represent a diverse set of genres: a newspaper, an economy forum, a celebrity magazine, a literature blog, and a consumer magazine  We then ask native Spanish annotators to generate an entailment, a neutral and a contradiction hypothesis for each premise  We collect a total of 2490 examples using this procedure, which is the same size as the XNLI development set  Finally, we create a human translated and a machine translated English version of the dataset using professional translators from Gengo and our machine translation system described in §SECREF8, respectively  We report results for the best epoch checkpoint on each set 
As shown in Table TABREF18, both BT-XX and MT-XX clearly outperform Orig in all test sets created through translation, which is consistent with our previous results  In contrast, the best results on the original English set are obtained by Orig, and neither BT-XX nor MT-XX obtain any clear improvement on the one in Spanish either  This confirms that the underlying phenomenon is limited to translated test sets  In addition, it is worth mentioning that the results for the machine translated test set in English are slightly better than those for the human translated one, which suggests that the difficulty of the task does not only depend on the translation quality  Finally, it is also interesting that MT-ES is only marginally better than MT-FI in both Spanish test sets, even if it corresponds to the Translate-Train approach, whereas MT-FI needs to Zero-Shot transfer from Finnish into Spanish  This reinforces the idea that it is training on translated data rather than training on the target language that is key in Translate-Train 
NLI experiments ::: Stress tests
In order to better understand how systems trained on original and translated data differ, we run additional experiments on the NLI Stress Tests BIBREF19, which were designed to test the robustness of NLI models to specific linguistic phenomena in English  The benchmark consists of a competence test, which evaluates the ability to understand antonymy relation and perform numerical reasoning, a distraction test, which evaluates the robustness to shallow patterns like lexical overlap and the presence of negation words, and a noise test, which evaluates robustness to spelling errors  Just as with previous experiments, we report results for the best epoch checkpoint in each test set 
As shown in Table TABREF23, Orig outperforms BT-FI and MT-FI on the competence test by a large margin, but the opposite is true on the distraction test  In particular, our results show that BT-FI and MT-FI are less reliant on lexical overlap and the presence of negative words  This feels intuitive, as translating the premise and hypothesis independently—as BT-FI and MT-FI do—is likely to reduce the lexical overlap between them  More generally, the translation process can alter similar superficial patterns in the data, which NLI models are sensitive to (§SECREF2)  This would explain why the resulting models have a different behavior on different stress tests 
NLI experiments ::: Output class distribution
With the aim to understand the effect of the previous phenomenon in cross-lingual settings, we look at the output class distribution of our different models in the XNLI development set  As shown in Table TABREF28, the predictions of all systems are close to the true class distribution in the case of English  Nevertheless, Orig is strongly biased for the rest of languages, and tends to underpredict entailment and overpredict neutral  This can again be attributed to the fact that the English test set is original, whereas the rest are human translated  In particular, it is well-known that NLI models tend to predict entailment when there is a high lexical overlap between the premise and the hypothesis (§SECREF2)  However, the degree of overlap will be smaller in the human translated test sets given that the premise and the hypothesis were translated independently, which explains why entailment is underpredicted  In contrast, BT-FI and MT-FI are exposed to the exact same phenomenon during training, which explains why they are not that heavily affected 
So as to measure the impact of this phenomenon, we explore a simple approach to correct this bias: having fine-tuned each model, we adjust the bias term added to the logit of each class so the model predictions match the true class distribution for each language  As shown in Table TABREF29, this brings large improvements for Orig, but is less effective for BT-FI and MT-FI  This shows that the performance of Orig was considerably hindered by this bias, which BT-FI and MT-FI effectively mitigate 
NLI experiments ::: Comparison with the state-of-the-art
So as to put our results into perspective, we compare our best variant to previous work on the XNLI test set  As shown in Table TABREF31, our method improves the state-of-the-art for both the Translate-Test and the Zero-Shot approaches by 4 3 and 2 8 points, respectively  It also obtains the best overall results published to date, with the additional advantage that the previous state-of-the-art required a machine translation system between English and each of the 14 target languages, whereas our method uses a single machine translation system between English and Finnish (which is not one of the target languages)  While the main goal of our work is not to design better cross-lingual models, but to analyze their behavior in connection to translation, this shows that the phenomenon under study is highly relevant, to the extent that it can be exploited to improve the state-of-the-art 
QA experiments
So as to understand whether our previous findings apply to other tasks besides NLI, we run additional experiments on QA  As shown in Table TABREF32, BT-FI and BT-ES do indeed outperform Orig for the Translate-Test approach on MLQA  The improvement is modest, but very consistent across different languages, models and runs  The results for MT-ES and MT-FI are less conclusive, presumably because mapping the answer spans across languages might introduce some noise  In contrast, we do not observe any clear improvement for the Zero-Shot approach on this dataset  Our XQuAD results in Table TABREF33 are more positive, but still inconclusive 
These results can partly be explained by the translation procedure used to create the different benchmarks: the premises and hypotheses of XNLI were translated independently, whereas the questions and context paragraphs of XQuAD were translated together  Similarly, MLQA made use of parallel contexts, and translators were shown the sentence containing each answer when translating the corresponding question  As a result, one can expect both QA benchmarks to have more consistent translations than XNLI, which would in turn diminish this phenomenon  In contrast, the questions and context paragraphs are independently translated when using machine translation, which explains why BT-ES and BT-FI outperform Orig for the Translate-Test approach  We conclude that the translation artifacts revealed by our analysis are not exclusive to NLI, as they also show up on QA for the Translate-Test approach, but their actual impact can be highly dependent on the translation procedure used and the nature of the task 
Discussion
Our analysis prompts to reconsider previous findings in cross-lingual transfer learning as follows:
Discussion ::: The cross-lingual transfer gap on XNLI was overestimated 
Given the parallel nature of XNLI, accuracy differences across languages are commonly interpreted as the loss of performance when generalizing from English to the rest of languages  However, our work shows that there is another factor that can have a much larger impact: the loss of performance when generalizing from original to translated data  Our results suggest that the real cross-lingual generalization ability of XLM-R is considerably better than what the accuracy numbers in XNLI reflect 
Discussion ::: Overcoming the cross-lingual gap is not what makes Translate-Train work 
The original motivation for Translate-Train was to train the model on the same language it is tested on  However, we show that it is training on translated data, rather than training on the target language, that is key for this approach to outperform Zero-Shot as reported by previous authors 
Discussion ::: Improvements previously attributed to data augmentation should be reconsidered 
The method by BIBREF4 combines machine translated premises and hypotheses in different languages (§SECREF2), resulting in an effect similar to BT-XX and MT-XX  As such, we believe that this method should be analyzed from the point of view of dataset artifacts rather than data augmentation, as the authors do  From this perspective, having the premise and the hypotheses in different languages can reduce the superficial patterns between them, which would explain why this approach is better than using examples in a single language 
Discussion ::: The potential of Translate-Test was underestimated 
The previous best results for Translate-Test on XNLI lagged behind the state-of-the-art by 4 6 points  Our work reduces this gap to only 0 8 points by addressing the underlying translation artifacts  The reason why Translate-Test is more severely affected by this phenomenon is twofold: (i) the effect is doubled by first using human translation to create the test set and then machine translation to translate it back to English, and (ii) Translate-Train was inadvertently mitigating this issue (see above), but equivalent techniques were never applied to Translate-Test 
Discussion ::: Future evaluation should better account for translation artifacts 
The evaluation issues raised by our analysis do not have a simple solution  In fact, while we use the term translation artifacts to highlight that they are an unintended effect of translation that impacts final evaluation, one could also argue that it is the original datasets that contain the artifacts, which translation simply alters or even mitigates  In any case, this is a more general issue that falls beyond the scope of cross-lingual transfer learning, so we argue that it should be carefully controlled when evaluating cross-lingual models  In the absence of more robust datasets, we recommend that future multilingual benchmarks should at least provide consistent test sets for English and the rest of languages  This can be achieved by (i) using original annotations in all languages, (ii) using original annotations in a non-English language and translating them into English and other languages, or (iii) if translating from English, doing so at the document level to minimize translation inconsistencies 
Conclusions
In this paper, we have shown that both human and machine translation can alter superficial patterns in data, which requires reconsidering previous findings in cross-lingual transfer learning  Based on the gained insights, we have improved the state-of-the-art in XNLI for the Translate-Test and Zero-Shot approaches by a substantial margin  Finally, we have shown that the phenomenon is not specific to NLI but also affects QA, although it is less pronounced there thanks to the translation procedure used in the corresponding benchmarks  So as to facilitate similar studies in the future, we release our NLI dataset, which, unlike previous benchmarks, was annotated in a non-English language and human translated into English 
Acknowledgments
We thank Nora Aranberri and Uxoa Iñurrieta for helpful discussion during the development of this work, as well as the rest of our colleagues from the IXA group that worked as annotators for our NLI dataset 
This research was partially funded by a Facebook Fellowship, the Basque Government excellence research group (IT1343-19), the Spanish MINECO (UnsupMT TIN2017‐91692‐EXP MCIU/AEI/FEDER, UE), Project BigKnowledge (Ayudas Fundación BBVA a equipos de investigación científica 2018), and the NVIDIA GPU grant program 

ground_truth: English
French
Spanish
German
Greek
Bulgarian
Russian
Turkish
Arabic
Vietnamese
Thai
Chinese
Hindi
Swahili
Urdu
Finnish

English, Spanish, Finnish

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was their highest MRR score?
Context 2: Introduction
BioASQ is a biomedical document classification, document retrieval, and question answering competition, currently in its seventh year  We provide an overview of our submissions to semantic question answering task (7b, Phase B) of BioASQ 7 (except for 'ideal answer' test, in which we did not participate this year)  In this task systems are provided with biomedical questions and are required to submit ideal and exact answers to those questions  We have used BioBERT BIBREF0 based system , see also Bidirectional Encoder Representations from Transformers(BERT) BIBREF1, and we fine tuned it for the biomedical question answering task  Our system scored near the top for factoid questions for all the batches of the challenge  More specifially, in the third test batch set, our system achieved highest ‘MRR’ score for Factoid Question Answering task  Also, for List-type question answering task our system achieved highest recall score in the fourth test batch set  Along with our detailed approach, we present the results for our submissions and also highlight identified downsides for our current approach and ways to improve them in our future experiments  In last test batch results we placed 4th for List-type questions and 3rd for Factoid-type questions )
The QA task is organized in two phases  Phase A deals with retrieval of the relevant document, snippets, concepts, and RDF triples, and phase B deals with exact and ideal answer generations (which is a paragraph size summary of snippets)  Exact answer generation is required for factoid, list, and yes/no type question 
BioASQ organizers provide the training and testing data  The training data consists of questions, gold standard documents, snippets, concepts, and ideal answers (which we did not use in this paper, but we used last year BIBREF2)  The test data is split between phases A and B  The Phase A dataset consists of the questions, unique ids, question types  The Phase B dataset consists of the questions, golden standard documents, snippets, unique ids and question types  Exact answers for factoid type questions are evaluated using strict accuracy (the top answer), lenient accuracy (the top 5 answers), and MRR (Mean Reciprocal Rank) which takes into account the ranks of returned answers  Answers for the list type question are evaluated using precision, recall, and F-measure 
Related Work ::: BioAsq
Sharma et al  BIBREF3 describe a system with two stage process for factoid and list type question answering  Their system extracts relevant entities and then runs supervised classifier to rank the entities  Wiese et al  BIBREF4 propose neural network based model for Factoid and List-type question answering task  The model is based on Fast QA and predicts the answer span in the passage for a given question  The model is trained on SQuAD data set and fine tuned on the BioASQ data  Dimitriadis et al  BIBREF5 proposed two stage process for Factoid question answering task  Their system uses general purpose tools such as Metamap, BeCas to identify candidate sentences  These candidate sentences are represented in the form of features, and are then ranked by the binary classifier  Classifier is trained on candidate sentences extracted from relevant questions, snippets and correct answers from BioASQ challenge  For factoid question answering task highest ‘MRR’ achieved in the 6th edition of BioASQ competition is ‘0 4325’  Our system is a neural network model based on contextual word embeddings BIBREF1 and achieved a ‘MRR’ score ‘0 6103’ in one of the test batches for Factoid Question Answering task 
Related Work ::: A minimum background on BERT
BERT stands for "Bidirectional Encoder Representations from Transformers" BIBREF1 is a contextual word embedding model  Given a sentence as an input, contextual embedding for the words are returned  The BERT model was designed so it can be fine tuned for 11 different tasks BIBREF1, including question answering tasks  For a question answering task, question and paragraph (context) are given as an input  A BERT standard is that question text and paragraph text are separated by a separator [Sep]  BERT question-answering fine tuning involves adding softmax layer  Softmax layer takes contextual word embeddings from BERT as input and learns to identity answer span present in the paragraph (context)  This process is represented in Figure FIGREF4 
BERT was originally trained to perform tasks such as language model creation using masked words and next-sentence-prediction  In other words BERT weights are learned such that context is used in building the representation of the word, not just as a loss function to help learn a context-independent representation  For detailed understanding of BERT Architecture, please refer to the original BERT paper BIBREF1 
Related Work ::: A minimum background on BERT ::: Comparison of Word Embeddings and Contextual Word Embeddings
A ‘word embedding’ is a learned representation  It is represented in the form of vector where words that have the same meaning have a similar vector representation  Consider a word embedding model 'word2vec' BIBREF6 trained on a corpus  Word embeddings generated from the model are context independent that is, word embeddings are returned regardless of where the words appear in a sentence and regardless of e g  the sentiment of the sentence  However, contextual word embedding models like BERT also takes context of the word into consideration 
Related Work ::: Comparison of BERT and Bio-BERT
‘BERT’ and BioBERT are very similar in terms of architecture  Difference is that ‘BERT’ is pretrained on Wikipedia articles, whereas BioBERT version used in our experiments is pretrained on Wikipedia, PMC and PubMed articles  Therefore BioBERT model is expected to perform well with biomedical text, in terms of generating contextual word embeddings 
BioBERT model used in our experiments is based on BERT-Base Architecture; BERT-Base has 12 transformer Layers where as BERT-Large has 24 transformer layers  Moreover contextual word embedding vector size is 768 for BERT-Base and more for BERT-large  According to BIBREF1 Bert-Large, fine-tuned on SQuAD 1 1 question answering data BIBREF7 can achieve F1 Score of 90 9 for Question Answering task where as BERT-Base Fine-tuned on the same SQuAD question answering BIBREF7 data could achieve F1 score of 88 5  One downside of the current version BioBERT is that word-piece vocabulary is the same as that of original BERT Model, as a result word-piece vocabulary does not include biomedical jargon  Lee et al  BIBREF0 created BioBERT, using the same pre-trained BERT released by Google, and hence in the word-piece vocabulary (vocab txt), as a result biomedical jargon is not included in word-piece vocabulary  Modifying word-piece vocabulary (vocab txt) at this stage would loose original compatibility with ‘BERT’, hence it is left unmodified 
In our future work we would like to build pre-trained ‘BERT’ model from scratch  We would pretrain the model with biomedical corpus (PubMed, ‘PMC’) and Wikipedia  Doing so would give us scope to create word piece vocabulary to include biomedical jargon and there are chances of model performing better with biomedical jargon being included in the word piece vocabulary  We will consider this scenario in the future, or wait for the next version of BioBERT 
Experiments: Factoid Question Answering Task
For Factoid Question Answering task, we fine tuned BioBERT BIBREF0 with question answering data and added new features  Fig  FIGREF4 shows the architecture of BioBERT fine tuned for question answering tasks: Input to BioBERT is word tokenized embeddings for question and the paragraph (Context)  As per the ‘BERT’ BIBREF1 standards, tokens ‘[CLS]’ and ‘[SEP]’ are appended to the tokenized input as illustrated in the figure  The resulting model has a softmax layer formed for predicting answer span indices in the given paragraph (Context)  On test data, the fine tuned model generates $n$-best predictions for each question  For a question, $n$-best corresponds that $n$ answers are returned as possible answers in the decreasing order of confidence  Variable $n$ is configurable  In our paper, any further mentions of ‘answer returned by the model’ correspond to the top answer returned by the model 
Experiments: Factoid Question Answering Task ::: Setup
BioASQ provides the training data  This data is based on previous BioASQ competitions  Train data we have considered is aggregate of all train data sets till the 5th version of BioASQ competition  We cleaned the data, that is, question-answering data without answers are removed and left with a total count of ‘530’ question answers  The data is split into train and test data in the ratio of 94 to 6; that is, count of '495' for training and '35' for testing 
The original data format is converted to the BERT/BioBERT format, where BioBERT expects ‘start_index’ of the actual answer  The ‘start_index corresponds to the index of the answer text present in the paragraph/ Context  For finding ‘start_index’ we used built-in python function find()  The function returns the lowest index of the actual answer present in the context(paragraph)  If the answer is not found ‘-1’ is returned as the index  The efficient way of finding start_index is, if the paragraph (Context) has multiple instances of answer text, then ‘start_index’ of the answer should be that instance of answer text whose context actually matches with what’s been asked in the question 
Example (Question, Answer and Paragraph from BIBREF8):
Question: Which drug should be used as an antidote in benzodiazepine overdose?
Answer: 'Flumazenil'
Paragraph(context):
"Flumazenil use in benzodiazepine overdose in the UK: a retrospective survey of NPIS data  OBJECTIVE: Benzodiazepine (BZD) overdose (OD) continues to cause significant morbidity and mortality in the UK  Flumazenil is an effective antidote but there is a risk of seizures, particularly in those who have co-ingested tricyclic antidepressants  A study was undertaken to examine the frequency of use, safety and efficacy of flumazenil in the management of BZD OD in the UK  METHODS: A 2-year retrospective cohort study was performed of all enquiries to the UK National Poisons Information Service involving BZD OD  RESULTS: Flumazenil was administered to 80 patients in 4504 BZD-related enquiries, 68 of whom did not have ventilatory failure or had recognised contraindications to flumazenil  Factors associated with flumazenil use were increased age, severe poisoning and ventilatory failure  Co-ingestion of tricyclic antidepressants and chronic obstructive pulmonary disease did not influence flumazenil administration  Seizure frequency in patients not treated with flumazenil was 0 3%" 
Actual answer is 'Flumazenil', but there are multiple instances of word 'Flu-mazenil'  Efficient way to identify the start-index for 'Flumazenil'(answer) is to find that particular instance of the word 'Flumazenil' which matches the context for the question  In the above example 'Flumazenil' highlighted in bold is the actual instance that matches question's context  Unfortunately, we could not identify readily available tools that can achieve this goal  In our future work, we look forward to handling these scenarios effectively 
Note: The creators of 'SQuAD' BIBREF7 have handled the task of identifying answer's start_index effectively  But 'SQuAD' data set is much more general and does not include biomedical question answering data 
Experiments: Factoid Question Answering Task ::: Training and error analysis
During our training with the BioASQ data, learning rate is set to 3e-5, as mentioned in the BioBERT paper BIBREF0  We started training the model with 495 available train data and 35 test data by setting number of epochs to 50  After training with these hyper-parameters training accuracy(exact match) was 99 3%(overfitting) and testing accuracy is only 4%  In the next iteration we reduced the number of epochs to 25 then training accuracy is reduced to 98 5% and test accuracy moved to 5%  We further reduced number of epochs to 15, and the resulting training accuracy was 70% and test accuracy 15%  In the next iteration set number of epochs to 12 and achieved train accuracy of 57 7% and test accuracy 23 3%  Repeated the experiment with 11 epochs and found training accuracy to be 57 7% and test accuracy to be same 22%  In the next iteration we set number of epochs to '9' and found training accuracy of 48% and test accuracy of 15%  Hence optimum number of epochs is taken as 12 epochs 
During our error analysis we found that on test data, model tends to return text in the beginning of the context(paragraph) as the answer  On analysing train data, we found that there are '120'(out of '495') question answering data instances having start_index:0, meaning 120( 25%) question answering data has first word(s) in the context(paragraph) as the answer  We removed 70% of those instances in order to make train data more balanced  In the new train data set we are left with '411' question answering data instances  This time we got the highest test accuracy of 26% at 11 epochs  We have submitted our results for BioASQ test batch-2, got strict accuracy of 32% and our system stood in 2nd place  Initially, hyper-parameter- 'batch size' is set to ‘400’  Later it is tuned to '32'  Although accuracy(exact answer match) remained at 26%, model generated concise and better answers at batch size ‘32’, that is wrong answers are close to the expected answer in good number of cases 
Example (from BIBREF8)
Question: Which mutated gene causes Chediak Higashi Syndrome?
Exact Answer: ‘lysosomal trafficking regulator gene’ 
The answer returned by a model trained at ‘400’ batch size is ‘Autosomal-recessive complicated spastic paraplegia with a novel lysosomal trafficking regulator’, and from the one trained at ‘32’ batch size is ‘lysosomal trafficking regulator’ 
In further experiments, we have fine tuned the BioBERT model with both ‘SQuAD’ dataset (version 2 0) and BioAsq train data  For training on ‘SQuAD’, hyper parameters- Learning rate and number of epochs are set to ‘3e-3’ and ‘3’ respectively as mentioned in the paper BIBREF1  Test accuracy of the model boosted to 44%  In one more experiment we trained model only on ‘SQuAD’ dataset, this time test accuracy of the model moved to 47%  The reason model did not perform up to the mark when trained with ‘SQuAD’ alongside BioASQ data could be that in formatted BioASQ data, start_index for the answer is not accurate, and affected the overall accuracy 
Our Systems and Their Performance on Factoid Questions
We have experimented with several systems and their variations, e g  created by training with specific additional features (see next subsection)  Here is their list and short descriptions  Unfortunately we did not pay attention to naming, and the systems evolved between test batches, so the overall picture can only be understood by looking at the details 
When we started the experiments our objective was to see whether BioBERT and entailment-based techniques can provide value for in the context of biomedical question answering  The answer to both questions was a yes, qualified by many examples clearly showing the limitations of both methods  Therefore we tried to address some of these limitations using feature engineering with mixed results: some clear errors got corrected and new errors got introduced, without overall improvement but convincing us that in future experiments it might be worth trying feature engineering again especially if more training data were available 
Overall we experimented with several approaches with the following aspects of the systems changing between batches, that is being absent or present:
training on BioAsq data vs  training on SQuAD
using the BioAsq snippets for context vs  using the documents from the provided URLs for context
adding or not the LAT, i e  lexical answer type, feature (see BIBREF9, BIBREF10 and an explanation in the subsection just below) 
For Yes/No questions (only) we experimented with the entailment methods 
We will discuss the performance of these models below and in Section 6  But before we do that, let us discuss a feature engineering experiment which eventually produced mixed results, but where we feel it is potentially useful in future experiments 
Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative)
During error analysis we found that for some cases, answer being returned by the model is far away from what it is being asked in the Question 
Example: (from BIBREF8)
Question: Hy's law measures failure of which organ?
Actual Answer: ‘Liver’ 
The answer returned by one of our models was ‘alanine aminotransferase’, which is an enzyme  The model returns an enzyme, when the question asked for the organ name  To address this type of errors, we decided to try the concepts of ‘Lexical Answer Type’ (LAT) and Focus Word, which was used in IBM Watson, see BIBREF11 for overview; BIBREF10 for technical details, and BIBREF9 for details on question analysis  In an example given in the last source we read:
POETS & POETRY: He was a bank clerk in the Yukon before he published "Songs of a Sourdough" in 1907 
The focus is the part of the question that is a reference to the answer  In the example above, the focus is "he" 
LATs are terms in the question that indicate what type of entity is being asked for 
The headword of the focus is generally a LAT, but questions often contain additional LATs, and in the Jeopardy! domain, categories are an additional source of LATs 
(   ) In the example, LATs are "he", "clerk", and "poet" 
For example in the question "Which plant does oleuropein originate from?" (BIBREF8)  The LAT here is ‘plant’  For the BioAsq task we did not need to explicitly distinguish between the focus and the LAT concepts  In this example, the expectation is that answer returned by the model is a plant  Thus it is conceivable that the cosine distance between contextual embedding of word 'plant' in the question and contextual embedding for the answer present in the paragraph(context) is comparatively low  As a result model learns to adjust its weights during training phase and returns answers with low cosine distance with the LAT 
We used Stanford CoreNLP BIBREF12 library to write rules for extracting lexical answer type present in the question, both 'parts of speech'(POS) and dependency parsing functionality was used  We incorporated the Lexical Answer Type into one of our systems, UNCC_QA1 in Batch 4  This system underperformed our system FACTOIDS by about 3% in the MRR measure, but corrected errors such as in the example above 
Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative) ::: Assumptions and rules for deriving lexical answer type 
There are different question types: ‘Which’, ‘What’, ‘When’, ‘How’ etc  Each type of question is being handled differently and there are commonalities among the rules written for different question types  Question words are identified through parts of speech tags: 'WDT', 'WRB' ,'WP'  We assumed that LAT is a ‘Noun’ and follows the question word  Often it was also a subject (nsubj)  This process is illustrated in Fig FIGREF15 
LAT computation was governed by a few simple rules, e g  when a question has multiple words that are 'Subjects’ (and ‘Noun’), a word that is in proximity to the question word is considered as ‘LAT’  These rules are different for each "Wh" word 
Namely, when the word immediately following the question word is a Noun, window size is set to ‘3’  The window size ‘3’ means we iterate through the next ‘3’ words to check if any of the word is both Noun and Subject, If so, such word is considered the ‘LAT’; else the word that is present very next to the question word is considered as the ‘LAT’ 
For questions with words ‘Which’ , ‘What’, ‘When’; a Noun immediately following the question word is very often the LAT, e g  'enzyme' in Which enzyme is targeted by Evolocumab?  When the word immediately following the question word is not a Noun, e g  in What is the function of the protein Magt1? the window size is set to ‘5’, and we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject  If present, the word is considered as the ‘LAT’; else, the Noun in close proximity to the question word and following it is returned as the ‘LAT’ 
For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself  For the word ‘How', e g  in How many selenoproteins are encoded in the human genome?, we look at the adjective and if we find one, we take it to be the LAT, otherwise the word 'How' is considered as the ‘LAT’ 
Perhaps because of using only very simple rules, the accuracy for ‘LAT’ derivation is 75%; that is, in the remaining 25% of the cases the LAT word is identified incorrectly  Worth noting is that the overall performance the system that used LATs was slightly inferior to the system without LATs, but the types of errors changed  The training used BioBERT with the LAT feature as part of the input string  The errors it introduces usually involve finding the wrong element of the correct type e g  wrong enzyme when two similar enzymes are described in the text, or 'neuron' when asked about a type of cell with a certain function, when the answer calls for a different cell category, adipocytes, and both are mentioned in the text  We feel with more data and additional tuning or perhaps using an ensemble model, we might be able to keep the correct answers, and improve the results on the confusing examples like the one mentioned above  Therefore if we improve our ‘LAT’ derivation logic, or have larger datasets, then perhaps the neural network techniques they will yield better results 
Our Systems and Their Performance on Factoid Questions ::: Impact of Training using BioAsq data (slightly negative)
Training on BioAsq data in our entry in Batch 1 and Batch 2 under the name QA1 showed it might lead to overfitting  This happened both with (Batch 2) and without (Batch 1) hyperparameters tuning: abysmal 18% MRR in Batch 1, and slighly better one, 40% in Batch 2 (although in Batch 2 it was overall the second best result in MRR but 16% lower than the highest score) 
In Batch 3 (only), our UNCC_QA3 system was fine tuned on BioAsq and SQuAD 2 0 BIBREF7, and for data preprocessing Context paragraph is generated from relevant snippets provided in the test data  This system underperformed, by about 2% in MRR, our other entry UNCC_QA1, which was also an overall category winner for this batch  The latter was also trained on SQuAD, but not on BioAsq  We suspect that the reason could be the simplistic nature of the find() function described in Section 3 1  So, this could be an area where a better algorithm for finding the best occurrence of an entity could improve performance 
Our Systems and Their Performance on Factoid Questions ::: Impact of Using Context from URLs (negative)
In some experiments, for context in testing, we used documents for which URL pointers are provided in BioAsq  However, our system UNCC_QA3 underperformed our other system tested only on the provided snippets 
In Batch 5 the underperformance was about 6% of MRR, compared to our best system UNCC_QA1, and by 9% to the top performer 
Performance on Yes/No and List questions
Our work focused on Factoid questions  But we also have done experiments on List-type and Yes/No questions 
Performance on Yes/No and List questions ::: Entailment improves Yes/No accuracy
We started by answering always YES (in batch 2 and 3) to get the baseline performance  For batch 4 we used entailment  Our algorithm was very simple: Given a question we iterate through the candidate sentences and try to find any candidate sentence is contradicting the question (with confidence over 50%), if so 'No' is returned as answer, else 'Yes' is returned  In batch 4 this strategy produced better than the BioAsq baseline performance, and compared to our other systems, the use of entailment increased the performance by about 13% (macro F1 score)  We used 'AllenNlp' BIBREF13 entailment library to find entailment of the candidate sentences with question 
Performance on Yes/No and List questions ::: For List-type the URLs have negative impact
Overall, we followed the similar strategy that's been followed for Factoid Question Answering task  We started our experiment with batch 2, where we submitted 20 best answers (with context from snippets)  Starting with batch 3, we performed post processing: once models generate answer predictions (n-best predictions), we do post-processing on the predicted answers  In test batch 4, our system (called FACTOIDS) achieved highest recall score of ‘0 7033’ but low precision of 0 1119, leaving open the question of how could we have better balanced the two measures 
In the post-processing phase, we take the top ‘20’ (batch 3) and top 5 (batch 4 and 5), predicted answers, tokenize them using common separators: 'comma' , 'and', 'also', 'as well as'  Tokens with characters count more than ‘100’ are eliminated and rest of the tokens are added to the list of possible answers  BioASQ evaluation mechanism does not consider snippets with more than ‘100’ characters as a valid answer  Considering lengthy snippets in to the list of answers would reduce the mean precision score  As a final step, duplicate snippets in the answer pool are removed  For example, consider these top 3 answers predicted by the system (before post-processing):
{
"text": "dendritic cells",
"probability": 0 7554540733426441,
"start_logit": 8 466046333312988,
"end_logit": 9 536355018615723
},
{
"text": "neutrophils, macrophages and
distinct subtypes of dendritic cells",
"probability": 0 13806867348304214,
"start_logit": 6 766478538513184,
"end_logit": 9 536355018615723
},
{
"text": "macrophages and distinct subtypes of dendritic",
"probability": 0 013973475271178242,
"start_logit": 6 766478538513184,
"end_logit": 7 24576473236084
},
After execution of post-processing heuristics, the list of answers returned is as follows:
["dendritic cells"],
["neutrophils"],
["macrophages"],
["distinct subtypes of dendritic cells"]
Summary of our results
The tables below summarize all our results  They show that the performance of our systems was mixed  The simple architectures and algorithm we used worked very well only in Batch 3  However, we feel we can built a better system based on this experience  In particular we observed both the value of contextual embeddings and of feature engineering (LAT), however we failed to combine them properly 
Summary of our results ::: Factoid questions ::: Systems used in Batch 5 experiments
System description for ‘UNCC_QA1’: The system was finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data 
System description for ‘QA1’ : ‘LAT’ feature was added and finetuned with SQuAD 2 0  For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data 
System description for ‘UNCC_QA3’ : Fine tuning process is same as it is done for the system ‘UNCC_QA1’ in test batch-5  Difference is during data preprocessing, Context/paragraph is generated from the relevant documents for which URLS are included in the test data 
Summary of our results ::: List Questions
For List-type questions, although post processing helped in the later batches, we never managed to obtain competitive precision, although our recall was good 
Summary of our results ::: Yes/No questions
The only thing worth remembering from our performance is that using entailment can have a measurable impact (at least with respect to a weak baseline)  The results (weak) are in Table 3 
Discussion, Future Experiments, and Conclusions ::: Summary:
In contrast to 2018, when we submitted BIBREF2 to BioASQ a system based on extractive summarization (and scored very high in the ideal answer category), this year we mainly targeted factoid question answering task and focused on experimenting with BioBERT  After these experiments we see the promise of BioBERT in QA tasks, but we also see its limitations  The latter we tried to address with mixed results using feature engineering  Overall these experiments allowed us to secure a best and a second best score in different test batches  Along with Factoid-type question, we also tried ‘Yes/No’ and ‘List’-type questions, and did reasonably well with our very simple approach 
For Yes/No the moral worth remembering is that reasoning has a potential to influence results, as evidenced by our adding the AllenNLP entailment BIBREF13 system increased its performance 
All our data and software is available at Github, in the previously referenced URL (end of Section 2) 
Discussion, Future Experiments, and Conclusions ::: Future experiments
In the current model, we have a shallow neural network with a softmax layer for predicting answer span  Shallow networks however are not good at generalizations  In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span  The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network  The mentioned dense layered question answering neural network need to be tuned for finding right hyper parameters  An example of such architecture is shown in Fig FIGREF30 
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network  By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy  Adding ‘LAT’ feature this way instead of feeding this word piece embedding directly to the BioBERT (as we did in our above experiments) would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'  Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy 
We also see potential for incorporating domain specific inference into the task e g  using the MedNLI dataset BIBREF14  For all types of experiments it might be worth exploring clinical BERT embeddings BIBREF15, explicitly incorporating domain knowledge (e g  BIBREF16) and possibly deeper discourse representations (e g  BIBREF17) 
APPENDIX
In this appendix we provide additional details about the implementations 
APPENDIX ::: Systems and their descriptions:
We used several variants of our systems when experimenting with the BioASQ problems  In retrospect, it would be much easier to understand the changes if we adopted some mnemonic conventions in naming the systems  So, we apologize for the names that do not reflect the modifications, and necessitate this list 
APPENDIX ::: Systems and their descriptions: ::: Factoid Type Question Answering:
We preprocessed the test data to convert test data to BioBERT format, We generated Context/paragraph by either aggregating relevant snippets provided or by aggregating documents for which URLS are provided in the BioASQ test data 
APPENDIX ::: Systems and their descriptions: ::: System description for QA1:
We generated Context/paragraph by aggregating relevant snippets available in the test data and mapped it against the question text and question id  We ignored the content present in the documents (document URLS were provided in the original test data)  The model is finetuned with BioASQ data 
data preprocessing is done in the same way as it is done for test batch-1  Model fine tuned on BioASQ data 
‘LAT’/ Focus word feature added and fine tuned with SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA_1:
System is finetuned on the SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
‘LAT’/ Focus word feature added and fine tuned with SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
The System is finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA3:
System is finetuned on the SQuAD 2 0 [reference] and BioASQ dataset[] For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
Fine tuning process is same as it is done for the system ‘UNCC_QA_1’ in test batch-5  Difference is during data preprocessing, Context/paragraph is generated form from the relevant documents for which URLS are included in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA2:
Fine tuning process is same as for ‘UNCC_QA_1 ’  Difference is Context/paragraph is generated form from the relevant documents for which URLS are included in the test data  System ‘UNCC_QA_1’ got the highest ‘MRR’ score in the 3rd test batch set 
APPENDIX ::: Systems and their descriptions: ::: System description for FACTOIDS:
The System is finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: List Type Questions:
We attempted List type questions starting from test batch ‘2’  Used similar approach that's been followed for Factoid Question answering task  For all the test batch sets, in the data pre processing phase Context/ paragraph is generated either by aggregating relevant snippets or by aggregating documents(URLS) provided in the BioASQ test data 
For test batch-2, model (System: QA1) is finetuned on BioASQ data and submitted top ‘20’ answers predicted by the model as the list of answers  system ‘QA1’ achieved low F-Measure score:‘0 0786’ in the second test batch  In the further test batches for List type questions, we finetuned the model on Squad data set [reference], implemented post processing techniques (refer section 5 2) and achieved a better F-measure score: ‘0 2862’ in the final test batch set 
In test batch-3 (Systems : ‘QA1’/’’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’) top 20 answers returned by the model is sent for post processing and in test batch 4 and 5 only top 5 answers are sent for post processing  System UNCC_QA2(in batch 3) for List type question answering, Context is generated from documents for which URLS are provided in the BioASQ test data  for the rest of the systems (in test batch-3) for List Type question answering task snippets present in the BioaSQ test data are used to generate context 
In test batch-4 (System : ‘FACTOIDS’/’UNCC_QA_1’/’UNCC_QA3’’) top 5 answers returned by the model is sent for post processing  In case of system ‘FACTOIDS’ snippets in the test data were used to generate context  for systems ’UNCC_QA_1’ and ’UNCC_QA3’ context is generated from the documents for which URLS are provided in the BioASQ test data 
In test batch-5 ( Systems: ‘QA1’/’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’ ) our approach is the same as that of test batch-4 where top 5 answers returned by the model is sent for post processing  for all the systems (in test batch-5) context is generated from the snippets provided in the BioASQ test data 
APPENDIX ::: Systems and their descriptions: ::: Yes/No Type Questions:
For the first 3 test batches, We have submitted answer ‘Yes’ to all the questions  Later, we employed ‘Sentence Entailment’ techniques(refer section 6 0) for the fourth and fifth test batch sets  Our Systems with ‘Sentence Entailment’ approach (for ‘Yes’/ ‘No’ question answering): ‘UNCC_QA_1’(test batch-4), UNCC_QA3(test batch-5) 
APPENDIX ::: Additional details for Yes/No Type Questions
We used Textual Entailment in Batch 4 and 5 for ‘Yes’/‘No’ question type  The algorithm was very simple: Given a question we iterate through the candidate sentences, and look for any candidate sentences contradicting the question  If we find one 'No' is returned as answer, else 'Yes' is returned  (The confidence for contradiction was set at 50%) We used AllenNLP BIBREF13 entailment library to find entailment of the candidate sentences with question 
Flow Chart for Yes/No Question answer processing is shown in Fig FIGREF51
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions
There are different question types, and we distinguished them based on the question words: ‘Which’, ‘What’, ‘When’, ‘How’ etc  Each type of question is being handled differently and there are commonalities among the rules written for different question types  How are question words identified? question words have parts of speech(POS): 'WDT', 'WRB', 'WP' 
Assumptions:
1) Lexical answer type (‘LAT’) or focus word is of type Noun and follows the question word 
2) The LAT word is a Subject  (This clearly not always true, but we used a very simple method)  Note: ‘StanfordNLP’ dependency parsing tag for identifying subject is 'nsubj' or 'nsubjpass' 
3) When a question has multiple words that are of type Subject (and Noun), a word that is in proximity to the question word is considered as ‘LAT’ 
4) For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself that is, ‘When’, ‘Who’, ‘Why’ respectively 
Rules and logic flow to traverse a question: The three cases below describe the logic flow of finding LATs  The figures show the grammatical structures used for this purpose 
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-1:
Question with question word ‘How’ 
For questions with question word 'How', the adjective that follows the question word is considered as ‘LAT’ (need not follow immediately)  If an adjective is absent, word 'How' is considered as ‘LAT’  When there are multiple words that are adjectives, a word in close proximity to the question word and follows it is returned as ‘LAT’  Note: The part of speech tag to identify adjectives is 'JJ'  For Other possible question words like ‘whose’  ‘LAT’/Focus word is question words itself 
Example Question: How many selenoproteins are encoded in the human genome?
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-2:
Questions with question words ‘Which’ , ‘What’ and all other possible question words; a 'Noun' immediately following the question word 
Example Question: Which enzyme is targeted by Evolocumab?
Here, Focus word/LAT is ‘enzyme’ which is both Noun and Subject and immediately follows the question word 
When the word immediately following the question word is a noun, the window size is set to ‘3’  This size ‘3’ means that we iterate through the next ‘3’ words (if present) to check if any of the word is both 'Noun' and 'Subject', If so, the word is considered as ‘LAT’/Focus Word  Else the word that is present very next to the question word is considered as ‘LAT’ 
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-3:
Questions with question words ‘Which’ , ‘What’ and all other possible question words; word immediately following the question word is not a 'Noun' 
Example Question: What is the function of the protein Magt1?
Here, Focus word/LAT is ‘function ’ which is both Noun and Subject and does not immediately follow the question word 
When the very next word following the question word is not a Noun, window size is set to ‘5’  Window size ‘5’ corresponds that we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject  If present, the word is considered as ‘LAT’  Else, the 'Noun' close proximity to the question word and follows it is returned as ‘LAT’ 
Ad we mentioned earlier, the accuracy for ‘LAT’ derivation is 75 percent  But clearly the simple logic described above can be improved, as shown in BIBREF9, BIBREF10  Whether this in turn produces improvements in this particular task is an open question 
APPENDIX ::: Proposing Future Experiments
In the current model, we have a shallow neural network with a softmax layer for predicting answer span  Shallow networks however are not good at generalizations  In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span  The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network  The mentioned dense layered question answering Neural network need to be tuned for finding right hyper parameters  An example of such architecture is shown in Fig FIGREF30 
In another experiment we would like to only feed contextual word embeddings for Focus word/ ‘LAT’, paragraph/ Context as input to the question answering neural network  In this experiment we would neglect all embeddings for the question text except that of Focus word/ ‘LAT’  Our assumption and idea for considering focus word and neglecting remaining words in the question is that during training phase it would make more precise for the model to identify the focus of the question and map answers against the question’s focus  To validate our assumption, we would like to take sample question answering data and find the cosine distance between contextual embedding of Focus word and that of the actual answer and verify if the cosine distance is comparatively low in most of the cases 
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network  By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy  Adding ‘LAT’ feature this way instead of feeding Focus word’s word piece embedding directly (as we did in our above experiments) to the BioBERT would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'  Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy 

ground_truth: 0.5115

0.6103

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. where did they obtain the annotated clinical notes from?
Context 3: Introduction
Medical search engines are an essential component for many online medical applications, such as online diagnosis systems and medical document databases  A typical online diagnosis system, for instance, relies on a medical search engine  The search engine takes as input a user query that describes some symptoms and then outputs clinical concept entries that provide relevant information to assist in diagnosing the problem  One challenge medical search engines face is the segmentation of individual clinical entities  When a user query consists of multiple clinical entities, a search engine would often fail to recognize them as separate entities  For example, the user query “fever joint pain weight loss headache” contains four separate clinical entities: “fever”, “joint pain”, “weight loss”, and “headache”  But when the search engine does not recognize them as separate entities and proceeds to retrieve results for each word in the query, it may find "pain" in body locations other than "joint pain", or it may miss "headache" altogether, for example  Some search engines allow the users to enter a single clinical concept by selecting from an auto-completion pick list  But this could also result in retrieving inaccurate or partial results and lead to poor user experience 
We want to improve the medical search engine so that it can accurately retrieve all the relevant clinical concepts mentioned in a user query, where relevant clinical concepts are defined with respect to the terminologies the search engine uses  The problem of extracting clinical concept mentions from a user query can be seen as a variant of the Concept Extraction (CE) task in the frequently-cited NLP challenges in healthcare, such as 2010 i2b2/VA BIBREF0 and 2013 ShARe/CLEF Task 1 BIBREF1  Both CE tasks in 2010 i2b2/VA and 2013 ShARe/CLEF Task 1 ask the participants to design an algorithm to tag a set of predefined entities of interest in clinical notes  These entity tagging tasks are also known as clinical Named Entity Recognition (NER)  For example, the CE task in 2010 i2b2/VA defines three types of entities: “problem”, “treatment”, and “test”  The CE task in 2013 ShARe/CLEF defines various types of disorder such as “injury or poisoning”, "disease or syndrome”, etc  In addition to tagging, the CE task in 2013 ShARe/CLEF has an encoding component which requires selecting one and only one Concept Unique Identifier (CUI) from Systematized Nomenclature Of Medicine Clinical Terms (SNOMED-CT) for each disorder entity tagged  Our problem, similar to the CE task in 2013 ShARe/CLEF, also contains two sub-problems: tagging mentions of entities of interest (entity tagging), and selecting appropriate terms from a glossary to match the mentions (term matching)  However, several major differences exist  First, compared to clinical notes, the user queries are much shorter, less technical, and often less coherent  Second, instead of encoding, we are dealing with term matching where we rank a few best terms that match an entity, instead of selecting only one  This is because the users who type the queries may not have a clear idea about what they are looking for, or could be laymen who know little terminology, it may be more helpful to provide a set of likely results and let the users choose  Third, the types of entities are different  Each medical search engine may have its own types of entities to tag  There is also one minor difference in the tagging scheme between our problem and the CE task in 2013 ShARe/CLEF - We limit our scope to dealing with entities of consecutive words and not disjoint entities   We use only Beginning, Inside, Outside (BIO) tags  Given the differences listed above, we need to customize a framework consisting of an entity tagging and term matching component for our CE problem 
Related Work
An effective model that has been commonly used for NER problem is a Bi-directional LSTM with a Conditional Random Field (CRF) on the top layer (BiLSTM-CRF), which is described in the next section  Combining LSTM’s power of representing relations between words and CRF’s capability of accounting for tag sequence constraints, Huang et al  BIBREF2 proposed the BiLSTM-CRF model and used handcrafted word features as the input to the model  Lample et al  BIBREF3 used a combination of character-level and word-level word embeddings as the input to BiLSTM-CRF  Since then, similar models with variation in types of word embeddings have been used extensively for clinical CE tasks and produced state-of-the-art results BIBREF4, BIBREF5, BIBREF6, BIBREF7  Word embeddings have become the cornerstone of the neural models in NLP since the famous Word2vec BIBREF8 model demonstrated its power in word analogy tasks  One well-known example is that after training Word2vec on a large amount of news data, we can get word relations such as $vector(^{\prime }king^{\prime }) - vector(^{\prime }queen^{\prime }) + vector(^{\prime }woman^{\prime }) \approx vector(^{\prime }man^{\prime })$  More sophisticated word embedding technique emerged since Word2vec  It has been shown empirically that better quality in word embeddings leads to better performance in many downstream NLP including entity tagging BIBREF9, BIBREF10  Recently, contextualized word embeddings generated by deep learning models, such as ELMo BIBREF11, BERT BIBREF12, and Flair BIBREF13, have been shown to be more effective in various NLP tasks  In our project, we make use of a fine-tuned ELMo model and a fine-tuned Flair model in the medical domain  We experiment with the word embeddings from the two fine-tuned models as the input to the BiLSTM-CRF model separately and compare the results 
Tang et al  BIBREF14 provided straightforward algorithm for term matching  The algorithm starts with finding candidate terms that contain ALL the entity words, with term frequency - inverse document frequency (tf-idf) weighting  Then the candidates are ranked based on the pairwise cosine distance between the word embeddings of the candidates and the entity 
Framework
We adopt the tagging - encoding pipeline framework from the CE task in 2013 ShARe/CLEF  We first tag the clinical entities in the user query and then select relevant terms from a glossary in dermatology to match the entities 
Framework ::: Entity Tagging
We use the same BiLSTM-CRF model proposed by Huang et al  BIBREF2  An illustration of the architecture is shown in Figure FIGREF6   Given a sequence (or sentence) of n tokens, $r = (w_1, w_2,   , w_n)$, we use a fine-tuned ELMo model to generate contextual word embeddings for all the tokens in the sentence, where a token refers to a word or punctuation  We denote the ELMo embedding, $x$, for a token $w$ in the sentence $r$ by $x = ELMo(w|r)$  The notation and the procedure described here can be adopted for Flair embeddings or other embeddings  Now, given a sequence of tokens in ELMo embeddings, $X =(x_1, x_2,    , x_n)$, the BiLSTM layer generates a matrix of scores, $P(\theta )$ of size $n \times k$, where $k$ is the number of tag types, and $\theta $ is the parameters of the BiLSTM  To simplify notation, we will omit the $\theta $ and write $P$  Then, $P_{i,j}$ denotes the score of the token, $x_i$, being assigned to the $j$th tag  Since certain constraints may exist in the transition between tags, an "O" tag should not be followed by an "I" tag, for example, a transition matrix, $A$, of dimension $(k+2)\times (k+2)$, is initialized to model the constraints  The learnable parameters, $A_{i,j}$, represent the probability of the $j$th tag follows the $i$th tag in a sequence  For example, if we index the tags by: 1:“B”, 2:“I”, and 3:“O”, then $A_{1,3}$ would be the probability that an “O” tag follows a “B” tag  A beginning transition and an end transition are inserted in $A$ and hence $A$ is of dimension $(k+2)\times (k+2)$ 
Given a sequence of tags, $Y=(y_1,y_2,   ,y_n)$, where each $y_i$, $1\le i \le n$, corresponds to an index of the tags, the score of the sequence is then given by
The probability of the sequence of tags is then calculated by a softmax,
where $\lbrace Y_x\rbrace $ denotes the set of all possible tag sequences  During training, the objective function is to maximize $\log (P(Y|X))$ by adjusting $A$ and $P$ 
Framework ::: Term Matching
The term matching algorithm of Tang et al  BIBREF14 is adopted with some major modifications  First, to identify candidate terms, we use a much looser string search algorithm where we stem the entity words with snowball stemmer and then find candidate terms which contain ANY non-stopword words in the entity  The stemming is mainly used to represent a word in its stem  For example, “legs” becomes “leg”, and “jammed” becomes “jam”  Thus, stemming can provide more tolerance when finding candidates  Similarly, finding candidates using the condition ANY (instead of ALL) also increases the tolerance  However, if the tagged entity contains stopwords such as “in”, “on”, etc , the pool of candidates will naturally grow very large and increase the computation cost for later part, because even completely irrelevant terms may contain stopwords  Therefore, we match based on non-stopwords only  To illustrate the points above, suppose a query is tagged with the entity
Ex  3 1 “severe burns on legs”,
and one relevant term is “leg burn”  After stemming, “burns” and “legs” in Ex UNKREF12 become “burn” and “leg”, respectively, allowing "leg burn" to be considered as a candidate  Although the word “severe” is not in the term “leg burn”, the term is still considered a candidate because we selected using ANY  The stopword “on” is ignored when finding candidate terms so that not every term that contains the word “on” is added to the candidate pool  When a candidate term, $C$, is found in this manner for the tagged entity, $E$, we calculate the semantic similarity score, $s$, between $C$ and $E$ in two steps  In the first step, calculate the maximum similarity score for each word in $C$ as shown in Figure FIGREF10  Given a word in the candidate term, $C_i$ ($1 \le i \le m$, $m$ is the number of words in the candidate term) and a word in the tagged entity, $E_j$ Their similarity score, $s_{ij}$ (shown as the element in the boxed matrix in Figure FIGREF10), is given by
where $ELMo(C_i|C)$ and $ELMo(E_j|E)$ are the ELMo embeddings for the word $C_i$ and $E_j$, respectively  The ELMo embeddings have the same dimension for all words when using the same fine-tuned ELMo model  Thus, we can use a distance function (e g , the cosine distance), denoted $d(\cdot )$ in equation DISPLAY_FORM13, to compute the semantic similarity between words  In step 2, we calculate the candidate-entity relevance score (similarity) using the formula
where $s_c$ is a score threshold, and $\mathbb {I} \lbrace max(\vec{S_i}) > s_c\rbrace $ is an indicator function that equals 1 if $max(\vec{S_i}) > s_c$ or equals 0 if not  In equation DISPLAY_FORM14 we define a metric that measures “information coverage” of the candidate terms with respect to a tagged entity  If the constituent words of a candidate term are relevant to the constituent words in the tagged entity, then the candidate term offers more information coverage  Intuitively, the more relevant words present in the candidate term, the more relevant the candidate is to the tagged entity  The purpose of the cutoff, $s_c$, is to screen the $(C_i,E_j)$ word pairs that are dissimilar, so that they do not contribute to information coverage  One can adjust the strictness of the entity - terminology matching by adjusting $s_c$  The higher we set $s_c$, the fewer candidate terms will be selected for a tagged entity  A normalization factor, $\frac{1}{m}$, is added to give preference to more concise candidate terms given the same amount of information coverage 
We need to create an extra stopword list to include words such as “configuration” and “color”, and exclude these words from the word count for a candidate term  This is because the terms associated with the description of color or configuration usually have the word “color” or “configuration” in them  On the other hand, a user query normally does not contain such words  For example, a tagged entity in a user query could be “round yellow patches”, for which the relevant terminologies include “round configuration” and “yellow color”  Since we applied a normalization factor, $\frac{1}{m}$, to the relevance score, the word “color” and “configuration” would lower the relevance score because they do not have a counterpart in the tagged entity  Therefore, we need to exclude them from word count  Once the process is complete, calculate $s(C,E)$ for all candidate terms and then we can apply a threshold on all $s(C,E)$ to ignore candidate terms with low information coverage  Finally, rank the terms by their $s(C,E)$ and return the ranked list as the results 
Experiments ::: Data
Despite the greater similarity between our task and the 2013 ShARe/CLEF Task 1, we use the clinical notes from the CE task in 2010 i2b2/VA on account of 1) the data from 2010 i2b2/VA being easier to access and parse, 2) 2013 ShARe/CLEF containing disjoint entities and hence requiring more complicated tagging schemes  The synthesized user queries are generated using the aforementioned dermatology glossary  Tagged sentences are extracted from the clinical notes  Sentences with no clinical entity present are ignored  22,489 tagged sentences are extracted from the clinical notes  We will refer to these tagged sentences interchangeably as the i2b2 data  The sentences are shuffled and split into train/dev/test set with a ratio of 7:2:1  The synthesized user queries are composed by randomly selecting several clinical terms from the dermatology glossary and then combining them in no particular order  When combining the clinical terms, we attach the BIO tags to their constituent words  The synthesized user queries (13,697 in total) are then split into train/dev/test set with the same ratio  Next, each set in the i2b2 data and the corresponding set in the synthesized query data are combined to form a hybrid train/dev/test set, respectively  This way we ensure that in each hybrid train/dev/test set, the ratio between the i2b2 data and the synthesized query data is the same 
The reason for combining the two data is their drastic structural difference (See figure FIGREF16 for an example)  Previously, when trained on the i2b2 data only, the BiLSTM-CRF model was not able to segment clinical entities at the correct boundary  It would fail to recognize the user query in Figure FIGREF16(a) as four separate entities  On the other hand, if the model was trained solely on the synthesized user queries, we could imagine that it would fail miserably on any queries that resemble the sentence in Figure FIGREF16(b) because the model would have never seen an “O” tag in the training data  Therefore, it is necessary to use the hybrid training data containing both the i2b2 data and the synthesized user queries 
To make the hybrid training data, we need to unify the tags  Recall that in Section SECREF1 we point out that the tags are different for the different tasks and datasets  Since we use custom tags for dermatology glossary in our problem, we would need to convert the tags used in 2010 i2b2/VA  But this would be an infeasible job as we need experts to manually do that  An alternative is to avoid distinguishing the tag types and label all tags under the generic BIO tags 
Experiments ::: Setup
To show the effects of using the hybrid training data, we trained two models of the same architecture and hyperparameters  One model was trained on the hybrid data and will be referred to as hybrid NER model  The other model was trained on clinical notes only and will be referred to as i2b2 NER model  We evaluated the performance of the NER models by micro-F1 score on the test set of both the synthesized queries and the i2b2 data 
We used the BiLSTM-CRF implementation provided by the flair package BIBREF16  We set the hidden size value to be 256 in the LSTM structure and left everything else at default values for the SequenceTagger model on flair  For word embeddings, we used the ELMo embeddings fine-tuned on PubMed articles and flair embeddings BIBREF13 trained on $5\%$ of PubMed abstracts , respectively  We trained models for 10 epochs and experimented with different learning rate, mini batch size, and dropouts  We ran hyperparameter optimization tests to find the best combination  $S_c$ is set to be 0 6 in our experiment 
Experiments ::: Hyperparameter Tuning
We defined the following hyperparameter search space:
embeddings: [“ELMo on pubmed”, “stacked flair on pubmed”],
hidden_size: [128, 256],
learning_rate: [0 05, 0 1],
mini_batch_size: [32, 64, 128] 
The hyperparameter optimization was performed using Hyperopt   Three evaluations were run for each combination of hyperparameters  Each ran for 10 epochs  Then the results were averaged to give the performance for that particular combination of hyperparameters 
Experiments ::: Results
From the hyperparameter tuning we found that the best combination was
embeddings: “ELMo on pubmed”,
hidden_size: 256,
learning_rate: 0 05,
mini_batch_size: 32 
With the above hyperparameter setting, the hybrid NER model achieved a F1 score of $0 995$ on synthesized queries and $0 948$ on clinical notes while the i2b2 NER model achieved a F1 score of $0 441$ on synthesized queries and $0 927$ on clinical notes (See Table TABREF23) 
Since there was no ground truth available for the retrieved terms, we randomly picked a few samples to assess its performance  Some example outputs of our complete framework on real user queries are shown in Figure FIGREF24  For example, from the figure we see that the query "child fever double vision dizzy" was correctly tagged with four entities: "child", "fever", "double vision", and "dizzy"  A list of terms from our glossary was matched to each entity  In real world application, the lists of terms will be presented to the user as the retrieval results to their queries 
Discussion
In most real user queries we sampled, the entities were tagged at the correct boundary and the tagging was complete (such as the ones shown in Figure FIGREF24)  Only on a few user queries the tagging was controversial  For example, the query “Erythematous blanching round, oval patches on torso, extremities” was tagged as “Erythematous blanching” and “oval patches on torso”  The entity “extremities” was missing  The segmentation was also not correct  A more appropriate tagging would be “Erythematous blanching round, oval patches”, “torso”, and “extremities”  The tagging could be further improved by synthesizing more realistic user queries  Recall that the synthesized user queries were created by randomly combining terminologies from the dermatology glossary, which, while providing data that helped the model learn entity segmentation, did not reflect the co-occurrence information in real user queries  For example, there could be two clinical entities that often co-occur or never co-occur in a user query  But since the synthesized user queries we used combined terms randomly, the co-occurrence information was thus missing 
The final retrieval results of our framework were not evaluated quantitatively in terms of recall and precision, due the the lack of ground truth  When ground truth becomes available, we will be able to evaluate our framework more thoroughly  Recently, a fine-tuned BERT model in the medical domain called BioBERT BIBREF17 has attracted some attention in the medical NLP domain  We could experiment with BioBERT embeddings in the future  We could also include query expansion technique for term matching  When finding candidate terms for an entity, our first step was still based on string matching  Given that there might be multiple entities that could be matched to the same term, it could be hard to include all these entities in the glossary and hard to match terms to these entities 
Conclusion
In this project, we tackle the problem of extracting clinical concepts from user queries on medical search engines  By training a BiLSTM-CRF model on a hybrid data consisting of synthesized user queries and sentences from clinical note, we adopt a CE framework for clinical user queries with minimal effort spent on annotating user queries  We find that the hybrid data enables the NER model perform better on both tagging the user queries and the clinical note sentences  Furthermore, our framework is built on an easy-to-use deep learning NLP Python library, which lends it more prospective value to various online medical applications that employ medical search engines 
Acknowledgment
This paper results from a technical report of a project the authors have worked on with visualDx, a healthcare informatics company that provides web-based clinical decision support system  The authors would like to thank visualDx for providing them the opportunity to work on such an exciting project  In particular, the authors would like to thank Roy Robinson, the Vice President of Technology and Medical Informatics at visualDx, for providing the synthesized user queries, as well as preliminary feedback on the performance of our framework 

ground_truth: clinical notes from the CE task in 2010 i2b2/VA

clinical notes from the CE task in 2010 i2b2/VA