Introduction
Question answering (QA) is the task of retrieving answers to a question given one or more contexts. It has been explored both in the open-domain setting BIBREF0 as well as domain-specific settings, such as BioASQ for the biomedical domain BIBREF1 . The BioASQ challenge provides $\approx 900$ factoid and list questions, i.e., questions with one and several answers, respectively. This work focuses on answering these questions, for example: Which drugs are included in the FEC-75 regimen? $\rightarrow $ fluorouracil, epirubicin, and cyclophosphamide.
We further restrict our focus to extractive QA, i.e., QA instances where the correct answers can be represented as spans in the contexts. Contexts are relevant documents which are provided by an information retrieval (IR) system.
Traditionally, a QA pipeline consists of named-entity recognition, question classification, and answer processing steps BIBREF2 . These methods have been applied to biomedical datasets, with moderate success BIBREF3 . The creation of large-scale, open-domain datasets such as SQuAD BIBREF4 have recently enabled the development of neural QA systems, e.g., wang2016machine, dcn, seo2016bidirectional, weissenborn2017fastqa, leading to impressive performance gains over more traditional systems.
However, creating large-scale QA datasets for more specific domains, such as the biomedical, would be very expensive because of the need for domain experts, and therefore not desirable. The recent success of deep learning based methods on open-domain QA datasets raises the question whether the capabilities of trained models are transferable to another domain via domain adaptation techniques. Although domain adaptation has been studied for traditional QA systems BIBREF5 and deep learning systems BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , it has to our knowledge not yet been applied for end-to-end neural QA systems.
To bridge this gap we employ various domain adaptation techniques to transfer knowledge from a trained, state-of-the-art neural QA system (FastQA, weissenborn2017fastqa) to the biomedical domain using the much smaller BioASQ dataset. In order to answer list questions in addition to factoid questions, we extend FastQA with a novel answering mechanism. We evaluate various transfer learning techniques comprehensively. For factoid questions, we show that mere fine-tuning reaches state-of-the-art results, which can further be improved by a forgetting cost regularization BIBREF9 . On list questions, the results are competitive to existing systems. Our manual analysis of a subset of the factoid questions suggests that the results are even better than the automatic evaluation states, revealing that many of the "incorrect" answers are in fact synonyms to the gold-standard answer.
Model
Our network architecture is based on FastQA BIBREF15 , a state-of-the-art neural QA system. Because the network architecture itself is exchangeable, we treat it as a black box, with subtle changes at the input and output layer as well as to the decoding and training procedure. These changes are described in the following. See Figure 1 for an overview of the system.
Input Layer
In a first step, words are embedded into a high-dimensional vector space. We use three sources of embeddings, which are concatenated to form a single embedding vector:
GloVe embeddings: 300-dimensional GloVe vectors BIBREF14 . These are open-domain word vectors trained on 840 billion tokens from web documents. The vectors are not updated during training.
Character embeddings: As used in FastQA BIBREF15 and proposed originally by seo2016bidirectional, we employ a 1-dimensional convolutional neural network which computes word embeddings from the characters of the word.
Biomedical Word2Vec embeddings: 200-dimensional vectors trained using Word2Vec BIBREF18 on about 10 million PubMed abstracts BIBREF19 . These vectors are specific to the biomedical domain and we expect them to help on biomedical QA.
As an optional step, we add entity tag features to the token embeddings via concatenation. Entity tags are provided by a dictionary-based entity tagger based on the UMLS Metathesaurus. The entity tag feature vector is a 127-dimensional bit vector that for each of the UMLS semantic types states whether the current token is part of an entity of that type. This step is only applied if explicitly noted.
Finally, a one-hot encoding of the question type (factoid or list) is appended to all the input vectors. With these embedding vectors as input, we invoke FastQA to produce start and end scores for each of the $n$ context tokens. We denote start scores by $y_{start}^{i}$ and end scores conditioned on a predicted start at position $i$ by $y_{end}^{i, j}$ , with start index $i \in [1, n]$ and end index $j \in [i, n]$ .
Output Layer
In our adapted output layer, we convert the start and end scores to span probabilities. The computation of these probabilities is independent of the question type. The interpretation, however, depends on the question type: While for factoid questions, the list of answer spans is interpreted as a ranked list of answer candidates, for list questions, answers above a certain probability threshold are interpreted as the set of answers to the question.
Given the start scores $y_{start}^1, ..., y_{start}^n$ and end scores $y_{end}^{i, 1}, ..., y_{end}^{i, n}$ , we compute the start and end probabilities as follows:
$$p_{start}^i = \sigma (y_{start}^i)$$   (Eq. 16)
$$p_{end}^{i, \cdot } = \operatorname{softmax}(y_{end}^{i, \cdot })$$   (Eq. 17)
where $\sigma (x)$ is the sigmoid function. As a consequence, multiple tokens can be chosen as likely start tokens, but the network is expected to select a single end token for a given start token, hence the $\operatorname{softmax}$ function. Finally, the probability that a given span $(i, j)$ answers the question is $p_{span}^{i, j} = p_{start}^{i} \cdot p_{end}^{i, j}$ . This extension generalizes the FastQA output layer such that multiple answer spans with different start positions can have a high probability, allowing us to retrieve multiple answers for list questions.
Decoding
Given a trained model, start probabilities can be obtained by running a forward pass and computing the start probability as in Equation 16 . For the top 20 starts, we compute the end probabilities as given by Eq. 17 . From the start and end probabilities, we extract the top 20 answer spans ranked by $p_{span}^{i, j}$ . As a simple post-processing step, we remove duplicate strings and retain only those with the highest probability.
For factoid questions, we output the 5 most likely answer spans as our ranked list of answers. For list questions, we learn a probability cutoff threshold $t$ that defines the set of list answers $A = \lbrace (i, j) | p_{span}^{i, j} \ge t\rbrace $ . We choose $t$ to be the threshold that optimizes the list F1 score on the respective development set.
Domain Adaptation
Our training procedure consists of two phases: In the pre-training phase, we train the model on SQuAD, using a token F1 score as the training objective as by weissenborn2017fastqa. We will refer to the resulting parameters as the base model. In the fine-tuning phase, we initialize the model parameters with the base model and then continue our optimization on the BioASQ dataset with a smaller learning rate.
To avoid catastrophic forgetting during fine-tuning as a means to regularize our model, we optionally add an additional forgetting cost term $L_{fc}$ , as proposed by riemer2017forgettingcost. It is defined as the cross-entropy loss between the current predictions and the base model's predictions.
We also add an L2 loss term $L_{l2}$ which penalizes deviations from the base model's parameters. Note that a more advanced approach would be to apply this loss selectively on weights which are particularly important in the source domain BIBREF10 . The final loss is computed as $L_{final} = L_{original} + C_{fc} \cdot L_{fc} + C_{l2} \cdot L_{l2}$ where $C_{fc}$ and $C_{l2}$ are hyperparameters which are set to 0 unless otherwise noted.
In this section, we evaluate various domain adaptation techniques. The results of the experiments are summarized in Table 1 .
As a baseline without transfer learning, Experiment 1 trains the model on BioASQ only. Because the BioASQ dataset by itself is very small, a dropout rate of $0.7$ was used, because it worked best in preliminary experiments. We observe a rather low performance, which is expected when applying deep learning to such a small dataset.
Experiments 2 and 3 evaluate the pure fine-tuning approach: Our base model is a system trained on SQuAD only and tested on BioASQ (Experiment 2). For Experiment 3, we fine-tuned the base model on the BioASQ4B training set. We observe that performance increases significantly, especially on list questions. This increase is expected, because the network is trained on biomedical- and list questions, which are not part of the SQuAD dataset, for the first time. Overall, the performance of the fine-tuned model on both question types is much higher than the baseline system without transfer learning.
In order to evaluate the impact of using biomedical word embeddings, we repeat Experiment 3 without them (Experiment 4). We see a factoid and list performance drop of $3.3$ and $1.2$ percentage points, respectively, showing that biomedical word embeddings help increase performance.
In Experiment 5, we append entity features to the word vector, as described in Section "Input Layer" . Even though these features provide the network with domain-specific knowledge, we found that it actually harms performance on factoid questions. Because most of the entity features are only active during fine-tuning with the small dataset, we conjecture that the performance decrease is due to over-fitting.
We continue our study with techniques to combat catastrophic forgetting as a means to regularize training during fine-tuning. In Experiment 6 of Table 1 we fine-tune the base model on a half-half mixture of BioASQ and SQuAD questions (BioASQ questions have been upsampled accordingly). This form of joint training yielded no significant performance gains. Experiment 7 regularizes the model via an additional forgetting cost term, as proposed by riemer2017forgettingcost and explained in Section "Domain Adaptation" . We generally found that this technique only increases performance for factoid questions where the performance boost was largest for $C_{fc} = 100.0$ . The fact that the forgetting loss decreases performance on list questions is not surprising, as predictions are pushed more towards the predictions of the base model, which has very poor performance on list questions.
Experiment 8 adds an L2 loss which penalizes deviations from the base model's parameters. We found that performance decreases as we increase the value of $C_{l2}$ which shows that this technique does not help at all. For the sake of completeness we report results for $C_{l2} = 0.3$ , the lowest value that yielded a significant drop in performance.
Datasets
SQuAD BIBREF4 is a dataset of $\approx 100,000$ questions with relevant contexts and answers that sparked research interest into the development of neural QA systems recently. The contexts are excerpts of Wikipedia articles for which crowd-source workers generated questions-answer pairs. Because of the large amount of training examples in SQuAD, it lends itself perfectly as our source dataset.
The BioASQ challenge provides a biomedical QA dataset BIBREF1 consisting of questions, relevant contexts (called snippets) from PubMed abstracts and possible answers to the question. It was carefully created with the help of biomedical experts.
In this work, we focus on Task B, Phase B of the BioASQ challenge, in which systems must answer questions from gold-standard snippets. These questions can be either yes/no questions, summary questions, factoid questions, or list questions. Because we employ an extractive QA system, we restrict this study to answering factoid and list questions by extracting answer spans from the provided contexts.
The 2017 BioASQ training dataset contains $1,799$ questions, of which 413 are factoid and 486 are list questions. The questions have $\approx 20$ snippets on average, each of which are on average $\approx 34$ tokens long. We found that around $65\%$ of the factoid questions and around $92\%$ of the list questions have at least one extractable answer. For questions with extractable answers, answers spans are computed via a simple substring search in the provided snippets. All other questions are ignored during training and treated as answered incorrectly during evaluation.
Training
We minimize the cross-entropy loss for the gold standard answer spans. However, for multiple answer spans that refer to the same answer (e.g. synonyms), we only minimize the loss for the span of the lowest loss. We use the ADAM BIBREF20 for optimization on SQuAD with a learning rate starting at $10^{-3}$ which is halved whenever performance drops between checkpoints. During the fine-tuning phase, we continue optimization on the BioASQ dataset with a smaller learning rate starting at $10^{-4}$ . During both phases, the model is regularized by variational dropout of rate $0.5$ BIBREF21 .
Evaluation
The official evaluation measures from BioASQ are mean reciprocal rank (MRR) for factoid questions and F1 score for list questions . For factoid questions, the list of ranked answers can be at most five entries long. The F1 score is measured on the gold standard list elements. For both measures, case-insensitive string matches are used to check the correctness of a given answer. A list of synonyms is provided for all gold-standard answers. If the system's response matches one of them, the answer counts as correct.
For evaluation, we use two different fine-tuning datasets, depending on the experiment: BioASQ3B, which contains all questions of the first three BioASQ challenges, and BioASQ4B which additionally contains the test questions of the fourth challenge. BioASQ4B is used as the training dataset for the fifth BioASQ challenge whereas BioASQ3B was used for training during the fourth challenge.
Because the datasets are small, we perform 5-fold cross-validation and report the average performance across the five folds. We use the larger BioASQ4B dataset except when evaluating the ensemble and when comparing to participating systems of previous BioASQ challenges.
All models were implemented using TensorFlow BIBREF22 with a hidden size of 100. Because the context in BioASQ usually comprises multiple snippets, they are processed independently in parallel for each question. Answers from all snippets belonging to a question are merged and ranked according to their individual probabilities.
Ensemble
Model ensembles are a common method to tweak the performance of a machine learning system. Ensembles combine multiple model predictions, for example by averaging, in order to improve generalization and prevent over-fitting. We evaluate the utility of an ensemble by training five models on the BioASQ3B dataset using 5-fold cross-validation. Each of the models is evaluated on the 4B test data, i.e., data which is not included in BioASQ3B.
During application, we run an ensemble by averaging the start and end scores of individual models before they are passed to the sigmoid / softmax functions as defined in Eq. 16 and 17 . In Table 2 we summarize the average performance of the five models, the best performance across the five models, and the performance of the ensemble. We observe performance gains of 3 percentage points on factoid questions and a less than 1 percentage point on list questions, relative to the best single model. This demonstrates a small performance gain that is consistent with the literature.
Comparison to competing BioASQ systems
Because the final results of the fifth BioASQ challenge are not available at the time of writing, we compare our system to the best systems in last year's challenge . For comparison, we use the best single model and the model ensemble trained on BioASQ3B (see Section "Ensemble" ). We then evaluate the model on the 5 batches of last year's challenge using the official BioASQ evaluation tool. Each batch contains 100 questions of which only some are factoid and list questions. Note that the results underestimate our system's performance, because our competing system's responses have been manually evaluated by humans while our system's responses are evaluated automatically using string matching against a potentially incomplete list of synonyms. In fact, our qualitative analysis in Section "Qualitative Analysis" shows that many answers are counted as incorrect, but are synonyms of the gold-standard answer. The results are summarized in Table 3 and compared to the best systems in the challenge in each of the batches and question type categories.
With our system winning four out of five batches on factoid questions, we consider it state-of-the-art in biomedical factoid question answering, especially when considering that our results might be higher on manual evaluation. The results on list questions are slightly worse, but still very competitive. This is surprising, given that the network never saw a list question prior to the fine-tuning phase. Due to small test set sizes, the sampling error in each batch is large, causing the single model to outperform the model ensemble on some batches.
Qualitative Analysis
In order to get a better insight into the quality of the predictions, we manually validated the predictions for the factoid questions of batch 5 of the fourth BioASQ challenge as given by the best single model (see Table 3 ). There are in total 33 factoid questions, of which 23 have as the gold standard answer a span in one of the contexts. According to the official BioASQ evaluation, only 4 questions are predicted correctly (i.e., the gold standard answer is ranked highest). However, we identified 10 rank-1 answers which are not counted as correct but are synonyms to the gold standard answer. Examples include "CMT4D disease" instead of "Charcot-Marie-Tooth (CMT) 4D disease", "tafazzin" instead of "Tafazzin (TAZ) gene", and " $\beta $ -glucocerebrosidase" instead of "Beta glucocerebrosidase". In total, we labeled 14 questions as correct and 24 questions as having their correct answer in the top 5 predictions.
In the following, we give examples of mistakes made by the system. Questions are presented in italics. In the context, we underline predicted answers and present correct answers in boldface.
We identified eight questions for which the semantic type of the top answer differs from the question answer type. Some of these cases are completely wrong predictions. However, this category also includes subtle mistakes like the following:
In which yeast chromosome does the rDNA cluster reside?
The rDNA cluster in Saccharomyces cerevisiae is located 450 kb from the left end and 610 kb from the right end of chromosome XII...
Here, it predicted a yeast species the rDNA cluster is located in, but ignored that the question is asking for a chromosome.
Another type of mistakes is that the top answer is somewhat correct, but is missing essential information. We labeled four predictions with this category, like the following example:
How early during pregnancy does non-invasive cffDNA testing allow sex determination of the fetus?
Gold Standard Answer: "6th to 10th week of gestation" or "first trimester of pregnancy"
Given Top Answer: "6th-10th"
In summary, to our judgment, 14 of 33 questions ( $42.4\%$ ) are answered correctly, and 24 of 33 questions ( $72.7\%$ ) are answered correctly in one of the top 5 answers. These are surprisingly high numbers considering low MRR score of $23.7\%$ of the automatic evaluation (Table 3 ).
Discussion and future work
The most significant result of this work is that state-of-the-art results in biomedical question answering can be achieved even in the absence of domain-specific feature engineering. Most competing systems require structured domain-specific resources, such as biomedical ontologies, parsers, and entity taggers. While these resources are available in the biomedical domain, they are not available in most domains.
Our system, on the other hand, requires a large open-domain QA dataset, biomedical word embeddings (which are trained in an unsupervised fashion), and a small biomedical QA dataset. This suggests that our methodology is easily transferable to other domains as well.
Furthermore, we explored several supervised domain adaptation techniques. In particular, we demonstrated the usefulness of forgetting cost for factoid questions. The decreased performance on list questions is not surprising, because the model's performance on those questions is very poor prior to fine-tuning which is due to the lack of list questions in SQuAD. We believe that large scale open-domain corpora for list questions would enhance performance further.
Unsupervised domain adaptation could be an interesting direction for future work, because the biomedical domain offers large amounts of textual data, some of which might even contain questions and their corresponding answers. We believe that leveraging these resources holds potential to further improve biomedical QA.
Conclusion
In this paper, we described a deep learning approach to address the task of biomedical question answering by using domain adaptation techniques. Our experiments reveal that mere fine-tuning in combination with biomedical word embeddings yield state-of-the-art performance on biomedical QA, despite the small amount of in-domain training data and the lack of domain-dependent feature engineering. Techniques to overcome catastrophic forgetting, such as a forgetting cost, can further boost performance for factoid questions. Overall, we show that employing domain adaptation on neural QA systems trained on large-scale, open-domain datasets can yield good performance in domains where large datasets are not available.
Acknowledgments
This research was supported by the German Federal Ministry of Education and Research (BMBF) through Software Campus project GeNIE (01IS12050).

Question: Among various transfer learning techniques, which technique yields to the best performance?
Answer: Unanswerable