Supervised and Unsupervised Transfer Learning for Question Answering

Abstract: Although transfer learning has been shown to be successful for tasks like object and speech recognition, its applicability to question answering (QA) has yet to be well-studied. In this paper, we conduct extensive experiments to investigate the transferability of knowledge learned from a source QA dataset to a target dataset using two QA models. The performance of both models on a TOEFL listening comprehension test (Tseng et al., 2016) and MCTest (Richardson et al., 2013) is significantly improved via a simple transfer learning technique from MovieQA (Tapaswi et al., 2016). In particular, one of the models achieves the state-of-the-art on all target datasets; for the TOEFL listening comprehension test, it outperforms the previous best model by 7%. Finally, we show that transfer learning is helpful even in unsupervised scenarios when correct answers for target QA dataset examples are not available.

One of the most important characteristics of an intelligent system is to understand stories like humans do. A story is a sequence of sentences, and can be in the form of plain text BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 or spoken content BIBREF0 , where the latter usually requires the spoken content to be first transcribed into text by automatic speech recognition (ASR), and the model will subsequently process the ASR output. To evaluate the extent of the model's understanding of the story, it is asked to answer questions about the story. Such a task is referred to as question answering (QA), and has been a long-standing yet challenging problem in natural language processing (NLP).
Several QA scenarios and datasets have been introduced over the past few years. These scenarios differ from each other in various ways, including the length of the story, the format of the answer, and the size of the training set. In this work, we focus on context-aware multi-choice QA, where the answer to each question can be obtained by referring to its accompanying story, and each question comes with a set of answer choices with only one correct answer. The answer choices are in the form of open, natural language sentences. To correctly answer the question, the model is required to understand and reason about the relationship between the sentences in the story.
Transfer learning BIBREF7 is a vital machine learning technique that aims to use the knowledge learned from one task and apply it to a different, but related, task in order to either reduce the necessary fine-tuning data size or improve performance. Transfer learning, also known as domain adaptation, has achieved success in numerous domains such as computer vision BIBREF8 , ASR BIBREF9 , BIBREF10 , and NLP BIBREF11 , BIBREF12 . In computer vision, deep neural networks trained on a large-scale image classification dataset such as ImageNet BIBREF13 have proven to be excellent feature extractors for a broad range of visual tasks such as image captioning BIBREF14 , BIBREF15 , BIBREF16 and visual question answering BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , among others. In NLP, transfer learning has also been successfully applied to tasks like sequence tagging BIBREF21 , syntactic parsing BIBREF22 and named entity recognition BIBREF23 , among others.
The procedure of transfer learning in this work is straightforward and includes two steps. The first step is to pre-train the model on one MCQA dataset referred to as the source task, which usually contains abundant training data. The second step is to fine-tune the same model on the other MCQA dataset, which is referred to as the target task, that we actually care about, but that usually contains much less training data. The effectiveness of transfer learning is evaluated by the model's performance on the target task.
In supervised transfer learning, both the source and target datasets provide the correct answer to each question during pre-training and fine-tuning, and the QA model is guided by the correct answer to optimize its objective function in a supervised manner in both stages.
We also consider unsupervised transfer learning where the correct answer to each question in the target dataset is not available. In other words, the entire process is supervised during pre-training, but unsupervised during fine-tuning. A self-labeling technique inspired by BIBREF26 , BIBREF24 , BIBREF25 is used during fine-tuning on the target dataset. We present the proposed algorithm for unsupervised transfer learning in Algorithm "Conclusion and Future Work" . [!htbp] Unsupervised QA Transfer Learning [1] Source dataset with correct answer to each question; Target dataset without any answer; Number of training epochs. Optimal QA model $M^{*}$ Pre-train QA model $M$ on the source dataset. For each question in the target dataset, use $M$ to predict its answer. For each question, assign the predicted answer to the question as the correct one. Fine-tune $M$ on the target dataset as usual. Reach the number of training epochs.
Although transfer learning has been successfully applied to various applications, its applicability to QA has yet to be well-studied. In this paper, we tackle the TOEFL listening comprehension test BIBREF0 and MCTest BIBREF1 with transfer learning from MovieQA BIBREF2 using two existing QA models. Both models are pre-trained on MovieQA and then fine-tuned on each target dataset, so that their performance on the two target datasets are significantly improved. In particular, one of the models achieves the state-of-the-art on all target datasets; for the TOEFL listening comprehension test, it outperforms the previous best model by 7%.
Transfer learning without any labeled data from the target domain is referred to as unsupervised transfer learning. Motivated by the success of unsupervised transfer learning for speaker adaptation BIBREF24 , BIBREF25 and spoken document summarization BIBREF26 , we further investigate whether unsupervised transfer learning is feasible for QA.
Although not well studied in general, transfer Learning for QA has been explored recently. To the best of our knowledge, BIBREF27 is the first work that attempted to apply transfer learning for machine comprehension. The authors showed only limited transfer between two QA tasks, but the transferred system was still significantly better than a random baseline. BIBREF28 tackled a more specific task of biomedical QA with transfer learning from a large-scale dataset. The work most similar to ours is by BIBREF29 , where the authors used a simple transfer learning technique and achieved significantly better performance. However, none of these works study unsupervised transfer learning, which is especially crucial when the target dataset is small. BIBREF30 proposed a two-stage synthesis network that can generate synthetic questions and answers to augment insufficient training data without annotations. In this work, we aim to handle the case that the questions from the target domain are available.
Among several existing QA settings, in this work we focus on multi-choice QA (MCQA). We are particularly interested in understanding whether a QA model can perform better on one MCQA dataset with knowledge transferred from another MCQA dataset. In Section "Question Answering Experiments" , we first formalize the task of MCQA. We then describe the procedures for transfer learning from one dataset to another in Section "Conclusion and Future Work" . We consider two kinds of settings for transfer learning in this paper, one is supervised and the other is unsupervised.
In MCQA, the inputs to the model are a story, a question, and several answer choices. The story, denoted by $\mathbf {S}$ , is a list of sentences, where each of the sentences is a sequence of words from a vocabulary set $V$ . The question and each of the answer choices, denoted by $\mathbf {Q}$ and $\mathbf {C}$ , are both single sentences also composed of words from $V$ . The QA model aims to choose one correct answer from multiple answer choices based on the information provided in $\mathbf {S}$ and $\mathbf {Q}$ .
We used MovieQA BIBREF2 as the source MCQA dataset, and TOEFL listening comprehension test BIBREF0 and MCTest BIBREF1 as two separate target datasets. Examples of the three datasets are shown in Table 1 .
Among numerous models proposed for multiple-choice QA BIBREF32 , BIBREF33 , BIBREF0 , we adopt the End-to-End Memory Network (MemN2N) BIBREF34 and Query-Based Attention CNN (QACNN) BIBREF35 , both open-sourced, to conduct the experiments. Below we briefly introduce the two models in Section "End-to-End Memory Networks" and Section "Query-Based Attention CNN" , respectively. For the details of the models, please refer to the original papers.
An End-to-End Memory Network (MemN2N) first transforms $\mathbf {Q}$ into a vector representation with an embedding layer $B$ . At the same time, all sentences in $\mathbf {S}$ are also transformed into two different sentence representations with two additional embedding layers $A$ and $C$ . The first sentence representation is used in conjunction with the question representation to produce an attention-like mechanism that outputs the similarity between each sentence in $\mathbf {S}$ and $\mathbf {Q}$ . The similarity is then used to weight the second sentence representation. We then obtain the sum of the question representation and the weighted sentence representations over $\mathbf {S}$ as $\mathbf {Q}^\prime $ . In the original MemN2N, $\mathbf {Q}^\prime $ is decoded to provide the estimation of the probability of being an answer for each word within a fixed set. The word with the highest probability is then selected as the answer. However, in multiple-choice QA, $B$0 is in the form of open, natural language sentences instead of a single word. Hence we modify MemN2N by adding an embedding layer $B$1 to encode $B$2 as a vector representation $B$3 by averaging the embeddings of words in $B$4 . We then compute the similarity between each choice representation $B$5 and $B$6 . The choice $B$7 with the highest probability is then selected as the answer.
A Query-Based Attention CNN (QACNN) first uses an embedding layer $E$ to transform $\mathbf {S}, \mathbf {Q}$ , and $\mathbf {C}$ into a word embedding. Then a compare layer generates a story-question similarity map $\mathbf {SQ}$ and a story-choice similarity map $\mathbf {SC}$ . The two similarity maps are then passed into a two-stage CNN architecture, where a question-based attention mechanism on the basis of $\mathbf {SQ}$ is applied to each of the two stages. The first stage CNN generates a word-level attention map for each sentence in $\mathbf {S}$ , which is then fed into the second stage CNN to generate a sentence-level attention map, and yield choice-answer features for each of the choices. Finally, a classifier that consists of two fully-connected layers collects the information from every choice answer feature and outputs the most likely answer. The trainable parameters are the embedding layer $E$ that transforms $\mathbf {S}, \mathbf {Q},$ and $\mathbf {C}$ into word embeddings, the two-stage CNN $\mathbf {S}, \mathbf {Q}$0 and $\mathbf {S}, \mathbf {Q}$1 that integrate information from the word to the sentence level, and from the sentence to the story level, and the two fully-connected layers $\mathbf {S}, \mathbf {Q}$2 and $\mathbf {S}, \mathbf {Q}$3 that make the final prediction. We mention the trainable parameters here because in Section "Question Answering Experiments" we will conduct experiments to analyze the transferability of the QACNN by fine-tuning some parameters while keeping others fixed. Since QACNN is a newly proposed QA model has a relatively complex structure, we illustrate its architecture in Figure 1 , which is enough for understanding the rest of the paper. Please refer to the original paper BIBREF35 for more details.
For pre-training MemN2N and QACNN on MovieQA, we followed the exact same procedure as in BIBREF2 and BIBREF35 , respectively. Each model was trained on the training set of the MovieQA task and tuned on the dev set, and the best performing models on the dev set were later fine-tuned on the target dataset. During fine-tuning, the model was also trained on the training set of target datasets and tuned on the dev set, and the performance on the testing set of the target datasets was reported as the final result. We use accuracy as the performance measurement.
Table 2 reports the results of our transfer learning on TOEFL-manual, TOEFL-ASR, MC160, and MC500, as well as the performance of the previous best models and several ablations that did not use pre-training or fine-tuning. From Table 2 , we have the following observations.
Rows (a) and (g) show the respective results when the QACNN and MemN2N are trained directly on the target datasets without pre-training on MovieQA. Rows (b) and (h) show results when the models are trained only on the MovieQA data. Rows (c) and (i) show results when the models are trained on both MovieQA and each of the four target datasets, and tested on the respective target dataset. We observe that the results achieved in (a), (b), (c), (g), (h), and (i) are worse than their fine-tuned counterparts (d), (e), (f), and (j). Through transfer learning, both QACNN and MemN2N perform better on all the target datasets. For example, QACNN only achieves 57.5% accuracy on MC160 without pre-training on MovieQA, but the accuracy increases by 18.9% with pre-training (rows (d) vs. (a)). In addition, with transfer learning, QACNN outperforms the previous best models on TOEFL-manual by 7%, TOEFL-ASR BIBREF33 by 6.5%, MC160 BIBREF36 by 1.1%, and MC500 BIBREF32 by 1.3%, and becomes the state-of-the-art on all target datasets.
For the QACNN, the training parameters are $E, W_{CNN}^{(1)}, W_{CNN}^{(2)}, W_{FC}^{(1)}$ , and $W_{FC}^{(2)}$ (Section "Query-Based Attention CNN" ). To better understand how transfer learning affects the performance of QACNN, we also report the results of keeping some parameters fixed and only fine-tuning other parameters. We choose to fine-tune either only the last fully-connected layer $W_{FC}^{(2)}$ while keeping other parameters fixed (row (d) in Table 2 ), the last two fully-connected layers $W_{FC}^{(1)}$ and $W_{FC}^{(2)}$ (row (e)), and the entire QACNN (row (f)). For TOEFL-manual, TOEFL-ASR, and MC500, QACNN performs the best when only the last two fully-connected layers were fine-tuned; for MC160, it performs the best when only the last fully-connected layer was fine-tuned. Note that for training the QACNN, we followed the same procedure as in BIBREF35 , whereby pre-trained GloVe word vectors BIBREF37 were used to initialize the embedding layer, which were not updated during training. Thus, the embedding layer does not depend on the training set, and the effective vocabularies are the same.
It is interesting to see that fine-tuning the entire QACNN doesn't necessarily produce the best result. For MC500, the accuracy of QACNN drops by 4.6% compared to just fine-tuning the last two fully-connected layers (rows (f) vs. (e)). We conjecture that this is due to the amount of training data of the target datasets - when the training set of the target dataset is too small, fine-tuning all the parameters of a complex model like QACNN may result in overfitting. This discovery aligns with other domains where transfer learning is well-studied such as object recognition BIBREF38 .
We expected to see that a MemN2N, when trained directly on the target dataset without pre-training on MovieQA, would outperform a MemN2N pre-trained on MovieQA without fine-tuning on the target dataset (rows (g) vs. (h)), since the model is evaluated on the target dataset. However, for the QACNN this is surprisingly not the case - QACNN pre-trained on MovieQA without fine-tuning on the target dataset outperforms QACNN trained directly on the target dataset without pre-training on MovieQA (rows (b) vs. (a)). We attribute this to the limited size of the target dataset and the complex structure of the QACNN.
We conducted experiments to study the relationship between the amount of training data from the target dataset for fine-tuning the model and the performance. We first pre-train the models on MovieQA, then vary the training data size of the target dataset used to fine-tune them. Note that for QACNN, we only fine-tune the last two fully-connected layers instead of the entire model, since doing so usually produces the best performance according to Table 2 . The results are shown in Table 3 . As expected, the more training data is used for fine-tuning, the better the model's performance is. We also observe that the extent of improvement from using 0% to 25% of target training data is consistently larger than using from 25% to 50%, 50% to 75%, and 75% to 100%. Using the QACNN fine-tuned on TOEFL-manual as an example, the accuracy of the QACNN improves by 2.7% when varying the training size from 0% to 25%, but only improves by 0.9%, 0.5%, and 0.7% when varying the training size from 25% to 50%, 50% to 75%, and 75% to 100%, respectively.
We also vary the size of MovieQA for pre-training to study how large the source dataset should be to make transfer learning feasible. The results are shown in Table 4 . We find that even a small amount of source data can help. For example, by using only 25% of MovieQA for pre-training, the accuracy increases 6.3% on MC160. This is because 25% of MovieQA training set (2,462 examples) is still much larger than the MC160 training set (280 examples). As the size of the source dataset increases, the performance of QACNN continues to improve.
We are interested in understanding what types of questions benefit the most from transfer learning. According to the official guide to the TOEFL test, the questions in TOEFL can be divided into 3 types. Type 1 questions are for basic comprehension of the story. Type 2 questions go beyond basic comprehension, but test the understanding of the functions of utterances or the attitude the speaker expresses. Type 3 questions further require the ability of making connections between different parts of the story, making inferences, drawing conclusions, or forming generalizations. We used the split provided by BIBREF33 , which contains 70/18/34 Type 1/2/3 questions. We compare the performance of the QACNN and MemN2N on different types of questions in TOEFL-manual with and without pre-training on MovieQA, and show the results in Figure 2 . From Figure 2 we can observe that for both the QACNN and MemN2N, their performance on all three types of questions improves after pre-training, showing that the effectiveness of transfer learning is not limited to specific types of questions.
So far, we have studied the property of supervised transfer learning for QA, which means that during pre-training and fine-tuning, both the source and target datasets provide the correct answer for each question. We now conduct unsupervised transfer learning experiments described in Section "Conclusion and Future Work" (Algorithm "Conclusion and Future Work" ), where the answers to the questions in the target dataset are not available. We used QACNN as the QA model and all the parameters $(E, W_{CNN}^{(1)}, W_{CNN}^{(2)}, W_{FC}^{(1)},$ and $W_{FC}^{(2)})$ were updated during fine-tuning in this experiment. Since the range of the testing accuracy of the TOEFL-series (TOEFL-manual and TOEFL-ASR) is different from that of MCTest (MC160 and MC500), their results are displayed separately in Figure UID29 and Figure UID30 , respectively.
From Figure UID29 and Figure UID30 we can observe that without ground truth in the target dataset for supervised fine-tuning, transfer learning from a source dataset can still improve the performance through a simple iterative self-labeling mechanism. For TOEFL-manual and TOEFL-ASR, QACNN achieves the highest testing accuracy at Epoch 7 and 8, outperforming its counterpart without fine-tuning by approximately 4% and 5%, respectively. For MC160 and MC500, the QACNN achieves the peak at Epoch 3 and 6, outperforming its counterpart without fine-tuning by about 2% and 6%, respectively. The results also show that the performance of unsupervised transfer learning is still worse than supervised transfer learning, which is not surprising, but the effectiveness of unsupervised transfer learning when no ground truth labels are provided is validated.
To better understand the unsupervised transfer learning process of QACNN, we visualize the changes of the word-level attention map during training Epoch 1, 4, 7, and 10 in Figure 4 . We use the same question from TOEFL-manual as shown in Table 1 as an example. From Figure 4 we can observe that as the training epochs increase, the QACNN focuses more on the context in the story that is related to the question and the correct answer choice. For example, the correct answer is related to “class project”. In Epoch 1 and 4, the model does not focus on the phrase “class representation”, but the model attends on the phrase in Epoch 7 and 10. This demonstrates that even without ground truth, the iterative process in Algorithm "Conclusion and Future Work" is still able to lead the QA model to gradually focus more on the important part of the story for answering the question.
In this paper we demonstrate that a simple transfer learning technique can be very useful for the task of multi-choice question answering. We use a QACNN and a MemN2N as QA models, with MovieQA as the source task and a TOEFL listening comprehension test and MCTest as the target tasks. By pre-training on MovieQA, the performance of both models on the target datasets improves significantly. The models also require much less training data from the target dataset to achieve similar performance to those without pre-training. We also conduct experiments to study the influence of transfer learning on different types of questions, and show that the effectiveness of transfer learning is not limited to specific types of questions. Finally, we show that by a simple iterative self-labeling technique, transfer learning is still useful, even when the correct answers for target QA dataset examples are not available, through quantitative results and visual analysis.
One area of future research will be generalizing the transfer learning results presented in this paper to other QA models and datasets. In addition, since the original data format of the TOEFL listening comprehension test is audio instead of text, it is worth trying to initialize the embedding layer of the QACNN with semantic or acoustic word embeddings learned directly from speech BIBREF39 , BIBREF40 , BIBREF41 instead of those learned from text BIBREF42 , BIBREF37 .

Figure and table captions:
Table 1: Example of the story-question-choices triplet from MovieQA, TOEFL listening comprehension test, and MCTest datasets. S,Q, and Ci denote the story, question, and one of the answer choices, respectively. For MovieQA, each question comes with five answer choices; and for TOEFL and MCTest, each question comes with only four answer choices. The correct answer is marked in bold.
Figure 1: QACNN architecture overview. QACNN consists of a similarity mapping layer, a query-based attention layer, and a prediction layer. The two-stage attention mechanism takes place in the query-based attention layer, yielding word-level and sentence-level attention map, respectively. The trainable parameters, including E,W (1)CNN ,W (2)
Table 2: Results of transfer learning on the target datasets. The number in the parenthesis indicates the accuracy increased via transfer learning (compared to rows (a) and (g)). The best performance for each target dataset is marked in bold. We also include the results of the previous best performing models on the target datasets in the last three rows.
Table 3: Results of varying sizes of the target datasets used for fine-tuning QACNN. The number in the parenthesis indicates the accuracy increases from using the previous percentage for fine-tuning to the current percentage.
Table 4: Results of varying sizes of the MovieQA used for pre-training QACNN. The number in the parenthesis indicates the accuracy increases from using the previous percentage for pre-training to the current percentage.
Figure 2: The performance of QACNN and MemN2N on different types of questions in TOEFL-manual with and without pre-training on MovieQA. ‘No’ in the parenthesis indicates the models are not pre-trained, while ‘Yes’ indicates the models are pre-trained on MovieQA.
Figure 3: The figures show the results of unsupervised transfer learning. The x-axis is the number of training epochs, and the y-axis is the corresponding testing accuracy on the target dataset. When training epoch = 0, the performance of QACNN is equivalent to row (b) in Table 2. The horizontal lines, where each line has the same color to its unsupervised counterpart, are the performances of QACNN with supervised transfer learning (row (e) in Table 2), and are the upperbounds for unsupervised transfer learning.
Figure 4: Visualization of the changes of the word-level attention map in the first stage CNN of QACNN in different training epochs. The more red, the more the QACNN views the word as a key feature. The input story-question-choices triplet is same as the one in Table 1.

Questions about this paper:
How different is the dataset size of source and target?

Answers:
the training dataset is large while the target dataset is usually much smaller