Title: Finding a Jack-of-All-Trades: An Examination of Semi-supervised Learning in Reading Comprehension
Authors: Rudolf Kadlec
Ondřej Bajgar
Peter Hrincar
Jan Kleindienst
Venue: ICLR 2017
Keywords: Natural language processing
Semi-Supervised Learning
Deep learning
Transfer Learning

Abstract: Deep learning has proven useful on many NLP tasks including reading
comprehension. However it requires a lot of training data which are not
available in some domains of application. Hence we examine the possibility
of using data-rich domains to pre-train models and then apply them in
domains where training data are harder to get. Specifically, we train a
neural-network-based model on two context-question-answer datasets, the
BookTest and CNN/Daily Mail, and we monitor transfer to subsets of bAbI,
a set of artificial tasks designed to test specific reasoning abilities, and of
SQuAD, a question-answering dataset which is much closer to real-world
applications. Our experiments show very limited transfer if the model isn’t
shown any training examples from the target domain however the results
are promising if the model is shown at least a few target-domain examples.
Furthermore we show that the effect of pre-training is not limited to word
embeddings.

TL;DR: We examine effect of transfer learning in AS Reader model from two source domains (CNN/DM and BookTest) to two target domains (bAbI and SQuAD).

Review discussion:
- have you tried characterizing the errors made by models with/without pretraining? it would be helpful to observe if what percentage of the errors are the same/different.
- can you add a horizontal line to figure2a and 2b so that we can see the performance of 0-shot model and the model trained on only full target task's data?
One difference we observed between the questions that only pre-trained models can answer and the ones which even random models can answer was that the pre-trained models were better able to answer questions which contained rare words.
So far we have not found any other clear pattern.
What concerns the figures, we added 0 to the x axes - this shows the 0-shot performances. The performance of a randomly-initialized model trained on the full training data for the target task are shown in Figure 2 (b) for SQuAD as the right-most end of the "Randomly initialized" curve. The value for training on the 10k training set on bAbI for some tasks is also shown in Figure 4. For tasks where the 10k performance is not included, the model reached a near-perfect accuracy already on the 1k or 5k training set, so experiments on 10k seem unnecessary.
Since these are values for a particular training set size on the x axis, we think it is methodologically cleaner to include them as points plotted against the appropriate x-axis value rather than a horizontal line across all x-axis values.
- It seems that most of the pre-training effects is contained by the word embeddings. One interesting baseline could be the performance by simply initializing with word embeddings trained on any corpus (like Glove or Word2vec vectors). Did you try?
- An important feature both in bAbI tasks (at least task 1) and SQuAD is to use whether there is word match between the question and a sentence. Did you characterize if the fully pre-trained models (no adjustment) were not actually just learning that?
The pre-training effects are both in the embeddings and in the recurrent encoders. Note that the increase in accuracy from the pre-trained embedding model to the fully pre-trained one is at least comparable to the increase from a randomly initialized model to the one with pre-trained embeddings.
We provided also the numbers for models with only pre-trained encoders which, in some cases, do not improve that much compared to the randomly initialized models. We explain this weak pre-training effect by the fact that the recurrent layer gets the embeddings as inputs, and hence if we randomly re-initialize the embeddings the encoder can easily get confused by the unexpected input.
Hence we consider the gaps random->pre-trained embedding and pret. emb. -> fully pretrained as more informative.
We tried using the Word2Vec embeddings and their advantage was weaker than the one of BookTest embeddings, if there was any improvement at all (details were added to Table 2). The advantage of BookTest-trained embeddings probably comes from them being trained directly for a task similar to the target task, instead of using the w2v training objective which is somewhat different.
Regarding your second question, we examined numerous examples from bAbI and it seems that the non-adjusted model is able to predict mostly the correct word class to fill in the blank (e.g. if the question is "John is in the XXXXX ." the model tends to highlight kitchen, hallway, garden,...) - it does this fairly reliably while it does actually fail on some exact-match examples. This emphasizes the point we make in the article - the transfer without target adjustment is very poor even for a model with state-of-the-art results on CBT.
Very close to transfer learning there is multi-task learning. Did you know about works like this one:
R. Collobert and J. Weston. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In International Conference on Machine Learning, ICML, 2008.
or did you purposely left it out and why?
Collobert and Weston is definitely relevant and we are aware of this.
However, we already cite Mou 2016 that builds on top of Collobert and Weston and sets it in a more recent context.
You seem to say in the paper that Mou et Al 2015 is about text classification. Collobert and Weston 2008, it seems to me, was about more than text classification. If I remember correctly it tackled the 'Semantic Role Labeling' task.
You are right that we have described previous work too briefly. There are more papers that study transfer learning in other NLP tasks besides text classification, including papers submitted to this conference (like Hashimoto et al.). We will expand this section in possible future versions of the paper.
For us Mou et al was more relevant since it studies transfer on multiple levels of abstraction (word embeddings, hidden layers, output layer) whereas Collobert and Weston used only shared word embeddings.
This work investigates the performance of transfer learning from resource-rich setup (BookTest, CNN/Daily Mail corpora) to low-resource (bAbI, SQuAD benchmarks) settings. Experiments show poor improvements in 0-shot learning.
However, when the model is exposed to few training instances some improvements are observed.
The claims made here require a more comprehensive analysis. I criticize the use of bAbI as a low-resource real-world scenario. bAbI is designed as a unit test and is far from representing many natural language phenomena.
Thus, the claims related to bAbI can only be weak evidence for questioning transfer learning high-resource to low-resource in real-world scenarios. I highly recommend using recently proposed real-world scenarios [1,2].
More importantly, the work does not explain why and how do we get improvement using transfer learning. They remotely address this by hypothesizing the knowledge of transfer is not just encoded in embeddings but also in the model. Considering the related work [3], these claims bring a marginal novelty and still "how and why" should be central in this work.
[1] http://www.msmarco.org/dataset.aspx
[2] https://datasets.maluuba.com/NewsQA
[3] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.149.8551&rep=rep1&type=pdf
Thank you for the review, however we find that some of its points are not entirely appropriate.
Firstly the review criticizes us for using bAbI as a low-resource real-world scenario while they were designed as unit tests.
However we do explicitly refer to the tasks as an artificial toy dataset (top of page 3) which we put into contrast with the "real-world" SQuAD. We do use them as unit tests for specific reasoning skills - this purpose is made clear already in point 1. on page 2 and, as the review says, this is what they were designed for.
Next the review recommends using two real-world scenarios, none of which was however available at the time of submission.
Instead we use SQuAD which we evaluated as the best "real-world" task at the time of performing the experiments. Even now SQuAD seems like a reasonable alternative to the suggested datasets. Why would you highly recommend using the two new datasets as opposed to SQuAD?
(We have actually tried running the experiments on MS Marco and observed behaviour very similar to the one on SQuAD.)
Finally the "how and why" is certainly an interesting question. We did try to partly address this in our last experiment. We did not just "hypothesize" that the knowledge is not only in the embeddings but also in the encoder - we performed an experiment and did verify this hypothesis. We certainly admit that this is only a small part of the why and how - however tackling this question more generally opens the whole topic of model interpretability, which is one of the biggest current challenges in Machine Learning and addressing it more deeply is beyond the scope of this work.
One difference between our work and [3] is that we pre-train both word embeddings and document/question encoder whereas [3] used just shared word embeddings. See discussion in thread "multi-task learning" on this page.
This paper proposes a study of transfer learning in the context of QA from stories. A system is presented with a a short story and has to answer a question about it. This paper studies how a system trained to answer questions on a dataset can eventually be used to answer questions from another dataset. The results are mostly negative: transfer seems almost non-existant.
This paper is centered around presenting negative results.
Indeed the main hypothesis of transferring between QA datasets with the attention sum reader turns out impossible and one needs a small portion of labeled data from the target dataset to get meaningful performance.
Having only negative results could be fine if the paper was bringing some value with a sharp analysis of the failure modes and of the reasons behind it. Because this might indicate some research directions to follow.
However, there is not much of that.
The answers to the pre-review questions actually start to give some insights: typing seems to be transferred for instance. How about the impact of syntax (very different between bAbI, Gutenberg books, and CNN news articles)? And the word/entity/ngrams distributions overlap between the 3 datasets?
Unfortunately, there is not much to take-away from this paper.
Thank you for your review. The paper presents three kinds of experiments. The results of the first experiment (transfer without training on the target dataset) are indeed negative - almost no transfer.
However this is not the case for the other two experiments.
The results of the second experiment show that pre-training can significantly improve the model's performance if it is shown at least a few examples from the target domain. On bAbI we see this only on some of the examined tasks;
however on SQuAD we see a jump from 31% of accuracy with a randomly initialized model to 48% with a pre-trained one (Table 2). We do see this as a positive result.
Also the third experiment does bring another positive observation - this improvement is not just thanks to pre-trained word embeddings (whose usefulness is well established), but also thanks to the pre-trained recurrent encoder.
And where we see the take-away from the paper? We see the negative result as a warning that what models are learning on some of the recently popular tasks may not be easily transferable to other settings. Since practical usefulness of machine learning models largely relies on this transferability, we consider this a prompt for our community, which often doesn't address this point at all in research papers, that it cannot be taken for granted and we may need to start paying more attention to this aspect.
The second experiment then shows that while transfer cannot be taken for granted it is possible under certain conditions, and the third experiment further examines the nature of this transfer.
A more specific take-away is that it is possible to transfer knowledge via pre-training a recurrent encoder in a way similar to re-using word-embeddings. This principle may be relevant to wide variety of models since some form of recurrent encoder is being used by almost all reading-comprehension models.
Thanks for you answer.
> "We see the negative result as a warning that what models are learning on some of the recently popular tasks may not be easily transferable to other settings."
Indeed, current best models do not transfer well to other settings but AFAIK the papers do not claim that they do.
Overall, it is not common belief that transfer works well. It is actually the opposite. In that sense, I do not think that the main paper result as stated in the quote above brings much.
First I would like to apologize for the delay in reviewing.
summary : This work explores several experiments to transfer training a specific model of reading comprehension ( AS Reader), in an artificial and well populated dataset in order to perform in another target dataset.
Here is what I understand are their several experiments to transfer learning, but I am not 100% sure.
1. The model is trained on the big artificial dataset and tested on the small target datasets (section 4.1)
2. The model is pre-trained on the big artificial dataset like before, then fine-tuned on a few examples from the target dataset and tested on the remaining target examples. Several such models are trained using different sub-sets of fine-tuning examples. The results are tested against the performance of randomly intialized then fine-tuned models (section 4.2).
3. The model is pre-trained on the big artificial dataset like before. The model is made of an embedding component and an encoder  component.
Alternatively, each component is reset to a random initialization, to test the importance of the pre-training in each component.
Then the model is fine-tuned on a few examples from the target dataset and tested on the remaining target examples. (section 4.3)
I think what makes things difficult to follow is the fact that the test set is composed by several sub tasks, and sometimes what is reported is the mean performance across the tasks, sometimes the performance on a few tasks. Sometimes what we see is the mean performance of several models? You should report standard deviations also. Could you better explain what you mean by best validation ?
Interesting and unpretentious work. The clarity of the presentation could be improved maybe by simplifying the experimental setup? The interesting conclusion I think is reported at the end of the section 4.1, when the nuanced difference between the datasets are exposed.
Minor: unexplained acronyms: GRU, BT, CBT.
benfits p. 2
subsubset p. 6
Thank you for the review.
Since generalization is one of the central topics of the article, we wanted to perform the experiments on a variety of tasks. Since behaviour was different on each task, we decided to provide plots of a few selected tasks in the main body of the article, leaving the rest for the appendix. To capture the remainder of the experiments in the main body in a compact form, we decided to provide a plot of the means across all tasks. We are sorry to hear that this had an impact on clarity and will bear this problem in mind when designing and describing future experiments.
As you requested, we have now added a table describing the standard deviations for each of the experiments, and perhaps more interestingly, added a table with significance levels of whether the pre-trained model was better than the randomly initialized one.
By best validation model we mean the model with the best accuracy on the validation dataset for each task. In the case of bAbI this was a set of 100 examples taken aside from the original training set. In the case of SQuAD this was the SQuAD development dataset filtered in the same way as the training dataset.
We have also corrected the typos and the use of acronyms, thank you for pointing them out.

Meta-review:
role: AC
data: comment: Dear authors and reviewers, this paper is currently very close to the decision boundary for acceptance and would benefit from a bit more discussion.


role: Author
data: value: comment: Thank you for alerting us. Based on the reviews we were already expecting a rejection rather than the subject being open to discussion.

We are now addressing some specific reviewers' concerns below, however let us summarize here what we see as the main contributions of the paper. 

What we see as a trend in Reading Comprehension research is further and further specialization of models on one particular narrow subtask (often associated with a specific dataset). That also means that the structures of models being applied to different subtasks are growing further apart since teams often focus on exploiting features particular to each dataset. However this narrow focus makes the resulting models less suitable for being applied to other datasets. 
(to be more specific, for example Henaf (2016) while setting a state of the art on the bAbI tasks performs rather poorly on the Children's Book Test - and that is still a positive exception in that it does provide results on other datasets. Fore instance none of the models that present results on the SQuAD datasets do report their results on any other dataset (e.g. Wang and Jiang (2016), Yu et al. (2016), Xiong (2016)) and the situation is very similar for publications on other datsets.)

We believe that if the final aim of Reading Comprehension research is to find a system that would be useful in the real world, the ability to generalize will be essential. This paper shows that being able to solve even a supposedly "real-world" task, the Children's Book Test, with state-of-the-art accuracy does not guarantee a model to be able to solve even simple toy tasks. This puts into question the usefulness of the wide-spread focus on narrow tasks, unless it is combined with studying the generalization potential.  

Apart from pointing out that models' ability to generalize certainly cannot be taken for granted, we also illustrate a few possible ways how we could examine models' ability to generalize. These can serve as an inspiration of how the important topic of generalization can be empirically examined.

We think that while many other papers present interesting incremental improvements on specific tasks, this paper, even while being only an unpolished preliminary exploration of the topic of generalization, would bring value to the ICLR community by opening an important direction of integrative research that aims to build models that perform well on multiple different datasets at the same time or can even transfer skills gained on one dataset to other data.



References:
Mikael Henaff, Jason Weston, Arthur Szlam, Anoine Bordes and Yann LeCun, Tracking the World State with Recurrent Entity Networks, arXiv:1612.03969
Shuogang Wang and Jing Jiang; Machine Comprehension Using Match-LSTM and Answer Pointer, arXiv:1608.07905
Yang Yu, Wei Zhang, Kazi Hasan, Mo Yu, Bing Xiang and Bowen Zhou; End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension, arXiv:1610.09996
Caiming Xiong, Victor Zong, Richard Socher; Dynamic Coattention Networks for Question Answering, arXiv:1611.01604
role: Reviewer
data: value: comment: First of all, I favor analysis papers like this one over architecture engineering approaches. These type of papers have a more ambitious goal of expanding our understanding the nature of tasks or models we use. However, this type of studies requires comprehensive and tedious analyses. 

This paper moves in the right direction however, needs more work because:
- bAbI is ridiculously small and I am not convinced that it reflects any language related phenomena. We are merely trying to learn inverse of a python function. Thus, on my side, any claims related to bAbI is obsolete.

- although there is a nice breakdown for bAbI we see no such analysis for SQUAD. For instance, a categorical breakdown of the type of answers (pos tag, question type etc., position in text etc.) would inform us for which categories transfer learning helps and when. Maybe more importantly, how much data is enough to get accurate answers for category X which we all care.

- There is a nagging simplification of using only one word answers for SQUAD because the model at hand only supports one-word answers. However, it would be great to see results with models supporting multi-word answers and see another categorical breakdown how transfer learning helps depending on the answer size. 

- Now that we have more realistic datasets compared to CNN/Daily Mail (I listed in my review) where in that work they have the previously mentioned nagging detail of replacing multiword expressions with entity ids, authors can improve the quality of this work by investigating the other dimensions such as domain type, annotation style etc.


role: PC
data: comment: The area chair agrees with reviewers 1 and 3 that the paper does not meet the bar for ICLR. Reviewer 3 in particular points out how the paper can be strengthened for future revisions.

Decision: Reject