Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: First I would like to apologize for the delay in reviewing.

summary : This work explores several experiments to transfer training a specific model of reading comprehension ( AS Reader), in an artificial and well populated dataset in order to perform in another target dataset. 

Here is what I understand are their several experiments to transfer learning, but I am not 100% sure.
1. The model is trained on the big artificial dataset and tested on the small target datasets (section 4.1)
2. The model is pre-trained on the big artificial dataset like before, then fine-tuned on a few examples from the target dataset and tested on the remaining target examples. Several such models are trained using different sub-sets of fine-tuning examples. The results are tested against the performance of randomly intialized then fine-tuned models (section 4.2).
3. The model is pre-trained on the big artificial dataset like before. The model is made of an embedding component and an encoder  component. Alternatively, each component is reset to a random initialization, to test the importance of the pre-training in each component. Then the model is fine-tuned on a few examples from the target dataset and tested on the remaining target examples. (section 4.3)

I think what makes things difficult to follow is the fact that the test set is composed by several sub tasks, and sometimes what is reported is the mean performance across the tasks, sometimes the performance on a few tasks. Sometimes what we see is the mean performance of several models? You should report standard deviations also. Could you better explain what you mean by best validation ?

Interesting and unpretentious work. The clarity of the presentation could be improved maybe by simplifying the experimental setup? The interesting conclusion I think is reported at the end of the section 4.1, when the nuanced difference between the datasets are exposed.

Minor: unexplained acronyms: GRU, BT, CBT.
benfits p. 2
subsubset p. 6
review_title: Interesting ; needs to improve clarity

final_score: rating: null
review_content: - have you tried characterizing the errors made by models with/without pretraining? it would be helpful to observe if what percentage of the errors are the same/different.


- can you add a horizontal line to figure2a and 2b so that we can see the performance of 0-shot model and the model trained on only full target task's data?
review_title: qualitative analysis

final_score: rating: null
review_content: Very close to transfer learning there is multi-task learning. Did you know about works like this one:
R. Collobert and J. Weston. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In International Conference on Machine Learning, ICML, 2008. 

or did you purposely left it out and why?


review_title: multi-task learning

final_score: rating: 3: Clear rejection
review_content: This work investigates the performance of transfer learning from resource-rich setup (BookTest, CNN/Daily Mail corpora) to low-resource (bAbI, SQuAD benchmarks) settings. Experiments show poor improvements in 0-shot learning. However, when the model is exposed to few training instances some improvements are observed.

The claims made here require a more comprehensive analysis. I criticize the use of bAbI as a low-resource real-world scenario. bAbI is designed as a unit test and is far from representing many natural language phenomena. Thus, the claims related to bAbI can only be weak evidence for questioning transfer learning high-resource to low-resource in real-world scenarios. I highly recommend using recently proposed real-world scenarios [1,2].

More importantly, the work does not explain why and how do we get improvement using transfer learning. They remotely address this by hypothesizing the knowledge of transfer is not just encoded in embeddings but also in the model. Considering the related work [3], these claims bring a marginal novelty and still "how and why" should be central in this work.  

[1] http://www.msmarco.org/dataset.aspx
[2] https://datasets.maluuba.com/NewsQA
[3] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.149.8551&rep=rep1&type=pdf
review_title: needs more thorough analysis

final_score: rating: null
review_content: - It seems that most of the pre-training effects is contained by the word embeddings. One interesting baseline could be the performance by simply initializing with word embeddings trained on any corpus (like Glove or Word2vec vectors). Did you try?
- An important feature both in bAbI tasks (at least task 1) and SQuAD is to use whether there is word match between the question and a sentence. Did you characterize if the fully pre-trained models (no adjustment) were not actually just learning that?
review_title: Questions

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a study of transfer learning in the context of QA from stories. A system is presented with a a short story and has to answer a question about it. This paper studies how a system trained to answer questions on a dataset can eventually be used to answer questions from another dataset. The results are mostly negative: transfer seems almost non-existant.

This paper is centered around presenting negative results. Indeed the main hypothesis of transferring between QA datasets with the attention sum reader turns out impossible and one needs a small portion of labeled data from the target dataset to get meaningful performance.

Having only negative results could be fine if the paper was bringing some value with a sharp analysis of the failure modes and of the reasons behind it. Because this might indicate some research directions to follow. However, there is not much of that. 

The answers to the pre-review questions actually start to give some insights: typing seems to be transferred for instance. How about the impact of syntax (very different between bAbI, Gutenberg books, and CNN news articles)? And the word/entity/ngrams distributions overlap between the 3 datasets?

Unfortunately, there is not much to take-away from this paper.
 
review_title: review

Meta-review: The area chair agrees with reviewers 1 and 3 that the paper does not meet the bar for ICLR. Reviewer 3 in particular points out how the paper can be strengthened for future revisions.
Decision: Reject