Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Did you check the variance that you obtain with different runs, in Table 3?

review_title: variance in experiments

final_score: rating: 7: Good paper, accept
review_content: Authors' response well answered my questions. Thanks!
Evaluation not changed.

###

This paper proposes a hierarchical framework of transfer learning for sequence tagging, which is expected to help the target task with the source task, by sharing as many levels of representation as possible. It is a general framework for various neural models. The paper has extensive and solid experiments, and the performance is competitive with the state of the art on multiple benchmark datasets. The framework is clear by itself, except that more details about training procedure, i.e. sec-3.3, need to be added. 

The experimental results show that for some task pairs {s,t}, this framework can help low-resource target task t, and the improvement increases with more levels of representations can be shared. Firstly, I suggest that the terms *source* and *target* should be more precisely defined in the current framework, because, due to Sec-3.3, the s and t in each pair are sort of interchangeable. That is, either of them can be the *source* or *target* task, especially when p(X=s)=p(X=t)=0.5 is used in the task sampling. The difference is: one is low-resourced and the other is not. Thus it could be thought of as multi-tasking between tasks with imbalanced resource. So one question is: does this framework simultaneously help both tasks in the pair, by learning more generalizable representations for different domains/applications/languages? Or is it mostly likely to only help the low-resourced one? Does it come with sacrifice on the high-resourced side? 

Secondly, as the paper shows that the low-resourced tasks are improved for the selected task pairs, it would also be interesting and helpful to know how often this could happen. That is, when the tasks are randomly paired (one chosen from a low-resource pool and the other from a high resource pool), how often could this framework help the low-resourced one?

Moreover, the choice of T-A/T-B/T-C lies intuitively in how many levels of representation *could* be shared as possible. This implicitly assumes share more, help more. Although I tend to believe so, it would be interesting to have some empirical comparison. For example, one could perhaps select some cross-domain pair, and see if T-A > T-B > T-C on such pairs, as mentioned in the author’s answer to the pre-review question. 

In general, I think this is a solid paper, and more exploration could be done in this direction. So I tend to accept this paper. 
review_title: Accept

final_score: rating: null
review_content: 
1. 
As the training framework is important in this paper, can you elaborate on the training details? 
How the task pair <s,t> is sampled given the binomial distribution? Do you sample <s,t> out of a pool of {<s,t>} or sample s and t separately out of {s} and {t} and then pair them? Are the representations shared across the pairs, e.g., the char/word embeddings that are the same for different pairs <s_1,t_1> and <s_2,t_2>?
What is the stop criterion? With many <s,t> pairs, certain pair <s_i,t_i> may converge earlier than others. So how this issue is dealt?
2. 
Although the way of selecting T-A/T-B/T-C is reasonable, it will be interesting to show some results when one certain task which is best-suited for T-A is treated as T-B/T-C.
review_title: Details about training framework and experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose transfer learning variants for neural-net-based models, applied to a bunch of NLP tagging tasks.

The field of multi-tasking is huge, and the approaches proposed here do not seem to be very novel in terms of machine learning: parts of a general architecture for NLP are shared, the amount of shared "layers" being dependent of the task of interest.

The novelty lies in the type of architecture which is used in the particular setup of NLP tagging tasks.

The experimental results show that the approach seems to work well when there is not much labeled data available (Figure 2). Table 3 show some limited improvement at full scale.

Figure 2 results are debatable though: it seems the authors fixed the architecture size while varying the amount of labeled data; it is very likely that tuning the architecture for each size would have led to better results.

Overall, while the paper reads well, the novelty seems a bit limited and the experimental section seems a bit disappointing.
review_title: Interesting research direction, but the scientific advances are limited and the experiments are not very convincing.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a clear hierarchical taxonomy of transfer learning methods as applicable to sequence tagging problems. This contextualizes and unifies previous work on specific instances of this taxonomy. Moreover, the paper shows that previously unexplored places in this taxonomy are competitive with or superior to the state of the art in key benchmark problems.

It'd be nice to see this explored further, such as highlighting what is the loss as you move from the more restrictive to the less restrictive transfer learning approaches, but I believe this paper is interesting and acceptable as-is.
review_title: Accept

final_score: rating: null
review_content: I am curious about what happens if the same task which could be implemented as T-A is treated as T-B or T-C. I'd expect the performance to fall, but unclear by how much.
review_title: Are the different transfer models ever compared on the same task?

Meta-review: One weak and one positive review without much concrete substance. The third review is positive, but the experiments are not that convincing: the gains from transfer are small in table 3 and in table 2 it is unclear how strong the baselines are. Given how competitive ICLR is, the area chair has no alternative than to unfortunately reject this paper.
Decision: Accept (Poster)