Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a transfer learning approach applied to a number of NLP tasks; the set of tasks appear to have an order in terms of complexity (from easy syntactic tasks to somewhat harder semantic tasks).

Novelty: the way the authors propose to do transfer learning is by plugging models corresponding to each task, in a way that respects the known hierarchy (in terms of NLP "complexity") of those tasks. In that respect, the overall architecture looks more like a cascaded architecture than a transfer learning one. There are some existing literature in the area (first two Google results found: https://arxiv.org/pdf/1512.04412v1.pdf, (computer vision) and https://www.aclweb.org/anthology/P/P16/P16-1147.pdf (NLP)). In addition to the architecture, the authors propose a regularization technique they call "successive regularization".

Experiments:

- The authors performed a number of experimental analysis to clarify what parts of their architecture are important, which is very valuable;

- The information "transferred" from one task to the next one is represented both using a smooth label embedding and the hidden representation of the previous task. At this point there is no analysis of which one is actually important, or if they are redundant (update: the authors mentioned they would add something there). Also, it is likely one would have tried first to feed label scores from one task to the next one, instead of using the trick of the label embedding -- it is unclear what the latter is actually bringing.

- The successive regularization does not appear to be important in Table 8; a variance analysis would help to conclude.
review_title: A joint model that actually works, limited novelty, a lot of experiments but possibly missing few important points.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduce a way to train joint models for many NLP tasks. Traditionally, we treat these tasks as “pipeline” — the later tasks will depending on the output of the previous tasks. Here, the authors propose a neural approach which includes all the tasks in one single model. The higher level tasks takes (1) the predictions from the lower level tasks and (2) the hidden representations of the lower level tasks. Also proposed in this paper, is the successive regularization. Intuitively, this means that, when training the high level tasks, we don’t want to change the model in the lower levels by too much so that the lower level tasks can keep a reasonable accuracy of prediction.

On the modeling side, I think the proposed model is very similar comparing to (Zhang and Weiss, ACL 2016) and SPINN (Bowman et al, 2016) in a even simpler way. The number of the experiments are good. But I am not sure I am convinced by the numbers in Table 1 since the patterns are not very clear there — sometimes, the performance of the higher level tasks even goes down when training with more tasks (sometimes it does go up, but also not very significant and stable). The dependency scores, although I don’t think this is a serious problem, comparing the UAS/LAS when the output is not guaranteed to be a well-formed tree isn’t strictly speaking fair.

I admit that the successive regularization make sense intuitively and is a very interesting direction to try. However, without a careful study of the training schema of such model, the current results on successive regularization do not convince me that it should be the right thing to do in such models (the current results are not strong enough to show that). The training methods need to be explored here including things as iteratively train on different tasks, and the relationship between the number of training iterations of a task and it’s training set size (and loss on this task etc).
review_title: review

final_score: rating: null
review_content: 1. From table 1, it seems training with high level tasks (jointly) will always hurt the low level tasks performance. Any explanation for that?
2. From table 1, training with only DE seems outperform training the JMT_all system. The POS/parsing information doesn't help the high level task?
3. The training/test sets in POS/parsing are different (but overlapped). Is it possible for parsing to get gold POS information indirectly in this training method?
review_title: questions

final_score: rating: null
review_content: questions:
- without output label embeddings how do you connect two tasks?
- depth and shortcut connections effect relatedness and entailment tasks too much. how do you explain this? could this be related to using the same corpus (PTB) for all the previous tasks?
- for the first three tasks section splittings are not the same to compare with literature (I guess). Would not it be safer to use the same section split so that a gold-tag information would not flow one task to the other?
- table 12,13 reports higher numbers for chunkings. I guess those are word level accuracies not F1 scores. is this correct?

suggestions:
- related work would be more thorough if you include joint learning studies from NLP literature such as https://transacl.org/ojs/index.php/tacl/article/viewFile/412/81
- predicting the head and the dependency label does not guarantee producing well-formed dependency trees. did you try using a decoder like eisner's algorithm or maximum-spanning-tree algorithm?
review_title: questions & suggestions

final_score: rating: null
review_content: Did the authors try to directly feed the label scores (instead of building an embedding)? -- Table 10 reports results without LE, but it is likely that one would try first this kind of approach by simply feeding the scores in the next architectures;
Also, have you tried with the label information, but not the hidden state one? I wonder which one is actually important, or if they are redundant (in that respect, the drop observed in Table 10 is minimal when removing LE).

review_title: alternative to label embeddings

final_score: rating: 3: Clear rejection
review_content: this work investigates a joint learning setup where tasks are stacked based on their complexity. to this end, experimental evaluation is done on pos tagging, chunking, dependency parsing, semantic relatedness, and textual entailment. the end-to-end model improves over models trained solely on target tasks.

although the hypothesis of this work is an important one, the experimental evaluation lacks thoroughness:

first, a very simple multi-task learning baseline [1] should be implemented where there is no hierarchy of tasks to test the hypothesis of the tasks should be ordered in terms of complexity.

second, since the test set of chunking is included in training data of dependency parsing, the results related to chunking with JMT_all are not informative. 

third, since the model does not guarantee well-formed dependency trees, thus, results in table 4 are not fair. 

minor issue:
- chunking is not a word-level task although the annotation is word-level. chunking is a structured prediction task where we would like to learn a structured annotation over a sequence [2].

[1] http://ronan.collobert.com/pub/matos/2011_nlp_jmlr.pdf
[2] http://www.cs.cmu.edu/~nasmith/LSP/
review_title: experimental setup should be improved

Meta-review: There is a bit of spread in the reviewer scores, but ultimately the paper does not meet the high bar for acceptance to ICLR. The lack of author responses to the reviews does not help either.
Decision: Reject