Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper is on an improvement of multi-task learning by considering the input tasks at two levels: (1) at task level, i.e. the relationship between the tasks and (2) by the data associated with each task. Their major argument is that most current methods hold the assumption that the tasks are correlated with each other but they conjecture that in the real-world this is not necessarily true and try to model the relationship between the input tasks at these two levels and incorporate that in the learning framework. To show effectiveness of their approach they test their method on differently oriented public datasets representing graphs, nodes and text and compare performance with some of the recent approaches to multi-task learning.

Comments to authors
1. Overall while one could get the gist of the arguments in the paper, it was not thoroughly reviewed by the authors for grammar, so it was hard to follow the finer points of the arguments. There are several grammatical mistakes and errors, on every page, it'd be too cumbersome to point them all out.
2. The distinction between the "general task dependency" and the "data dependency" does not seem significant enough. The data-dependent task dependency actually depends on the "general task dependency" as stated in the paper. This is probably manifested in the relatively slight improvement of the method compared with the SOTA. Perhaps more clarity on the difference and contribution of each "level" would make the significance stand out  clearer.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors propose a multi-task learning method that uses attention mechanism to identify relations between the tasks.  

Method:

- The authors motivate the use of attention mechanism for identifying a sample-dependent measure  of task relatedness by that "task dependency can be different for different data samples.." At the introduction stage this argument was not clear to me. I would suggest to expand this part of the paper by providing a stronger motivation for the proposed approach.
- what kind of mappings are used in (2)?

Experiments:

- it seems that all the datasets used are for multi-label learning. Thus, in the evaluation procedure, could the same input X appear both in the training and the test data sets (but in different tasks)? If yes, I believe it might make the evaluation less thorough. In either case it would be helpful to have this information in the description of the setting
- since use of attention is the main contribution of this work, but not the only part of the method,  I would recommend adding to the evaluation a method which is equivalent to the proposed one, but doesn't involve attention (i.e. only uses D). 

Additional comments:

- in its current form the manuscript is rather hard to follow, it requires a thorough proof-reading
- it is unclear what Figure 1 on page 2 is for
- on page 2 phrase "... the label ratio is imbalanced." is confusing. I believe the authors meant that the data (not label) proportions between the tasks are uneven
- on page 3 the authors say that minimisation of the empirical risk (eq. (1)) is "the goal of multi-task learning". This sentence needs rephrasing, because from the point of view of empirical risk minimisation any multi-task approach is worse than the corresponding single-task version (i.e. its empirical risk is higher). Only in terms of the generalization performance one can argue that information sharing is beneficial.
- notation in (3) is confusing - index i is used in two meanings
- it's unclear what k in eq. (4) is
- it seems that a few references are broken

To my knowledge the idea of making amount of transfer between tasks dependent on the particular sample at hand is new. Therefore, in my opinion, with improved presentation (and in particular motivation at the beginning of the manuscript) and additional evaluation demonstrating effects of the attention component the manuscript could be recommended for acceptance.

---------------------------------------------------------------

I thank the authors for their comments. The quality fo the manuscript has indeed improved and the differences with the existing methods are clearer. However, in light of the reviewers' comments, I agree that at least the experimental section needs to be extended by adding relevant baselines. In particular, comparison to "End-to-end multi-task learning with attention" is needed to demonstrate importance of the task-level dependence measure. If direct comparison is not possible (or in addition to it) I would suggest to evaluate a modification of L2MITTEN with matrix D being equal to all 1s.  
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes a ‘Learning to Transfer via Modeling Multi-level Task Dependency’ for multi-task learning’, which uses the attention mechanism to learn task dependency.

In the introduction, authors claim ‘most of the current multitask learning framework rely on the assumption that all the tasks are highly correlated’. I don’t think this claim is correct. In fact, most state-of-the-art multi-task learning models can learn task dependency via different forms.

In the proposed network, different tasks have their own encoder, which leads to a large number of model parameters especially when there are a large number of tasks. This situation becomes even worse when each task has a limited number of labeled samples.

The attention has been used in multi-task learning. Authors can google ‘multi-task learning attention’ to find related works. Of course authors need to compare with those related works.

A typo: “is theposition-wise mutual attention between”
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The submission argues for the modeling the relationships between different tasks and incorporating such relationships when training multi-task frameworks. Though the basic concept (usefulness of modeling and incorporating the relationships among tasks) is valid, the submission has a number of critical issues, namely missing prior work that did that that already, missing critical specifics of the method, and unnecessary mix of different concepts. 

Elaborated comments: 

A) Authors seem to be unaware of critically related prior work that specially modeled task relationships and did much of what's proposed in this submission, especially "Taskonomy: Disentangling task transfer learning". The "relationship among tasks" that this submission frequently talks about is the main concept in taskonomy 2018 paper (see their abstract). Besides the apparent similarities (eg the fig 1 of this submission vs fig 1&2 of taskonomy or fig 4 of this submission vs fig 13&7 of taskonomy), the formulation has strong similarities too (transferring from "task-specific" encoders of source tasks to target tasks using transfer readout functions, or ensembling multiple task-specific representations which seem to be the same as taskonomy's higher order transfer). This submission should be majorly revised in light of prior work and the critically relevant ones should be discussed and experimentally compared to. 

B) The presentation suffers from missing critical specifics. For instance, the "general task dependency" matrix  shown in Fig 3 and mentioned in page 4, which seem to be the same concept as taskonomy's task affinity matrix, is only mentioned in passing. While that seem to be one of the most important components of the method and its definition and extraction method should be discussed. 

C) Inline with the point B above, the presentation of the "Transfer Block" and what the authors refer to as "Point-wise Mutual Attention Mechanism" has issues and missing details. This block could potentially have new points in it, but it's not feasible to judge that and its technical correctness given the current disposition. For instance eq 2 seem to suggest the authors develop a universal representation space where all task-specific representations get mapped to and all target tasks can be inferred from (to reduce T^2 complexity to 2T). The rest of the section does not provide a clear implementation of this and add mathematical/notation confusions. Eg H_i_j is defined to be the task-specific representation of the source task i but is indexed over both tasks i and j where j is the target, or there is a E_j(X_j) where both indexes are j while E's index is over tasks and X's index is over datapoints. 

Similarly the submission seem to jump over certain concepts/terms e.g. "multi-view task dependency" in page 4 vs"multi-level task dependency" in the title, etc. What exactly "view" or "level" mean here? Are those phrases really needed? Dropping any loosely grounded phrase would be a useful practice toward a clearer presentation.

Overall, unfortunately the submission suffers from serious issues in its current shape. 


----
Comments after rebuttal stage: 

Thanks to the authors for the rebuttal. It provided some help, but unfortunately it doesn't resolve the majority of the issues, as most of them are too major. A clear discussion on how the proposal is different and why it is better than the recent works that were not cited would be needed, and likely authors needed strong experimental comparison with some of them, eg to prove both general and data-specific task dependency is needed. 

I also didn't find the hierarchical justification clear or convincing "The reason is that different from the image which [1] focuses on, text and graph data are hierarchical: word -> sentence and node -> graph. The task dependency at the basic level (word and node) may be different from the general task dependency". 

review_title: Official Blind Review #2

Meta-review: In this work, the authors address a multi-task learning setting and propose to enhance the estimation of task dependency with an attention mechanism capturing sample-dependant measure of task relatedness. All reviewers and AC agree that the current manuscript lacks clarity and convincing empirical evaluations that clearly show the benefits of the proposed approach w.r.t. state-of-the-art methods. Specifically, the reviewers raised several important concerns that were viewed by AC as critical issues:
(1) the empirical evaluations need to be significantly strengthened to show the benefits of the proposed methods over SOTA -- see R2’s request to empirically compare with the related recent work [Taskonomy, 2018] and R4’s request to compare with the work [End-to-end multi-task learning with attention, 2018]. R4 also suggested to include an ablation study to assess the benefits of the attention mechanism. Pleased to report that the authors addressed the ablation study in their rebuttal and confirmed that the proposed attention mechanism plays an important role in the performance of the proposed method. 
(2) All reviewers see an issue with the presentation clarity of the conceptual and technical contributions  -- see R4’s and R2’s detailed comments and questions regarding technical contributions; see R3’s and R4’s comments that the distinction between the general task dependency and the data-driven dependency is either not significant or is not clearly articulated; finding better examples to illustrate the difference (instead of reiterating the current ones) would strengthen the clarity and conceptual contributions.  
A general consensus among reviewers and AC suggests, in its current state the manuscript is not ready for a publication. It needs more clarifications, empirical studies and polish to achieve the desired goal.


Decision: Reject