Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a relational multi-task learning setting and design a MetaLink model.
main_review: According to the setting introduced in Section 2, it seems that the proposed setting is just a meta learning setting but not a multi-task learning setting. The name of relational multi-task learning seems a bit misleading.

In the first paragraph of Section 3.1, the claim is not general, since in many cases, the task head consists of multiple layers instead of only one layer. In the multi-layer case, based on the second paragraph of Section 3.2, the proposed MetaLink model seems not applicable.

The knowledge graph stated in this paper is just a graph. This is different from the actual ‘knowledge graph’ and this seems misleading.

The heterogeneous GNN used in Section 3.2 seems no difference with the conventional GNN. I did not see significant novelty in the proposed MetaLink model.

In experiments, only multi-label datasets are used. Authors should experiment on multi-task benchmark datasets such as CityScape, NYUv2, Pascal-Context, Taskonomy, Office-31, Office-Home.
summary_of_the_review: The proposed setting seems a bit misleading. The proposed model has no significant novelty. More standard benckmark datasets should be used.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a novel multi-task learning framework called MetaLink that takes the opportunity to utilize the auxiliary task labels by constructing a knowledge base with the tasks and data(instances) as nodes and auxiliary labels as the edge labels between them. This method shows significant empirical success 
main_review: Strengths:
1. MetaLink is able to leverage the correlations among tasks successfully.
2. I find the idea of creating the knowledge graph for harnessing the relational information about the data and tasks very intreging. 
3. The usage of the labels of the auxiliary task is well motivated.

Weakness: 
1. The writing could have been more precise in some parts of the paper.
summary_of_the_review: Concerns:
1. The introduction could have been more concise, I feel the authors could have avoided explaining the technicalities of the knowledge graph in the introduction. Rather, it would help the reader to be better situated in your research space if you discuss the fundamental difference between some of the related models and yours at an early stage of the paper. For example, the authors could have mentioned Wang et al. 2016 and ML-GCN  (Chen et al.) in the introduction itself. and explained why their work is different from the others. 

2. Are the comparisons fair? The GraphConv( ) step uses additional parameters, especially when the number of layers (iteration) increases. Could you include the number of parameters for all the baselines and your method?

3. The tasks are very specific to multilabel classification. What happens when objectives are different for each task? How would MetalLink adapt to that?

minor: In the paragraph: "Results on biochemical datasets.", Table 5 ---> Table 1.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper exploits the multi-task modeling using a heterogenous Graph Neural Network. The key contribution of the work is having both data (sample) and task nodes in the same graph, focusing on data-task relation, accommodating for sparse task labels by design.

Experimental results show relevant improvements with the proposed relational model on biomedical datasets and Imagenet splits for few-shot.

main_review: Strengths
- Innovative formulation
-- a wide “range of heterogeneity” in the graph: a node for each sample in the support set a node for each task - backed up by relevant experimental improvements
-- dynamical graph structure based on the existing labels for the current entry
- Experimental results are strong for the few-shot setup and medical datasets
- Ablations are very informative: higher improvement in more correlated tasks and using more auxiliary tasks come with higher performance

Weaknesses
- Table 1. Why are there so many variations of the KG layer (both inside the same dataset and per metric)? Is there some consistent explanation for this hyperparameter?
- Even though it is quite simplistic, Figure 2 is confusing. The authors should highlight somehow that the edges show the conceptual connections between nodes (and not the real link in the algorithm flow). Such an example: in c) and d) why are data nodes (the ones from the triangle structures) useful if they are already connected via known label edges?
- Ablations on the graph structure decision would be useful (the ones from eq. 2 and 3, sustaining the “special GNN design”)
- Prior work is focused on GNN approaches (even in the multi-task paragraph), comparison from the design point of view with prior-art approaches in relational multi-task graphs could offer a more complete view (eg. Robust Learning Through Cross-Task Consistency - https://arxiv.org/abs/2006.04096, Self-Supervised Learning in Multi-Task Graphs through Iterative Consensus Shift - https://arxiv.org/abs/2103.14417)

Questions
- How many parameters does MetaLink learn?
- How could the authors adapt the proposed MetaLink to work with dense predictions rather than 1-D tasks? What would be the implications?

Others (minor, eg. typos)
- KG layer (knowledge graph) is never expanded
- Please mention what you show in the tables (top5 performance)
- “where T = {1, 2, ..., m} is integers between 1 and m.”
- Eq. 1 should have an (l-1) instead of (l) - it uses only h(l)

summary_of_the_review: The paper is sound, introducing and validating an interesting heterogeneous graph design focused on each task and data sample as a different node. The work also offers an approach for the common labeling limitation in multi-task problems (by working with sparse labels in a dynamic graph architecture ). I recommend acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, they introduce relational multi-task learning in which they construct a graph with the data points and task and labels as edges. They represent the data points as the embedding of NN model and task as the last layer of NN model for that task. Then they solve the link label prediction problem between each node and task by using GNN and heterogeneous message passing method to predict the label of test data point on a new task at inference time.
main_review: Strengths:
- The paper is very well-organized and well-written. Algorithm 1 is self-explanatory.

- I really liked the idea of creating a knowledge graph and then solve the link label prediction problem.

- The experiments are thorough, I enjoyed reading the experiments section; they cover ablation studies, and answer several question in there.  Specifically how relational ML could take advantage of more related tasks. In addition, the improvements are noticeable (Tables 1-4).


Notes:
- In figure 1, in the right panel, I would put 1's and 0's in a right place, e.g., 1's for the solid lines or 0's with the dashed line?!

- I know that I have seen figure 2 somewhere before, but do not remember the paper, if you have seen this figure from another paper, that will be a good idea to put the reference.

- I would recommend defining use the name of your model in figure 4 instead of using 'ours'
summary_of_the_review: In general, I believe the paper is innovative and descent. The backed their proposal with thorough experiments. I think this work will be useful for researchers working in the area of multi-tasking.
review_title: null

Meta-review: The paper describes a novel learning scenario where there are many related tasks, some seen at test time, and some seen only at training time, where additionally the task labels can be hidden or present.  This approach generalizes both a "relational setting" (where auxiliary task labels could be used as features) and a "meta setting" (where new tasks need to be solved in a zero-shot setting using data from related tasks only).  The idea behind the method is to do MTL with a common representation and a set of task-specific heads, and build a graph where (1) tasks are nodes associated with the parameters of their task-specific "heads" and (2) edges link examples to tasks with known labels.  A GNN method is then used to find regularities in the graph.

Pros
 - The setting is innovative and the approach is novel
 - The experimental results are strong

Cons
 - Some of the terminology seems awkward and/or strained (eg "knowledge graph" for the task-example graph)
Decision: Accept (Spotlight)