Relational Multi-Task Learning: Modeling Relations between Data and Tasks
Authors: Kaidi Cao
Jiaxuan You
Jure Leskovec
Venue: ICLR 2022 Spotlight (2022)
Keywords: Graph Neural Networks
Relational Representation Learning
Multi-task Learning
Meta Learning

Abstract: A key assumption in multi-task learning is that at the inference time the multi-task model only has access to a given data point but not to the data point’s labels from other tasks. This presents an opportunity to extend multi-task learning to utilize data point’s labels from other auxiliary tasks, and this way improves performance on the new task. Here we introduce a novel relational multi-task learning setting where we leverage data point labels from auxiliary tasks to make more accurate predictions on the new task. We develop MetaLink, where our key innovation is to build a knowledge graph that connects data points and tasks and thus allows us to leverage labels from auxiliary tasks. The knowledge graph consists of two types of nodes: (1) data nodes, where node features are data embeddings computed by the neural network, and (2) task nodes, with the last layer’s weights for each task as node features. The edges in this knowledge graph capture data-task relationships, and the edge label captures the label of a data point on a particular task. Under MetaLink, we reformulate the new task as a link label prediction problem between a data node and a task node. The MetaLink framework provides flexibility to model knowledge transfer from auxiliary task labels to the task of interest. We evaluate MetaLink on 6 benchmark datasets in both biochemical and vision domains. Experiments demonstrate that MetaLink can successfully utilize the relations among different tasks, outperforming the state-of-the-art methods under the proposed relational multi-task learning setting, with up to 27% improvement in ROC AUC.

Published as a conference paper at ICLR 2022
RELATIONAL MULTI-TASK LEARNING: MODELING RELATIONS BETWEEN DATA AND TASKS
Kaidi Cao"
Jiaxuan You
Jure Leskovec
Department of Computer Science, Stanford University (kaidicao, jiaxuan, jure (ccs stanford. edu
ABSTRACT
A key assumption in multi-task learning is that the inference time the multi-task model only has access to given data point but not to the data point's labels from other tasks. This presents an opportunity to extend multi-task learning to utilize data point's labels from other auxiliary tasks, and this way improves performance on the new task. Here we introduce a novel relational multi-task learning setting where we leverage data point labels from auxiliary tasks make more accurate predictions on the new task. We develop MctaLink, where our key innovation is to build a knowledge graph that connects data points and tasks and thus allows us to leverage labels from auxiliary tasks. The knowledge graph consists of two types of nodes: (1) data nodes, where node features are data embeddings computed by the neural network, and (2) task nodes, with the last layer's weights for each task as node features. The edges in this knowledge graph capture data-task relationships, and the edge label captures the label of data point on particular task. Under MetaLink, we reformulate the new task and a task node. The Metal Link framework provides flexibility to model knowledge transfer from auxiliary task labels to the task of interest. We evaluate Metal.nnk on 6 benchmark datasets in both biochemical and vision domains. Experiments demonstrate that MetaLink can successfully utilize the relations among different tasks, outperforming the state-of-the-ar methods under the proposed relational multi-task learning setting, with up to 27% improvement in ROC AUC.
link label prediction problem between a data node
INTRODUCTION
The general idea of learning from multiple tasks has been explored under different settings, including multi-task learning (Caruama, 1997), meta learning (Finn et al., 2017), and few-shot learning (Vinyals al., 2016; Cao al., 2020a;b). While these learning settings have inspired models that can utilize relationships among tasks (Chen et al., 2019; 7amir et al., 201 8; Sener & Koltun, 2018; Lin et al., 2019; Ma al., 2020), they are not able to capture the full complexity of real-world machine learning applications. Concretely, when learning from multiple tasks, current approaches assume that the test data points have no access to the labels from other tasks when making predictions on a new task. However, this assumption oversimplifies potential useful knowledge in many real-world applications.
For example, multi-task learning studies simultaneously learning multiple predictive tasks to exploit commonalities between the tasks. At the test time the multi-task model predicts labels of a given data point for the tasks of interests, e.g., predicting whether the chemical compound x is non-toxic. At the same time, one may also have access to the data point's labels for some other auxiliary tasks, e.g., the compound x has a positive result on two toxicology tests. Such auxiliary task labels could greatly help with improved predictions.
However, current deep learning architectures cannot model such knowledge transfer between auxiliary tasks/labels and the target tasks. Naively concatenating the known labels to the input features has its limitations, especially since such labels are sparsely available, and it is also unclear how to use the approach for new and unseen tasks. Another potential solution to model such flexible and conditional inference is through generative models (Dempster et al., 1977; Koller & Friedman, 2009). Although generative models are powerful, they are notoriously data-hungry, thus it very difficult to construct. and train generative model for high dimensional data (Turhan & Bilge, 2018).
"The two first authors made equal contributions.

---

Published as a conference paper at ICLR 2022
Input Data
Default Approach
MetaLink
f1
f2
x(1)
1
0
Feature extractor
x2)
?
1
fo
x 1
?
Figure I: In the relational multi-task setting, the model learns to incorporate auxiliary knowledge in making predictions to achieve data efficiency. Concretely, given observations x( and their labels (u (0/1 in this example) on subsets of tasks { the goal is to build model that can harness the auxiliary task labels {u, (1) and make predictions on new task 4n- A standard approach is to build multi-head deep neural network, with prediction head for each individual task fz. However, such approach cannot utilize auxiliary labels. In contrast, our proposed MetaLink reinterprets the last layer's 's weights of each task as task nodes and creates a knowledge graph where data points and tasks nodes and labeled edges provide information about labels of data points on tasks. Then, when predicting data point's ' label for a given task tj. MetaLink uses labels from other tasks to improve predictive performance.
Here we propose a new multi-task learning setting called relational multi-task learning. In our setting distinguish between the target tasks (e.g., predicting molecule toxicity), which are tasks we aim to predict, and auxiliary tasks, which are tasks for which data point's labels are available at the inference time. Note that under our setting, each data point might have labels available for a different subset of auxiliary tasks. The goal then is to achieve strong predictive performance through leveraging labels of given data point on some subset of auxiliary tasks.
To tackle the relational multi-tast task learning, we propose MetaLink', a general discriminative model that can explicitly incorporate the knowledge from auxiliary tasks. Our key innovation is build a knowledge graph that connects different tasks ty and data points x()) (Figure 1). The first step of our approach is to take input data points x and the feature extractor (i.e., neural network) fo to get its embedding (6), Then we build the knowledge graph which consists of two types of nodes: task node fú if data point (()
data nodes (+) and task nodes fj- A data node (+) connects
participates in task ln and the edge is annotated with the label y of (() on task Lj- We initialize
data node features be the last layer embedding ( in the feature extractor fe, and task node 4 features are instantiated as the last layer's weights wy-
Given our knowledge graph, we reformulate the multi-task learning problem link-label prediction problem between data nodes and task nodes. This means that at the inference time MetaLink is able to use all the information about a given data point x (including its labels on auxiliary tasks {t,) to predict its label on new task bn- We solve this link label prediction learning task via Graph Neural Network (GNN) (Hamilton et al., 2017; He et al., 2019; Xue et al., 2021). Unlike previous works, e.g.. ML-GCN (Chen et al., 2019), that solely model relationships among tasks, MetaLink allows flexible and automatic modeling for data-task. data-data and task-task relationships.
We evaluate MetaLink on six benchmark datasets in both biochemical and vision domains under various settings. Empirical demonstrate that MetaLink can successfully utilize the relations among different tasks, outperforming the state-of-ther methods under the proposed relational multi-task learning setting, with up to 27% improvement in ROC AUC.
2 RELATIONAL MULTI-TASK LEARNING SETTINGS
Here we first formally introduce the settings for relational multi-task learning. Suppose we have m machine learning tasks [e, where T 21, m) integers between and m. We propose to categorize different settings in two dimensions: (1) whether the task is relational task, i.e. if auxiliary task labels can be used at inference time; and, (2) whether the task is a meta task, i.e., if the
Source code available https: htta:///iithii httmm//aaaaataafoor/rraaapgoo
2

---

Published as a conference paper at ICLR 2022
(a) Standard: Supervised Setting
(b) New:
(c) Standard:
(d) New:
Relational Setting
Meta Setting
Relational Meta Setting
Training stage
Test stage
Seen Data Node
Unseen Data Node
Known Label during model inference
Seen Task Node
0 Unseen Task Node
Unknown Label during model inference
Figure 2: Our MetaLink framework allows for modeling four different multi-task learning settings: O represent data nodes and D represent task nodes. Blue represents the data/tasks seen in the training stage and white denotes the data/tasks seen only in the test stage. During model inference (for both the training and the test stage), the label of data-task pair with solid line is known, while we want to predict labels of the data-task pairs with dotted lines.
task at test time has been seen at the training time. Altogether, there are four possible task settings, which are illustrated in Figure 2 and below.
Standard supervised setting. Let x denote the input and v/ e denote the corresponding label associated with task i.e. y) tg. Standard supervised multi-task learning can be represented as
Train: (x(0 Luj (2) ~ ()EET}
Test: {x - Luj) lybeer}
where connects the input and the output. Training and test sets cover non-overlapping data points. (+) We use yj to concisely represent u/ ~ t, later on.
(8)
Relational setting. In relational setting. in addition to input (+), we assuiee we also have access to auxiliary task labels when making predictions. Taux and Tistt are partitions of integers (I) that relates to subsets of tasks. Specifically, Taxx refers to the indices of tasks that input x has access to, and Tiss are the indices of tasks that wish to predict; these two sets non-overlapping. i.e., Taux n Tiest = 0, and
i.e., the partition
T:(1) and Ti(2) can be different. The
input-output pairs are now in the form of
Train: {(x²), fu, (9)) ser
(0),
Test: {<x(), {1,5) JETS)) + {B, (0) Jerr
Meta setting. In the meta setting, we want to learn to predict unseen tasks at the test time. Formally,
let T,, Tu denote the set of partitions for seen tasks (used at training time) and unseen tasks (used at test time), where T2 n T. 0. Here, we have access to a batch of samples with labels as the support set S and the objective is to correctly predict samples in the query set Q.
Train: given S - (((x()) (v,)) (1) ) } predict Q = (xx(0 + {u]) (1) bsen }
Test: given S (x()) (v,5) ) }. predict Q = (x(0) - (v, (2)
Relational meta setting. Relational meta setting combines the features of relational setting and
meta setting. Similar the meta setting, we aim to predict unseen tasks Tū at test time; meanwhile, similar to the relational setting, we also assume having labels on a limited number of auxiliary tasks Tax C T3 to harness. Formally, we have a support set and a query set the form of
Train: given S = {(x()) fo,") ) predict Q = {(x(), (u)" (1) DETS)) + {0, ())
Test: given S = ((xe0)
( (,//)) ( )) predict Q = {(x(0) (u,5" )ETT)) - (v/)' (/,5"))))))))) )ee!"
3

---

Published as a conference paper at ICLR 2022
3 METALINK FRAMEWORK
Next, we describe our MetaLink framework, which allows us to formulate the above multi-task learning settings single framework. In particular, MetaLink formulates them as a link label prediction task on a heterogeneous knowledge graph, this way, MetaLink can harness the relational information about data and tasks.
3.1
BUILD A KNOWLEDGE GRAPH ON TASK HEADS
We first recap the general formulation of a neural network. Given data points {x()) and labels (u, ) a deep learning model can be formulated as a parameterized embedding function So (which is deep) and a task head fw. consisting of only single weight matrix w. fo maps a data point x(1) vector embedding space, fo(x(0) 2 E RP. fw then maps an embedding (x)) to prediction j(t) TR2 (z(+)) w (1) =(((), When task head involves multi-layer transformation, we have (z()) wT g((()) ((و), where g(-) can be arbitrary function. In multi-task learning settings, people usually assign multiple task heads to a neural network. Suppose we have m tasks {t, 1/=1 then there are task heads such that 0/5" fw (z)).
Here, our observation is that the weights in task heads Wi...., w,, and the data embeddings (X)) play symmetric roles in multi-task prediction task (due to the dot product). Therefore, instead of viewing weights Wi,.... : W as parameters in a neural network, we propose to represent Wi...., w as another type of input that supports the prediction. Essentially, we reformulate task head from This new perspective, where both task weights w, and embedding 2 viewed as input, enables to build more sophisticated predictive model fe- fa in the sense that it contains two main steps, i.e., GraphConv(-) and EdgePred(()). In general, EdgePred(-) has similar model complexity as sw,, whereas GraphConv()) provides additional expressiveness.
0/ - fw, (z to of =
In MetaLink, we propose to build a knowledge graph over task weights w, and embedding ( By building this knowledge graph, we can succinctly represent relationships between data points and tasks, as well as different multi-task learning settings. Concretely, a knowledge graph helps us easily express any data-task relationship (e.g.. data point has label given task), data-data relationship (e.g., two data points are similar or not), or task-task relationship (e.g., hierarchy of different tasks). Moreover, knowledge graph greatly simplifies the representation of all the multi-task learning settings that we outlined in Section 2: in fact, all the settings can be viewed as link label prediction tasks where different portions of the knowledge graph can be constructed, illustrated in Figure 2.
We define the knowledge graph as G = (V, where V is the node set and E C V x V is the edge set. We define two types of nodes, data nodes Va = [x(11) ,.... x(m))
} and task nodes V, = (tm We can then define edges between data and task nodes Est C V/ V,, within data nodes Eda S Vi x Va. and within task nodes En S V2 x Vz. MetaLink framework can work with all three types of edges; however, since most benchmark datasets do not have information on data- data or task-task relationship, we focus on data-task relationship Eat in the remaining discussions.
(8) Specifically, we define Ex based on task labels. Edt { (x(1), .t) VjeTm. }. i.e., we connect a data node (6) with a task node ty label v exists.
3.2 LEARN FROM THE KNOWLEDGE GRAPH VIA HETEROGENEOUS GNN
Given the constructed graph, we then discuss how MetaLink learns from the built knowledge graph.
Initialize node/edge features. First, we initialize features for the knowledge graph. Concretely, we initialize data node features to be the data embeddings (6) computed from the feature extractor fo(.). h, (0) (1), We initialize the seen task node embeddings using the weights in task heads w,, h (0) wj- In meta settings where unseen task nodes appear during test time, we will initialize those nodes by a constant vector of I; such initialization ensures that MctaLink can generalize to unseen tasks, since the node feature construction process is inductive.
Without loss of generality, we assume task is binary classification 1-D regression
4

---

Published as a conference paper at ICLR 2022
Predict via a heterogeneous GNN. We implement the predictive model fo over data nodes and task nodes as a GNN. The goal of GNN is to learn expressive node embeddings h, based on an iterative aggregation of local network neighborhoods. The 1-th iteration of GraphConv(.), or the I-th layer, can be written as:
h AGG!)) (loo(((MMss ((MSO u No(t)).
where h is the node embedding after iterations, h have been initialized as explained above, and (v) denotes the direct neighbors of v. AGG is the acronym for aggregation function and MSG the acronym for message function. We perform I. GNN layers on top of the knowledge graph G that we have built. After updating data and task node embeddings, we can make predictions on the given task through EdgePred(-) in the form of Új MLP (CONCAT(H) (T) ))
In general, Metal Link should work with any GNN architecture that follows the definition in Equation 1. We use the GraphSAGE layer (Hamilton et al., 2017) in MetaLink (W(), UO)) are trainable):
h U CONCAT |MEAN (RELU ng-"") E ((t)) n(
((o))).
Next, we discuss the special GNN design used in MetaLink which has shown to be successful.
Special GNN designs in MetaLink. We make three extensions in the formulation in Equation 1. First, since there are two rypes of nodes our knowledge graph, we define different message passing functions for different message types, i.e.. the message from data nodes task nodes, and the message from task nodes to data nodes. Second, we include edge features in the message computation. This is especially important for formulation, since the task label values y/ included edge features, and should be considered during GNN message passing. Concretely, we extend Equation 2 into:
h = T (MEAN( (RELU 1 h (1-1) + y E.(()))) n
(1-1)
where 1[v E Va, u Vi]] indicates the message type (whether from task data, data to task), and o( an additional trainable weight that allows task label ((()) participates in message passing. Finally, we let each GNN layer make a prediction and sum them up the final prediction; this way, the final prediction is obtained from mixed information from different hops of node neighbors. We observe this multi-layer ensemble technique can help MetaLink make robust predictions.
MetaLink for relational meta setting. Here we provide detailed description how to ap- ply MetaLink for the relational meta setting in Algorithm 1. At training time, since most of the existing multi-label datasets are not designed for meta setting, we manually simulate such setting by sampling mini batch with support S = { {x)(), {v, (t) )en(t)) and query set
(xx)) (0,5' (ers9)
fu," (i) (edo J. We make sure the sampled meta tasks T.( and auxiliary
knowledge tasks (1) have no intersection. We first use the feature extractor fo to get data embeddings and use the embeddings initialize data nodes. To initialize task nodes, we either: (1) use 1 to initialize if the task is a meta task; Or, (2) use trained weights otherwise. We construct the edge set {of} (0) jETi))
E by connecting each lata-task pair based on {v,) (0) )e7 in the support set and
(we eliminate the relations we want to predict) in query set. Now that have all the components of the knowledge graph, we can apply the predictive graph model fo to learn expressive data and task node embeddings and make predictions.
At test time, we use the same pipeline as in training to construct the knowledge graph and run inference. Please refer to Appendix A for the pipelines of relational or meta settings.
4 EXPERIMENTS
Here experimentally show that our proposed MetaLink flexibly handles different settings and leverages knowledge from auxiliary tasks. We first evaluate our algorithms on Tox21 (Huang et al., 6), Sider (Kuhn et al., 2016), ToxCast (Richard al., 2016), and MS-COCO (Lin et al.,
5

---

Published as a conference paper at ICLR 2022
Algorithm 1 MetaLink Training in Relational Meta Setting
Require: Dataset Dusin {xx 3) A parameterized embedding function fo. Last layer weights for each task wj. A parameterized heterogeneous GNN Sa. Number of GNN layers L.
1: for each iteration do
2:
S,Q < SampleMinnBatch(DDeg
D. Simulate meta setting in training
3:
{z) - fx(x) for (S, Q)
d v-(0) = [h; + z for z E {z}}
4:
> Initialize data nodes
V((0) = -[h; (0) f- 1 if meta else Wy for each wz}
5:
b Initialize task nodes
E {eey f (x) for (6) E (S,)))
6:
b Initialize edges
7:
for l 1 L do
Vd V. - GraphConv(V2 (V (v(l-1) v!ll1),, F) with Ss
8:
9: logits Edge v!( with fo
Backward (Criterion(logite (fo) (00
1eo) } Q))
2014) datasets with various controllable settings on relational multi-task learning. To additionally demonstrate the advantage of MetaLink, we also include experiments on a well-studied task: few-shot learning. Our core algorithm is developed using PyTorch (Paszke et al., 2017). We use one NVIDIA RTX 8000 GPU for each experiment and the most time-consuming one (MS-COCO) takes less than 24 hours. Please refer to Appendix A for additional low-level implementation details.
4.1 EVALUATING METALINK ON RELATIONAL MULTI-TASK SETTINGS
Datasets. We simulate four relational multi-task settings (Figure 2) using four widely-used multi- label datasets. Tox21 (Huang et al., 2016) contains 12 different toxicological experiments for each sample with binary labels (active/inactive). Sider (Kuhn et al., 2016) 6) is database of marketed drugs and adverse drug reactions (ADR), grouped into 27 tasks. ToxCast (Richard et al., 2016) contains about 8K pairs of molecular graphs and corresponding 61 617-dimensional binary vectors that represent different experimental results. Microsoft COCO (Common Objects in Context) (Lin et al., 2014) is originally a large-scale object detection, segmentation dataset. By counting whether each type of object exists in a scene as a single task, it also serves as the default large-scale dataset for benchmarking multi-label classification vision. There are 80 binary classification tasks with an average of 2.9 positive labels per image.
Experimental setting. We evaluate MetaLink on all four settings described in Figure 2, and we summarize the detailed configurations as follows. STANDARD SETTING: standard supervised learning. In order to have fair comparison, we evaluate the same set of tasks with unknown labels as we sampled in relational settings. RELATIONAL SETTING: we assume each example has access to labels of 20% of all tasks in each dataset. We evaluate on the rest of the tasks with unknown labels. META SETTING: we hold out 20% of the tasks at the training time. We only evaluate on the held-out tasks the test time. We use 256-shot setting, meaning at test time, to initialize the prototypes of unseen tasks. The reason why the number of shots is much larger than what is commonly used in few-shot learning (I-shot, 5-shot) is that positive labels are sometimes sparse in certain tasks. RELATIONAL META SETTING: we hold out the same 20% of the tasks at the training time as in the meta setting. At test time, we assume each unseen task has access to 20% of the labels of seen tasks. We will release our splits of meta setting to promote reproducibility.
use 256 data points
support set
Though the main motivation of our work is to utilize auxiliary task labels, we still include few baselines under the standard supervised setting in order to benchmark ofate-of-the-ar results. The simplest one is (1) Empirical risk minimization (ERM): we train the network with cross-entropy loss under the standard supervised setting; (2) Various commonly used graph neural network architectures designed for molecules, MPNN (Gilmer et al., 2017), DMPNN (Yang et al., 2019), MGCN (Lu et al., 2019), AttentiveFP (Xiong et al., 2019); (3) GROVER (Rong et al., 2020) integrates Message Passing Networks into the Transformer-str architecture. By leverage pretraining, it achieves ofte-of-the-ar art results on the aforementioned molecule datasets; (4) Baseline++ (Chen et al., 2018): since there is no prior work addressing meta setting for multi-task learning, adapt Baseline++ from few-shot
Baselines.

---

Published as a conference paper at ICLR 2022
Table 1: Results of different multi-task learning settings on graph classification tasks, measured in ROC AUC. MetaLink can successfully utilize the relations among different tasks, outperforming the state-of-the-ar method GROVER (Rong et al., 2020) under the relational setting.
Method
Secting
Tox21 (12 tasks)
Sider (27 tusks)
ToxCast (617 tasks)
MPNN (Gilmer et al., 2017)
80.8:22.4
59.5+13.0
69.1
DMPNN (Yang et al., 2019)
82.61:233
63.2:t:.3
71.81:11
MGCN (Lu et al. 2019)
70,7+116
55,2+t188
66.3+0.9
Standard
AttentivFF (Xiong et al., 2019)
80.7:4:.00
60.5:16.0
$7.9+1.0
(ROVER(48M) (Rong al., 2020)
81,9+2200
65.6:20.6
12.3++10
GROVER 100M) (Rong al., 20100)
83,14-2.5
65.84-2.3
73.7++1.0
Standard
82.3:42.2
syer
60.9:1:2.4
jKG a)
Relational
83.7:1199
IKG koeeee))))
76,8:13.0
(KG layer
79.4=110 (KG le)ar-))
MetaLink
Meta
77.5422.1
(KG (KG =2)
57,945500 (KG liyer 2)
71.3+222 (KG lagar==2
Relational+Met:
79.2:12.9
(KG logor
65.4:14.3
(KG u)
84.3=1.2 (KGilyyrr
learning to this setting. We first train feature extractor on the training set. At test time we use the support set to train a linear classification layer.
Results on biochemical datasets. Table 5 summarizes the results on Tox21, Sider, and ToxCast datasets. We first clarify that when performing MetaLink under the standard setting, where there is no auxiliary task label to leverage, the knowledge graph we built degenerates to a trivial set since the edge set is empty. It has negligible difference compared with training a vanilla network with cross-entropy loss. Thus, it is reasonable that MetaLink under the standard setting has similar performance with the baselines. GROVER has better performance because leverages large-scale pretraining. In terms of comparing relational setting with standard setting, conclude that the proposed MetaLink successfully leverages the auxiliary knowledge for each sample. Notably, with auxiliary labels from 20% of the tasks, Metalinkk surpassed GROVER with large-scale pretraining by a large margin. In addition, we observe similar improvements between meta and relational meta settings. The empirical results for meta or relational meta setting are very close the performance in standard setting, proving the potential of data-efficient research in biochemical domains.
Results on MS-COCO. Table 2 summarizes the results on MS-COCO. Note that as also mentioned in the settings, we need large support set to ensure each task in the meta testing stage has least one positive label. This blocks the feasibility of using large model and input size as it is done in common benchmarks. Thus, we use ResNet-50 with an input size of 224 for all the experiments. We observe consistent improvements for both relational setting and relational meta setting. Metal ink also outperforms Bascline++ in the meta setting. Note that since label distribution for most of the tasks is highly skewed, mAP degrades more in the meta setting than for the biochemical datasets.
Table 2: Results on MS-COCO (80 tasks) dataset. We report the average accuracy and standard deviation over 5 runs on the validation set. We use ResNet-50 and our input size is 224. MetaLink achieves the best performance relational settings through harnessing the auxiliary labels efficiently.
Method
Setting
mAP
ML-GCN (Chen et al., 2019)'
Standard
69.15 + 0.19
BRM
Standard
71.22 + 0.15
Baselinc++ (Chen et al., 2018)
Meta
30.46 = 0.69
Standard
71.58 + 0.16
Relational
75.36 1 0.16
MetaL.nk
Meta
41.75 = 0.92
Relational+Meta
49.73 = 0.88
4.2 ABLATION STUDIES
Does Mnk truly learn to utilize correlations between tasks? To better understand the im- provement of our algorithm, we first plot the Pearson correlation heat map on Sider (Figure 3). We
'https://gt thub. com//eevvi/iiiiii Nanj waw.iingg/iiiee GCN
7

---

Published as a conference paper at ICLR 2022
Figure 3: Pearson correlation heat map on 27 tasks of Sider dataset.
Table 3: ROC AOC 3 tasks with the lowest
average Pearson correlation to other tasks (tasks 2, 17, 22), as well as tasks with the highest average Pearson correlation to other tasks (tasks 5, 20, 21) on the Sider dataset. We observe MetaLink gains more improvement on the tasks with high Pearson correlation.
Correlation Map
0.2
0.1
Setting
Low Corr Tasks
Hägh Corr Tasks
Standard
57.5 1 8,4
61.0 4.1
Relational
70.1 1: 7.8
80.6 + 3,3
Improvement
21.9%
32.1%
30
24
then find the top tasks with the highest and lowest average correlation with respect to the rest of the tasks. We report the performance of MetaLink on these two subsets of tasks, respectively (Table 3). We observe that MetaLink demonstrates larger improvement tasks with higher correlations. This experiment verifies that Metal iink learns to utilize correlations between tasks as expected.
How does MetaLink perform with varying ratio of auxiliary task labels? We vary the ratio of additional labels per test point and report the results in Figure 4. We observe consistent improvement as we gradually increase the number of auxiliary labels each dataset. This experiment demonstrates that MetaLink can successfully utilize marginal information whenever auxiliary task labels are added the dataset. The improvement is usually prominent for the first few added labels.
Tox21 (12 tasks)
SIDER (27 tasks)
ToxCast (617 tasks)
0.90
0.84
0.88
0.85
0.82
ROC AUC
0.86
0.80
0.80
0.78
0.75
0.84
0.76
0.70
0.74
0.82
0.65
0.72
MetaLink
MetaLint GROVER
MetaLink
0.80
GROVER
0.60
0.70
0.68
0.78
0.55
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6 0.8
0.0
0.2
0.4
0.6
0.8
Ratio additional label per test point
Figure 4: We vary the ratio of auxiliary labels per test point and plot the ROC AUC with the error bar. also plot the state-of-the-art method (GROVER) as the dash line which cannot utilize additional auxiliary task labels. The performance of MetaLink consistently improves more tasks utilized. In Sider, MetaLink can outperform GROVER by up to 27% when 90% auxiliary labels are provided.
4.3 EVALUATING METALINK ON STANDARD FEW-SHOT LEARNING DATASETS
Since the relational multi-tash setting is novel, there is very small number of baselines that we can compare against. The major aim of this section is to show the advantage MetaLink well-studied problem: few-shot learning. Note that there is slight difference between the meta setting above and few-shot learning setting here. For few-shot learning, for all the link label prediction tasks related to an input, there will be only positive link. This inductive bias could be easily incorporated by modeling all the link label predictions together using cross-entropy loss.
Experimental setting. We evaluate performance on two standard benchmarks: mini- ImageNet (Vinyals et al., 2016) and tiered-Imagelt (Ren al., 2018). We compare against MatchNet (Vinyals et al., 2016), Baseline++ (Chen et al., 2018), MetaOptNet (Lee et al., 2019), and Meta- Haaa-Baeeline (Chen et al., 2020b), who only assume the input is vector.
8

---

Published as a conference paper at ICLR 2022
Table 4: Results on 5-way, 5-shot classification on mini-ImageNet and fiered-ImageNe datasets. We
report the average accuracy and standard deviation over 800 randomly sampled episodes. MetaLink demonstrates consistent improvement from stacking KG layer =
Method
MatchNet (Vinyals al., 2016)
78.72 + 0.15
80.60 1: 0.71
Baseline++ (Chen et al., 2018)
77.76 + 0.17
83.74 + 0.18
MetaOptNet (Lee et al., 2019)
78.63 1 0.46
81.56 1 0.53
Meta-Baseline (Chen et al., 2020b) (KG layer
79.26 + 0.17
83.29 + 0.18
layer- 1) MetaLink (KG layer
MetaLink
79.86 + 0.18
83.91 + 0.17
81.13 1 0.17
84.68 1 0.17
Results. Table 4 shows that our MetaLink outperforms the standard few-shot learning benchmarks. The
Note that if we set KG Layer O, the proposed Metalink degenerates to experiments clearly demonstrate the benefits of building a knowledge graph on the last layer. Fur- thermore, as an ablation study we manipulate the number of KG layers and find that in the few-shot image recognition setting, there's an improvement for stacking 2 KG layers instead of I, meaning non-linearity is useful. We do not observe further improvements for more than 3 layers.
5 RELATED WORK
Multi-task learning. Multi-task learning is a learning paradigm that jointly optimizes a set of tasks with shared parameters. It is generally believed that relations across different tasks can improve the overall performance. Some works cast it as a multi-objective optimization problem and introduce multiple gradient-based methods to reduce negative transfer among tasks (Fliege & Vaz, 2016; Lin et al., 2019). Other works assign (adaptive) weights for different tasks using certain heuristics (Kendall al., 2018; Chen et al., 2018). Our empirical study also closely related to multi-label learning, where the problem is usually decomposed into multiple binary classification tasks (Tsoumakas & Katakis, 2007). There is line of work learning to leverage the relationship among tasks (Haller et al., 2021: Zamir et al., 2020). Wang et al. (2016) utilized recurrent neural networks to transform labels into embedded label vectors to learn the correlation among labels. The most recent work is ML-GCN (Chen et al., 2019), which uses GCN to map label graph into set of inter-dependent classifiers. Although the major motivation of our work is also about leveraging correlation among tasks, our problem formulation is new and thus yields a novel algorithm.
Exploring graph structure for data and tasks. There are prior works exploring graph structure for data points or tasks, and graph structure was proven to be effective in some tasks. Satorras & Estrach (2018) explores graph neural representations over data points only for few-shot learning. Besides, some works study how to transfer knowledge among tasks through constructing a graph over task nodes/classifiers only (Liu et al., 2019; Chen et al., 2020a). Along this direction, instead of building fully connected graph, recent works utilize auxiliary task structure/knowledge graph to build the graph (Chen et al., 2019; Lee et al., 2018). In contrast, and orthogonal to the papers above, we focus on modeling data-task relationships with the reinterpretation of the last layer. In addition, with data-task relationships solely, we are still able to capture data-data, task-task relationships implicitly through higher-order message passing.
CONCLUSION
We introduced relational multi-task settings in which the methods are required learn to leverage labels on auxiliary tasks to predict on the new task. These settings are impactful in the biomedical domain where labels of different tasks are often scarcely available. To address these settings, we propose MetaLink, which is general enough to allow us formulating the above settings in a single framework. We demonstrated that MetaLink can successfully utilize the relations among tasks, outperforming the state-of-the-art methods under the proposed relational multi-task learning setting, with up to 27% improvement in ROC AUC. We limit our focus to model data-task relationships since most benchmark datasets do not have information MetaLink is expressive enough to model such relationships. We leave extending MetaLink to more complex relationships or tasks for future work.
data-data or task-task relationships, though

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
We want to give special thanks to Hamed Nilforoshan, Michihiro Yasunaga. Weihua Hu, Paridhi Maheshwari, Xuechen Li, Wengong Jin, Octavian Ganea, Tommi Jaakkola, and Regina Barzilay for the thoughtful discussions. We gratefully acknowledge the support of DARPA under Nos. HR001 12190039 (TAMI, N660011924033 (MCS); ARO under Nos. 19911N6-6-100342 (MURI, W91 16-1 1-0171 (DURIP); NSF under Nos. 1A3-1835598 (CINES), OAC-1934578 (HDR), CCF-1918940 (Expeditions), IIS-2030477 (RAPID), NIH under No. R56LM013365; Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Chan Zuckerberg Biohub, Amazon, JPMorgan Chase, Docomo, Hitachi, Intel, KDDI, Toshiba, NEC, and UnitedHealth Group. Jiaxuan You is supported by JPMC PhD Fellowship and Baidu Scholarship.
REFERENCES
Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. In International Conference on Learning Representations, 2020a.
Kaidi Cao. Jingwei Ji, Zhangjie Cao, Chien- Yi Chang, and Juan Carlos Niebles. Few-shot video classification via temporal alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10618-10627, 2020b.
Rich Caruana. Multitask learning, Muchine learning, 28(1):41-75, 1997.
Riquan Chen, Tianshui Chen, Xiaolu Hui, Hefeng Wu, Guanbin Li, and Liang Lin. Knowledge graph transfer network for few-shot recognition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 10575-10582, 2020a.
Yu Chen, Yen-Cheng Liu, 7solt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2018.
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. A new meta-baseline for few-shot learning. arXiv preprint arXi::2003.04:: 000.:00000 2020b.
Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5177-5186, 2019.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1126-1135, PMLR, 2017.
Jorg Fliege and A Ismael F Vaz. method for constrained multiobjective optimization based on sqp techniques. SIAM Journal on Optimization, 26(4):2091-2119, 2016
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning (ICML), 2017.
Emanuela Haller, Elena Burceanu, and Marius Leordeanu. Unsupervised domain adaptation through iterative consensus shift in multi-task graph. arXiv preprini arXiv:2 210.:::201...11177 14417, 2021.
William Hamilton, Rex Ying, and Jure Leskovec, Inductive representation learning on large graphs In Proceedings of the 3 International Conference on Neural Information Processing Systems, pp. 1025-1035 2017.
Chaoyang He, Tian Xie, Yu Rong, Wenbing Huang, Yanfang Li, Junzhou Huang, Xiang Ren, and Cyrus Shahabi. Bipartite graph neural networks for efficient node representation learning. arXiv e-prints, pp. arXiv-1906, 2019.
10

---

Published as a conference paper at ICLR 2022
Ruili Huang, Menghang Xia, Dac-Trung Nguyen, Tongan Zhao, Srilatha Sakamuru, Jinghua Zhao, Sampada Shahane, Anna Rossoshek, and Anton Simeonov. Tox2 Tox2Ichalee Ichalleng to build predictive models of nuclear receptor and stress response pathways mediated by exposure environmental chemicals and drugs. Frontiers in Environmental Science, 3:85, 2016.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7491, 2018.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. The sider database of drugs and side effects. Nucleic acids research, 44(D1):D1075-DD) -1079,, 2016.
Chung- Wei Lee, Wei Fang, Chih-Kuan Yeh, and Yu-Chiang Frank Wang. Multi-label zero-shot learning with structured knowledge graphs. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10657-10665, 2019.
Tsung-Yr Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Kwong, Pareto multi-task learning. In NIPS, 2019.
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang, Learning propagate for graph meta-learning 2019.
Chengqiang Lu, Qi Liu, Chao Wang, Zhenya Huang. Peize Lin, and Lixin He. Molecular property prediction: multilevel quantum interactions modeling perspective. AAAI Conference on Artificial Intelligence (AAAI), 2019.
Pingehuan Ma, Tao Du, and Wojeiech Matusik. Efficient continuous pareto exploration in multi-task learning. International Conference on Machine Learning, pp. 6522-6531. PMLR, 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVitto, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard Zemel. Meta-learning for semi-supervised few-shot classification. In International Conference on Learning Representations, 2018.
Ann M Richard, Richard S Judson, Keith A Houck, Christopher M Grulke, Patra Volarath, Inthirany Thillanaddarajah, Chihae Yang. James Rathman, Matthew Martin, John F Wambaugh, et al. Toxcast chemical landscape: paving the road to 21 century toxicology. Chemical research in toxicology, 2985) 2016.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. sele-supe graph transformer on large-scale molecular data. Advances Neural Information Processing Systems (NeurIPS), 2020.
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In International Conference on Learning Representations, 2018.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In NIPS, 2018.
Grigorios Tsoumakas and Ioannis Katakis. Multi-label classification: An overview. International Journal of Data Warehousing and Mining (IJDWM). 3(3):1113, 2007.
11

---

Published as a conference paper at ICLR 2022
Ceren Güzel Turhan and Hasan Sakir Bilge. Recent trends in deep generative models: review. In 2018 3rd International Conference on Computer Science and Engineering (UBMK), pp. 574-579. IEEE, 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 3637-3645, 2016.
Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang. and Wei Xu. Cnn-rnn: unified framework for multi-label image classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2285-2294 2016.
Zhaoping Xiong, Dingyan Wang. Xiaohong Liu, Feisheng Zhong, Xiaozhe Wan, Xutong Li, Zhaojun Li, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, 2019.
Hansheng Xue, Luwei Yang, Vaibhav Rajan, Wen Jiang, Yi Wei, and Yu Lin. Multiplex bipartite network embedding using dual hypergraph convolutional networks. In Proceedings of the Web Conference 2021, pp. 1649-1660, 2021.
Kevin Yang. Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 2019.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3712-3722, 2018
Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas Guibas. Robust learning through cross-task consistency. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11197-11206 2020.
12

---

Published as a conference paper at ICLR 2022
A ADDITIONAL IMPLEMENTATION DETAILS
Implementation details for biomedical datasets. We use an adapted version of GraphSAGE Hamilton et al. (2017) as the base neural architecture, and implement MetaLink based on the descriptions the main paper. We use Adam optimizer, with initial learning of 0.001 and cosine learning rate scheduler. The model is trained with batch size of 128 for 50 epochs. We search over the number layers of [2, 3, 4, 5]. and report the test set performance when the best validation set performance is reached.
Implementation details for MS-COCO. We ResNet-50 with input size of x24x224 as the base neural architecture, and implement MetaLink based on the descriptions in the main paper. We perform standard data augmentation. including random crop and horizontal flipping. We use Adam optimizer with an initial learning rate of 0.002, weight decay of x 10-1. The model is trained with batch size of 256 for 40 epochs. We use the same set of hyper-parameters for all the four settings mentioned in the main paper.
Implementation details for few-shot learning, We use the widely adopted backbones ResNet-12 12 with an input size of 84 84 (Chen al., 2020b). We perform standard data augmentation, including random crop and horizontal flipping. We use SGD optimizer with an initial learning rate of 0.001, momentum of 0.9, weight decay of x 10-4 The model is trained with a batch size of for 20 epochs. Notice one batch includes support set (with 5x5 examples) and a query set(with 15 examples). In a nutshell, we use the same set of hyperparameters (Chen al., 2020b), which is exactly the degenerated model if we set KG layer = 0.
Additional algorithms. We summarize the steps of MetaLink for relational setting in Algorithm 2, for meta setting in Algorithm 3.
Code release. We will make all the source code public at the time of publication.
Algorithm 2 Metal.nk Training in Relational Setting
Require: Dataset Dirain {xx, v) A parameterized embedding function fo. Last layer weights for each task [w,}. A parameterized heterogeneous GNN fg. Number of GNN layers L.
1: for each iteration do
2:
{(x, {u Jer
J. {v je }) + SampleMiniBatc((P.. (Derii))
(4)
3:
{z} + fo(x) for x E {xx, (2 Serti
JEnti J. (v)
4:
v = (h. (0) t z forz E {z}}
> Initialize data nodes
5:
V(() = [h, + wj for w; E [w,]
b Initialize task nodes
6:
E {ei f (x,), for 11) E (ujerm.
b Initialize edges
7:
for To To do
v2 vi GraphConv(V (1-1) v!ll1) E) with fo
8:
9: logits + EdgePred(V v.( with fo
Backward (Criterion (Lv) (8) )ETTO)
B ADDITIONAL DISCUSSIONS
Limitations. i) In principle, MetaLink leverages task correlation to achieve data efficiency/gain better performance. Thus, it is less effective when tasks are not correlated. We believe it is fair to assume a multi-task learning system will work worse when tasks as not correlated. ii) Though our method is general to model diverse multi-task scenarios, limit our empirical study to labti-label learning mostly due to the limitation of dataset availability. We hope that the proposed MetaLink can inspire researchers collect more multi-task learning datasets with diverse use cases.
13

---

Published as a conference paper at ICLR 2022
Algorithm 3 MetaLink Training in Meta Setting
Require: Dataset Duain {(x. y)}. A parameterized embedding function fo- A parameterized
heterogeneous GNN fa- Number of GNN layers L. 1: for each iteration do
2: S,Q + GampleMiniBatch((Pere
D. Simulate meta setting in training
3:
{z) t fx(x) for (S, Q)
v-(0) = {h; + z for Z E {z}}
4:
d
> Initialize data nodes
V.(O) = [hh (0) +1
5:
b Initialize task nodes
6:
E {ei f (x)), for y (6) E
b Initialize edges
7:
for 1 L do
Vd V. + GraphConv(Vá (V(l-1) v!ll1), R) with So
8:
9: logits Ed v!l)) with fo
Backward Crriterion (logits, (fg) Hoo herd Q))
Social impact. In practice, collecting high-quality datasets often expensive and time-comeenn mome-conuuming In the biomedical domain, labeling requires domain expertise and hence is very resource- poourcc-intensive.. In some applications, requires long-term experiments to get the ground truth labels. The approach that we proposed could reduce the high labeling cost by utilizing auxiliary task labels that already available. In principle, Metal attempts to leverage correlation among tasks. Thus, when comes future deployment, we suggest being careful about defining tasks. Because might come across similar issues related to fairness as in other supervised learning problems, where the algorithm learns biased correlations from the datasets that are not considered to be appropriate. The datasets used for experiments are among the most widely-used benchmarks, which should not contain any undesirable bias
C ADDITIONAL RESULTS
Advantages on using heterogeneous weights. As discussed in Section 3, we proposed to learn from the knowledge graph constructed via heterogeneous GNN. Here we empirically demonstrate the improvement of using distinct weights for updating data nodes and task nodes. Results are summarized in Table
Table 5: Results of ROC AUC (for biomedical) and mAP (for MS-COCO) datasets.
Sig
We Weights
Tox21 tasks)
Sider (27 tasks)
ToxCast (617 tasks)
MS-COCO tasks)
Shared
83.1 = 1.6
75.8 :: 2.5
78.1 + 1.3
74.02 :: 0.16
Relational
Hetero
83.7 = 1,9
76.8 + 3.0
79.4 + 1.0
75.36 0.16
14


Reviews:
[{"review_id": "va8fCQw6D9", "reviewer": "Reviewer_2M5P", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Innovative formulation\n-- a wide “range of heterogeneity” in the graph: a node for each sample in the support set a node for each task - backed up by relevant experimental improvements\n-- dynamical graph structure based on the existing labels for the current entry\n- Experimental results are strong for the few-shot setup and medical datasets\n- Ablations are very informative: higher improvement in more correlated tasks and using more auxiliary tasks come with higher performance\n\nWeaknesses\n- Table 1. Why are there so many variations of the KG layer (both inside the same dataset and per metric)? Is there some consistent explanation for this hyperparameter?\n- Even though it is quite simplistic, Figure 2 is confusing. The authors should highlight somehow that the edges show the conceptual connections between nodes (and not the real link in the algorithm flow). Such an example: in c) and d) why are data nodes (the ones from the triangle structures) useful if they are already connected via known label edges?\n- Ablations on the graph structure decision would be useful (the ones from eq. 2 and 3, sustaining the “special GNN design”)\n- Prior work is focused on GNN approaches (even in the multi-task paragraph), comparison from the design point of view with prior-art approaches in relational multi-task graphs could offer a more complete view (eg. Robust Learning Through Cross-Task Consistency - https://arxiv.org/abs/2006.04096, Self-Supervised Learning in Multi-Task Graphs through Iterative Consensus Shift - https://arxiv.org/abs/2103.14417)\n\nQuestions\n- How many parameters does MetaLink learn?\n- How could the authors adapt the proposed MetaLink to work with dense predictions rather than 1-D tasks? What would be the implications?\n\nOthers (minor, eg. typos)\n- KG layer (knowledge graph) is never expanded\n- Please mention what you show in the tables (top5 performance)\n- “where T = {1, 2, ..., m} is integers between 1 and m.”\n- Eq. 1 should have an (l-1) instead of (l) - it uses only h(l)\n", "summary_of_the_paper": "The paper exploits the multi-task modeling using a heterogenous Graph Neural Network. The key contribution of the work is having both data (sample) and task nodes in the same graph, focusing on data-task relation, accommodating for sparse task labels by design.\n\nExperimental results show relevant improvements with the proposed relational model on biomedical datasets and Imagenet splits for few-shot.\n", "summary_of_the_review": "The paper is sound, introducing and validating an interesting heterogeneous graph design focused on each task and data sample as a different node. The work also offers an approach for the common labeling limitation in multi-task problems (by working with sparse labels in a dynamic graph architecture ). I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "deMsadjhC4", "reviewer": "Reviewer_jGkV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. MetaLink is able to leverage the correlations among tasks successfully.\n2. I find the idea of creating the knowledge graph for harnessing the relational information about the data and tasks very intreging. \n3. The usage of the labels of the auxiliary task is well motivated.\n\nWeakness: \n1. The writing could have been more precise in some parts of the paper.", "summary_of_the_paper": "The paper introduces a novel multi-task learning framework called MetaLink that takes the opportunity to utilize the auxiliary task labels by constructing a knowledge base with the tasks and data(instances) as nodes and auxiliary labels as the edge labels between them. This method shows significant empirical success ", "summary_of_the_review": "Concerns:\n1. The introduction could have been more concise, I feel the authors could have avoided explaining the technicalities of the knowledge graph in the introduction. Rather, it would help the reader to be better situated in your research space if you discuss the fundamental difference between some of the related models and yours at an early stage of the paper. For example, the authors could have mentioned Wang et al. 2016 and ML-GCN  (Chen et al.) in the introduction itself. and explained why their work is different from the others. \n\n2. Are the comparisons fair? The GraphConv( ) step uses additional parameters, especially when the number of layers (iteration) increases. Could you include the number of parameters for all the baselines and your method?\n\n3. The tasks are very specific to multilabel classification. What happens when objectives are different for each task? How would MetalLink adapt to that?\n\nminor: In the paragraph: \"Results on biochemical datasets.\", Table 5 ---> Table 1.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R4_FEOw2FVj", "reviewer": "Reviewer_ijJF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is very well-organized and well-written. Algorithm 1 is self-explanatory.\n\n- I really liked the idea of creating a knowledge graph and then solve the link label prediction problem.\n\n- The experiments are thorough, I enjoyed reading the experiments section; they cover ablation studies, and answer several question in there.  Specifically how relational ML could take advantage of more related tasks. In addition, the improvements are noticeable (Tables 1-4).\n\n\nNotes:\n- In figure 1, in the right panel, I would put 1's and 0's in a right place, e.g., 1's for the solid lines or 0's with the dashed line?!\n\n- I know that I have seen figure 2 somewhere before, but do not remember the paper, if you have seen this figure from another paper, that will be a good idea to put the reference.\n\n- I would recommend defining use the name of your model in figure 4 instead of using 'ours'", "summary_of_the_paper": "In this paper, they introduce relational multi-task learning in which they construct a graph with the data points and task and labels as edges. They represent the data points as the embedding of NN model and task as the last layer of NN model for that task. Then they solve the link label prediction problem between each node and task by using GNN and heterogeneous message passing method to predict the label of test data point on a new task at inference time.", "summary_of_the_review": "In general, I believe the paper is innovative and descent. The backed their proposal with thorough experiments. I think this work will be useful for researchers working in the area of multi-tasking.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HfjvV94qC-V", "reviewer": "Reviewer_QALx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "According to the setting introduced in Section 2, it seems that the proposed setting is just a meta learning setting but not a multi-task learning setting. The name of relational multi-task learning seems a bit misleading.\n\nIn the first paragraph of Section 3.1, the claim is not general, since in many cases, the task head consists of multiple layers instead of only one layer. In the multi-layer case, based on the second paragraph of Section 3.2, the proposed MetaLink model seems not applicable.\n\nThe knowledge graph stated in this paper is just a graph. This is different from the actual ‘knowledge graph’ and this seems misleading.\n\nThe heterogeneous GNN used in Section 3.2 seems no difference with the conventional GNN. I did not see significant novelty in the proposed MetaLink model.\n\nIn experiments, only multi-label datasets are used. Authors should experiment on multi-task benchmark datasets such as CityScape, NYUv2, Pascal-Context, Taskonomy, Office-31, Office-Home.", "summary_of_the_paper": "This paper proposes a relational multi-task learning setting and design a MetaLink model.", "summary_of_the_review": "The proposed setting seems a bit misleading. The proposed model has no significant novelty. More standard benckmark datasets should be used.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a novel learning scenario where there are many related tasks, some seen at test time, and some seen only at training time, where additionally the task labels can be hidden or present.  This approach generalizes both a "relational setting" (where auxiliary task labels could be used as features) and a "meta setting" (where new tasks need to be solved in a zero-shot setting using data from related tasks only).  The idea behind the method is to do MTL with a common representation and a set of task-specific heads, and build a graph where (1) tasks are nodes associated with the parameters of their task-specific "heads" and (2) edges link examples to tasks with known labels.  A GNN method is then used to find regularities in the graph.

Pros
 - The setting is innovative and the approach is novel
 - The experimental results are strong

Cons
 - Some of the terminology seems awkward and/or strained (eg "knowledge graph" for the task-example graph)
Decision: Accept (Spotlight)