Graph Representation Learning for Multi-Task Settings: a Meta-Learning Approach
Authors: Davide Buffelli
Fabio Vandin
Venue:  (2021)
Keywords: Graph Representation Learning
Multi-Task Learning
Meta-Learning
Graph Neural Networks

Abstract: Graph Neural Networks (GNNs) have become the state-of-the-art method for many applications on graph structured data. GNNs are a framework for graph representation learning, where a model learns to generate low dimensional node embeddings that encapsulate structural and feature-related information. GNNs are usually trained in an end-to-end fashion, leading to highly specialized node embeddings. While this approach achieves great results in the single-task setting, generating node embeddings that can be used to perform multiple tasks (with performance comparable to single-task models) is an open problem. We propose a novel representation learning strategy, based on meta-learning, capable of producing multi-task node embeddings. Our method avoids the difficulties arising when learning to perform multiple tasks concurrently by, instead, learning to quickly (i.e. with a few steps of gradient descent) adapt to multiple tasks singularly.  We show that the embeddings produced by our method can be used to perform multiple tasks with comparable or higher performance than both single-task and multi-task end-to-end models. Our method is model-agnostic and task-agnostic and can hence be applied to a wide variety of multi-task domains.

Under review as a conference paper at ICLR 2021
GRAPH REPRESENTATION LEARNING FOR MULTI- TASK SETTINGS: A META-LEARNING APPROACH
Anonymous authors
Paper under double-blind review
ABSTRACT
Graph Neural Networks (GNNs) have become the state-of-the-art method for many applications on graph structured data. GNNs are a framework for graph representation learning, where model learns to generate low dimensional node embeddings that encapsulate structural and feature-related information. GNNs are usually trained in end-to-end fashion, leading highly specialized node embeddings. While this approach achieves great results in the single-task setting, generating node embeddings that be used perform multiple tasks (with per- formance comparable to single-task models) is an open problem. We propose a novel representation learning strategy, based on meta-learning, capable of produc- ing multi-task node embeddings. Our method avoids the difficulties arising when learning to perform multiple tasks concurrently by, instead, learning to quickly (i.e. with few steps of gradient descent) adapt to multiple tasks singularly. We show that the embeddings produced by our method can used to perform multi- ple tasks with comparable or higher performance than both single-task and multi- task end-to-end models. Our method is model-agnostic and task-agnostic and can hence be applied to wide variety of multi-task domains.
INTRODUCTION
Graph Neural Networks (GNNs) are deep learning models that op- erate graph structured data, and have become one of the main top- ies of the deep learning research community. Part of their success given by great empirical per- formance on many graph-related tasks. Three tasks in particular, with many practical applications, have received the most attention: graph classification, node classifi- cation, and link prediction. GNNs centered around the concept of node representation learning, and typically follow the same archi- tectural pattern with an encoder- decoder structure (Hamilton al. 2017 Chami et al., 22200) Wu et al. 2020 The encoder produces node embeddings (low-dimensional vec- tors capturing relevant structural and feature-related information about each node), while the de- coder uses the embeddings to carry out the desired downstream task. The model is then trained in an end-to-end manner, giving rise to highly specialized node embeddings. While this can lead to state-oftte afrthe-art performance, it also affects the generalization and reusability of the embeddings. In fact, taking the encoder from a GNN trained on a given task and using its node embeddings to train decoder for different task leads to substantial performance loss, shown in Figure 1
0.3
Ckas Ecation
Grahh Gautication
0.99
Fidician
0.3
BOC AUC
0.25
0.79
0.4%
0.83
0.99
0.3
0.33
(b)
Figure 1: Performance drop when transferring node embed- dings between tasks on (a) Node Classification (NC), (b) Graph Classification (GC), and (c) Link Prediction (L.P) on the ENZYMES dataset. On the horizontal axis, "x ->y" indi- cates that the embeddings obtained from model trained on task x are used to train network for task

---

Under review as a conference paper at ICLR 2021
The low transferability of node embeddings requires the of one specialized encoder and one specialized decoder for each considered task. However, many practical machine learning appli- cations operate in rssource-constranned environments where being able to share part of the model architecture between tasks is of great importance. Furthermore, the training signal from multiple related tasks can lead to higher generalization. Nevertheless, making sure tasks do not negatively interfere with each other is not trivial (Standley et al. (020). The problem of learning models that can perform multiple tasks is known as Multi- Task Learning (MTL), and is an open area of research, attracting many researchers in the deep learning community (Vandenhende et al. 2020).
MTL on graphs has not received much attention, and no single model capable of performing the three most common graph-related tasks has yet been proposed. In fact, we notice that training a multi-head model with the classical procedure, i.e. by performing multiple tasks concurrently on each graph, and updating the parameters with some form of gradient descent to mínimize the sum of the single-task losses, can lead performance loss with respect to single-task models. Thus, we propose a novel optimization-base meta-learning (Finn al. 2017) procedure with focus on representation learning that can generate node embeddings that generalize across tasks.
Our proposed meta-learning procedure produces tusk-generulizing node embeddings by not aiming at a setting of the parameters that can perform multiple tasks concurrently (like a classical method would do), to setting that allows fast multi-task adaptation (like traditional meta-learning to a setting that can easily be adapted to perform each of the tasks singularly. In fact, our meta- learning procedure aims at setting of the parameters where few steps of gradient descent given task, can lead to good performance on that task, hence removing the burden of directly learning solve multiple tasks concurrently.
but
We summarize our contributions as follows:
We propose novel method for learning representations that can generalize to multiple tasks. We apply it to the challenging setting of graph MTL. and show that a GNN trained with our method produces higher quality node embeddings with respect classical end-to- end training procedures. Our method is based on meta-learning and is model-agnostic and task-agnostic, which makes easily applicable
wide range of multi-task task domains.
To the best of our knowledge, we are the first to propose a GNN model generating a single set of node embeddings that can be used to perform the three most common graph-related tasks (graph classífication, node classification, and link prediction). In particular, our em- beddings lead to comparable higher performance with respect to single-task models even when used as input to simple linear classifier.
We show that the episodic training strategy at the base of our proposed meta-learning pro- cedure leads to better node embeddings even for models trained on a single task. This un- expected finding provides interesting directions that we believe can be useful to the whole deep representation learning community.
2 RELATED WORK
GNNs, MTL, and meta-learning are very active areas of research. We highlight works that are at the intersections of these subjects, and point the interested reader to comprehensive reviews of each field. To the best of our knowledge there is no work using meta-learning for graph MTL, or proposing a GNN performing graph classification, node classification. and link prediction concurrently.
Graph Neural Networks. GNNs have a long history (Scarselli et al., 2009), but in the past few
years the field has grown exponentially; we refer the reader to Chami et al. (2020); Wu et al. (2020) for a thorough review of the field. The first popular GNN approaches were based on filters in the graph spectral domain (Bronstein et al., 2017), and presented many challenges including high computational complexity. |Defferrard et al. (016) introduced ChebNet, which uses Chebyshev polynomials to produce localized and efficient filters in the graph spectral domain. Graph Convo- lutional Networks (Kipf & Welling 2017) then introduced a localized first-order approximation of spectral graph convolutions which was then extended to include attention mechanisms (Veličkovid et al. 2018). Recently, Xu et al. (2019) provides theoretical proofs for the expressivity of GNNs.
2

---

Under review as a conference paper at ICLR 2021
Multi-Task Learning. Works at the intersection of MTL and GNNs have mostly focused on multi- head architectures. These models are all composed of a series of GNN layers followed by multiple heads that perform the desired downstream tasks. In this category, Montanari et al. (2019) propose
a model for the prediction of physico-chemical properties. Holtz et al. (2119) and Xie et al. (2020) propose multi-task models for concurrently performing node and graph classification. Finally, Ave- lar et al. (2019) introduce multi-head GNN for learning multiple graph centrality measures, and Li & Ji (2019) propose MTL method for the extraction of multiple biomedical relations. The work by (Haonan et al. 2019) introduces a model that can be trained for several tasks singularly, hence, unlike the previously mentioned approaches and our proposed method, it can not perform multiple tasks concurrently. There are also some works that use GNNs as tool for MTL: Liu et al. (2019b) use GNNs to allow communication between tasks, while Zhang et al. (0118) use GNNs to estimate the test error of a MTL. model. We further mention the work by Wang et al. (2020) which considers the task of generating "general" node embeddings, however their method is not based on GNNs, does not consider node attributes (unlike our method), and is not focused on the three most common graph related tasks, which we consider. For an exhaustive review of deep MTL techniques we refer the reader to Vandenhende et al. (2020).
Meta-Learning, Meta-Leaming, consists in learning to learn. Many methods have been proposed (see the review by |Hospedales al. (2020)) specially in the area of few-shol learning. Garcia & Bruna (2018) frame the few-shot learning problem with a partially observed graphical model and use GNNs as an inference algorithm. Liu et al. (2019a) use GNNs to propagate messages between class prototypes and improve existing few-shot learning methods, while et al. (2020) use GNNs in the form of graphs. There are also several works that use meta- learning to train GNNs in few-shot learning scenarios with applications to node classification (Zhou al.) 2019: Yao et al. 2020). edge labelling (Kim et al., (2019), link prediction (Alet et al., 2019] Bose et al. 2019). and graph regression (Nguyen al. (020). Finally, other combinations of meta- learning and GNNs involve adversarial attacks on GNN models (Zügner & Günnemann 2019) and
introduce
active learning (Madhawa & Murata 2020).
3
PRELIMINARIES
3.1 GRAPH NEURAL NETWORKS
Many popular state-of-the-an trrt--r art GNN models follow the message-passing paradigm (Gilmer et al, 2017 Let represent graph G (A, X) with an adjacency matrix A E {0, 1)ax", and node feature matrix X E R.ndd where the v-th row X, represents the d dimensional feature vector of node u. Let ( E Randd be the matrix containing the node representations at layer f. A message passing layer updates the representation of every node v as follows:
msg? (0) AGGREGATE (H))) Yu E (.))
H (++1) UPDATE(H (2) !
where (0) No is the set of neighbours of node v, AGGREGATE is a permutation invariant function, and UPDATE usually a neural network. After L message-passing layers, the final node embeddings (L) are used to perform a given task, and the network is trained end-to-end.
3.2 MODEL-AGNOSTIC META-LEARNING AND ANIL
MAML (Model- Meta- Meta- Learning) is optimization-basee meta- leeaale strategy proposed by Finn et al. (2017). Let fo be a deep learning model, where # represents its parameters. Let p(E) be a distribution over episoded[] where an episode & p(E) is defined as a tuple containing a loss function, a support set, and a larget set: E, = (Læ, (-), Se,, Te,), ). where support and target sets are simply sets of labelled examples. MAML's goal is find a value of that can quickly, i.c. in a few steps of gradient descent, be adapted to new episodes. This is done with a nested loop optimization procedure: an inner loop adapts the parameters to the support set of an episode by performing some steps of gradient descent, and an outer loop updates the initial parameters aiming at setting that
'The meta-learning literature usually derives episodes from tasks (i.c. tuples containing a dataset and loss function). We focus episades avoid using the term task for both a MTI. task, and meta-learning task.
3

---

Under review as a conference paper at ICLR 2021
Support
Target
E BC
x
&C K
6C
GCN
LP
Muttihead
Node Embeddings
Embeddings
Output
Layer
in inner www.
Adapt
Update loop
Update outor
inner loop
(b)
(c)
Figure 2: (a) Schematic representation of a multi-task episode. For each task, support and target set are designed to be as the training and validation sets for single-task training. (b) Scheme of iSAME: both the backbone GNN and the task-specific output layers are adapted (one a time) in the inner loop of our meta-learning procedure. (c) Scheme of eSAME: only the task-specific output layers are adapted (one at a time) in the inner loop of our meta-learning procedure.
allows fast adaptation. Formally, by defining (1) as the parameters after adaptation steps on the support set of episode E,, express the computations in the inner loop
(t) = (t(t - a Veg (Je;(e-1) Se,), with (0()) =
where L(Je((- 1), ) indicates the loss over the support set Se, of the model with parameters (t 1), and is the learning rate. The meta-objective that the outer loop tries to minimize is defined as Lmela = Eemptee Le, (fo Te,), which leads to the following parameter update"
= - Lmeu = 0 BV
Σ Le (fo Te,). S)pp(E)
Raghu et al. (2020) showed that feature reuse is the dominant factor in MAML: in the adaptation loop, only the last layer(s) in the network are updated, while the first layer(s) remain almost un- changed. The authors then propose ANIL (Almost No Inner Loop) where they split the parameters in two sets: one that is used for adaptation in the inner loop, and one that is only updated in the outer loop. This simplification leads computational improvements while maintaining performance.
OUR METHOD
Our novel representation learning technique, based meta-learning, is built three insights:
(i) optimization- meta-learning is implicitly learning robust representations. The findings by |Raghu et al. (2020) suggest that in a model trained with MAML, the first layer(s) learn features that are reusable across episodes, while the last layer(s) are set up for fast adaptation. MAML is then implicitly focusing on learning reusable representations that generalize across episodes.
(ii) meta-learning episodes can be designed to encourage generalization. If we design support and target set mimic the training and validation sets of classical training procedure, then the meta-lcarning procedure is effectively optimizing for generalization.
(iii) meta-learning can learn to quickly adapt to multiple tasks singularly, without having to learn to solve multiple tasks concurrently. We can design the meta-learning procedure so that, for each considered task, the inner loop adapts the parameters to a task-specific support set, and tests the adaptation this fast multiple single-task adaptation. This procedure is effectively searching for a parameter setting that can be easily adapted to obtain good single-task performance, without learning solve multiple tasks concurrently. This procedure differs from classical training methods (which aim at solving multiple tasks concurrently), and from traditional meta- meta-lcarning approaches (which aim at parameters that allow fast multi-task adaptation, inheriting the problems classical methods
a task-specific target set. The outer loop then updates the parameters to allow
We limit ourself to one step of gradient descent for clarity, but any optimization strategy could be used. We provide detailed discussion the differences with traditional methods in Appendix

---

Under review as a conference paper at ICLR 2021
Based on (it) and (iii,, develop novel meta-learning procedure where the inner loop adapts to multiple tasks singularly, each time with the goal of single-task generalization. Using an encoder- decoder architecture, (i) suggests that this procedure leads to encoder that learns features that reusable across episodes. Furthermore, in each episode, the learner is adapting to multiple tasks, hence the encoder is learning features that general across multiple tasks.
Intuition. Training multi-task models very challenging (Standley et al. 2020). as some losses may dominate over others, gradients for different tasks may point in opposite directions. Some methods have been proposed to counteract this issues (Kendall et al. 2018 Chen et al. 20188) but they are not always effective and it is not clear how one should choose which method to apply Vandenhende et al., 2020). We design a meta-lcarning procedure where the learner does not have to find configuration of the parameters that concurrently performs all tasks, but it has to find configuration that can easily be adapted to perform each of the tasks singularly. By then leveraging the implicit/explicit robust representation learning that happens with MAML and ANIL,, can extract an encoder capable of generating node representations that generalize across tasks.
In the rest of this section, we formally present our novel meta-learning procedure for multi-task graph representation learning. There are three aspects that we need to define: (1) Episode Design: how is a an episode composed, (2) Model Architecture Design: what is the architecture of our model, (3) Meta-Trainie Design: how, and which, parameters adapted/updated.
4.1 EPISODE DESIGN
In our case, an episode becomes a multi-task episode (Figure (a)). To formally introduce the concept, let us consider the case where the tasks are graph classification (GC), node classification (NC), and link prediction (LP). We define multi-task episode c! ~ p(g(m)) as tuple
e( (CE)) (m) s(m) (Té. (m))
C = (GC) (((0)) + + c + (LP) (LP)
[L.
((P)) s(()) = {5600) s(Nc) Se.
Té. = (Te, (GC)
(T())) Te, .ee ((P))
(NC)
where X are balancing coefficients. The meta-objective of our method then becomes:
(m)
Σ El (("">~p(E (p(m)))
(GG) (00) ((NC) (NC) + (1.P) (LP)
metio
Support and target sets are set up to resemble a training and a validation set. This way the outer loop's objective becomes to maximize the performance hence pushing towards generalization. In more detail, given a batch of graphs, we divide it in equally sized splits (one per task), and create support and target sets follows:
validation set, given training set,
Graph Classification:
s! (GC) and Té. (GC) contain labeled graphs, obtained with random split.
Node Classification: <(C)) and Téd (NC) are composed of the same graphs, with different labelled
St nodes. We mimic the common semi-supervised setting (Kipf & Welling 2017) where
feature vectors available for all nodes, and only small subset of nodes labelled.
Link Prediction:
s(.P' and T!! are composed of the same graphs, with different query edges. In every graph we randomly remove some edges, used as positive examples together with non- removed edges, and randomly sample couples of non-adjacent nodes as negative examples.
The full algorithm for the creation of multi-task episodes is provided in Appendix
4.2 MODEL ARCHITECTURE DESIGN
We use an encoder- encoder-decoder model with a multi-head architecture. The backbone (which represents the encoder) is composed of 3 GCN (Kipf & Welling 2017) layers with ReLU non-linearities and residual connections (He et al. 2016). The decoder is composed three heads. The node classi- fication head is single layer neural network with Sofimx activation that is shared across nodes
5

---

Under review as a conference paper at ICLR 2021
and maps node embeddings to class predictions. In the graph classification head, first single layer neural network (shared across nodes) performs a linear transformation (followed by a ReLU acti- vation) of the node embeddings. The transformed node embeddings single layer neural network with Softmax activation outputs the class predictions. The link predic- tion head is composed of single layer neural network with ReLU non-linearity that transforms node embeddings, and another single layer neural network that takes as input the concatenation of two node embeddings and outputs the probability of a link between them.
then averaged and a final
4.3
DESIGN
We first present our meta-learning training procedure, and successively describe which parameters are adapted/updated in the inner and outer loops.
Meta-Learninng Training Procedure. To avoid the problems arising from training model that performs multiple tasks concurrently, we design a meta-learning procedure where the inner loop adaptation and the mela-objective computation involves a single task a time. Only the parameter update performed to minimize the meta-objective involves multiple tasks, but, crucially, it does not aim at a setting of parameters that can solve, or quickly adapt to, multiple tasks concurrently, but to a setting that allows multiple fast single-task adaptation.
The pseudocode of our procedure is in Algorithm| [ ADAPT performs a few steps of gradient descent on a task specific loss function and support set, TEST computes the value of the meta- objective component on task spe- eific loss function and target set, and UPDATE optimizes the parameters by minimizing the meta-objective. No- tice how the multiple heads of the de- coder in our model are never used con- currently.
Algorithm 1: Proposed Meta-Learning Procedure
Input: Model fo: Episodes & = {E,,..,En init(0)
for & in & do
c_loss +0 for t in (GC, NC, LP) do
Ør(t) t
(r(t) + ADAPT(fa. s(t) c((
o_loss + oolss T(t)) 2(b))
end
Parameter Update in Inner/Outer Loop. Let partition the parame- ters of our model in four sets: = end (OGeN, Oxc, 0cc.0.p) representing the parameters of the backbone (#GCN),
0 t UPDATE o ((0)) (1000,0(N0)) g(LP))
node classification head (0xc), graph classification head (Occ), and link prediction head (0LP). We name our proposed meta-learning strategy SAME (Single-Task Adaptation for Multi-Task Embeddings), and present two variants (F)
Implicit SAME (iSAME): all the parameters 9 are used for adaptation. This strategy makes use of the implicit feature-reuse factor of MAML, leading to parameters OGEN that are general across multi-task episodes.
Explicit SAME (eSAME): only the head parameters ONC, fcc, GuP are used for adaptation. Con- trary to the previous, this strategy explicitly aims learning the parameters fGGCN to be general across multi-task episodes by only updating them in the outer loop.
5 EXPERIMENTS
Our goal is to assess the quality of the representations leamned by our proposed methods. In more detail, we aim to answer the following questions:
Q1: Do our proposed meta-learning procedures lead to high quality node embeddings in the single- task setting?
Q2: Do our proposed meta-learning procedures lead to high quality node embeddings in the multi- task setting?

---

Under review as a conference paper at ICLR 2021
Table 1: Results for a single-task model trained in a classical supervised manner (CI), and linear classifier trained on the embeddings produced by our meta-lcarning strategies (iSAME, eSAME).
Task
Model
Dataset
ENZYMES
PROTEINS
DHFR
COX2
CI
87.5 1.9
72.3 4.4
97.3 1 0.2
96.4 0.3
NC
iSAME
87.3 + 0.8
96. 1.
96.1
eSAME
87.8
8.
96.8
96.5
Cl
5.2
3.6
76.7 2.3
GC
iSAME
2.9
1.2
3.2
eSAME
5.01
1.
7.16 7..6
CI
3.0
0.8
0.8 9.7
LP
iSAME
81.7
8..0
99.2
99.1 = 0.5
eSAME
3.4
0.9
99.2 0.3
99.2 + 0.7
Q3: Do our proposed meta-learning procedures for multiple tasks extract information that is not captured by classically trained multi-task models?
Q4: Can the node embeddings learned using our proposed meta-learning procedures be used for multiple tasks with comparable better performance than classical multi-task models?
Throughout this section we use GC to refer to graph classification, NC for node classification, and LP for link prediction. Unless otherwise stated, accuracy (%) is used for NC and GC, while ROC AUC (%) is used for LP.
To perform multiple tasks, we consider datasets with graph labels, node attributes, and node labels from the TUDataset library (Morris et al., (220):: ENZYMES (Schomburg et al.) 2004). PROTEINS (Dobson & Doig 2003), DHth(Sutherland al.) 2003), and COX2 (Sutherland al. 2003 ENZYMES is a dataset of protein structures belonging to six classes. PROTEINS is dataset of chemical compounds with two classes (enzyme and non-enzyme). DHFR, and COX2 are datasets of chemical inhibitors which can be active inactive. In all datasets, each node has series of attributes containing physical and chemical measurements.
Datasets.
Experimental Setup. We perform 10-fold cross validation, and average results across folds. To ensure a fair comparison, we use the same architecture for all training strategies. We tested loss balancing techniques (e.g. uncertainty weights (Kendall et al. 2018), and gradnorm (Chen et al. 2018)) but found that they were not effective. In our experiments we notice that the losses do not need to be balanced, and we set (GC) X(NC) ( without performing any tuning. For more information we refer to Appendix and we provide source code as supplementary material.
Q1: For every task, we train a linear classifier on top of the embeddings produced by model trained using our proposed methods, and compare against network with the same architecture trained a classical manner. Results are shown in Table []] For all three tasks, a linear classifier on the embeddings produced by our methods achieves comparable, if not superior, performance end-to-end model. In fact, the linear classifier is never outperformed by more than 2%, and it can outperform the classical end-to-end model by up to 12%. These results show that our meta-learning procedures are learning high quality node embeddings.
Q2: We train model with our proposed methods, on all multi-task combinations, and use the em- beddings as the input for a linear classifier. We compare against models with the same task-specific architecture trained in the classical manner on single task, and with a fine-tuning baseline. The lat- ter is model that has been trained on all three tasks, and then fine-tuned on two specific tasks. The idea is that the initial training on all tasks should lead the model towards the extraction of features that it would otherwise not consider (by only seeing 2 tasks), and the fine-tuning process should then allow the model to use these features to target the specific tasks of interest. Results are shown in Table 2 omit standard deviation for space limitations). We notice that the embeddings produced by our procedures a multi- task setting, followed by a linear classifier, achieve comparable perfor- mance to end-to-end single-task models. In fact, the linear classifier is never outperformed by more than 3%, and in 50% of the cases it actually achieves higher performance. We further notice that the fine-tuning baseline severely struggles, and is almost always outperformed by both single-task
7

---

Under review as a conference paper at ICLR 2021
Table 2: Results for single-task model trained in classical supervised manner, fine-tuned model (trained on all three tasks, and fine-tuned on the two shown tasks), and a linear classifier trained on node embeddings learned with our proposed strategies (iSAME, eSAME) in a multi-task setting.
Task NC
Dataset
GC
LP
ENZYMES GC
PROTEINS
DHER
COX2 GC
NC
LP
GC
NC LP
GC
NC LP
NC
LP
Classical End-to-End Training
51.6
73.3
71.5
76.7
87.5
72.3
97.3
96.4
75.5
85.6
98.8
98.3
Fine- TuniTTunng
48.3
85.3
73.6
72.0
66.4
92.4
80.0
92.3
49.3
71.6
69.6
80.7
65.3
58.9
80.2
50.9
87.7
73.9
80.4
81.5
80.7
56.6
87.4
52.3
iSAME (ours)
50.1
86.1
73.1
76.6
71.6
94.8
75.2
95.4
50.7
83.1
73.4
85.2
71.6
99.2
77.5
98.9
86.3
83.4
79.4
87.7
96.5
99.3
95.5
99.0
50.0
86.5
82.3
71.4
76.6 87.3
71.2
95.5
99.5
75.4
95.2
99.2
eSAME (ours)
51.7
86.1
71.5
79.2
70.1
95.7
95.5 77.5
75.6
51.9
80.1
71.7
85.4
70.1
99.1
98.8
86.7
82.2
80.7
86.3
96.6
99.4
95.6
99.1
51.5
86.3
81.1
71.3
79.6
86.8
70.2
95.3
99.5
77.7
95.7
98.8
models, and our proposed methods. These results indicate that the episodic meta-learning procedure adopted by SAME extracting features that are otherwise not accessible with standard training techniques.
Q3: We train multi-task model, and we simple network (with
then train the same architecture as the heads de- scribed in Section 4.2), which refer to as classifier, on the embeddings to perform a task that was not seen dur- ing training. We compare the perfor- mance of the classifier on the embed- dings learned by a model trained in classical manner, and with our proposed procedure. Intuitively, this tests gives us a way to analyse the embeddings proposed approaches
Chooocal
ISAME
#SAME
0.8
0.8
Accuracy
0.55
0.50
0.0
0.35
0.4
0.4
0.30
(b)
(d)
learned by contain "more information" than em- beddings learned in classical manner. Results on the ENZYMES dataset are shown in Figure where notice that embeddings learned by our proposed ap- proaches lead to at least 10% higher per- formance. We observe an analogous trend on the other datasets, and report all results in Appendix D
Figure 3: Results for neural network, trained on the em- beddings generated by multi-task model, performing task that was not seen by the multi-task model. "r, y->z" indicates that xr y the tasks for training the multi-task model, and is the new task.
Q4: We train the same multi-task model, both in the classical supervised manner, and with our proposed approaches, on all multi-task combinations. For our approaches, we then train linear classifier on top of the node embeddings. We further consider the fine-tuning baseline introduced in Q2. We use the multi-task performance (Am)) metric (Maninis et al., (2019), defined as the average per-task drop with respect to the single-task baseline: Am = Σ 1 (Mm,i MM.s) /M where M m is the value of task's metric for the multi-task model, and M, is the value for the baseline.
8

---

Under review as a conference paper at ICLR 2021
Table 3: Am (%) results for classical multi-task model (CI), fine-tuned model (FT; trained all three tasks and fine-tuned on two) and linear classifier trained on the node embeddings learned using our meta-learning strategies (iSAME, eSAME) in multi-task setting.
Task
Model
Dataset
GC NC
LP
ENZYMES
PROTEINS
DHFR
COX2
CI
-0.1 0.5
1.0
0.3
0.1
FT
1.2
0.1
-7.4
0.1
iSAME
0.9
2.7
-.22
0.2
eSAME
0.8
3.2+144
0.3
-1.2
CI
3.2
-5.3112
4.3
-214
FT
1.9
1.5
2.8
iSAME
0.5
-0.2
0.2
0.5
eSAME
3.2
111
3.7
0.7
CI
7.2
6.9
29.1
2.5
FT
0.3
3.1
+.4
4.2
iSAME
4.1
6.1
+.2
2.66
eSAME
3.9 1.3
6.1
0.1
-.6
CI
1.3
2.9
1..
-16.9
iSAME
1.5
0.2
1.4
1.9
eSAME
1.9
2.8 0.2
1.70
1.24
Results are shown in Table We first notice that usually multi-task models achieve lower perfor- mance than specialized single-task ones. We then highlight that linear classifiers trained the embeddings produced by our procedures are not only comparable, but in many cases significantly superior classically trained multi-task models. In fact, multi-task model trained in classical manner is highly sensible to the tasks that are being learned (e.g. GC and LP negatively interfere with each other in every dataset), while methods seem much less sensible: the former has worst-case average drop in performance of 29%, while our method has a worst-case average drop of less than 3%. Finally, we also notice that the tine-tuning bascline generally performs worst than classically trained models, confirming that transferring knowledge in multi-task settings is not easy, and more advanced techniques, like our proposed method SAME, are needed.
6 CONCLUSIONS
In this work propose novel representation learning strategy for multi-task settings. Our method overcomes the problems that arise when learning to solve multiple tasks concurrently by optimizing for parameter setting that quickly, i.e. with few steps of gradient descent, be adapted for high single-task performance on multiple tasks. We apply our method to graph representation learning, and find that our training procedure leads higher quality node embeddings, both in the multi-task setting, and in the single-task setting. In fact, we show that a linear classifier trained on the em- beddings produced by our method has comparable supervised models. Furthermore, we find that the embeddings learned with our proposed procedures lead to higher performance on downstream tasks that were not seen during training. We believe this work can be highly useful to the whole deep representation learning community, as our method model agnostic and task agnostic, and can therefore applied on wide variety of multi-task domains.
better performance than classical end-to-end
REFERENCES
Ferran Alet, Erica Weng, Tomas Lozano-Perez, and L. Kaelbling. Neural relational inference with fast modular meta-learning. In NewrIPS, 2019.
Pedro Avelar, Henrique Lemos, Marcelo Prates, and Luis Lamb. Multitask learning graph ral networks: Learning multiple graph centrality measures with a unified network. ICANN Workhhop and Special Sessions. 2019.

---

Under review as a conference paper at ICLR 2021
Eytan Bakshy, Lili Dworkin, Brian Karrer, Konstantin Kashin, Benjamin Letham, Ashwin Murthy, and Shaun Singh. Ae: A Comain-agnostic platform for adaptive experimentation. In NeurIPS Systems for ML Workshop, 2018.
Avishek Joey Bose, Ankit Jain, Piero Molino, and William L Hamilton. Meta-graph: Few shot link prediction via meta learning. arXiv, 2019.
Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geo- metric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine, 2017.
Ines Chami, Sami Abu-El-Haija, Bryan Perozzi, Christopher Ré, and K. Murphy, Machine learning on graphs: A model and comprehensive taxonomy. arXiv, 2020.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. ICML, 2018.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In NeurlPS, 2016.
Tristan Deleu, Tobias Würfl, Mandana Samiei, Joseph Paul Cohen, and Yoshua Bengio. Torchmeta: A Meta-Learning library for PyTorch. arXiv, 2019.
Paul Dobson and Andrew J. Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of Molecular Biology, 2003.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometrie. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaminn for fast adaptation of deep networks. In ICML, 2017.
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. In ICLR, 2018.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In ICML, 2017.
William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. IEEE Data Engineering Bulletin, 2017.
Lu Haonan, Seth H. Huang, Tian Ye, and Guo Xiuyan. Graph star net for generalized multi-task learning. arXiv, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Chester Holtz, Onur Atan, Ryan Carey, and Tushit Jain. Multi-task learning on graphs with node and graph level labels. In NeurlPS Workshop on Graph Representation Learning. 2019.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-leamiin in neural networks: survey, arXiv, 2020.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. CVPR, 2018
Jongmin Kim, Taesup Kim, S. Kim, and C. Yoo. Edge-labeling graph neural network for few-shot learning, In CVPR, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Diya Li and Heng Ji. Syntax- multi-task graph convolutional networks for biomedical relation extraction. In LOUHI, 2019.
10

---

Under review as a conference paper at ICLR 2021
Lu Liu, Tianyi Zhou, Guodong Long. Jing Jiang, and Chengqi Zhang. Learning to propagate for graph meta-learning. In NeurlPS, 2019a.
Pengfei Liu, J. Fu, Y. Dong, Xipeng Qiu, and J. Cheung. Learning multi-task communication with message passing for sequence learning. In AAAI, 2019b.
Kaushalya Madhawa and Tsuyoshi Murata. Active learning graphs via meta learning. In ICML Workshop on Graph Representation Learning and Beyond, ICML, 2020.
Kevis- Prrissii Maninis, Ilija Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multi- ple tasks. In CVPR, 2019.
Floriane Montanari, Lara Kuhnke, Antonius Ter Laak, and Djork-Amé Clevert. Modeling physico- chemical ADMET endpoints with multitask graph convolutional networks. Molecules, 2019.
Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML
Workshop on Graph Representation Learning and Beyond, 2020.
Cuong Q. Nguyen, Constantine Kreatsoulas, and Branson Kim M. Meta-learning gnn initializations for low-resource molecular property prediction. In ICML Workshop on Graph Representation Learning and Beyond, ICML, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. NeurlPS. 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Couraapaa,, Matthieu Brucher, Matthieu Perrot, and Édouard Duch- esnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In ICLR, 2020.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.
The graph neural network model. IEEE Transactions Neural Networks, 2009.
Ida Schomburg. Antje Chang, Christian Ebeling, Marion Gremse, Christian Heldt, Gregor Huhn, and Dietmar Schomburg. Brenda, the enzyme database: updates and major new developments. Nucleic acids research, 2004.
Trevor Standley, Amir R. Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In ICML, 2020.
Qiuling Suo, Jingyuan Chou, Weida Zhong, and Aidong Zhang. Tadanet: Task-adaptive network for graph-enriched meta-learning. In ACM SIGKDD, 2020.
Jeffrey J. Sutherland, Lee A. O' O'Brien, and Donald F. Weaver. Spline-fittingg with genetic algorithm: A method for developing classification structure-activity relationships. Journal of Chemical In- formation and Computer Sciences, 2003.
Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, Dengxin Dai, and Lue Van Gool. Revisiting multi-task learning in the deep learning era. arXiv, 2020.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lið, and Yoshua Bengio. Graph Attention Networks. In ICLR, 2018.
Shanfeng Wang, Qixiang Wang, and Maoguo Gong. Multi-task task learning based network embedding. Frontiers in Neuroscience, 2020.
11

---

Under review as a conference paper at ICLR 2021
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Yu Xie, Maoguo Gong, Yuan Gao, A. K. Qin, and Xiaolong Fan. A multi-task representation learning architecture for enhanced graph classification. Frontiers in Neuroscience, 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ICLR, 2019.
Huaxiu Yao, Chuxu Zhang, Ying Wei, Meng Jiang, Suhang Wang, Junzhou Huang, Nitesh V. Chawla, and Zhenhui Li. Graph few-shot learning via knowledge transfer. In AAAI, 2020.
Yu Zhang, Ying Wei, and Qiang Yang. Learning to multitask. In NeurlPS, 2018.
Fan Zhou, Chengtai Cao, Kunpeng Zhang, Goce Trajcevski, Ting Zhong, and Ji Geng. Meta-gnn: On few-shot node classification in graph meta-learning In CIKM, 2019.
Daniel Zügner and Stephan Günnemann. Adversarial attacks on graph neural networks via meta learning. ICLR, 2019.
A EPISODE DESIGN ALGORITHM
Algorithm S contains the procedure for the creation the episodes for our meta-learning procedures. The algorithm takes as input batch of graphs (with graph labels, node labels, and node features) and the loss function balaneing weights, and outputs a multi-task episode. We assume that each graph has a set of attributes that can be accessed with dot-notation (like in most object-oriented programming languages).
Notice how the episodes are created so that only one task is performed on each graph. This is impor- tant as in the inner loop of our meta-learning procedure, the learner adapts and tests the adaptated parameters on one task at a time. The outer loop then updates the parameters, optimizing for repre- sentation that leads to fast single-task adaptation. This procedure bypasses the problem of learning parameters that directly solve multiple tasks, which can be very challenging.
Another important aspect to notice is that the support and target sets are designed as they were training and validation splits for training single-tasi taskeettsk model with the classical procedure. This way the oojaa-obeective becomes to train a model that can generalize well.
B ADDITIONAL EXPERIMENTAL DETAILS
In this section we provide additional information the implementation of the models used in our experimental section. We implement our models using PyToorch (Paszke et al., 2019), PyTorch Geometric (Fey & Lenssen 2019) and Torchmeta (Deleu et al.) 2019). For all models the number and structure of the layers is as described in Section 4.2 of the paper, where we use 256-dimensional node embeddings at every layer.
At every cross-validation fold the datasets are split into 70% for training, 10% for validation, and 20% for testing. For each model perform 100 iterations of hyperparameter optimization over the same search space (for shared parameters) using Ax (Bakshy et al. 2018).
We tried some sophisticated methods to balance the contribution of loss functions during multi-task training like GradNorm (Chen et al., 2018) and Uncertainty Weights (Kendall et al. 2018). but we saw that usually they do positively impact performance. Furthermore, in the few cases where they increase performance, they work for both classically trained models, and for models trained with our proposed procedures. We then set the balancing weights to (GC) (NC) (L.)) provide better comparisons between the training strategies.
12

---

Under review as a conference paper at ICLR 2021
Algorithm 2: Episode Design Algorithm
Input Batch of n randomly sampled graphs B = (G1..., 9n}
Loss weights A(GC), X(NC), X E [0, 1]
Output: Episode & (CE, (mm) m))
B(GO), B B(NC), B(LP) t equally divide the graphs in B in three sets
/* Graph Classification
(GC) TG() randomly divide B(GC) with 60/40 split
3, .Te.
Node Classification
for 9 in B(NC)
num_labellednnoes + Gi num_nodes x 0.3
divid nodes per class, then iteratively randomly sample one node per class without replacement and add it to N until [V] mum_labellednodes
G + (Cp
labe led nodes c- W; label led nodes Gi nodes W (NC) add(G.); ((NC)
& add((G))
end
/* Link Prediction
for G in (LPP) do
E/ + randomly pick negative samples (edges that are not in the graph; possibly in the same number the number of edges in the graph)
E.1NN , EE, divide E with 80/20 split
E + randomly remove 20% of the edges in G;
G(T)) + G. removed of E
(T2) copy(g("))
G( positive.edges ((() edges; " .pssitive.edges E (P)
g(()) negativeeedges + E. (,,( g( .ngattive +- E2 E2)(M)
(LL)) )dd(g Te. (LP)) add()
end
m)
- (S(0)) GG) (NC))
S s(IP)
E & (Té. GC) NC)) Tse (1P)
Te,
(G0)
T. +- Cross-Entropye
C + Cross
(LP)) < Binary Cross-Entrooyl
2 (G0))_1
(GC) (GC) + (NC)) (NC).
+ (LP)) (PP)
Return E (Chmm) Sst Téd
13

---

Under review as a conference paper at ICLR 2021
Linear Model. The linear model trained the embeddings produced by our proposed method is a standard linear SVM. In particular we use the implementation available in Scikit-learn (Pedregosal et al., 2011) with default hyperparameters. For graph classification, take the mean of the node embeddings as input. For link prediction we take the concatenation of the embeddings two nodes. For node classification we keep the emheddings unaltered.
Deep Learning Baselines. We train the single task models for 1000 epochs, and the multi-task models for 5000 epochs. with early stopping on the validation set (for multi-task models we use the sum of the task validation losses accuracies done using Adam (Kingma & Ba 2015). For node classification and link prediction we found that normalizing the node embeddings unit norm in between GCN layers helps performance.
metrics for early-stopping.. Optimization is
Our Meta-Learning Procedure. We train the single task models for 5000 epochs, and the multi- task models for 15000 epochs, with early stopping on the validation set (for multi-task models we the sum of the task validation losses accuracies as metrics for early-stopping), Early stopping is very important in this case as it is the only way to check if the meta-learned model is overfitting the training data. The inner loop adaptation consists of step of gradient descent. Optimization in the outer loop is done using Adam (Kingma & Ba, 2015). We found that normalizing the node embeddings to unit norm in between GCN layers helps performance.
C COMPARISON WITH TRADITIONAL TRAINING APPORACHES
Our proposed meta-learning approach is significantly different from the classical training strategy (Algorithm D). and the traditional meta-learning approaches (Algorithm 4
The classical training approach for multi-task models takes input batch of graphs, which is simply a set of graphs, where on each graph the model has to execute all the tasks. Based on the cumulative loss on all tasks
L= XGG) + AINC) + (LP)) C(LP)
for all the graphs in the batch, the parameters are updated with some form of gradient descent, and the procedure is repeated for each batch.
The traditional meta-learning approach takes as input an episode, like our approach, but for every graph in the episode all the tasks are performed. The support set and target set are single sets of graphs, where every task can be performed on all graphs. The support set is used to obtain the adapted parameters O', which have the goal of concurrently solving all tasks on all graphs in the target set. The loss functions, both for the inner loop and for the outer loop, are the same as the one used by the classical training approach. The outer loop then updates the parameters aiming at a setting that can easily, i.e. with few steps of gradient descent, be adapted to perform multiple tasks concurrently given a support set.
Algorithm 3: Classical Training
Algorithm 4: Traditional Meta-Learning
Input: Model fo; Batches B (Bi,., Br init(())
Input: Model fo: Episodes & = {E1, "i,,,..,,, En init((f)
for B, in B do
for &, in do
loss concurrently perform all tasks
i_loss f- concurrently perform all
on all graphs in B,
tasks on all support set graphs
0 - UPDATE((, loss)
0' <- ADAAPT(((, i.loss)
end
о ooss + concurrently perform all tasks all target set graphs using parameters 0'
+ UPDATE(0, o.lsssss)
end
14

---

Under review as a conference paper at ICLR 2021
Table 4: Results of neural network trained the embeddings generated by multi-task model, perform task that was not seen during training by the multi-task model. "z.y ->>" indicates that the multi-task model was trained on tasks and y, and the neural network performing task 2.
Task
Model
Dataset
ENZYMES
PROTEINS
DHFR
COX2
CI
56.9 + 3.9
1.
61.2 2.2
59.8 + 0.4
GC,NC -PP
iSAME
77.3 - 4.5
88.5
9
07.12.0
eSAME
78.9 - 2.8
1.5
9.2
95.8 3.3
CI
1.2
1.6
9.3
68.9 10.7
GC,LP->NC iSAME
7
2.5
78.1 1.6
eSAME
7
64.7
76.1
76.9 + 3.3
CI
47.1
1.5
77.
79.9 - 3.4
NC,LP->GC
iSAME
48.5 : 5.5
2.3
76.1
79.7 = 5.1
eSAME
56.6 3.1
2.76
77.1
79.3 = 6.2
D FULL RESULTS FOR THE GENERALIZATION OF NODE EMBEDDINGS
Table contains results for neural network, trained on the embeddings generated by multi-task model, perform a task that was not seen during the training of the multi-task model. Accuracy (%) is used for node classification (NC) and graph classification (GC); ROC AUC (%) is used for link prediction (LP). The embeddings produced by meta-leaminn methods lead to higher per- formance (up to 35%), showing that our procedures lead to the extraction of more informative node embeddings with respect the classical end-to-end training procedure.
15


Reviews:
[{"review_id": "x52RDBfnnLP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript proposes SAME, a model based on GNN and meta-learning for learning multi-task node embeddings. Unlike multi-task learning setting, SAME aims at learning to quickly adapt to multiple tasks. Two model variants iSAME and eSAME are proposed base on different settings in inner/outer loop of parameter update. Experiments on several datasets demonstrate the good performance of SAME. \n\nPros\n1. The problem is new and interesting. It is the first work to study single set of node embeddings for multi-tasks in graph. \n2. The presentation is overall good. The content is clear for me. \n3. Introduce a new model for learning multi-task node embeddings through meta-learning way. The model is simple yet interesting for new problem. \n\nCons/Questions\n1. The novelty of this work incremental. Despite the new problem and different task settings, the model framework adopts the similar procedure as general meta-learning procedure. The model is quite simple. I would like to see more discussion about the contribution and novelty of this work as well as the potential future study. \n\n2. This work follows the meta-learning setting. Besides studying different graph learning tasks, it is better to provide content and add experiment for the scenario of few-shot labeled data. If I did not miss it, there is no discussion and experiment about this. I would suggest the authors to add comparison experiments for different tasks where only few-shot supervised data are available. \n\n3. Reference format is not consistent, typos, etc. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "th96Uu1O5h6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper formulates the learning of three tasks, including graph classification, node classification and link prediction, as a multi-task learning problem and adopts a meta learning approach to learn the three tasks together in the spirit of the Model-Agnostic Meta Learning (MAML) method.\n\nActually, there have been some works to study the three tasks (i.e., graph classification, node classification and link prediction) as a multi-task learning problem. Authors need to discuss differences with those works and compare with them in experiments.\n\nThe proposed meta learning approach seems a direct application of the MAML method. I cannot see much difference with the MAML method.\n\nIn the meta-objective, how to set different \\lambda’s? This is more important to the performance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4nMX3y5g8J", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a multi-task framework to represent the node embedding for transferred knowledge. The methodology is based on the meta-learning, which is capable of producing multi-task node embedding. This paper is well-motivated and well-written. The experimental results illustrate the effectiveness of the model. I would like to recommend to accept this paper.\n\nMajor Concerns:\n1. The related work can be strengthened. In the current version, the related work seems to stack all the papers in sequence, which makes it tough to understand the development of this task. I suggest the authors to reformate this subsection to Mind Graph.\n2. Can you introduce the dataset more specifically?\n3. There shall be more baselines for the experiments.\n\nMinor Concern:\n1. Fig.1: the legend shall be outside the box.\n2. Sec 3 seems redundant to most related readers. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper experimentally observes the negative transfer in Multi-task Graph Representation Learning and proposes to solve the negative transfer with a novel Meta-Learning based training procedure. However, the proposed methods seems not technically sound. There are some concerns about this paper：1. The technique contribution of this paper is limited. The method proposed in this paper is just an application of MAML in Graph Representation Learning with a little variation. 2. This paper only compares SAME with the vanilla MTL method, which adopts the uniform weights. However, the vanilla MTL method commonly performs poorly. The state-of-the-art MTL methods should be taken into comparison, for example MGDA [1]. 3. The traditional Meta-Learning framework introduced in Algorithm 4 is misleading.  4. The experimental analysis of this paper is not sufficient. For example, the paper has not analyzed whether the improvement comes from the meta updating or comes from the singularly training strategy.

[1]. Sener, Ozan, and Vladlen Koltun. "Multi-task learning as multi-objective optimization." NIPS 2018.


Decision: Reject