Graph Representation Learning for Multi-Task Settings: a Meta-Learning Approach

1 INTRODUCTION Original Embeddings Transferred Embeddings NC GC- > NC LP- > NC0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 Ac cu ra cy - 13.21 % - 14.52 % Node Classification ( b ) GC NC- > GC LP- > GC0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 Ac cu ra cy - 21.29 % - 10.82 % Graph Classification ( c ) LP NC- > LP GC- > LP0.55 0.60 0.65 0.70 0.75 0.80 RO C AU C - 5.89 % - 4.43 % Link Prediction ( d ) Figure 1 : Performance drop when transferring node embeddings between tasks on ( a ) Node Classification ( NC ) , ( b ) Graph Classification ( GC ) , and ( c ) Link Prediction ( LP ) on the ENZYMES dataset . On the horizontal axis , “ x - > y ” indicates that the embeddings obtained from a model trained on task x are used to train a network for task y. Graph Neural Networks ( GNNs ) are deep learning models that operate on graph structured data , and have become one of the main topics of the deep learning research community . Part of their success is given by great empirical performance on many graph-related tasks . Three tasks in particular , with many practical applications , have received the most attention : graph classification , node classification , and link prediction . GNNs are centered around the concept of node representation learning , and typically follow the same architectural pattern with an encoderdecoder structure ( Hamilton et al. , 2017 ; Chami et al. , 2020 ; Wu et al. , 2020 ) . The encoder produces node embeddings ( low-dimensional vec- tors capturing relevant structural and feature-related information about each node ) , while the decoder uses the embeddings to carry out the desired downstream task . The model is then trained in an end-to-end manner , giving rise to highly specialized node embeddings . While this can lead to state-of-the-art performance , it also affects the generalization and reusability of the embeddings . In fact , taking the encoder from a GNN trained on a given task and using its node embeddings to train a decoder for a different task leads to substantial performance loss , as shown in Figure 1 . The low transferability of node embeddings requires the use of one specialized encoder and one specialized decoder for each considered task . However , many practical machine learning applications operate in resource-constrained environments where being able to share part of the model architecture between tasks is of great importance . Furthermore , the training signal from multiple related tasks can lead to higher generalization . Nevertheless , making sure tasks do not negatively interfere with each other is not trivial ( Standley et al. , 2020 ) . The problem of learning models that can perform multiple tasks is known as Multi-Task Learning ( MTL ) , and is an open area of research , attracting many researchers in the deep learning community ( Vandenhende et al. , 2020 ) . MTL on graphs has not received much attention , and no single model capable of performing the three most common graph-related tasks has yet been proposed . In fact , we notice that training a multi-head model with the classical procedure , i.e . by performing multiple tasks concurrently on each graph , and updating the parameters with some form of gradient descent to minimize the sum of the single-task losses , can lead to a performance loss with respect to single-task models . Thus , we propose a novel optimization-based meta-learning ( Finn et al. , 2017 ) procedure with a focus on representation learning that can generate node embeddings that generalize across tasks . Our proposed meta-learning procedure produces task-generalizing node embeddings by not aiming at a setting of the parameters that can perform multiple tasks concurrently ( like a classical method would do ) , or to a setting that allows fast multi-task adaptation ( like traditional meta-learning ) , but to a setting that can easily be adapted to perform each of the tasks singularly . In fact , our metalearning procedure aims at a setting of the parameters where a few steps of gradient descent on a given task , can lead to good performance on that task , hence removing the burden of directly learning to solve multiple tasks concurrently . We summarize our contributions as follows : • We propose a novel method for learning representations that can generalize to multiple tasks . We apply it to the challenging setting of graph MTL , and show that a GNN trained with our method produces higher quality node embeddings with respect to classical end-toend training procedures . Our method is based on meta-learning and is model-agnostic and task-agnostic , which makes it easily applicable to a wide range of multi-task domains . • To the best of our knowledge , we are the first to propose a GNN model generating a single set of node embeddings that can be used to perform the three most common graph-related tasks ( graph classification , node classification , and link prediction ) . In particular , our embeddings lead to comparable or higher performance with respect to single-task models even when used as input to a simple linear classifier . • We show that the episodic training strategy at the base of our proposed meta-learning procedure leads to better node embeddings even for models trained on a single task . This unexpected finding provides interesting directions that we believe can be useful to the whole deep representation learning community . 2 RELATED WORK . GNNs , MTL , and meta-learning are very active areas of research . We highlight works that are at the intersections of these subjects , and point the interested reader to comprehensive reviews of each field . To the best of our knowledge there is no work using meta-learning for graph MTL , or proposing a GNN performing graph classification , node classification , and link prediction concurrently . Graph Neural Networks . GNNs have a long history ( Scarselli et al. , 2009 ) , but in the past few years the field has grown exponentially ; we refer the reader to Chami et al . ( 2020 ) ; Wu et al . ( 2020 ) for a thorough review of the field . The first popular GNN approaches were based on filters in the graph spectral domain ( Bronstein et al. , 2017 ) , and presented many challenges including high computational complexity . Defferrard et al . ( 2016 ) introduced ChebNet , which uses Chebyshev polynomials to produce localized and efficient filters in the graph spectral domain . Graph Convolutional Networks ( Kipf & Welling , 2017 ) then introduced a localized first-order approximation of spectral graph convolutions which was then extended to include attention mechanisms ( Veličković et al. , 2018 ) . Recently , Xu et al . ( 2019 ) provides theoretical proofs for the expressivity of GNNs . Multi-Task Learning . Works at the intersection of MTL and GNNs have mostly focused on multihead architectures . These models are all composed of a series of GNN layers followed by multiple heads that perform the desired downstream tasks . In this category , Montanari et al . ( 2019 ) propose a model for the prediction of physico-chemical properties . Holtz et al . ( 2019 ) and Xie et al . ( 2020 ) propose multi-task models for concurrently performing node and graph classification . Finally , Avelar et al . ( 2019 ) introduce a multi-head GNN for learning multiple graph centrality measures , and Li & Ji ( 2019 ) propose a MTL method for the extraction of multiple biomedical relations . The work by ( Haonan et al. , 2019 ) introduces a model that can be trained for several tasks singularly , hence , unlike the previously mentioned approaches and our proposed method , it can not perform multiple tasks concurrently . There are also some works that use GNNs as a tool for MTL : Liu et al . ( 2019b ) use GNNs to allow communication between tasks , while Zhang et al . ( 2018 ) use GNNs to estimate the test error of a MTL model . We further mention the work by Wang et al . ( 2020 ) which considers the task of generating “ general ” node embeddings , however their method is not based on GNNs , does not consider node attributes ( unlike our method ) , and is not focused on the three most common graph related tasks , which we consider . For an exhaustive review of deep MTL techniques we refer the reader to Vandenhende et al . ( 2020 ) . Meta-Learning . Meta-Learning consists in learning to learn . Many methods have been proposed ( see the review by Hospedales et al . ( 2020 ) ) , specially in the area of few-shot learning . Garcia & Bruna ( 2018 ) frame the few-shot learning problem with a partially observed graphical model and use GNNs as an inference algorithm . Liu et al . ( 2019a ) use GNNs to propagate messages between class prototypes and improve existing few-shot learning methods , while Suo et al . ( 2020 ) use GNNs to introduce domain-knowledge in the form of graphs . There are also several works that use metalearning to train GNNs in few-shot learning scenarios with applications to node classification ( Zhou et al. , 2019 ; Yao et al. , 2020 ) , edge labelling ( Kim et al. , 2019 ) , link prediction ( Alet et al. , 2019 ; Bose et al. , 2019 ) , and graph regression ( Nguyen et al. , 2020 ) . Finally , other combinations of metalearning and GNNs involve adversarial attacks on GNN models ( Zügner & Günnemann , 2019 ) and active learning ( Madhawa & Murata , 2020 ) . 3 PRELIMINARIES . 3.1 GRAPH NEURAL NETWORKS . Many popular state-of-the-art GNN models follow the message-passing paradigm ( Gilmer et al. , 2017 ) . Let us represent a graph G = ( A , X ) with an adjacency matrix A ∈ { 0 , 1 } n×n , and a node feature matrix X ∈ Rn×d , where the v-th row Xv represents the d dimensional feature vector of node v. Let H ( ` ) ∈ Rn×d′ be the matrix containing the node representations at layer ` . A message passing layer updates the representation of every node v as follows : msg ( ` ) v = AGGREGATE ( { H ( ` ) u ∀u ∈ Nv } ) H ( ` +1 ) v = UPDATE ( H ( ` ) v , msg ( ` ) v ) where H ( 0 ) = X , Nv is the set of neighbours of node v , AGGREGATE is a permutation invariant function , and UPDATE is usually a neural network . After L message-passing layers , the final node embeddings H ( L ) are used to perform a given task , and the network is trained end-to-end . 3.2 MODEL-AGNOSTIC META-LEARNING AND ANIL . MAML ( Model-Agnostic Meta-Learning ) is an optimization-based meta-learning strategy proposed by Finn et al . ( 2017 ) . Let fθ be a deep learning model , where θ represents its parameters . Let p ( E ) be a distribution over episodes1 , where an episode Ei ∼ p ( E ) is defined as a tuple containing a loss function , a support set , and a target set : Ei = ( LEi ( · ) , SEi , TEi ) , where support and target sets are simply sets of labelled examples . MAML ’ s goal is to find a value of θ that can quickly , i.e . in a few steps of gradient descent , be adapted to new episodes . This is done with a nested loop optimization procedure : an inner loop adapts the parameters to the support set of an episode by performing some steps of gradient descent , and an outer loop updates the initial parameters aiming at a setting that 1The meta-learning literature usually derives episodes from tasks ( i.e . tuples containing a dataset and a loss function ) . We focus on episodes to avoid using the term task for both a MTL task , and a meta-learning task . allows fast adaptation . Formally , by defining θ′i ( t ) as the parameters after t adaptation steps on the support set of episode Ei , we can express the computations in the inner loop as θ′i ( t ) = θ ′ i ( t− 1 ) − α∇θ′i ( t−1 ) LEi ( fθ′i ( t−1 ) , SEi ) , with θ ′ i ( 0 ) = θ where L ( fθ′i ( t−1 ) , SEi ) indicates the loss over the support set SEi of the model with parameters θ′i ( t − 1 ) , and α is the learning rate . The meta-objective that the outer loop tries to minimize is defined as Lmeta = ∑ Ei∼p ( E ) LEi ( fθ′i ( t ) , TEi ) , which leads to the following parameter update 2 θ = θ − β∇θLmeta = θ − β∇θ ∑ Ei∼p ( E ) LEi ( fθ′i ( t ) , TEi ) . Raghu et al . ( 2020 ) showed that feature reuse is the dominant factor in MAML : in the adaptation loop , only the last layer ( s ) in the network are updated , while the first layer ( s ) remain almost unchanged . The authors then propose ANIL ( Almost No Inner Loop ) where they split the parameters in two sets : one that is used for adaptation in the inner loop , and one that is only updated in the outer loop . This simplification leads to computational improvements while maintaining performance .

Summary: The manuscript proposes SAME, a model based on GNN and meta-learning for learning multi-task node embeddings. Unlike multi-task learning setting, SAME aims at learning to quickly adapt to multiple tasks. Two model variants iSAME and eSAME are proposed base on different settings in inner/outer loop of parameter update. Experiments on several datasets demonstrate the good performance of SAME.