Metric Learning on Temporal Graphs via Few-Shot Examples

1 INTRODUCTION . Metric learning aims to learn a proper distance metric among data items in the input space , which reflects their underlying relationship . With the prevalence of graph data in many real-world applications , it is of key importance to design a good distance metric function for graph data , such that the output value of the function is small for similar graphs and large for dissimilar ones . Many downstream tasks on the graph data can benefit from such a distance metric . For example , it could lead to significantly improved classification accuracy for graph classification in many domains such as protein and drug discovery ( Schölkopf et al. , 2004 ; Dai et al. , 2016 ) , molecular property prediction ( Duvenaud et al. , 2015 ; Gilmer et al. , 2017 ) , and epidemic infectious pattern analysis ( Derr et al. , 2020 ; Oettershagen et al. , 2020 ) ; it could also speed up the labeling of graph data in an active learning framework ( Macskassy , 2009 ) . However , current graph metric learning methods ( Shaw et al. , 2011 ; Tsitsulin et al. , 2018 ; Bai et al. , 2019 ; Li et al. , 2019 ; Yoshida et al. , 2019 ) assume the input graph data as static and ignore evolution patterns of temporal graphs , which may also provide insights for identifying the graph property ( Isella et al. , 2011 ) . To best of our knowledge , there is currently no algorithm designed for learning metrics over temporal graphs to further involve evolution pattern consideration into the learned metric space . On the other hand , facing limited i.i.d . data , traditional metric learning methods ( Goldberger et al. , 2004 ; Salakhutdinov & Hinton , 2007 ) have been extended to the few-shot learning by transferring the learned metric across different tasks ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Oreshkin et al. , 2018 ; Allen et al. , 2019 ) . Label scarcity problem also occurs in the graph research community , because labeling graph data is typically expensive and requires background knowledge ( Hu et al. , 2020a ; b ; Qiu et al. , 2020 ) , especially for domain-specific applications such as biological graph data labeling ( Zitnik et al. , 2018 ) . Inspired by that , graph metric learning via few-shot examples has recently attracted many nascent researchers ’ attention . But , the majority has been devoted to the node-level metric learning ( Yao et al. , 2020 ; Suo et al. , 2020 ; Huang & Zitnik , 2020 ; Lan et al. , 2020 ; Wang et al. , 2020 ; Ding et al. , 2020 ) , only a few nascent efforts focus on the graph-level metrics ( Ma et al. , 2020 ; Chauhan et al. , 2020 ) , and all of them ignore the graph dynamics but take static graphs as input . To wrap up , these discussed-above observations bring three bottlenecks to present temporal graph metric learning algorithms : 1 ) How to learn a good metric over temporal graphs , especially on the entire graph level ( i.e. , accuracy of metrics ) ; 2 ) How to ensure the learning process only consumes less labelled temporal graph data ; and 3 ) How to smoothly apply that learned metric to identify unseen graphs ( i.e. , flexibility of metrics ) . In this paper , we wish to learn a distance metric only over fewer temporal graphs , which metric ( as shown in Figure 1 ) could not only help accurately classify seen temporal graphs during each metric learning task , but also be adapted smoothly to new metric learning tasks and converge fast ( i.e. , several training iterations ) to classify unseen temporal graphs by consuming a few labeled examples . Our main contributions can be summarized as : • To describe the evolving graph in a fine-grained manner , we propose the streaming-snapshot model that contains multiple time scales suitable for complex real-world scenarios and other merits are discussed in Section 3 . • To learn the metric over a bunch of streaming-snapshot modelled temporal graphs , we propose the prototypical temporal graph encoder to extract the lifelong evolution representation of a temporal graph with the proposed multi-scale time attention mechanism , such that temporal graphs from the same class share the similar encoded patterns ; To make the extracted metric rapidly adapt to unseen temporal graphs with only a few examples , we introduce a meta-learner to transfer and tailor knowledge and encapsulate it with the prototypical temporal graph encoder into an end-to-end model , called METATAG . • We conduct the temporal graph classification experiments on biological network domain and social network domain , which show the effectiveness of METATAG compared with state-of-the-art algorithms . Also , we analyze the convergence speed of METATAG during the meta-testing , the parameter sensitivity , and the ablation study of each part of METATAG . 2 PRELIMINARIES . Graph Metric Learning . Learning a distance metric is closely related to the feature extraction problem ( Globerson & Roweis , 2005 ; Salakhutdinov & Hinton , 2007 ) . To be specific , given any distance metric D , we can measure distance D ( xi , xj ) between two input feature vectors xi ∈ Rm and xj ∈ Rm by computing D′ ( fθ ( xi ) , fθ ( xi ) ) , where fθ is a learnable function mapping the input feature xi ∈ Rm into the latent feature hi = fθ ( xi ) ∈ Rf ( Salakhutdinov & Hinton , 2007 ) . The transformation function fθ could be linear or non-linear ( Wang & Sun , 2015 ) . When fθ is a linear function fθ ( xi ) = Wxi , learning a generalized Mahalanobis metric D can be expressed as follows . D ( xi , yj ) = √ ( xi − xj ) > M ( xi − xj ) = √ ( xi − xj ) > W > W ( xi − xj ) = √ ( Wxi −Wxj ) > ( Wxi −Wxj ) = D′ ( fθ ( xi ) , fθ ( xj ) ) ( 1 ) where M is some arbitrary positive semi-definite matrix to be determined for the Mahalanobis metric D , and M can be decomposed as M = W > W . Then the Mahalanobis metric D on the input feature space is equivalent to the Euclidean metric D′ on the hidden feature space , such that learning a undetermined metric D ( e.g. , Mahalanobis ) on input feature is equivalent to learning hidden features on a fixed metric D′ ( e.g. , Euclidean ) ( Globerson & Roweis , 2005 ; Salakhutdinov & Hinton , 2007 ; Wang & Sun , 2015 ; Snell et al. , 2017 ) . Also , fθ can be a non-linear transformation for involving more parameters to model higher-order correlations between input data dimensions than linear transformations ( Salakhutdinov & Hinton , 2007 ; Wang & Sun , 2015 ; Snell et al. , 2017 ) . Based on the above analysis , we are ready to model our graph metric learning problem : learning a `` good '' distance metric over pairs of graphs is to learn a `` good '' mapping function fθ of graphs in Euclidean space . The `` goodness '' is controlled by θ and we discuss how we define it in Section 3 . 3 STREAMING-SNAPSHOT MODEL AND PROBLEM SETUP . The table of symbols is summarized in Appendix . To specify , we use bold lowercase letters to denote column vectors ( e.g . a ) , bold capital letters to denote matrices ( e.g. , A ) , and A ( i , : ) to denote the i-th row of matrix A . Also , we let the parenthesized superscript denote the timestamp like A ( t ) . We use graph and network interchangeably in this paper . Streaming-Snapshot Model . In the streaming-snapshot model , there exists two kinds of timestamps , te ∈ { 0 , 1 , . . . , Te } denotes the edge timestamp and ts ∈ { 0 , 1 , . . . , Ts } denotes the snapshot timestamp . To be specific , we describe a temporal graph G as a sequence of timestamped snapshots { S ( ts ) } Tsts=0 , and each timestamped snapshot has a set of timestamped edges labeled as ( vi , vj , te , ts ) . Note that , these two timestamps are different measures , they do not need to have the comparison relationship . In Figure 2 , we provide a temporal graph example whose Te = 4 and Ts = 2 . The merits of describing the temporal graph within the streaming-snapshot model include : 1 ) Carrying multi-scale complex temporal information . Some social networks change rapidly in the microscopic view ( Leskovec et al. , 2008 ) , while some graphs like yeast metabolic graph ( Tu et al. , 2005 ) and repeating frames in video analysis ( Li et al. , 2020 ) change slowly in the macroscopic view ( Leskovec et al. , 2005 ) . If the input temporal graph has these two evolution patterns ( i.e. , edge timestamps and snapshot timestamps ) , our streaming-snapshot model could handle both of them simultaneously because streaming model could describe the interaction graph in a rapid and continuous manner and snapshots could compensate for the complement by modeling episodic , slowly-changing , and periodical patterns ( Aggarwal & Subbian , 2014 ) . If not , our streaming-snapshot is also viable by downgrading into a single streaming or a single snapshot model . 2 ) Saving computation memory . When we need to generate the graph-level embedding for a long lifetime temporal graph , we only need to load each snapshot embedding vector instead of loading every node embedding that appears in the whole temporal graph . ( The detail of how to generate a snapshot embedding through its relevant node embeddings is discussed in Section 4.1.1 , i.e. , Multi-Scale Time Attention Mechanism . ) Beyond recent temporal graph representation learning methods ( Pareja et al. , 2020 ; Xu et al. , 2020 ; Beladev et al. , 2020 ) that only focus on one time scale and ignore the whole lifetime evolution representation , our method can learn the lifelong evolution pattern of a temporal graph on different time scales . As for the data structure , we store each edge as ( vi , vj , te ) and each snapshot adjacency matrix as A ( ts ) ∈ R|V ( ts ) |×|V ( ts ) | , i.e. , V ( ts ) ⊆ V and |V ( ts ) | 6= |V ( ts+1 ) | is allowable . Although our method is readily designed for evolving input features according to different timestamps , for the notation clarity , we denote the node feature matrix X ∈ Rn×m , such that the input node feature of temporal graph G is already time-aware , and n = |V | and m denotes the dimension of features . Problem Setup . With the streaming-snapshot modelled temporal graphs , our goal is to learn a parameterized metric that could accurately classify seen temporal graphs and also be smoothly adapted to unseen temporal graphs . Based on above analysis , this problem can be solved by learning a `` good '' graph representation learning function fθ in Euclidean metric . To further achieve this `` goodness '' only with less labelled data , we formalize fθ into a bi-level meta-learning paradigm ( Finn et al. , 2017 ) . Given the streaming-snapshot modelled temporal graphs and corresponding labels G̃ = { ( G0 , y0 ) , ( G1 , y1 ) , . . . , ( Gn , yn ) } , we split G̃ into G̃train for meta-training and G̃test for metatesting , where the testing set only has unseen graph labels from the training set . We shuffle the training set G̃train to sample graph metric learning tasks following a distribution Ti ∼ P ( T ) , where each graph metric learning task Ti is realized by a K-way N -shot temporal graph classification task based on the graph representation fθi ( Gn ) . During each task Ti , we sample a support set G̃trainsupport and a query set G̃trainquery , such that the support set is used to train the graph representation function fθi to accurately predict the graph labels of the query set . At the meta-testing stage , we transfer the learned knowledge from each task ( i.e. , θi ) to the meta-learner ( i.e. , Θ ) , then we update Θ a few times by classifying unseen temporal graphs on support set G̃testsupport , finally we report the classification accuracy of fine-tuned Θ on query set G̃testquery . The concrete objective and loss function of each graph metric learning task Ti , i.e. , the `` goodness '' , is mathematically expressed in Section 4 .

Summary: This paper considers the graph metric learning problem where the underlying graphs are temporal. The key idea of obtaining a higher classification accuracy is to use a bi-level meta-learning paradigm. It essential contains two parts: 1) prototypical temporal graph encoder where the model uses multi-scale time attention to capture temporal information; and 2) meta-learner where it uses bi-level paradigm proposed in Finn et al., 2017. The authors apply the proposed method to the task of graph classification on two real-world datasets. Compared with other baseline methods, MetaTag achieves better performance in terms of classification accuracy.