Metric Learning on Temporal Graphs via Few-Shot Examples

1 INTRODUCTION . Metric learning aims to learn a proper distance metric among data items in the input space , which reflects their underlying relationship . With the prevalence of graph data in many real-world applications , it is of key importance to design a good distance metric function for graph data , such that the output value of the function is small for similar graphs and large for dissimilar ones . Many downstream tasks on the graph data can benefit from such a distance metric . For example , it could lead to significantly improved classification accuracy for graph classification in many domains such as protein and drug discovery ( Schölkopf et al. , 2004 ; Dai et al. , 2016 ) , molecular property prediction ( Duvenaud et al. , 2015 ; Gilmer et al. , 2017 ) , and epidemic infectious pattern analysis ( Derr et al. , 2020 ; Oettershagen et al. , 2020 ) ; it could also speed up the labeling of graph data in an active learning framework ( Macskassy , 2009 ) . However , current graph metric learning methods ( Shaw et al. , 2011 ; Tsitsulin et al. , 2018 ; Bai et al. , 2019 ; Li et al. , 2019 ; Yoshida et al. , 2019 ) assume the input graph data as static and ignore evolution patterns of temporal graphs , which may also provide insights for identifying the graph property ( Isella et al. , 2011 ) . To best of our knowledge , there is currently no algorithm designed for learning metrics over temporal graphs to further involve evolution pattern consideration into the learned metric space . On the other hand , facing limited i.i.d . data , traditional metric learning methods ( Goldberger et al. , 2004 ; Salakhutdinov & Hinton , 2007 ) have been extended to the few-shot learning by transferring the learned metric across different tasks ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Oreshkin et al. , 2018 ; Allen et al. , 2019 ) . Label scarcity problem also occurs in the graph research community , because labeling graph data is typically expensive and requires background knowledge ( Hu et al. , 2020a ; b ; Qiu et al. , 2020 ) , especially for domain-specific applications such as biological graph data labeling ( Zitnik et al. , 2018 ) . Inspired by that , graph metric learning via few-shot examples has recently attracted many nascent researchers ’ attention . But , the majority has been devoted to the node-level metric learning ( Yao et al. , 2020 ; Suo et al. , 2020 ; Huang & Zitnik , 2020 ; Lan et al. , 2020 ; Wang et al. , 2020 ; Ding et al. , 2020 ) , only a few nascent efforts focus on the graph-level metrics ( Ma et al. , 2020 ; Chauhan et al. , 2020 ) , and all of them ignore the graph dynamics but take static graphs as input . To wrap up , these discussed-above observations bring three bottlenecks to present temporal graph metric learning algorithms : 1 ) How to learn a good metric over temporal graphs , especially on the entire graph level ( i.e. , accuracy of metrics ) ; 2 ) How to ensure the learning process only consumes less labelled temporal graph data ; and 3 ) How to smoothly apply that learned metric to identify unseen graphs ( i.e. , flexibility of metrics ) . In this paper , we wish to learn a distance metric only over fewer temporal graphs , which metric ( as shown in Figure 1 ) could not only help accurately classify seen temporal graphs during each metric learning task , but also be adapted smoothly to new metric learning tasks and converge fast ( i.e. , several training iterations ) to classify unseen temporal graphs by consuming a few labeled examples . Our main contributions can be summarized as : • To describe the evolving graph in a fine-grained manner , we propose the streaming-snapshot model that contains multiple time scales suitable for complex real-world scenarios and other merits are discussed in Section 3 . • To learn the metric over a bunch of streaming-snapshot modelled temporal graphs , we propose the prototypical temporal graph encoder to extract the lifelong evolution representation of a temporal graph with the proposed multi-scale time attention mechanism , such that temporal graphs from the same class share the similar encoded patterns ; To make the extracted metric rapidly adapt to unseen temporal graphs with only a few examples , we introduce a meta-learner to transfer and tailor knowledge and encapsulate it with the prototypical temporal graph encoder into an end-to-end model , called METATAG . • We conduct the temporal graph classification experiments on biological network domain and social network domain , which show the effectiveness of METATAG compared with state-of-the-art algorithms . Also , we analyze the convergence speed of METATAG during the meta-testing , the parameter sensitivity , and the ablation study of each part of METATAG . 2 PRELIMINARIES . Graph Metric Learning . Learning a distance metric is closely related to the feature extraction problem ( Globerson & Roweis , 2005 ; Salakhutdinov & Hinton , 2007 ) . To be specific , given any distance metric D , we can measure distance D ( xi , xj ) between two input feature vectors xi ∈ Rm and xj ∈ Rm by computing D′ ( fθ ( xi ) , fθ ( xi ) ) , where fθ is a learnable function mapping the input feature xi ∈ Rm into the latent feature hi = fθ ( xi ) ∈ Rf ( Salakhutdinov & Hinton , 2007 ) . The transformation function fθ could be linear or non-linear ( Wang & Sun , 2015 ) . When fθ is a linear function fθ ( xi ) = Wxi , learning a generalized Mahalanobis metric D can be expressed as follows . D ( xi , yj ) = √ ( xi − xj ) > M ( xi − xj ) = √ ( xi − xj ) > W > W ( xi − xj ) = √ ( Wxi −Wxj ) > ( Wxi −Wxj ) = D′ ( fθ ( xi ) , fθ ( xj ) ) ( 1 ) where M is some arbitrary positive semi-definite matrix to be determined for the Mahalanobis metric D , and M can be decomposed as M = W > W . Then the Mahalanobis metric D on the input feature space is equivalent to the Euclidean metric D′ on the hidden feature space , such that learning a undetermined metric D ( e.g. , Mahalanobis ) on input feature is equivalent to learning hidden features on a fixed metric D′ ( e.g. , Euclidean ) ( Globerson & Roweis , 2005 ; Salakhutdinov & Hinton , 2007 ; Wang & Sun , 2015 ; Snell et al. , 2017 ) . Also , fθ can be a non-linear transformation for involving more parameters to model higher-order correlations between input data dimensions than linear transformations ( Salakhutdinov & Hinton , 2007 ; Wang & Sun , 2015 ; Snell et al. , 2017 ) . Based on the above analysis , we are ready to model our graph metric learning problem : learning a `` good '' distance metric over pairs of graphs is to learn a `` good '' mapping function fθ of graphs in Euclidean space . The `` goodness '' is controlled by θ and we discuss how we define it in Section 3 . 3 STREAMING-SNAPSHOT MODEL AND PROBLEM SETUP . The table of symbols is summarized in Appendix . To specify , we use bold lowercase letters to denote column vectors ( e.g . a ) , bold capital letters to denote matrices ( e.g. , A ) , and A ( i , : ) to denote the i-th row of matrix A . Also , we let the parenthesized superscript denote the timestamp like A ( t ) . We use graph and network interchangeably in this paper . Streaming-Snapshot Model . In the streaming-snapshot model , there exists two kinds of timestamps , te ∈ { 0 , 1 , . . . , Te } denotes the edge timestamp and ts ∈ { 0 , 1 , . . . , Ts } denotes the snapshot timestamp . To be specific , we describe a temporal graph G as a sequence of timestamped snapshots { S ( ts ) } Tsts=0 , and each timestamped snapshot has a set of timestamped edges labeled as ( vi , vj , te , ts ) . Note that , these two timestamps are different measures , they do not need to have the comparison relationship . In Figure 2 , we provide a temporal graph example whose Te = 4 and Ts = 2 . The merits of describing the temporal graph within the streaming-snapshot model include : 1 ) Carrying multi-scale complex temporal information . Some social networks change rapidly in the microscopic view ( Leskovec et al. , 2008 ) , while some graphs like yeast metabolic graph ( Tu et al. , 2005 ) and repeating frames in video analysis ( Li et al. , 2020 ) change slowly in the macroscopic view ( Leskovec et al. , 2005 ) . If the input temporal graph has these two evolution patterns ( i.e. , edge timestamps and snapshot timestamps ) , our streaming-snapshot model could handle both of them simultaneously because streaming model could describe the interaction graph in a rapid and continuous manner and snapshots could compensate for the complement by modeling episodic , slowly-changing , and periodical patterns ( Aggarwal & Subbian , 2014 ) . If not , our streaming-snapshot is also viable by downgrading into a single streaming or a single snapshot model . 2 ) Saving computation memory . When we need to generate the graph-level embedding for a long lifetime temporal graph , we only need to load each snapshot embedding vector instead of loading every node embedding that appears in the whole temporal graph . ( The detail of how to generate a snapshot embedding through its relevant node embeddings is discussed in Section 4.1.1 , i.e. , Multi-Scale Time Attention Mechanism . ) Beyond recent temporal graph representation learning methods ( Pareja et al. , 2020 ; Xu et al. , 2020 ; Beladev et al. , 2020 ) that only focus on one time scale and ignore the whole lifetime evolution representation , our method can learn the lifelong evolution pattern of a temporal graph on different time scales . As for the data structure , we store each edge as ( vi , vj , te ) and each snapshot adjacency matrix as A ( ts ) ∈ R|V ( ts ) |×|V ( ts ) | , i.e. , V ( ts ) ⊆ V and |V ( ts ) | 6= |V ( ts+1 ) | is allowable . Although our method is readily designed for evolving input features according to different timestamps , for the notation clarity , we denote the node feature matrix X ∈ Rn×m , such that the input node feature of temporal graph G is already time-aware , and n = |V | and m denotes the dimension of features . Problem Setup . With the streaming-snapshot modelled temporal graphs , our goal is to learn a parameterized metric that could accurately classify seen temporal graphs and also be smoothly adapted to unseen temporal graphs . Based on above analysis , this problem can be solved by learning a `` good '' graph representation learning function fθ in Euclidean metric . To further achieve this `` goodness '' only with less labelled data , we formalize fθ into a bi-level meta-learning paradigm ( Finn et al. , 2017 ) . Given the streaming-snapshot modelled temporal graphs and corresponding labels G̃ = { ( G0 , y0 ) , ( G1 , y1 ) , . . . , ( Gn , yn ) } , we split G̃ into G̃train for meta-training and G̃test for metatesting , where the testing set only has unseen graph labels from the training set . We shuffle the training set G̃train to sample graph metric learning tasks following a distribution Ti ∼ P ( T ) , where each graph metric learning task Ti is realized by a K-way N -shot temporal graph classification task based on the graph representation fθi ( Gn ) . During each task Ti , we sample a support set G̃trainsupport and a query set G̃trainquery , such that the support set is used to train the graph representation function fθi to accurately predict the graph labels of the query set . At the meta-testing stage , we transfer the learned knowledge from each task ( i.e. , θi ) to the meta-learner ( i.e. , Θ ) , then we update Θ a few times by classifying unseen temporal graphs on support set G̃testsupport , finally we report the classification accuracy of fine-tuned Θ on query set G̃testquery . The concrete objective and loss function of each graph metric learning task Ti , i.e. , the `` goodness '' , is mathematically expressed in Section 4 .

Summary: This authors present a novel method for learning representations for time-varying graphs which allows for incorporating information at different time-scales using their streaming-snapshot model. The streaming-snapshot model has the following parts: * Each snapshot $S^{(t_{s})} = (V^{(t_{s})}, E^{(t_{s})})$ has edges of the form $(v_i, v_j, t_e) \in E^{(t_{s})}$ where $t_e$ denotes the time at which edge was formed (and is present since then). * The snapshots $S^{(t_{s})}$ are at a different time-scale ($t_e$ and $t_s$ are not comparable) with the overall learning representation being ${\cal S} \to \Re^f$. * Learning this representation is used for downstream few-shot classification task (for dynamic graphs) and is evaluated on two scenarios - time-varying biological (protein-protein interaction) networks and time-varying social networks.  The MetaTag architecture has the following components:  * _Time-aware node representation_ The edge creation time $t_e$ is used to learn a time-aware node representation ${\bf u}^t_e$ using attention-based weighting of neighbouring nodes features concatenated with a learnable time kernel (Algorithm 1). - The snapshot feature matrix $U^{(t_{s})}$ takes the node representation by consider the latest edge for node $u$ and using attention mechanism above to get influence of earlier edges. * _Intra-snapshot representation_ This is constructed using standard representation loss using a GCN-based encoder-decoder architecture followed by permutation-invariant readout to obtain vector representation for snapshot. * _Overall representation_ The overall representation for the time-varying graph is  weighted average using attention pooling (learnt parameter) of different snapshot representations.   This representation is used downstream for classification task (classification-head) based on prototypical approach [Snell+, 2017] resulting in overall end-to-end differentiable model with weighted average of reconstruction loss and classification loss. Further, the model allows adaptation to new tasks (with different classification labels) by fine-tuning on small test set (few-shot learning).   Experiments are shown on biological and social network datasets (in appendix) showing efficacy of the approach compared to static graph representation methods as well as tdGraphEmbed (doc2vec style method for embedding temporal graphs) including augmentation with ProtoNet for few-shot learning comparison.