Adaptive Data Augmentation on Temporal Graphs

1 Introduction . Many real-world graphs are not static but evolving , where every edge ( or interaction ) has a timestamp to denote its occurrence time . These graphs are called temporal ( or dynamic ) graphs [ 39 ] . Recently , temporal graph networks ( TGNs ) [ 25 , 39 , 16 ] have been proposed to support learning on temporal graphs . Advanced TGNs utilize an RNN based memory module to represent a node ’ s history as a compact state ( see Fig . 1 ) , which is used to predict the node ’ s activities [ 28 ] . TGNs are capable of making predictions from complex graph topology and temporal information , thanks to their advanced representational power . However , the increased representational capacity comes with higher model complexity , which can induce over-fitting and weaken their generalization ability . In particular , a trained TGN may capture random noise instead of semantic information , which is not desired [ 41 ] . To combat over-fitting , data augmentation ( DA ) has been demonstrated to be effective [ 23 ] . Nevertheless , DA for the temporal graphs remains under-explored , of which the main challenges lie in highly irregular dynamic topology . DA applies transformations to input features so as to model realistic noise for enriching the input data . The magnitudes of the transformations , known as DA magnitudes , are controlled by hyper-parameters , which is positively related to the difference between the input features before and after DA [ 5 ] . Existing work on image and text data devises adaptive DA methods , which apply higher DA magnitudes to the less informative parts of input features , in order to effectively augment the input features while preserving the essential semantic information [ 38 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In order to design such an adaptive DA method on temporal graphs , we consider the informativeness of edges to a target node in terms of both time and topology when predicting the activities of the target node . On the time axis , more recent edges tend to be more informative for predicting the target node ’ s states than the earlier ones . For example , when a girl is going to watch a movie in the evening , the interaction between her and a cinema conductor a few minutes ago is much more informative to predict her behaviors than the academic discussion between her and a student in the morning . Similarly , in terms of topology , the edges directly connecting the target node provide more important information than edges a few hops away . Overall , the edges that are closer to the target node in time or topology tend to provide more important information , as validated in the existing work [ 25 ] . To devise an efficient adaptive DA method for temporal graphs , the central idea of this paper is to generate a few graphs with different DA magnitudes , and perform the message passing between these graphs to provide adaptively augmented inputs for every prediction . We encapsulate this idea in a novel module , called MeTA ( Memory Tower Augmentation ) : MeTA stacks a few levels of memory modules as a tower with weight sharing , where lower levels process the graphs of lower DA magnitudes ( see Fig . 2 ) . MeTA includes two message passing mechanisms across levels for adaptive DA . The first one is cross-level propagation , which propagates the features between nodes across levels , while the second one is memory transition , that transmits memory states of a node from higher levels to lower ones periodically . This design allows us to achieve the goal of adaptive DA , since for every prediction , the edges closer to the target node in time or topology are placed on lower levels corresponding to lower magnitudes , as demonstrated by the theoretical analysis ( see Sec . 3.4 ) . To complement MeTA , we propose three DA strategies augmenting both the temporal and topology features : ( i ) perturbing the edge time to simulate time shifts , ( ii ) removing edges , and ( iii ) adding edges to modify the topology . Our strategies effectively enrich the input data by modeling realistic noise intuitively and theoretically . MeTA is a general module that can be applied for training popular TGNs to enhance their performance . MeTA improves the effectiveness of TGNs without increasing their time complexity during training , as analyzed in Sec . 3.4 . Note that our methods do not induce any extra inference cost since DA is not applied during inference . We evaluate our MeTA on edge prediction and node classification tasks using the standard temporal graph datasets : Reddit [ 3 ] , Wikipedia [ 20 ] , MOOC [ 16 ] . We measure its performance through the metrics : test accuracy , average precision ( AP ) , and the area under the ROC accuracy curve ( AUC ) , under inductive and transductive settings . Overall , MeTA achieves substantial improvements for popular TGN models [ 25 , 39 ] and enhances them to outperform the baseline methods . 2 Related Work . Although many methods have been proposed for representation learning on static graphs [ 24 , 9 , 14 , 15 , 10 , 29 , 27 , 18 , 32 , 34 , 36 ] , the work on representation learning in temporal graphs is much sparser . There exist two main classes of temporal graphs : discrete-time dynamic graphs ( DTDG ) [ 17 ] and continuous-time dynamic graphs ( CTDG ) [ 25 ] . DTDG are sequences of static graph snapshots taken at intervals in time , while CTDG can be represented as timed lists of events . Representation learning on CTDG is more flexible , general and challenging , which is the focus of this paper . Early models for temporal graph learning focus on DTDGs [ 11 , 17 , 1 , 8 , 40 ] . Some recent work addresses the learning on CTDGs [ 16 , 28 , 25 , 39 , 30 ] based on the increased model complexity . For example , [ 31 ] proposes a temporal graph learning model based on causal anonymous walks , which uses RNNs to encode the temporal information . Our method is orthogonal to these work in the sense that we do not propose a new TGN model , but acts as a general DA module that can be flexibly incorporated into the training of popular TGNs to improve their effectiveness . Data augmentation plays a central role in training neural networks , of which the effectiveness has been validated on image data [ 42 , 26 , 7 ] . On the data augmentation for the static graph data , [ 35 ] proposes NodeAug to augment and utilize the unlabeled data in semi-supervised learning , and [ 33 ] crops the subgraphs to augment the input features for the static graph classification . In [ 37 ] , the authors propose the Mixup methods for static graphs , which interpolate the input features of both nodes and graphs in the semantic space as data augmentation . Different from them , our work focuses on the DA for temporal graphs . Our MeTA considers the edge importance related to the temporal and topology information and adaptively augment temporal graphs in an efficient manner , while our DA strategies effectively augment both the temporal and topological features . 3 Methodology . In this section , we propose MeTA ( Memory Tower Augmentation ) , an adaptive data augmentation ( DA ) approach for improving the temporal graph learning . A MeTA module can be applied to popular TGN models , and trains them with DA to improve its inference performance . MeTA includes two message passing mechanisms : cross-level propagation and memory transition , to provide adaptively augmented inputs for every prediction , as demonstrated by the theoretical analysis . To complement MeTA , we propose three DA strategies for augmenting temporal graphs by modifying the temporal and topological features , which effectively model the realistic noise to enrich the input data intuitively and theoretically . Using our methods to enhance the popular TGNs does not increase their time complexity and does not induce any inference cost . 3.1 Preliminaries . In a temporal graph G , an interaction between nodes i and j is denoted as a temporal edge eij ( t ) , t > 0 , where e is the edge attributes [ 25 ] , and t is the edge time . t = 0 is generally the time of starting observations . Advanced TGN models utilize a memory module to store and update the ( memory ) state si ( t ) of node i ( see Fig . 1 ) . The state of node i is expected to represent i ’ s history in a compressed format . Given the memory updater as mem , when an edge eij ( t ) connecting node i is observed , node i ’ s state is updated as : si ( t ) = mem ( si ( t − ) , eij ( t ) ‖sj ( t− ) ) , ( 1 ) where si ( t− ) is the memory state of node i just before time t , ‖ is the concatenation operator , and node j is i ’ s neighbor connected by eij ( t ) . si ( t ) is initialized to 0 values and not changed until the next edge involving node i happens . Edge feature eij ( t ) and neighbor state sj ( t− ) are concatenated as the input to update si ( t ) . All edges are fed to the memory modules in time order ( see Fig . 1 ) . The implementation of mem varies slightly for different TGN models , of which a common component is a recurrent neural network ( RNN ) such as LSTM [ 12 ] or GRU [ 4 ] . TGNs make the prediction for node i at time t using si ( t− ) , instead of si ( t ) , since when making predictions for time t , the interactions at time t have not happened and can not be observed . The number of hops is used to denote the distance on the topology [ 22 ] . From Eq . ( 1 ) , we observe that the edges connecting node i directly contributes to i ’ s state , while the edges , that are multiple hops away from i , have their features propagated to node i through i ’ s neighbors ’ states . 3.2 Memory Tower for Adaptive Data Augmentation on Temporal Graphs . Data augmentation ( DA ) aims at creating novel training data by applying transformation to input features [ 38 ] . The magnitudes of the transformations , known as DA magnitudes , are controlled by the hyper-parameters , which is positively related to the difference between the input features before and after DA [ 5 ] . To effectively augment the input features while preserving the essential semantic information , adaptive DA methods apply low DA magnitudes to the informative parts of the input features and higher magnitudes to the less informative parts [ 38 ] . For example , TF-IDF based word replacement , a DA method for natural language processing , is designed to remove less important keywords with higher probability [ 38 ] . This helps the model to generalize better to changes in the uninformative features , and effectively encode the essential features into the representations . In a temporal graph , when predicting the activities of a target node , different edges are of different informativeness for the target node . For example , an engineer is busy doing a project in the evening , the interaction between him and his colleague a few minutes ago is much more important for predicting his working status than the discussion between him and his daughter on a new movie in the morning . Similarly , the edges directly connecting the target node tend to be more informative than the ones that are multiple hops distant since the former directly happen on the target node . Overall , if an edge is more close to the target node in terms of time or topology , it is more informative , as validated in [ 25 ] . Applying adaptive DA magnitudes to edges of different informativeness is not trivial , since an edge ’ s informativeness changes with different predictions , i.e. , when making predictions for different target nodes , the distance from an edge to the target node changes . A simple method for adaptive DA is to separately augment the graph with adaptive magnitudes for every prediction , which leads to different input graphs for different predictions . Therefore , this method repeats feeding the graph edges to the model from the beginning time for every prediction , and is named the refeed method . Existing work [ 25 , 28 , 16 ] trains the memory module in a recursive manner for efficiency , i.e. , the predictions at time t take the former output at time τ ( ∀τ < t ) as inputs , as shown in Fig . 1 , since there is no DA and different predictions share the same input graph . The refeed method is much less efficient than recursive training , since in the former one , the memory states before the prediction time have to be recomputed for every prediction , while the number of predictions is generally the same as the number of edges during training , which is a large value in practice [ 16 ] . We propose the Memory Tower Augmentation ( MeTA ) , which offers appropriate DA magnitudes for every prediction in an efficient manner . MeTA stacks L levels of memory modules as a tower with weight sharing . For level l ( l = 1 , . . . , L ) , we augment the graph G as G ( l ) with the magnitude pl . pl increases as l increases , so that the DA magnitude is higher at a higher level . Denote the state of node i at time t on level l as s ( l ) ( t ) , and the edge connecting nodes i and j at time t on level l , i.e. , belonging to G ( l ) , as e ( l ) ij ( t ) . We feed edges at all levels in time order to the memory module , and always make the prediction using the memory states at the first level , i.e. , s ( 1 ) i ( t ) . MeTA includes two message passing mechanisms across levels to offer adaptive DA for every prediction : cross-level propagation and memory transition . With cross-level propagation , when observing an edge e ( l ) ij ( t ) involving node i at level l , node i ’ s state at level l is updated as : s ( l ) i ( t ) = mem ( s ( l ) i ( t − ) , e ( l ) ij ( t ) ‖s ( min ( l+1 , L ) ) j ( t − ) ) , ( 2 ) In this way , the edges that are more distant to the target node i in terms of the topology are augmented at a higher magnitude , since their features are propagated to the target node through the neighbors ’ states s ( min ( l+1 , L ) ) j ( t − ) that are located on higher levels ( see Proposition 1 ) . In addition to the distances on the topology , the time distances also influence the edge ’ s informativeness . Hence , the second message passing mechanism we design is the memory transition for nodes ’ Algorithm 1 Memory Tower for Adaptive Data Augmentation on Temporal Graphs Input : Original Temporal Graph G with the node set V and the edge sequence E in time order with timestamps : t1 , . . . , tN ( 0 ≤ t1 ≤ · · · ≤ tN ) , where N is the number of edges , a TGN model with the memory module mem , hyper-parameters L for the level number , T for the memory transition period , and the augmented graphs at different levels G ( l ) , l = 1 , . . . , L. Output : The predictions made by the TGN model on time t1 , . . . , tN : { p ( ti ) |i = 1 , . . . , N } 1 : Initialize states s ( l ) i ( 0 ) , ∀l = 1 , . . . , L , ∀i ∈ V , counter values ci , ∀i ∈ V , starting time t0 to 0 . 2 : for n← 1 to N do 3 : eij ( tn ) ← E [ n ] ( the nth element in E ) 4 : We order the edges from G ( l ) , l = 1 , . . . , L between tn−1 ( inclusive ) and tn as a sequence En with timestamps τ1 , . . . , τNn ( tn−1 ≤ τ1 ≤ · · · ≤ τNn < tn ) . 5 : for n′ ← 1 to Nn do 6 : e ( l ) km ( τn′ ) ← En [ n′ ] ( the n′th element in En ) 7 : s ( l ) k ( τn′ ) ← mem ( s ( l ) k ( τ − n′ ) , e ( l ) km ( τn′ ) ‖s ( min ( l+1 , L ) ) m ( τ − n′ ) ) 8 : s ( l ) m ( τ − n′ ) ← mem ( s ( l ) m ( τ − n′ ) , e ( l ) km ( τ − n′ ) ‖s ( min ( l+1 , L ) ) k ( τ − n′ ) ) 9 : end for 10 : Make prediction p ( tn ) related to edge eij ( tn ) through states on time t−n at level 1 . 11 : for o in { i , j } do 12 : co ← co + 1 13 : if co ≥ T then 14 : for l← 1 to L− 1 do 15 : s ( l ) o ( t−n ) ← s ( l+1 ) o ( t−n ) 16 : end for 17 : co ← 0 18 : end if 19 : end for 20 : end for states from higher levels to lower ones , i.e. , replacing the level-l states with the level-l + 1 ones : s ( l ) i ( t ) ← s ( l+1 ) i ( t ) , ∀l = 1 , . . . , L− 1 . ( 3 ) The transition is made in the memory tower periodically , so that when we make a prediction at level 1 , the more recent edges contributing to the prediction are at lower levels corresponding to lower DA magnitudes ( see Proposition 2 ) . We determine the transition time of node i by implementing a counter for node i , which is initialized to 0 and increments when the model makes a prediction related to node i . If the counter reaches T , we reinitialize the counter and perform this memory transition for node i , where T is a hyper-parameter for the transition period . Overall , we visualize our MeTA method in Fig . 2 and summarize it in Alg . 1 ( see Appendix ) . How to obtain the predictions from memory states and calculate the training loss differ slightly for the specific TGN models and downstream tasks , of which the details can be found in the corresponding work [ 16 , 28 , 25 ] .

Summary: The paper introduces Memory Tower Augmentation (MeTA) --- an adaptive data augmentation approach for temporal graph networks (TGNs). The proposal aims to tackle overfitting and improve learning in TGNs. MeTA applies a hierarchical message-passing scheme to update TGNs' memory states. In addition, MeTA considers three different data augmentation strategies: i) perturb time, ii) remove edges, and iii) add edges with perturbed time.  The experiments show that MeTA achieves higher accuracy than existing TGNs (without data augmentation) on future link prediction and node classification tasks.