Adaptive Data Augmentation on Temporal Graphs

1 Introduction . Many real-world graphs are not static but evolving , where every edge ( or interaction ) has a timestamp to denote its occurrence time . These graphs are called temporal ( or dynamic ) graphs [ 39 ] . Recently , temporal graph networks ( TGNs ) [ 25 , 39 , 16 ] have been proposed to support learning on temporal graphs . Advanced TGNs utilize an RNN based memory module to represent a node ’ s history as a compact state ( see Fig . 1 ) , which is used to predict the node ’ s activities [ 28 ] . TGNs are capable of making predictions from complex graph topology and temporal information , thanks to their advanced representational power . However , the increased representational capacity comes with higher model complexity , which can induce over-fitting and weaken their generalization ability . In particular , a trained TGN may capture random noise instead of semantic information , which is not desired [ 41 ] . To combat over-fitting , data augmentation ( DA ) has been demonstrated to be effective [ 23 ] . Nevertheless , DA for the temporal graphs remains under-explored , of which the main challenges lie in highly irregular dynamic topology . DA applies transformations to input features so as to model realistic noise for enriching the input data . The magnitudes of the transformations , known as DA magnitudes , are controlled by hyper-parameters , which is positively related to the difference between the input features before and after DA [ 5 ] . Existing work on image and text data devises adaptive DA methods , which apply higher DA magnitudes to the less informative parts of input features , in order to effectively augment the input features while preserving the essential semantic information [ 38 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In order to design such an adaptive DA method on temporal graphs , we consider the informativeness of edges to a target node in terms of both time and topology when predicting the activities of the target node . On the time axis , more recent edges tend to be more informative for predicting the target node ’ s states than the earlier ones . For example , when a girl is going to watch a movie in the evening , the interaction between her and a cinema conductor a few minutes ago is much more informative to predict her behaviors than the academic discussion between her and a student in the morning . Similarly , in terms of topology , the edges directly connecting the target node provide more important information than edges a few hops away . Overall , the edges that are closer to the target node in time or topology tend to provide more important information , as validated in the existing work [ 25 ] . To devise an efficient adaptive DA method for temporal graphs , the central idea of this paper is to generate a few graphs with different DA magnitudes , and perform the message passing between these graphs to provide adaptively augmented inputs for every prediction . We encapsulate this idea in a novel module , called MeTA ( Memory Tower Augmentation ) : MeTA stacks a few levels of memory modules as a tower with weight sharing , where lower levels process the graphs of lower DA magnitudes ( see Fig . 2 ) . MeTA includes two message passing mechanisms across levels for adaptive DA . The first one is cross-level propagation , which propagates the features between nodes across levels , while the second one is memory transition , that transmits memory states of a node from higher levels to lower ones periodically . This design allows us to achieve the goal of adaptive DA , since for every prediction , the edges closer to the target node in time or topology are placed on lower levels corresponding to lower magnitudes , as demonstrated by the theoretical analysis ( see Sec . 3.4 ) . To complement MeTA , we propose three DA strategies augmenting both the temporal and topology features : ( i ) perturbing the edge time to simulate time shifts , ( ii ) removing edges , and ( iii ) adding edges to modify the topology . Our strategies effectively enrich the input data by modeling realistic noise intuitively and theoretically . MeTA is a general module that can be applied for training popular TGNs to enhance their performance . MeTA improves the effectiveness of TGNs without increasing their time complexity during training , as analyzed in Sec . 3.4 . Note that our methods do not induce any extra inference cost since DA is not applied during inference . We evaluate our MeTA on edge prediction and node classification tasks using the standard temporal graph datasets : Reddit [ 3 ] , Wikipedia [ 20 ] , MOOC [ 16 ] . We measure its performance through the metrics : test accuracy , average precision ( AP ) , and the area under the ROC accuracy curve ( AUC ) , under inductive and transductive settings . Overall , MeTA achieves substantial improvements for popular TGN models [ 25 , 39 ] and enhances them to outperform the baseline methods . 2 Related Work . Although many methods have been proposed for representation learning on static graphs [ 24 , 9 , 14 , 15 , 10 , 29 , 27 , 18 , 32 , 34 , 36 ] , the work on representation learning in temporal graphs is much sparser . There exist two main classes of temporal graphs : discrete-time dynamic graphs ( DTDG ) [ 17 ] and continuous-time dynamic graphs ( CTDG ) [ 25 ] . DTDG are sequences of static graph snapshots taken at intervals in time , while CTDG can be represented as timed lists of events . Representation learning on CTDG is more flexible , general and challenging , which is the focus of this paper . Early models for temporal graph learning focus on DTDGs [ 11 , 17 , 1 , 8 , 40 ] . Some recent work addresses the learning on CTDGs [ 16 , 28 , 25 , 39 , 30 ] based on the increased model complexity . For example , [ 31 ] proposes a temporal graph learning model based on causal anonymous walks , which uses RNNs to encode the temporal information . Our method is orthogonal to these work in the sense that we do not propose a new TGN model , but acts as a general DA module that can be flexibly incorporated into the training of popular TGNs to improve their effectiveness . Data augmentation plays a central role in training neural networks , of which the effectiveness has been validated on image data [ 42 , 26 , 7 ] . On the data augmentation for the static graph data , [ 35 ] proposes NodeAug to augment and utilize the unlabeled data in semi-supervised learning , and [ 33 ] crops the subgraphs to augment the input features for the static graph classification . In [ 37 ] , the authors propose the Mixup methods for static graphs , which interpolate the input features of both nodes and graphs in the semantic space as data augmentation . Different from them , our work focuses on the DA for temporal graphs . Our MeTA considers the edge importance related to the temporal and topology information and adaptively augment temporal graphs in an efficient manner , while our DA strategies effectively augment both the temporal and topological features . 3 Methodology . In this section , we propose MeTA ( Memory Tower Augmentation ) , an adaptive data augmentation ( DA ) approach for improving the temporal graph learning . A MeTA module can be applied to popular TGN models , and trains them with DA to improve its inference performance . MeTA includes two message passing mechanisms : cross-level propagation and memory transition , to provide adaptively augmented inputs for every prediction , as demonstrated by the theoretical analysis . To complement MeTA , we propose three DA strategies for augmenting temporal graphs by modifying the temporal and topological features , which effectively model the realistic noise to enrich the input data intuitively and theoretically . Using our methods to enhance the popular TGNs does not increase their time complexity and does not induce any inference cost . 3.1 Preliminaries . In a temporal graph G , an interaction between nodes i and j is denoted as a temporal edge eij ( t ) , t > 0 , where e is the edge attributes [ 25 ] , and t is the edge time . t = 0 is generally the time of starting observations . Advanced TGN models utilize a memory module to store and update the ( memory ) state si ( t ) of node i ( see Fig . 1 ) . The state of node i is expected to represent i ’ s history in a compressed format . Given the memory updater as mem , when an edge eij ( t ) connecting node i is observed , node i ’ s state is updated as : si ( t ) = mem ( si ( t − ) , eij ( t ) ‖sj ( t− ) ) , ( 1 ) where si ( t− ) is the memory state of node i just before time t , ‖ is the concatenation operator , and node j is i ’ s neighbor connected by eij ( t ) . si ( t ) is initialized to 0 values and not changed until the next edge involving node i happens . Edge feature eij ( t ) and neighbor state sj ( t− ) are concatenated as the input to update si ( t ) . All edges are fed to the memory modules in time order ( see Fig . 1 ) . The implementation of mem varies slightly for different TGN models , of which a common component is a recurrent neural network ( RNN ) such as LSTM [ 12 ] or GRU [ 4 ] . TGNs make the prediction for node i at time t using si ( t− ) , instead of si ( t ) , since when making predictions for time t , the interactions at time t have not happened and can not be observed . The number of hops is used to denote the distance on the topology [ 22 ] . From Eq . ( 1 ) , we observe that the edges connecting node i directly contributes to i ’ s state , while the edges , that are multiple hops away from i , have their features propagated to node i through i ’ s neighbors ’ states . 3.2 Memory Tower for Adaptive Data Augmentation on Temporal Graphs . Data augmentation ( DA ) aims at creating novel training data by applying transformation to input features [ 38 ] . The magnitudes of the transformations , known as DA magnitudes , are controlled by the hyper-parameters , which is positively related to the difference between the input features before and after DA [ 5 ] . To effectively augment the input features while preserving the essential semantic information , adaptive DA methods apply low DA magnitudes to the informative parts of the input features and higher magnitudes to the less informative parts [ 38 ] . For example , TF-IDF based word replacement , a DA method for natural language processing , is designed to remove less important keywords with higher probability [ 38 ] . This helps the model to generalize better to changes in the uninformative features , and effectively encode the essential features into the representations . In a temporal graph , when predicting the activities of a target node , different edges are of different informativeness for the target node . For example , an engineer is busy doing a project in the evening , the interaction between him and his colleague a few minutes ago is much more important for predicting his working status than the discussion between him and his daughter on a new movie in the morning . Similarly , the edges directly connecting the target node tend to be more informative than the ones that are multiple hops distant since the former directly happen on the target node . Overall , if an edge is more close to the target node in terms of time or topology , it is more informative , as validated in [ 25 ] . Applying adaptive DA magnitudes to edges of different informativeness is not trivial , since an edge ’ s informativeness changes with different predictions , i.e. , when making predictions for different target nodes , the distance from an edge to the target node changes . A simple method for adaptive DA is to separately augment the graph with adaptive magnitudes for every prediction , which leads to different input graphs for different predictions . Therefore , this method repeats feeding the graph edges to the model from the beginning time for every prediction , and is named the refeed method . Existing work [ 25 , 28 , 16 ] trains the memory module in a recursive manner for efficiency , i.e. , the predictions at time t take the former output at time τ ( ∀τ < t ) as inputs , as shown in Fig . 1 , since there is no DA and different predictions share the same input graph . The refeed method is much less efficient than recursive training , since in the former one , the memory states before the prediction time have to be recomputed for every prediction , while the number of predictions is generally the same as the number of edges during training , which is a large value in practice [ 16 ] . We propose the Memory Tower Augmentation ( MeTA ) , which offers appropriate DA magnitudes for every prediction in an efficient manner . MeTA stacks L levels of memory modules as a tower with weight sharing . For level l ( l = 1 , . . . , L ) , we augment the graph G as G ( l ) with the magnitude pl . pl increases as l increases , so that the DA magnitude is higher at a higher level . Denote the state of node i at time t on level l as s ( l ) ( t ) , and the edge connecting nodes i and j at time t on level l , i.e. , belonging to G ( l ) , as e ( l ) ij ( t ) . We feed edges at all levels in time order to the memory module , and always make the prediction using the memory states at the first level , i.e. , s ( 1 ) i ( t ) . MeTA includes two message passing mechanisms across levels to offer adaptive DA for every prediction : cross-level propagation and memory transition . With cross-level propagation , when observing an edge e ( l ) ij ( t ) involving node i at level l , node i ’ s state at level l is updated as : s ( l ) i ( t ) = mem ( s ( l ) i ( t − ) , e ( l ) ij ( t ) ‖s ( min ( l+1 , L ) ) j ( t − ) ) , ( 2 ) In this way , the edges that are more distant to the target node i in terms of the topology are augmented at a higher magnitude , since their features are propagated to the target node through the neighbors ’ states s ( min ( l+1 , L ) ) j ( t − ) that are located on higher levels ( see Proposition 1 ) . In addition to the distances on the topology , the time distances also influence the edge ’ s informativeness . Hence , the second message passing mechanism we design is the memory transition for nodes ’ Algorithm 1 Memory Tower for Adaptive Data Augmentation on Temporal Graphs Input : Original Temporal Graph G with the node set V and the edge sequence E in time order with timestamps : t1 , . . . , tN ( 0 ≤ t1 ≤ · · · ≤ tN ) , where N is the number of edges , a TGN model with the memory module mem , hyper-parameters L for the level number , T for the memory transition period , and the augmented graphs at different levels G ( l ) , l = 1 , . . . , L. Output : The predictions made by the TGN model on time t1 , . . . , tN : { p ( ti ) |i = 1 , . . . , N } 1 : Initialize states s ( l ) i ( 0 ) , ∀l = 1 , . . . , L , ∀i ∈ V , counter values ci , ∀i ∈ V , starting time t0 to 0 . 2 : for n← 1 to N do 3 : eij ( tn ) ← E [ n ] ( the nth element in E ) 4 : We order the edges from G ( l ) , l = 1 , . . . , L between tn−1 ( inclusive ) and tn as a sequence En with timestamps τ1 , . . . , τNn ( tn−1 ≤ τ1 ≤ · · · ≤ τNn < tn ) . 5 : for n′ ← 1 to Nn do 6 : e ( l ) km ( τn′ ) ← En [ n′ ] ( the n′th element in En ) 7 : s ( l ) k ( τn′ ) ← mem ( s ( l ) k ( τ − n′ ) , e ( l ) km ( τn′ ) ‖s ( min ( l+1 , L ) ) m ( τ − n′ ) ) 8 : s ( l ) m ( τ − n′ ) ← mem ( s ( l ) m ( τ − n′ ) , e ( l ) km ( τ − n′ ) ‖s ( min ( l+1 , L ) ) k ( τ − n′ ) ) 9 : end for 10 : Make prediction p ( tn ) related to edge eij ( tn ) through states on time t−n at level 1 . 11 : for o in { i , j } do 12 : co ← co + 1 13 : if co ≥ T then 14 : for l← 1 to L− 1 do 15 : s ( l ) o ( t−n ) ← s ( l+1 ) o ( t−n ) 16 : end for 17 : co ← 0 18 : end if 19 : end for 20 : end for states from higher levels to lower ones , i.e. , replacing the level-l states with the level-l + 1 ones : s ( l ) i ( t ) ← s ( l+1 ) i ( t ) , ∀l = 1 , . . . , L− 1 . ( 3 ) The transition is made in the memory tower periodically , so that when we make a prediction at level 1 , the more recent edges contributing to the prediction are at lower levels corresponding to lower DA magnitudes ( see Proposition 2 ) . We determine the transition time of node i by implementing a counter for node i , which is initialized to 0 and increments when the model makes a prediction related to node i . If the counter reaches T , we reinitialize the counter and perform this memory transition for node i , where T is a hyper-parameter for the transition period . Overall , we visualize our MeTA method in Fig . 2 and summarize it in Alg . 1 ( see Appendix ) . How to obtain the predictions from memory states and calculate the training loss differ slightly for the specific TGN models and downstream tasks , of which the details can be found in the corresponding work [ 16 , 28 , 25 ] .

Summary: This paper proposes MeTA: multi-level module that processes the augmented graphs of different magnitudes on separate levels. Three DA strategies augmenting are adopted (perturbing the edge time to simulate time shifts; removing edges; adding edges). Experiments are conducted on edge prediction and node classification tasks.