Dynamic Graph Representation Learning with Fourier Temporal State Embedding

1 INTRODUCTION . Graph Representation Learning learns the graphs with low-dimensional vectors at nodes and graphs level . ( Perozzi et al. , 2014 ; Tang et al. , 2015 ; Wang et al. , 2016 ; Cao et al. , 2015 ; Ou et al. , 2016 ) In recent years , deep neural networks ( DNNs ) are extended to process graphical data and have been utilized in a plethora of real-time cases . ( Estrach et al. , 2014 ; Duvenaud et al. , 2015 ; Defferrard et al. , 2016 ; Li et al. , 2015 ; Gilmer et al. , 2017 ; Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Jin et al. , 2017 ; Chen et al. , 2018 ; Veličković et al. , 2018 ; Gao & Ji , 2019 ) . One of the most popular networks is Graph Convolution Neural Networks ( GCNs ) which originated from Spectral Graph Theory but developed into spatial-based varieties . GCNs are natural extensions of Convolution Neural Networks ( CNNs ) which has been widely studied and used in many applications in different fields of research . Traditional GCNs have achieved commendable performance on static graphs . However , many applications involved dynamic graphs where nodes and edges evolved steadily over time . For example , a social network is updated on a day-to-day basis as people developed new friends . The dynamic graph represented users ’ evolving social relationships . In financial networks , transactions between nodes naturally adopt time-stamps as temporal features . Transactions of different nature may perform differently in a financial network where our main focus is to find malicious parties . Learning the evolving nature of graphs is an important task where we predict future graphical features and classify nodes based on their past behaviors . Learning evolving graphs poses great challenges on traditional GCN as temporal features can not be easily incorporated into learning algorithms . The simple way of concatenating GCNs with RNNs is straight forward in handling dynamic graphs , but it suffered from many drawbacks . We can summarize them as three folds : Firstly , the embedding vector of each node is not static and will be evolving with time . Models need to be capable of capturing the evolving nature . Secondly , the memory and computation cost for batch training is huge to keep multiple graphs from different timesteps in the memory at the same time . Finally , the large number of timesteps within a single batch brings difficulties to high precision modeling . There is also a focus on using Deep Neural Networks to generate the graph embedding recently ( Trivedi et al. , 2018 ; Pareja et al. , 2020 ; Xu et al. , 2020 ) as another direction compared with traditional unsupervised dynamic graph embedding approaches ( Nguyen et al. , 2018 ; Li et al. , 2018 ; Goyal et al. , 2018 ; 2020 ) . Existing methods normally utilize Sequential Models ( e.g.Recurrent Neural Networks ( RNNs ) ) to learn temporal features . However , as the graph is a non-linear data structure , sequential model-based approaches are memory-costly to train and evaluate with the information from the whole graph as input . Meanwhile , pure GCN approaches built for static graphs are inefficient in capturing evolving features . Some approaches combining GCNs with RNNs ( Trivedi et al. , 2018 ; Pareja et al. , 2020 ) are costly to evaluate due to the high time complexity induced by repeated Graph Convolutions as well as high space complexity caused by a large number of network parameters . Meanwhile , the RNN-based method could only see through a fixed amount of history timesteps in training , which makes the prediction imprecise . We introduced Fourier temporal state embedding ( FTSE ) to address the above problem . Instead of using sequential models to model the evolving nature of edges , we formalize the existence of edges as a signal , transforming the original embedding problem into signal processing . We also designed a simple and efficient Edge-Convolution Network structure for FTSE and compared the complexity of it with RNN based approaches . Our main technique is Discrete-Time Fourier Transform ( DTFT for short ) , which transforms the discrete temporal signal into its continuous frequency domain . Therefore , we can embed history timesteps into a fixed-length vector , enlarging the receptive field in a single batch . Our empirical study shows that FTSE is an efficient method in modeling temporal graphs and a good approach to model the signal with high precision . We summarize the contribution of this work as follows : 1 . We proposed Fourier Temporal State Embedding ( FTSE ) to learn dynamic graph representation via transforming time-series signal into the frequency domain . FTSE directly modeled the harmonic component rather than timesteps . We also designed a simple but potent edge convolution network to model continuous-time dynamic graphs . FTSE is also the first GCN based approach capable of modeling continuous time dynamic graphs ( CTDGs ) . 2 . We studied the drawbacks of sequential-based methods in time and space complexity and justified that FTSE has much lower complexity and smaller parameter scale , making it a more efficient alternative to sequential-based approaches . This has also been proven with experiments . 3 . Extensive empirical study showed that FTSE significantly outperforms previous methods in convergence speed as well as model performance in Link Prediction , Node/Edge Classification , achieving more than 10 % improvement on some datasets . 2 RELATED WORK . Many static network embedding methods are proposed to map the nodes to low-dimensional vector representations while preserving network structure as well as node information . Both supervised and unsupervised techniques have been designed . Dynamic graph embedding methods are often extensions of their static counterparts . DANE ( Roweis & Saul , 2000 ; Belkin & Niyogi , 2002 ) used a matrix factorization-based approach to generate static node embeddings from eigenvectors of graph Laplacian matrix . This work was extended by ( Li et al. , 2017 ) by updating eigenvectors from previous ones to generate dynamic node embeddings . Random-walk based approaches ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ) used normalized inner products of node embedding to model the transition probabilities in random-walk . These two approaches are extended by CTDANE ( Nguyen et al. , 2018 ) , which proposes to walk on the temporal order . Deep learning approaches are also popular in this area thanks to the flourishing new models . DynGEM ( Kamra et al. , 2017 ) used an autoencoding approach which minimizes the reconstruction loss as well as the distance of connected nodes in the embedding space . The point process-based approach is also popular in dynamic knowledge graph modeling . KnowEvolve ( Trivedi et al. , 2018 ) and DyRep ( Trivedi et al. , 2018 ) model the occurrence of edges as a point-process and model the intensity function with neural networks . DynamicTriad ( Zhou et al. , 2018 ) focuses on the basic ’ triad closure process ’ where a group of three vertices is developed from unconnected vertices . They proved that this process is fundamental in graph evolving thereby making their model able to predict network dynamics . HTNE ( Zhou et al. , 2018 ) used the Hawkes process with the attention mechanism to determine the influence of historical neighbors . Point process based approaches are especially good at event time prediction . Another set of approaches comes from the combination of Graph Neural Networks ( GNNs ) with Sequential Networks ( e.g . RNNs ) . GNNs are used to digest topological information and RNNs are used to handle dynamism . Graph Convolution Neural Networks ( GCNs ) is one of the most explored GNNs in this setting . GCRN ( Seo et al. , 2018 ) proposed two methods . The first one is doing GCN first and feed its output to an LSTM to capture dynamism . The second one put LSTM as an alternative for Fully Connected Layer in GCN and feed it with node features . STGCN ( Yu et al. , 2018 ) proposed ST-Conv block which was composed of a 1D convolution on the temporal dimension of node features followed by a Graph Convolution layer and another 1D convolution . STGCN is designed for spatio-temporal traffic data , which had a fixed graph structure but dynamic node embeddings . EvolveGCN ( Pareja et al. , 2020 ) combines the two models together to form EvolveGCN-Unit . They propose two versions of it . In the first one , the GCN parameters are hidden state of a recurrent architecture that takes node embeddings as input . In the second one , the GCN parameters are the input of recurrent architecture . 3 PROBLEM FORMULATION . The dynamic graph has two notable varieties as has been formally defined in ( Kazemi & Goel , 2020 ) . Continuous-time dynamic graph ( CTDG ) can be represented as a pair ( G , O ) where G is a static graph representing the initial state of a dynamic graph at time t0 and O is a set of observations where each observation is a tuple of the form ( event type , event , timestep ) where an event can be a 1-step modification on the graph structure. ( e.g . edge addition , edge deletion , node addition ) At any point t ≤ t0 , we can obtain the snapshot Gt by updating G sequentially according to the observations O occured before time t. The discrete time dynamic graph ( DTDG ) is the sequence of snapshots coming from the dynamic graph sampled at fixed space . DTDG is defined as the set { G1 , G2 , ... GT } where Gt = ( Vt , Et ) is the snapshot at time t. Vt is the set of nodes in Gt , and Et is the set of edges in Gt . DTDG may lose information compared to their CTDG counterparts since it only includes snapshots at constant intervals . Models designed for CTDG are generally applicable to DTDG but the inverse is not necessarily true . CTDG problem can be approximated by DTDGs by aggregating G within a constant time period to form the snapshot . This time period is called granularity , denoting the length of time within a single timestep . The smaller the granularity , the better the approximation to the CTDG . Existing methods predominately focus on the DTDG problem and make approximations to the respective CTDG problem . FTSE , based on Fourier Transform , is capable of modeling CTDGs which we detailed in section 4 . For a more detailed discussion on the implementation of them , please refer to appendix B . The prediction problem could also be categorized into interpolation and extrapolation . Suppose we have information of G in time period [ t0 , tT ] . In interpolation problem , we make predictions on some time t such that t ∈ [ t0 , tT ] . It is also known as the completion problem as has been studied in ( Li et al. , 2017 ; Leblay & Chekol , 2018 ; Dasgupta et al. , 2018 ; Goel et al. , 2019 ) . In extrapolation problem , our goal is to make predictions at time t such that t ≥ tT . Extrapolation is a more challenging problem as it is trying to predict the future based on the past . In this work we focus on extrapolation problem . In some cases , the new observation is streamed to the model at a fast rate that we can not update the model in an on-line fashion . This is also called streaming scenario in ( Kazemi & Goel , 2020 ) . This concept is similar to that of inductive/transductive learning , where the difference lied in whether or not using the training set to do inference on the testing set . Compared with the non-streaming scenario where we are able to retrain the model once new data comes , this scenario poses greater challenges on the model ’ s capacity to generalize .

Summary: This paper presents a new method called Fourier temporal state embedding. The motivation for this approach is unclear and should be appropriately justified. In the abstract and introduction, the claim appears to be that previous methods are not time nor memory-efficient, and therefore FTSE is proposed. But this is obviously not true. So it is unclear what this new approach offers compared to the state-of-the-art. In Table 1, why not report performance for more standard baselines like CTDNE? The clarity and writing of this work require significant improvement. There are many incomplete and incorrect sentences throughout the paper that make it difficult if not impossible to understand. In the problem formulation, CTDG and DTDG were originally introduced in the CTDNE paper, but instead a more recent 2020 paper is referenced. Many of the ideas are never fully explained properly. The labels in nearly all the figures need to be appropriately sized, as they are impossible to read.