Online Learning of Graph Neural Networks: When Can Data Be Permanently Deleted

1 INTRODUCTION . Training of Graph Neural Networks ( GNNs ) on temporal graphs has become a hot topic . Recent works include combining GNNs with recurrent modules ( Seo et al. , 2018 ; Manessi et al. , 2020 ; Sankar et al. , 2020 ; Pareja et al. , 2020 ) and vertex embeddings as a function of time to cope with continuous-time temporal graphs ( da Xu et al. , 2020 ; Rossi et al. , 2020a ) . Concurrently , other approaches have been proposed to improve the scalability of GNNs . Those include sampling-based techniques ( Chiang et al. , 2019 ; Zeng et al. , 2020 ) and shifting expensive neighborhood aggregation into pre-processing ( Wu et al. , 2019 ; Rossi et al. , 2020b ) or post-processing ( Bojchevski et al. , 2020 ) . However , there are further fundamental issues with temporal graphs that are not properly answered yet . First , as new vertices and edges appear ( and disappear ) over time , so can new classes . This results in a distribution shift , which is particularly challenging in an online setting , as there is no finite , a-priori known set of classes that can be used for training and it is not known when a new class appears . Second , scalable techniques for GNNs address the increased size of the graph , but always operate on the entire graph and thus on the entire temporal duration the graph spans . However , training on the entire history of a temporal graph ( even in the context of scaling techniques like sampling ( Chiang et al. , 2019 ; Zeng et al. , 2020 ) ) may actually not be needed to perform tasks like vertex classification . Thus , it is important to investigate if , at some point in time , one can actually “ intentionally forget ” old data and still retain the same predictive power for the given task . In fact , is has been observed in other tasks such as stock-market prediction that too much history can even be counterproductive ( Ersan et al. , 2020 ) . Proposed Solution and Research Questions While we do not suggest to use an entirely new GNN architecture , we propose to adapt existing GNN architectures and scalable GNN techniques to the problem of distribution shift in temporal graphs . In essence , we propose a new evaluation procedure for online learning on the basis of the distribution of temporal differences , which assesses the nature of how vertices are connected in a temporal graph by enumerating the temporal differences of connected vertices along k-hop paths . This information is crucial for balancing between capturing the distribution shift while having sufficient vertices within the GNN ’ s receptive field . In summary , the central question we aim to answer is , whether we can intentionally forget old data without losing predictive power in an online learning scenario under presence of distribution shift . We simulate this scenario by applying temporal windows of different sizes over the temporal graph , as illustrated in Figure 1 . The window size c resembles how much history of the temporal graph is used for training , or with other words : which information we forget . In this example , data older than t − 2 is ignored . We evaluate the accuracy of representative GNN architectures and scalable GNN techniques trained on the temporal window , against training on the entire timeline of the graph ( full history ) . We evaluate the models by classifying the vertices at time step t , before we advance to the next time step . To answer the research question , we break it down into four specific questions Q1 to Q4 , each answered in a separate experiment . For Q1 : Distribution Shift under Static vs Incremental Training , we verify that incremental training is necessary to account for distribution shift , compared to using a once-trained , static model . Extending from Q1 , we investigate in Q2 : Training with Warm vs Cold Restarts whether it is preferable to reuse model parameters from the previous time step ( warm start ) or restart with newly initialized parameters at each time step ( cold start ) . In Q3 : Incremental Training on Different Window Sizes , we answer the question what influence different choices for the window sizes have , i. e. , how far do we need to look into the past such that a GNN trained on the window is still competitive to a model trained on the full graph . Question Q4 extends Q3 by considering Q4 : Incremental Training with Scalable GNN Methods , i. e. , how scalable GNN approaches compare to using the full history of the temporal graph and to which extent scaling techniques can be applied on top of the temporal window . New Datasets To enable an analysis with a controlled extent of distribution shift , we contribute three newly compiled temporal graph datasets based on scientific publications : two citation graphs based on DBLP and one co-authorship graph based on Web of Science . To determine candidate window sizes , we contribute a new measure to compute the distribution of temporal differences within the k-hop neighborhood of each vertex , where k corresponds to the number of GNN layers . We select the 25th , 50th , and 75th percentiles of this distribution as candidate window sizes . This results in window sizes of 1 , 3 , and 6 time steps for the two DBLP datasets , and 1 , 4 , 8 for the Web of Science dataset . Results We select three representative GNN architectures : GraphSAGE-Mean ( Hamilton et al. , 2017 ) , graph attention networks ( Veličković et al. , 2018 ) and jumping knowledge networks ( Xu et al. , 2018 ) along with graph-agnostic multi-layer perceptrons . As scalable GNN techniques , we consider GraphSAINT ( Zeng et al. , 2020 ) as well as simplified GCNs ( Wu et al. , 2019 ) . The results of our experiments show that already with a small window size of 3 or 4 time steps , GNNs achieve at least 95 % accuracy compared to using the full graph . With window sizes of 6 or 8 , 99 % accuracy can be retained . With a window size of 1 , for almost all experiments , a relative accuracy of no less than 90 % could be retained , compared to models trained on the full graph . Furthermore , our experiments confirm that incremental training is necessary to account for distribution shift in temporal graphs and we show that both reinitialization strategies are viable and differ only marginally , when the learning rates are tuned accordingly . Surprisingly , simplified GCNs perform notably well on the most challenging dataset DBLP-hard and are only outperformed by GraphSAGE-Mean . We outline the related work below . We provide a problem formalization and selection of GNNs for our experiments in Section 3 . We describe the experimental apparatus and datasets in Section 4 . The results of our experiments are reported in Section 5 and discussed in Section 6 , before we conclude . 2 RELATED WORK . In Rossi & Neville ( 2012 ) , the authors distinguish between tasks where the predicted attribute is static or changing over time . The dynamic graph problem is set up in a way that vertex and edge features may change over time and that edges may appear and disappear . This is conceptually different as it assumes a fixed vertex set , whereas in our case , the vertex set is changing over time . Furthermore , the predicted attribute is static in our case because it will not change after the respective vertex has appeared . Several recent works follow this setup and assume a fixed vertex set ( Trivedi et al. , 2017 ; Seo et al. , 2018 ; Kumar et al. , 2018 ; Trivedi et al. , 2019 ; Manessi et al. , 2020 ; Sankar et al. , 2020 ) . In Park et al . ( 2017 ) , the authors use vertex features concatenated with the adjacency vector and apply 1D-convolution . The experiments comprise link prediction and user state prediction . 1Dconvolution on the time axis can be regarded as a sliding window . However , the paper does not consider new classes during the evaluation time frame and does not analyze how much past training data would be required for up-training . In Fish & Caceres ( 2017 ) , the authors aim to find the optimal window size , given a dataset , a task , and a model . They treat the window size as a hyperparameter and propose an optimization algorithm which requires multiple runs of the model . This might be rather expensive . Furthermore , the study does not supply insights on how much predictive power can be preserved when selecting a nearoptimal but much smaller , and thus more efficient , window size . CTDNE ( Nguyen et al. , 2018 ) is an embedding method for continuous-time graphs introducing temporal random walks . This approach considers graphs with featureless vertices with the objective to learn a meaningful/useful vertex embedding . In a recent extension of CTDNE ( Lee et al. , 2020 ) , the method is applied to edge streams via up-training of the embedding . Comparing this approach to our work , we find that we have another task ( discrete-time online vertex classification vs continuoustime online vertex embedding ) , consider a different type of graph ( attributed vs featureless ) , and face different challenges ( adaption to new classes ) . Nevertheless , it would be an interesting direction of future work to apply our experimental procedure to ( streaming ) CTDNE . For discrete-time dynamic graphs involving new vertices , Goyal et al . ( 2018 ) proposes DynGEM as an autoencoder-like approach that jointly minimize reconstruction loss between t and t + 1 and embedding distance between connected vertices . In Dyngraph2vec ( Goyal et al. , 2020 ) , the authors extend this approach by additional variants such as recurrent decoders . EvolveGCN ( Pareja et al. , 2020 ) and T-GAT ( da Xu et al. , 2020 ) are both inductive approaches designed for attributed temporal graphs . EvolveGCN predicts the parameters of a GCN with an RNN by tying the RNN output or hidden state to the GCN parameters . T-GAT introduces a selfattention mechanism on the time axis . These approaches can cope with newly appearing vertices and are able to predict different labels for the same node at different times . They both require a sequence of graph snapshots for training . When new classes appear , these sequence-based models would need to be retrained . In our setting with limited window sizes , the sequence of snapshots within a window , i.e . the data available for retraining , might become very short : down to only one snapshot in the extreme case . Furthermore , these approaches focus on predicting future edges or predicting a label for each vertex at each time step . Therefore , the models serve a different purpose compared to the setting that we face , in which the label of each vertex is fixed . For these two reasons , we have focused on adapting and evaluating more efficient , static architectures as well as scalable GNN techniques , while leaving the adaption of T-GAT and EvolveGCN as future work . To summarize , most works on dynamic graphs assume a fixed vertex set , while considering dynamics within the vertex/edge features , and/or the edges themselves . Inductive approaches such as EvolveGCN and T-GAT do allow new nodes . CTDNE can deal with new nodes via up-training . Previous work on finding optimal window sizes proposes a hyperparameter tuning algorithm . However , none of these works specifically analyzes the problem of new classes appearing over time and how much past training data is necessary , or how few is enough , to maintain good predictive power .

Summary: This paper proposes a paradigm which speeds up the training time of GNNs while not compromising too much performance. The method adopts a layerwise training procedure. In particular, the authors inject a loss function at each layer while storing and fixing the feed-forward values of its previous layer. The training is then carried out along all layers parallelly, which allows the updating of paradigms to be decoupled and is not applicable in previous works. A further improvement (lazy-update) by not updating the feed-forward values of each layer is used to reduce the training time.