Online Learning of Graph Neural Networks: When Can Data Be Permanently Deleted
Authors: Lukas Paul Achatius Galke
Benedikt Franke
Tobias Zielke
Ansgar Scherp
Venue:  (2021)
Keywords: graph neural networks
online learning

Abstract: Online learning of graph neural networks (GNNs) faces the challenges of distribution shift and ever growing and changing training data, when temporal graphs evolve over time. This makes it inefficient to train over the complete graph whenever new data arrives. Deleting old data at some point in time may be preferable to maintain a good performance and to account for distribution shift. We systematically analyze these issues by incrementally training and evaluating GNNs in a sliding window over temporal graphs. We experiment with three representative GNN architectures and two scalable GNN techniques, on three new datasets. In our experiments, the GNNs face the challenge that new vertices, edges, and even classes appear and disappear over time.  Our results show that no more than 50% of the GNN's receptive field is necessary to retain at least 95% accuracy compared to training over a full graph. In most cases, i.e., 14 out 18 experiments, we even observe that a temporal window of size 1 is sufficient to retain at least 90%.

Under review as a conference paper at ICLR 2021
ONLINE LEARNING OF GRAPH NEURAL NETWORKS: WHEN CAN DATA BE PERMANENTLY DELETED?
Anonymous authors Paper under double-blind review
ABSTRACT
Online learning of graph neural networks (GNNs) faces the challenges of distribu- tion shift and ever gbv rowing and changing training data, when temporal graphs evolve over time. This makes it inefficient to train over the complete graph when- ever new data arrives. Deleting old data at some point time may be preferable to maintain a good performance and to account for distribution shift. We system- atically analyze these issues by incrementally training and evaluating GNNs in a sliding window over temporal graphs. We experiment with three representative GNN architectures and two scalable GNN techniques, on three new datasets. our experiments, the GNNs face the challenge that new vertices, edges, and even classes appear and disappear over time. Our results show that no more than 50% of the GNN's receptive field is necessary to retain at least 95% accuracy compared to training over a full graph. In most cases, i. c., 14 out 18 experiments. we even observe that temporal window of size 1 is sufficient retain at least 90%.
INTRODUCTION
Training of Graph Neural Networks (GNNs) on temporal graphs has become a hot topic. Recent works include combining GNNs with recurrent modules (Seo al.
2018 Manessi al. 2020 Sankar et al. 2020: Pareja et al. 2020) and vertex embeddings as a
function of time to cope with 2020a). Concurrently, other
continuous-time temporal graphs (da Xu et al. 2020 Rossi et al.]
approaches have been proposed to improve the scalability of GNNs. Those include sampling-based techniques (Chiang al.] 2019 Zeng al. 2020) and shifting expensive neighborhood aggregation into pre-processing [Wu et al. 2019 Rossi et al. (2200) or post-processing (Bojchevski et al.. 2020).
However, there are further fundamental issues with temporal graphs that are not properly answered yet. First, as new vertices and edges appear (and disappear) over time, so can new classes. This results in a distribution shift, which particularly challenging in an online setting, as there is no finite, a-priori known set of classes that can be used for training and it is not known when new class appears. Second, scalable techniques for GNNs address the increased size of the graph, but always operate the entire graph and thus the entire temporal duration the graph spans. However, training on the entire history of a temporal graph (even in the context of scaling techniques like sampling (Chiang al. 2019 Zeng et al. 2020 may actually not be needed to perform tasks like vertex classification. Thus, is important to investigate if, at some point time, one can actually "intentionally forget" old data and still retain the same predictive power for the given task. In fact, is has been observed in other tasks such as stock-marke prediction that too much history can even be counterproductive (Ersan al. 2020).
Proposed Solution and Research Questions While we do not suggest to use entirely new GNN architecture, we propose to adapt existing GNN architectures and scalable GNN techniques to the problem of distribution shift in temporal graphs. In essence, procedure for online learning on the basis of the distribution of temporal differences, which assesses the nature of how vertices connected in temporal graph by enumerating the temporal differences of connected vertices along k-hhpp paths. This information is crucial for balancing between capturing the distribution shift while having sufficient vertices within the GNN's receptive field.
propose a evaluation
In summary, the central question we aim to answer is, whether we can intentionally forget old data without losing predictive power in online learning scenario under presence of distribution shift.

---

Under review as a conference paper at ICLR 2021
We simulate this scenario by applying temporal windows of different sizes over the temporal graph, as illustrated in Figure []] The window size c resembles how much history of the temporal graph is used for training, with other words: which information we forget. In this example, data older than t 2 is ignored. We evaluate the accuracy of representative GNN architectures and scalable GNN techniques trained the temporal window, against training on the entire timeline of the graph (full history). We evaluate the models by classifying the vertices at time step t, before we advance to the next time step.
Temporal window size
Eval
t-4 t-3 1-2
t-1
I+1
Time
Figure 1: A temporal graph Ge where new vertices with potentially new classes appear over time. For example, class "c" emerged t and was subsequently added the class set C. Training is constrained on a temporal window to simulate intentional deletion of older data. The task is to label the new vertices marked with "?" time step t, before advancing to the next time step.
To answer the research question, we break it down into four specific questions QI to Q4, each an- swered in separate experiment. For Q1: Distribution Shift under Static vs Incremental Train- ing, we verify that incremental training is necessary to account for distribution shift, compared to using a once-trained, static model. Extending from QI, we investigate in Q2: Training with Warm vs Cold Restarts whether it is preferable to reuse model parameters from the previous time step (warm start) or restart with newly initialized parameters cremental Training on Different Window Sizes, we answer the question what influence different choices for the window sizes have, i.e., how far do we need to look into the past such that a GNN trained on the window is still competitive to model trained on the full graph. Question Q4 extends Q3 by considering Q4: Incremental Training with Scalable GNN Methods, i. e., how scalable GNN approaches compare to using the full history of the temporal graph and to which extent scal- ing techniques can be applied on top of the temporal window.
each time step (cold start). In Q3: In-
To enable analysis with a controlled extent of distribution shift, contribute three newly compiled temporal graph datasets based on scientific publications: two citation graphs based DBLP and one co-authorship graph based Web of Science. To determine candidate window sizes, we contribute a new measure to compute the distribution of temporal differences within the k-hop neighborhood of each vertex, where l corresponds to the number of GNN layers. We select the 25th, 50th, and 75th percentiles of this distribution as candidate window sizes. This results in window sizes of 1, 3, and time steps for the two DBLP datasets, and 4, for the Web of Science dataset.
New Datasets
Results We select three representative GNN architectures: GraphSAGE-Mean (Hamitton et al., 2017). graph attention networks (Veličković et al., 2018) and jumping knowledge networks (Xu al. 2018) along with craph-agnostic multi-layer perceptrons. As scalable GNN techniques, we consider GraphSAINT (Zegg et al., (020) as well as simplified GCNs (Wu et al. 2019). The results of our experiments show that already with small window size of 3 or 4 time steps, GNNs achieve at least 95% accuracy compared to using the full graph. With window sizes of 6 or 8, 99% accuracy can be retained. With a window size of 1, for almost all experiments, relative accuracy of no less than 90% could be retained, compared to models trained on the full graph. Furthermore, our experiments confirm that incremental training is necessary to account for distribution shift in temporal graphs and we show that both reinitialization strategies learning rates are tuned accordingly. Surprisingly, simplified GCNs perform notably well on the most challenging dataset DBL ---ad and are only outperformed by GraphSAGFF-Man..
viable and differ only marginally, when the
2

---

Under review as a conference paper at ICLR 2021
We outline the related work below. We provide a problem formalization and selection of GNNs for our experiments in S We describe the experimental apparatus and datasets in Section The results of our experiments reported in Section 5 and discussed in Section [ before conclude.
2 RELATED WORK
In Rossi & Neville (2012). the authors distinguish between tasks where the predicted attribute is static or changing over time. The dynamic graph problem is set up in a way that vertex and edge features may change over time and that edges may appear and disappear. This is conceptually different it assumes a fixed vertex set, whereas in our case, the vertex set is changing over time. Furthermore, the predicted attribute is static in our case because it will not change after the respective vertex has appeared. Several recent works follow this setup and assume fixed vertex set (Trivedi
et al. 2017 Seo et al. 2018: Kumar et al. 2018 Trivedi et al. 2019 Manessi et al. 2020 Sankar
et al.
2020
In Park et al. (2017), the authors use vertex features concatenated with the adjacency vector and apply ID-convolution. The experiments comprise link prediction and user state prediction. ID- convolution on the time axis can be regarded as a sliding window. However, the paper does not consider new classes during the evaluation time frame and does not analyze how much past training data would be required for up-training.
In Fish & Caceres (017), the authors aim to find the optimal window size, given a dataset, a task, and model. They treat the window size as hyperparameter and propose optimization algorithm which requires multiple runs of the model. This might be rather expensive. Furthermore, the study does not supply insights on how much predictive power be preserved when selecting a near- optimal but much smaller, and thus more efficient, window size.
CTDNE Nguuynn et al., 2018) is embedding method for continuous-time graphs introducing temporal random walks. This approach considers graphs with featureless vertices with the objective learn a meaningful/useful vertex embedding. In a recent extension of CTDNE (Lee al. 2020), the method is applied to edge streams via up-training of the embedding. Comparing this approach to our work, we find that we have another task diiscrete-time online vertex classification vs continuous- time online vertex embedding), consider a different type of graph (attributed vs featureless), and face different challenges (adaption to new classes). Nevertheless, would be an interesting direction of future work to apply our experimental procedure to (streaming) CTDNE.
For discrete-time dynamic graphs involving new vertices, Goyal et al. 2018) proposes DynGEM as an autoencoder-like approach that jointly minimize reconstruction loss between i and + 1 and embedding distance between connected vertices. In Dyngraph2vec (Goyal et al. 2020), the authors extend this approach by additional variants such recurrent decoders.
EvolveGCN (Pareja et al. 2020) and T-GAT (da Xu et al., 2020) are both inductive approaches designed for attributed temporal graphs. EvolveGCN predicts the parameters of GCN with an RNÑ by tying the RNN output or hidden state to the GCN parameters. T-GAT introduces a self- attention mechanism on the time axis. These approaches can cope with newly appearing vertices and are able to predict different labels for the same node at different times. They both require a sequence of graph snapshots for training. When new classes appear, these sequence-based models would need to be retrained. In our setting with limited window sizes, the sequence of snapshots within window, i.e. the data available for retraining, might become very short: down to only one snapshot in the extreme case. Furthermore, these approaches focus on predicting future edges or predicting a label for each vertex at each time step. Therefore, the models serve a different purpose compared to the setting that we face, in which the label of each vertex is fixed. For these two reasons, we have focused on adapting and evaluating more efficient, static architectures as well as scalable GNN techniques, while leaving the adaption of T-GAT and EvolveGCN as future work.
To summarize, most works on dynamic graphs assume a fixed vertex set, while considering dy- namics within the vertex/edge features, and/or the edges themselves. Inductive approaches such as EvolveGCN and T-GAT do allow new nodes. CTDNE can deal with new nodes via up-training. Pre- vious work on finding optimal window sizes proposes hyperparameter tuning algorithm. However, none of these works specifically analyzes the problem of new classes appearing over time and how much past training data is necessary, how few is enough, to maintain good predictive power.
3

---

Under review as a conference paper at ICLR 2021
3 PROBLEM FORMALIZATION AND SELECTED METHODS
Problem Formalization We consider a vertex-labeles temporal graph G. (V,, E,) with vertices Vi and edges Et, provided by sequence of snapshots ordered by E N. Thus, V, is the (finite) set of vertices that are in the graph at time step 1, and E, the corresponding set of edges at time step t. Furthermore, we define the set of all vertices V : UieN V2 and all edges E ::- Uien E, i. e..
g = (V, EV. Let tsmin V + N be function that returns for each vertex v E V the timestamp at which the vertex was first added to the graph, i. e., 18min v + min E N/ E V.). Finally, for each vertex e E V we have a feature vector Xx E TR. where D is the number of vertex features, and class label Yo E C with C' being the global set of classes C ::= UseN Cj-
In each time step t, previously unseen vertices and edges and even new classes may appear as illustrated in Figure For these temporal graphs, we investigate training graph neural networks for the vertex classification task, i. e.e assigning class labels y to previously unseen vertices based on vertex attributes X and connections to other vertices via edges. We denote the history of vertices and edges we take into account as the temporal window. The temporal window spans range of multiple time steps, which we denote as the temporal window size c.
Selected Graph Neural Networks Several works have been proposed that combine GNNs with recurrent neural networks to capture temporal dynamics (Seo et al. 2018 Manessi et al.
2020 Sankar et al.
2020
Pareja et al. 2020). Other works focus on continuous-time temporal
graphs (da Xu et al. 2020 Rossi et al. 2020a). Our work is orthogonal to those works as we focus on the distribution shift of temporal graphs and the question if and when old data be deleted without sacrificing predictive power: In the following, we introduce and motivate our choice of representative GNN architectures well as scalable GNN techniques for our experiments.
Dwivedi et al.] (2020) have introduced a benchmarking framework to re-evaluate several recent GNN variants. Dwivedi al. distinguish between isotropic and anisotropic GNN architectures. In isotropic GNNs, all edges are treated equally. Apart from graph convolutional networks (Kipf & Welling 2017), examples of isotropic GNNs are GraphSAGE-
(Hamilton al.] 2017). Diff- Pool Ying et al. 2018). and GIN (Xu et al.] 2019). In anisotropic GNNs, the weights for edges are computed dynamically. Instances of anisotropic GNNs include graph attention networks (Veličkovid
et al. 2018), GatedGCN (Bresson & Laurent 2017) and MoNet (Monti et al. 2017).
We select GraphSAGE-Mean (GS-Mean) (Hamilton et al. 2017 as a representative for isotropic GNNs because its special treatment of the vertices' sell-information has shown to be benefi- cial (Dwivedi et al. 2020). The representations from self-connections are concatenated to averaged neighbors' representations before multiplying the parameters. In GS-Mean, the procedure to obtain representations in layer 1+1 for vertex is given by the equations h1t11 h/|l deg. Ejer(s) h2, and
h/|11 = (U'((t1 where (i)) is the set of adjacent vertices to vertex i, U are the parameters of layer L, a is a non-linear activation function, and -||- is the concatenation.
We select Graph Attention Networks (GATs) by (Velicković al. 2018) as representative for the class of anisotropic GNNs. In GATs, representations in layer ++ for vertex i are computed as follows: h,411 =! + EjeN(e) why h², and h/#1 a (U'h h!t1), where the edge weights Wij and self-connection weights w; are computed by a self-attention mechanism based on the representations h; and h2, e., the softmax of a(U ((U',llU',, hy)) over edges, where a is a single-layer neural network with Lea activation.
Scaling Graph Neural Networks to Large Graphs Several approaches have been proposed to
scale GNNs to large graphs. In general, these approaches fall into two categories: sampling either locally (Hamilton et al. 2017 Huang et al.] 2018)), or globally (Chiang et al.) [019 Zeng et al,
2020 and separating neighborhood aggregation from the neural network component (Wu et al.
2019
Rossi et al. 20206 Bojchevski et al. 2020
From both categories, we select one representative for experiments. We use GraphSAINT Zeng
cet al. 2020) as sfate-of-the-arn sampling technique along with simplified GCNs (Wu et al. 2019) as representative for shifting the neighborhood aggregation into preprocessing step.
4

---

Under review as a conference paper at ICLR 2021
Simplified GCN (Wu et al.) 2019) is scalable variant of Graph Convolutional Networks (Kipf & Welling that admits regular mini-batch sampling. Simplified GCN removes nonlincarities and collapses consecutive weight matrices into single one. Thus, simplified GCN can be described by the equation SGC softmax (S X where the parameter K has similar effect the number of layers in a regular GCN, S is the normalized adjacency matrix and θ is the weight matrix. Instead of using multiple layers. the k-hop neighbourhood is computed by S* which can be precomputed. This makes Simplified GCN efficient to compute, while not necessarily harming the performance.
In GraphSAINT (Zeng et al. 2020), entire subgraphs are sampled for training GNNs. Subgraph sampling introduces a bias which is counteracted by normalization coefficients for the loss function. The authors propose different sampling methods: vertex sampling, edge sampling, and random-walk sampling. We use the best-pett test-performing random- aando--walk sampling for our experiments. The underlying GNN is exchangeable, yet the authors suggest to use Jumping Knowledge networks (JKNets) (Xul al. 2018). JKNets introduce skip-connection to GNNs: each hidden layer has direct connection to the output layer, in which the representations are aggregated, e. g., by concatenation. This enables the network to learn from representations corresponding different levels of the local neighborhood. To isolate the effect of GraphSAINT sampling, we also include JKNets in our comparison.
4 EXPERIMENTAL APPARATUS
Procedure For each evaluation time step E |saant: /ena), we construct a subgraph 9 = (V, E) of induced on V = (u E Vlt c I 18 (v) S f) and E {(u, u) E E u, v E V}. The parameter e denotes the window size, i. e., determines the time steps that the temporal window spans. Then, we supply the competing models with the subgraph &. the corresponding vertex features, and labels for vertices {M V (u) < t) along with an epoch budget for updating their parameters. The task is to predict the labels for vertices {u E V 18min ()) = 1). Finally, we evaluate the accuracy of the model before incrementing L. We provide an algorithmic view in Appendix A
When advancing from one time step to the next, we consider two options of initializing the model. Using cold restarts corresponds to randomly re-initializing each model in each time step and training it from scratch. In contrast, when using warm restarts, take the final weights of the previous time step as initialization for the next time step. In both cases, we initialize the additional parameters in the output layer randomly, when classes appear.
Novel Measure for Distribution of Temporal Differences In the following, we develop a novel dataset-agnostic measure for the distribution of temporal difference within the k-hop neighborhood of each vertex. When l graph convolution layers are used, the features within the k-hop neighbor- hood of each vertex are taken into account for its prediction. This k-hop neighborhood is referred to as the receptive field of GNN (Chen et al. al, 2018). When we incrementally train GNNs on a sliding window through time, the window size determines which vertices are available for training and for inference. Ideally, the temporal window covers all vertices within the GNN's receptive field, such that GNNs have access to all relevant information.
How many vertices of the receptive field are contained in temporal window of size depends on the characteristics of the datasets. Therefore, we introduce a new measure for the distribution of temporal differences tdiff, within the receptive field of a k-layer GNN. Let N*((u) be the k-hop neighborhood of u, i. e., the set of vertices that are reachable from u by traversing at most edges. Then, we define (d) to be the multiset of time differences to past vertices:
udiff' (9) : [1Smin (u) - (m)inn Ju E V A E A USmin (u) 2 (U))
Please note that this is measure to determine comparable window sizes over different datasets and different granularities. It needs to be computed only once per dataseoncet, prior to any training itera- tions. When we consider a GNN with l graph convolution layers, the distribution tdiff,, & enumerates the temporal differences within the receptive field of the GNN. In experiments, we will use the 25th, 50th, and 75th percentiles of this distribution for amalyzing the effect of the temporal window size. This choice corresponds an average receptive field coverage of 25%, 50%, and 75%.
Newly Compiled Datasets Pre-compiled temporal graph datasets for our real-world scenario are surprisingly rare. Therefore contribute three new temporal graph datasets based on scientific
5

---

Under review as a conference paper at ICLR 2021
Table 1: Total number of vertices [V,, number of edges | excluding self-loops, est. power law exponent number of features D and number of classes [C], number of newly appearing classes [Cnew] within the evaluation time steps, the 25.50,75 perreenties of the distribution of temporal dif- ferences 2 along with the total number of time steps T for our datasets.
Dataset
[V
EE|
D
|C|
tdiff, @ (25, 50, 75 %
T
PharmaBio
68,068
2,1M
4,829
7
1, 4, 8
21
DBLP-easy
45,407
112,131
2,278
12 (4 new)
1, 3,
DBLP-hard 198,675
643,734
4,043
73 (23 new)
1, 3, 6
25
publications: one temporal co-authorship graph dataset (PharmaBio) well as two newly compiled temporal citation graph datasets based on DBLP (DBLP-easy and DBLP-hard These new datasets enable us to simulate real-world scenario, in which not only new vertices but also new classes (venues) appear over time. Table summarizze the basic characteristics of the datasets and Figure shows the distribution of temporal differences tdiff,, for different values of k. For details on the dataset creation procedure well as degree and label distributions, we refer Appendix A.2
difference in timesteps
dataset DBLP-easy
dataset DBLP-hard
dataset PharmaBio
1
2
3
max hops
max hops
max hops
Figure 2: Distributions tdiff,, of time differences (y-axis) for PharmaBio (left), DBLP-easy (center) and DBLP-hard (right) within the k-hop neighborhood of each vertex for k (1,2,3) (x-axis).
Evaluation Measures As our datasets have imbalanced classes, one could argue to use Micro Macro F1-score evaluation measure. However,
primarily interested in the relative performance between limited-window training and training on the full graph. Motivated by real- world scenarios, we chose sample-based Fl-score as evaluation measure (equivalent to accuracy in single-label scenarios). When aggregating results over time, we use the unweighted average.
EXPERIMENTAL RESULTS
We report the results of our experiments along the research questions stated in the introduction.
Q1: Distribution Shift under Static Incremental Training In this experiment, compare a once-trained static model against incrementally trained models. We train the static models for 400 epochs on the data before the first evaluation time step. which comprises 25% of the total vertices. We train incremental models for 200 epochs temporal windows of time steps (4 on PharmaBio dataset) before evaluating each time step. models have comparable capacity.
Figure 3 shows the results. We see that the accuracy of the static models decreases over time on DBLP-casy and DBLP-hard, where new classes appear over time. On PharmaBio, the accuracy of the static models plateaus, while the accuracy of incrementally trained models increases. That confirms our expectations as PharmaBio does not have any new classes, and incrementally trained models merely benefit from the increased amount of training data, while DBLP-est Pasy and DBLP- hard do have new classes appearing during the evaluation time frame. In the following experiments, only incrementally trained models because they outperform static models in all cases.

---

Under review as a conference paper at ICLR 2021
675
672
140
0.0
045
Fubo
631
033
2010 20000 2012 2015.
7054 Jear
year
J0ar
Figure 3: Results for QI: Distribution Shift under Static vs Incremental Training. Comparison of static models (solid lines) and incrementally trained models (dashed lines) on PharmaBio (left), DBLP-Easy (center), and DBLP-Hard (right). Average accuracy of 10 runs (y-axis) per time step (x-axis). Error regions are 95% confidence intervals computed with 1 1,000 bootstrap iterations.
Q2: Training with Warm vs Cold Restarts We compare reusing the parameters of the model from the previous time step (warm restart) against randomly re-initializing the model parameters for each temporal window (cold restart). In both cases, we impose 200 epoch budget per time step- The window size is set to for PharmaBio and for the two DBLP datasets, corresponding to 50% coverage of the GNNs' receptive field. All models have comparable capacity.
1.7
014-
6TT
4/2
redd
042
626
200
035
335
202655 20053 NOTE 2300a 20065 20150
2004
2006
2003
2010
26.4
2000
2002
2014
you
Figure 4: Results for Q2: Training with Warm Cold Restarts in online scenario with 200 epochs training over the window per time step. Average accuracy of 10 runs (y-axis) per time step (X-axis). Error regions are 95% confidence intervals computed with 1,000 bootstrap iterations.
Figure 4 shows the results. We observe that the results obtained by GNNs using warm and cold restarts are close to each other. On DBLP-hard with 23 new classes appearing during the evaluation steps, GS-Mean seems to benefit from warm restarts, while GATs yield better scores when cold restarts are used. On PharmaBio with fixed class set, both GNNs benefit from reusing parameters from previous iterations. For now, we conclude that both reinitialization strategies are viable and we proceed by running both variants for the next experiments Q3 and Q4.
We compare the models trained on win- dows of different sizes and compare with a model trained on all available data, e.. the full graph, which is our baseline. We select three window sizes per dataset based on the distribution of temporal differences tdiff, (see Section 4.. These window sized correspond to quartiles, i.e., the windows cover 25%, 50%, and 75% of the GNNs receptive field (RF) (see Table I]. Thus, window sizes across datasets with different characteristics, i. e., connectivity patterns through time and total number of time steps. The epoch budget is 200 and all models have comparable capacity.
Q3: Incremental Training on Different Window Sizes
compare
Table (top) shows the results. We observe that those GNN variants trained on the full timeline of the graph yield the highest scores on DBLP-casy and DBLP-hard There, GNNs with window size (25% RF) yield lower scores than training with larger window sizes (50% and 75% RF). On all datasets, the scores for training with limited window sizes larger than are close to the ones of

---

Under review as a conference paper at ICLR 2021
Table Results for Q3: Incremental Training Different Window Sizes (top), and, Q4: Incre- mental Training with Scalable GNN methods (bottomm.. Average accuracy across different runs and time steps with varying temporal window sizes (column c), 95% CI are computed based on sample variance. We list only the best performing variants of cold (c) and warm (w) restarts for each configuration. We compare each average accuracy to the average accuracy obtained by training on the full graph (see column relative performance).
GAT
GS-Mean
MLP
accuracy
relative
accuracy
relative
accuracy
relative
Dataset
1
w: .649+.00
92%
w: .652+t.00
91%
w: .622++.0
98%
3
wt -601++00
98%
w: .693++.0
97%
w: .629+.00
99%
dblp-easy
c: .703+.00
100%
c: .7112.00
99%
c: .6272.00
99%
full
w: .702+.00
100%
c: .7162.00
100%
c: .634..00
100%
e: .3941.00
86%
w: 4000E.00
85%
w: .3831.00
100%
dblp-hard 3
c: 4.0+.00
96%
w: ..51=.00
96%
w: .389+.00
102%
w: .453.000
99%
w: .467..00
99%
c: .392+.00
103%
full
.564..0
100%
..71
100%
.382+.00
100%
w: .654..01
100%
w: .6861.01
99%
w: .6631.01
101%
4
.653..01
100%
.6902.01
100%
.663..01
101%
pharmabio
w: .614..01
100%
w: .6902.01
100%
w: .6532.01
100%
full
.654+.01
100%
C: .690#.01
100%
.654+.01
100%
Simplified GCN
GraphSAINT
Jumping Knowledge accuracy
accuracy
relative
accuracy
relative
relative
w: .628:..0
90%
w: .64331.0
93%
w .626::.00
88%
3
.672=.00
96%
.671=.00
97%
.681
96%
dblp-easy
c: .6922.00
99%
w: .684=.00
100%
w: .700=.00
99%
full
.699+.00
100%
.685=.00
100%
.708+.00
100%
w: .383..0
90%
w: .372+.00
93%
w: .342+.00
80%
3
.17++000
99%
w: .405+.00
101 %
.426+.00
100%
dblp-hard
6
w: .4244.00
100%
w: .413=.00
103%
w: .4311.00
102%
full
w: .4244.00
100%
w: .402..00
100%
w: .4241.00
100%
w: .642..00
100%
w: .673#.00
104%
W: .650#.00
103%
w: .6477.00
100%
w: .672..00
104%
W: .634+.00
101%
pharmabio 8
.657+.00
102%
..66+.00
103%
.641+.00
102%
full
c: .644+.00
100%
W: .649
100%
W: .630+.00
100%
full- uull-graph training. In summary, window sizes that cover 50% of the receptive field, GNNs and also MLPs achieve at least 95% classification accuracy compared to full-graph training. When 75% of the receptive field is covered by the temporal window, least 99% accuracy could retained in all datasets. We refer to Appendix A.4 for extended results including both reinitialization strategies.
Q4: Incremental Training with Scalable GNN Methods Similarly to Q3, we again compare different window sizes against training on the full graph. This time, we focus on using scalable GNN techniques and aim lear how they perform in conjunction with temporal windows. We further alleviate the fixed-capacity constraint of previous experiments and tune the hidden size as an additional hyperparameter. We refer to Appendix A.3 for details hyperparameter choices.
We compare Simplified GCN and GraphSAINT, while including JKNet to isolate the effect of observe that, again, limiting the
GraphSAINT sampling. Table 2 (bottom) shows the results. window size cover 50% of the GNN's receptive field leads to at least 95% relative accuracy, com- pared to full graph training. As expected, GraphSAINT sampling (with JKNets as a base model) yields slightly lower scores than full-batch JKNets. On DBLP-hard, simplified GCN outperforms the other, more complex models. terms of relative performance, limiting the receptive field does negatively impact GraphSAINT DBLP -BL.-hard and PharmaBio.
8

---

Under review as a conference paper at ICLR 2021
6 DISCUSSION
We have created a new experimental procedure for temporal graphs with new classes appearing over time, for which we contribute three newly compiled datasets with controlled degrees of distribution shift, In this online learning setup, we have evaluated three representative GNN architectures as well as two GNN scaling techniques. With the goal of generalizable results, we have introduced new measure for the distribution of temporal differences tdiff based on which we have selected the temporal window sizes. Our results show that past data can be permanently deleted very early without diminishing the performance of an online vertex classification model. This has direct con- sequences for online learning of GNNs on temporal graphs and, thus, impacts how GNNs can be employed for numerous real-world applications.
Our main result that incremental training with limited window sizes is as good as incremental training over the full timeline of the graph (see Q3 and Q4). With window sizes of 3 or 4 (50% receptive field coverage), GNNs achieve at least 95% accuracy compared to using all available data for incremental training. With window sizes of 6 or 8 (75% receptive field coverage), at least 99% accuracy can be retained. This result holds not only for standard GNN architectures but also when scaling techniques such subgraph sampling are applied on-top of the temporal window. Finally, in almost all experiments, at least 90% of relative accuracy is reached with a window of size 1.
Furthermore, we have verified that incremental training helps to account for distribution shift com- pared to once-trained, static models (see Q1). We have further investigated on reusing parameters from previous iterations (Q2). Our results show that both strategies are viable, when learning rates are tuned accordingly. During hyperparameter optimization for Q4, in which we alleviated the exed-capacity constraint, we further noticed that warm restarts are more suitable for higher capacity models with low learning rates, while using cold restarts admits using lower capacity models and higher learning rates (the details of hyperparameter optimization can be found in Appendix AA.3).
Even though it was not our main objective to compare the absolute performances of the models, it is noteworthy that simplified GCNs perform surprisingly well on DBLP-hard. Despite the simplicity of the approach, the model yields higher scores than GraphSAINT, JK JKNets and fixed-capacity GAT's, and are only outperformed by GraphSAGE-mean
A limitation of the present work is that we assume that the true labels of each time step become available as training data for the next time step. In practice, however, only a small fraction of vertices might come with labels for training, while the larger part could be annotated by the model itself. Adapting our experimental procedure to use only a small fraction of true labels in each time step would be interesting direction of future work.
One could further argue that deleting data that is not linked to the most recent data points would be a viable alternative to deletion based on a fixed time difference. However, this approach would be only feasible in retrospect because, in woal-world scenarios, it is impossible to know whether a future data will include link to a past data point. Still, future work could involve employing other methods to determine what data to delete, such as the personalized PageRank score (Bojchevski et al. 2020))
7 CONCLUSION
Temporal graphs occur in many real-world scenarios such as citation graphs, transaction graphs, and social graphs. Practitioners face a trade-off between memory requirements, which are tied to the temporal window size, and expected accuracy of their models. Until now, it was not clear, how GNNs can be efficiently trained in those online scenarios, especially when distribution shift becomes an issue. We demonstrate that high level of accuracy can be retained, when training only on a fraction of the temporal graph, determined by a temporal window. The results of this paper serve guidelines for training GNNs temporal graphs, particularly regarding the intentional forgetting of data while retaining a certain percentage of predictive power. For researchers, we supply our newly compiled datasets along with implementation of the experimental procedure.
We will make the code and data available to reviewers during the peer-reviewing process as sug-
gested in the ICLR 2021 author's guide.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Aleksandar Bojchevski, Johannes Klicpera, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózemberczki, Michal Lukasik, and Stephan Günnemann. Scaling graph neural networks with approximate pagerank. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery Data Mining, KDD '20, pp. 2464-2473, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984, wwt.o//.........1.../11. URL ULL/NEEp.///oo..ccc/..111///////
doi: 10.1145/3394486.34033366
3403296
Xavier Bresson and Thomas Laurent. Residual gated graph convnets. CoRR, abs/ 1ols//771..753,, 2017.
Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction. In ICML, 2018.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceed- ings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Min- ing, KDD '19, pp. 257-266, New York, NY, USA, 2019. Association for Computing Machin- ery. ISBN 9781 1781145362016 doi: 0144/33292500.3300222 1145/ 3330925
www.s:ttppp/.mmmm //to///oii/// /10
da Xu, chuanwei ruan, evren korpeoglu, sushant kumar, and kannan achan. Inductive representation learning on temporal graphs. In International Conference on Learning Representations, 2020. URL httptttss///ppeeeee//e
Vijay Prakash Dwivedi, Chaitanya K. Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. CoRR, abs/2003.00982, 2020.
D. Ersan, C. Nishioka, and A. Scherp. Comparison of machine learning methods for financial time series forecasting the examples of over 10 years of daily and hourly data of dax 30 and s&p 500. J Compur Soc Sc, 3:103-133 133, 2020.
Matthias Fey and Jan Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Benjamin Fish and Rajmonda S Caceres. task-driven approach time scale detection in dy- namic networks. In Proceedings of the 13th International Workshop Mining and Learning with Graphs (MLG') 17). htlp://www. migworkshop. 0rg20177juppr//M.2207 -ppprr_7pppp1111 pdf, 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS 2010, volume
JMLR Proceedings. JMLR.org, 2010.
Palash Goyal, Nitin Kamra, Xinran He, and Yan Liu. Dyngem: Deep embedding method for dy- namic graphs. CoRR, abs/ 805. 1273, 2018.
Palash Goyal, Sujit Rokka Chhetri, and Arquimedes Canedo. dyngraph2vec: Capturing network dynamics using dynamic graph representation learning. Knowl.-Based Syst., 187, 2020.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS, 2017.
Wen-bing Huang, Tong Zhang. Yu Rong, and Junzhou Huang. Adaptive sampling towards fast graph representation learning. In NewrIPS, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/ (bb//1412.690,, 2014.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Srijan Kumar, Xikun Zhang, and Jure Leskovec. Learning dynamic embeddings from temporal interactions. arXiv preprint arXiv: 1ete.8/22222 2222222 2018.
10

---

Under review as a conference paper at ICLR 2021
John Boaz Lee. Giang Nguyen, Ryan A Rossi, Nesreen K Ahmed, Eunyee Koh, and Sungchul Kim. Dynamic node embeddings from edge streams. IEEE Transactions on Emerging Topics in Computational Intelligence. 2020.
Franco Manessi, Alessandro Rozza, and Mario Manzo. Dynamic graph convolutional networks. Pattern Recognition, 97, 2020.
T. Melnychuk, A. Wirsich, and C. Schultz. Effects of sniversity-indusstie collaborations in basic re- search on different stages of pharmaceutical new product development. In Innovation and Product Development Management Conference, 2019.
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolå, Jan Svoboda, and Michael M. Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In CVPR. IEEE Computer Society, 2017.
Mark EJ Newman. Power laws, pareto distributions and zipf's law. Contemporary physics, 46(5), 2005.
Giang Hoang Nguyen, John Boaz Lee, Ryan A. Rossi, Nesreen K. Ahmed, Eunyee Koh, and Sungchul Kim. Continuous-time dynamic network embeddings. In WWW (Companion Volume), pp. 969-976. ACM, 2018.
Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma. Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao B. Schardl, and Charles E. Leiserson. Evolvegen: Evolving graph convolutional networks for dynamic graphs. AAAI, pp. 5363-5370. AAAI Press, 2020.
Hogun Park, John Moore, and Jennifer Neville, Deep dynamic relational classifiers: Exploiting dy- namic neighborhoods in complex networks. In Proceedings of MAISoN Workshop in International Conference on Web Search and Data Mining (WSDM), 2017.
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint NKKV:20006106637 2020a.
Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael Bronstein, and Fed- erico Monti. Sign: Scalable inception graph neural networks. 2020b.
Ryan A. Rossi and Jennifer Neville. Time-evolving relational classification and ensemble methods. In PAKDD (1), volume 7301 of Lecture Notes in Computer Science, pp. 1-13. Springer, 2012.
Gerard Salton and Christopher Buckley. Term-weighting approaches in automatic text retrieval. Information processing & management, 24(5), 1988.
Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang, and Hao Yang. Dysat: Deep neural repre- sentation learning on dynamic graphs via self-attention networks. In WSDM, 2020.
Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In ICONIP. Springer, 2018
Jie Tang, Jing Zhang. Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Areeminer: Extraction and mining of academic social networks. In KDD '08. ACM, 2008.
Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In ICML, 2017.
Rakshit Trivedi, Mehrdad Farajtabar, Prasenject Biswal, and Hongyuan Zha. Dyrep: Learning rep- resentations over dynamic graphs. In ICLR, 2019.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICIR, 2018.
Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma. et al. Deep graph library: Towards efficient and scalable deep learning graphs. arXiv preprint 199iv:999 2019.

---

Under review as a conference paper at ICLR 2021
Felix Wu, Tianyi Zhang, Amauri H. Souza Jr., Christopher Fifty, Tao Yu, and Kilian Q. Wein- berger. Simplifying graph convolutional networks. CoRR, abs/1902.07153, 2019. URL http:
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie
Jegelka. Representation learning on graphs with jumping knowledge networks. In ICML, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ICLR, 2019.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Williamm L. Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In NeurlPS, 2018.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. Graphsaint: Graph sampling based inductive learning method. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020. URL htttp::/ net fort/formm?al BJe
12

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1
ALGORITHM FOR OUR EXPERIMENTAL PROCEDURE
Algorithm Daaihmm@one outlines our incremental training and evaluation procedure.
Data:
Temporal graph G, features X, labels y, time steps t, temporal window size c, epoch
budget flepochs
Result: Predicted class labels for vertices in each time step of the graph
anown_classed 0;
s initialize pparliieeeaaameeer
for ts txart to fend do
ğ < subgraph of induced on vertices where 1 1:8mi (u) st;;
Yunin + Yiu. where 18mmn < t";
do do-ccol-resstrr then
// Cold restart: re re-initialize all parameters initialize-pareen paramleezepp
else
// Warm restart: initialize new parameters, copy others
tmp + cl)n
10
0 - initialize parametersO;
11
kloses f- tmp clossnclasses>
12
end
0 train(O, S. X, Frrain) for Repochs epochs;
13
Ppred < predict(0. G. X)) for vertices u, where t8 (u) t*: known classes +- known classes L (Bhain):
14
15
16
end
Algorithm 1: Incremental training procedure of our experimental apparatus
A.2 DATASET DETAILS
In the following, outline the dataset compilation procedure and supply further descriptive statis- ties of the resulting datasets.
PharmaBio
To compile the PharmaBio dataset, we use the metadata of 543,853 papers by Pharma and Biotech companies from Web of Science (Melnychuk al. 2019). After removing duplicates, our data cleaning procedure ensures that there is a certain amount of labels for each class per year and that each paper connected
least other paper by same-author edge. More specifically, we: (1) Keep only papers that are in journal category with at least 20 papers per year, (2) Keep only papers where least one of the authors has least two papers per year; (3) Create vocabulary of words (regular expression: \w A w+) that appear in at least 20 papers globally and keep only papers with at least one of these words. We iterate steps 1-3 until no further paper has been removed in one pass. We end up with 68,068 papers from 23,689 authors working for 68 companies. These papers are distributed across 2,818 journals which are, in turn, categorized into seven journal categories. During preprocessing, each paper becomes vertex in the graph. The class of the paper is the category of the journal in which it was published. We insert an edge between two vertices, if they share at least one common author (based on string comparison).
DBLP-cast To compile these datasets, use the DBLP Citation Network dataset (version I0 Tang et al. 2008) as basis. comprises 3M citing documents and 25M citations to 2M distinct cited documents, ranging between years. We venues (conferences or journals) class labels and use citations as edges. First, we select the subset from 1990 until 2015. Then, we follow similar procedure above: (1) Keep only papers from venues that have least Tvenue papers in each year they occur (may be only every second year). (2) Keep only papers that stand in at least
https://aminer..oggg/
13

---

Under review as a conference paper at ICLR 2021
citation relation to another paper. (3) Remove papers from venues that occur only in single year. (4) Keep only papers with at least one word from a vocabulary of words that are in at least Twards papers. We iterate steps 1-4 until further paper has been removed in one pass.
Dog oe citriuutcc
2000
1000
(a) PharmaBio paper count per year (b) PharmaBio degree distribution (c) PharmaBio label distribution, (log-log scales), power law coeffi entropy H 2.34 bit
(log scale)
cient 1.4592
A005
(d) DBLP-easy paper count per (e) DBLP-easy degree distribution (f) DBLP-easy label distribution, (log-log scales), power law coeffi- entropy 3.25 bit cient 1.9300
year (log scale)
Dog CIRFBUICe
282
(g) DBLP-hard paper count per (h) DBLP- degree distribution (i) hard label distribution, year (log scale)
(log-log scales), power law coeffi- entropy H 5.52 bit
cient a = 1.7345
Figure 5: Distribution of vertices per year, degree distributions, label distributions, for our temporal graph datasets.
DBLP-hard The difference between DBLP-easy and DBLP-hard is that Tvemue == 100 papers in the easy variant and Tvenue == 45 papers in the hard variant. The minimum word occurrence threshold Twords is set to 20 for DBLP-casy and 40 for DBLP-hard. Finally, construet the graph with papers as vertices, citations as edges, and venues as classes.
For all three datasets, vertex features based the corresponding papers' title. We estimate the power law coefficient a via
122-ormalized if-idf (Salton & Buckley 1988) representations as
maximum likelihood (Newman 2005) PharmaBio).
n Euev In
deg
where degmin is 1 (2 for
degmio
In Figure we visualize the degree distribution. label distribution. the distribution years, as well as the distributions of temporal differences (as described in Section ( All compiled datasets seem to follow a power law distribution, which is typical for citation and co-authorship graphs.
14

---

Under review as a conference paper at ICLR 2021
Figure 6: Hyperparameter choices for experiment Q1. Static methods were trained for 400 epochs on 25% of the data before the first evaluation time step. Incremental methods were trained with warm restarts for 200 epochs per time step using a window size of (DBLP) and (PharmaBio)
Method
Training
Layers
Hidden Size
Learning Rate
MLP MLP
static
2
64
10
[10 10
incremental
2
64
10-3
110-4,10- 10-
GS-Mean
static
2
64
10
[10-4,10 10
GS-Mean
incremental
2
64
10-3
[10
11004,10-1
GAT
static
2
64
10 3
[10-4,1001]
[10
10
GAT
incremental
2
64
10-2
[10-4,10-] 10
10
For each dataset, we chose the boundaries for our evaluation time steps [/san, lend], such that 25% of the total number of vertices lie before (aart. and lend is the final time step. For PharmaBio (1985- 2016), that is fstart = 1999, and for both DBLP variants (1990-2015), that is Istan 2004. Data before start may be used for training, depending on the window size. Regarding changes in the class set (distribution shift), DBLP-easy has 12 venues in total, including one bi-annual conference and four new venues appearing in 2005, 2006, 2007, and 2012. DBLP-hard has 73 venues, including one discontinued, nine bi-annual, six irregular venues, and 23 new venues.
A.3 IMPLEMENTATION DETAILS AND HYPERPARAMETERS
We tune the hyperparameters separately for each window size and each restart configuration. We tune the hyperparameters on DBLP-easy and use the same set of hyperparameters for DBLP-hard and PharmaBio.
For experiments QI-Q3, we design the models to have a comparable capacity: one hidden layer with 64 hidden units. We ReLU activation on the hidden layer of MLP and GS-Mean. GS-Mean has one hidden layer, e. two graph convolutional layers, with 32 units for self-connections and 32 units for aggregated neighbor representations. GAT has one hidden layer composed of attention heads and 8 hidden units per head, along with one attention head for the output layer. We initialize the model parameters according to Glorot and Bengio (Glorot & Bengio 2010). For both GS GS-Meen and GAT, the output of the second layer corresponds to the number of classes. We use dropout probability 0.5 on the hidden units for all models in experiment Q3. We use Adam (Kingma & Baj 2014) to optimize for cross-entropy. We tune the learning rates on DBLP-easy with search space of 1110-1,5-000 10-2, 10-2,05-00 5. 10-3 10-3,,5 50-4, 10-4) and re-use these learning rates for the other datasets. The learning rates are tuned separately for each model, each parameter reinitialization strat- egy, and each window size. We do not use weight decay because it did not increase the performance (search space (0, 10-3 5- 10-1 10-1,5 5. 10-3 10-5)). 9)) The optimal learning rates can be found in Figure 66 for Q1, Figure 7 for Q2, and Figure g for Q3. For implementation of GraphSAGE-mean and GAT', we use DeepGraphLibrary (Wang al. 2019). All methods are trained transductively: for each new snapshot, the new vertices are inserted into the graph without their labels, then, the models are allowed (up-)train before making predictions.
For the experiment Q4, we use two hidden layers with 64 hidden units each to make use of jumping knowledge (Xu al., (2018). as suggested as base architecture in GraphSAINT (Zeng et al. 2020). The learning rate is tuned in the space of {0.0001, 0.001, 0.01, 0.1}. Dropout probability is set to 0.2. We do not use weight decay. We also tune the batch size of GraphSAINT in the range of (256, 512, 2048, 4096), subgraph size is an important hyperparameter. For simplified GCN, we tune the learning rate in the range of {0.0005, 0.001, 0.005, 0.01, 0.053 and we set the neighbor- hood aggregation parameter K to 2, corresponding to two-layer aggregation. For implementation of GraphSAINT and JKNet, we use rameter values well the respective search spaces for experiment Q4 can be found in Figure JKNets and simplified GCNs are trained transductively, while GraphSAINT is trained inductively suggested by the original work (Zeng et al. 2020
(Fey & Lenssen 2019). The optimal hyperpa-
15

---

Under review as a conference paper at ICLR 2021
Figure 7: Hyperparameter choices for experiment Q2. All methods supplied with 200 epochs per time step. Learning rate optimization is performed on DBLP-easy.
Method
Restarts
Layers
Hidden Size
Learning Rate
MLP
cold
2
64
10-3 10 10-3
10-3
MLP
warm
2
64
[10 10
GS-Mean
cold
2
64
5 10-3
10-4.10-1
GS-Mean
warm
2
64
10-3
[10-4,10-]
GAT
cold
2
64
5 10
[10 100".100"
GAT
warm
2
64
10-2
[10 [100,1101
Figure 8: Hyperparameter choices for experiment Q3. All methods are supplied with 200 epochs per time step. We separately optimize hyperparameters for each window size and for each restart configuration on DBLP-easy.
Method
Window Size
Restarts
Layers
Hidden Size
Learning Rate
MLP
cold warm
10-3 3 [10 11004,10-
10
MLP
64
5 10-1 11011,00
MILP
3 4
cold
64
10-3 110-11000
MLP
3/4
warm
10-3 10-31004,10"
MLP
6/8
cold
64
10-3100-110-
MLP
6/8
warm
64
10 10-01111100
MLP
full
cold
64
50 10 10.0004100
MLP
full
warm
64
10-3 3 10 10010-4 4.10
GS-Mean
cold
64
10-3 3 10
GS-Mean
warm
64
5.10 10 10
GS-Mean
3/4
cold
64
5. 100
GS- GS-Mean
3/4
warm
64
10
4,10
GS-Mean
6/8
cold
64
1.130
GS-Mean
6/8
warm
64
10 10-410 1
GS-Mean
full
cold
64
10-2 10-4 10
GS-Mean
full
warm
64
10 10 10
GAT
cold
64
5 10
1001 10
GAT
warm
64
1
10
10 4.10
GAT
3/ /
cold
64
5.
4,010
GAT
3/4
warm
64
10-2 10-4
GAT
6/8
cold
64
5. 10-4
GAT
6/8
warm
64
20-2 10 10-400
GAT
full
cold
64
5.
GAT
full
warm
64
5-10-2
10-4
16

---

Under review as a conference paper at ICLR 2021
Table 3: Hyperparameter choices for experiment Q4. All methods are supplied with 200 epochs per time step. We separately optimize hyperparameters for each window size and for each restart configuration on DBLP-easy. GraphSAINT and Jumping Knowledge use 2 hidden layers with 64 hidden units each.
Method
Window Size
Restarts
Batch Size
Learning Rate
Simplified GCN
cold warm cold warm
5 10 5- 10-2
Simplified GCN
5 50-3 5-10-55-100-
Simplified GCN
3
10
Simplified GCN
3/4
5 1.-1---10----1000
Simplified GCN
6/8
cold
10255-10"..-10-
Simplified GCN
6/8
warm
6.10-35-11--5-11
Simplified GCN
full
cold
Simplified GCN
full
warm
50 10
GraphSAINT
cold
1024
[1024, 4096]
5 10 5.0
GraphSAINT
warm
1024
[1024, 4096
5.25--10-,
GraphSAINT
3/4
cold
1024
[1024,4996
1.-135511--555
GraphSAINT
3/4
warm
4096
[1024, 4096
10-25-10-<55-1
GraphSAINT
6/8
cold
4096 [1024, 4096]
10 50-25-10-4.-110
GraphSAINT
6/8
warm
4096
(1024,
501-5-11-
GraphSAINT
full
cold
4096
[1024, 4096]
GraphSAINT
full
warm
4096
[1024, 4096]
10 5-1044,5-
Jumping Knowledge
cold
10 5-1
Jumping Knowledge
warm
5-100 4.5.100-
Jumping Knowledge
3/4
cold
2.10-
Jumping Knowledge
3/4
warm
10-25-100-,--10-
Jumping Knowledge
6/8
cold
5 10-2 10-25--10--.-110
Jumping Knowledge
6/8
warm
10 10-25-10-<.5-
Jumping Knowledge
full
cold
10-25.10-15-10
Jumping Knowledge
full warm
5 10-3 5-10-4.5-10-
17

---

Under review as a conference paper at ICLR 2021
0439
236
350
0397
18600
28025
044
045
3.0
064
042
044
035
330
TEOO%
J007
year
Figure 9: Detailed results per time step for experiment Q3. Comparison of different temporal window sizes in an online scenario with 200 incremental training epochs per time step with either cold (Top) or warm restarts (Bottom) and varying temporal window sizes. 95% CI not shown for reasons of better visualization. Average accuracy of 10 different runs (y-axis) per timestep (x-axis).
A.4 EXTENDED RESULTS
Table 4 shows the full results table with both warm and cold restarts for experiment Q3. Table shows the full results table with both warm and cold restarts for experiment Q4. Figure 9 visualizes the results for each time step of experiment Q3.
18

---

Under review as a conference paper at ICLR 2021
Table 4: Extended results for experiment Q3. Average accuracy across seeds and time steps with varying temporal window sizes, 95% confidence intervals are computed based on sample variance (N=10,080), Window size is listed in the column c, warm restarts (w) and cold restarts (c) listed the column r. We compare each average accuracy to the average accuracy obtained by training on the full graph (see column relative performance).
GAT
GS-Mean
MLP
accuracy
relative
accuracy
relative
accuracy
relative
dataset
-0
87%
.606+..0
85%
.5611.00
88%
1
w
.647=.00
92%
.643+.00
90%
.6131.01
100%
.688=.00 00
98%
.688#.00
96%
.6091.00
96%
3
dblp-easy
w
.691=
98%
.688+.00
96%
.623#.00
102%
.703
100%
7711+.00
99%
.627+.00
99%
w
.704 .00
100%
705+
99%
.621+.00
101 %
.702-000
100%
7177..00
100%
.6341.00
100%
full
w
.702-00
100%
.714..00
100%
.613+.00
100%
86%
.600E.00
77%
.1611.00
83%
1
.390= 00
85%
.355.00
79%
.31
97%
c
.440 00
96%
.474.00
95%
.377.00
88%
.434
.00
95%
.34+.00
92%
.83
105%
dblp-hard
c
.450 000
99%
4634.00
98%
.922.00
103%
w
.448-00
98%
.664..0
99%
.3994.00
104%
.457=.00
100%
.70.E.00
100%
3814.00
100%
full
w
.456 .456=.00 000
100%
.72++00
100%
364+.00
100%
.616=.01
94%
.655+.01
95%
.627#.01
96%
w
.654-.01
100%
.691..01
101%
.670..01
104%
.645
99%
.680+.01
99%
.6631.01
101 %
4
pharmabio
w
.652
100%
.696L.01
102%
.657..01
102%
.651=.01
100%
.6922.01
100%
.6431.01
98%
8
w
.656=.01
100%
.688..01
101%
.6534.01
101 %
.654
01
100%
.690
100%
.654+.01
100%
full
w
.654--00 01
100%
.682
100%
.644+.01
100%
19

---

Under review as a conference paper at ICLR 2021
Table 5: Extended results for experiment Q4. Average accuracy across seeds and time steps with varying temporal window sizes, 95% confidence intervals are computed based on sample variance. Window size is listed in the column warm restarts (w) and cold restarts (c) are listed in the column r. We compare each average accuracy to the average accuracy obtained by training on the full graph (see column relative performance).
Simplified GCN accuracy
GraphSAINT
Jumping Knowledge accuracy
relative
accuracy
relative
relative
dataset
.585+.00
84%
.6211.00
92%
.500.00
82%
w
..28
92%
.643+.00
94%
.626=.00
88%
c
.670+.00
96%
.6631.00
99%
.666=.00
95%
.672+.00
98%
.671#.00
97%
681=.00
96%
dblp-easy
.692+.00
99%
6682+.00
101%
.697 00
99%
+00
100%
664+.00
100%
.700 00
99%
full
c
.6999.00
100%
672.1.00
100%
.704-.00 00
100%
.686+1.0
100%
.685..00
100%
.708 00
100%
.3191.00
91%
.34.1.00
97%
290 00
83%
.383+.00
90%
.32++00
93%
.322 00
80%
c
.357+.00
103%
3194.00
102%
405= 00
116%
.417+.00
98%
4.5
101%
.426 .00
100%
dblp-hard
.363-.00
105%
.314..0
109%
.411 .00
118%
.424+.00
100%
.134.00
103%
.431 .00
102%
.3474.00
100%
3134.00
100%
.349-.0 00
100%
full
.424+.00
100%
402+-00
100%
.424 .00
100%
.6144.00
114%
.634+.00
103%
.623=
100%
.642+.00
100%
.673..00
104%
.650 00
103%
.630+1.0
117%
.6381.00
104%
.586-.00 .00
94%
.647-.00
100%
.672..00
104%
.634-600 .00
101%
pharmabio
.6324.00
117%
.33$..00
103%
.548.00
88%
8
w
.6574.00
102%
.6661.00
103%
.641=.00
102%
.538 .00
100%
.615-.00
100%
.622 .00
100%
full
W
.644+.00
100%
.649+.00
100%
.630=.0
100%
20


Reviews:
[{"review_id": "k9HtI_JnmPs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper proposes a paradigm which speeds up the training time of GNNs while not compromising too much performance. The method adopts a layerwise training procedure. In particular, the authors inject a loss function at each layer while storing and fixing the feed-forward values of its previous layer. The training is then carried out along all layers parallelly, which allows the updating of paradigms to be decoupled and is not applicable in previous works. A further improvement (lazy-update) by not updating the feed-forward values of each layer is used to reduce the training time.\n\n### Reasons for Score\nThe paper discusses the important topic of layer updating and proposes a decoupling strategy that enables a layerwise parallel updating scheme. However, the lazy-update technique that has been argued as another important point is not fully justified in its memory usage, higher parameter setting, and performance consistency between different experiments.\n\n### Pros\nThe paper tackles the problem of layer decoupling in GNN training, which is an important problem when training large-scale networks. The decoupling training approach is useful if the memory can hold multiple feed-forward layers’ outputs, which is not the case with previous methods [1].\n\n### Cons\nThe paper proposes two strategies: decoupling technique and lazy-update. While I think the first strategy is a good supplement to the previous work, I do feel there are some points that are not properly justified in the arguments and experiments of lazy-update.\n\n1. The actual memory used in LU-DGL-GCN lazy-update. It is stated that in Algorithm 2, $\\hat{H}^{(l)}$ is used for each layer, which means that there need to be at least $LNK$ extra space needed for memorizing these parameters as $\\hat{H}^{(l)}$ can not be computed on-the-fly due to its lazy-update nature (e.g. $T_{lazy}$ is too large, when we are at Epoch $t$ updating $W^{(l)}$, $\\hat{H}^{(l-1)}$ may refer to $FH^{(l-1)}$ where $H^{(l-1)}$ holds the value multiple epochs ago). With this being said, it’s hard to figure out why LU-DGL-GCN still has similar memory usage as with L2GCN and is drastically different from the normal GCN.\n\n2. The balance between stableness of LU-DGL-GCN and the large value of $T_{lazy}$ is hard to find. In Fig. 3, the authors show that the framework is sensitive to $T_{lazy}$ and can be highly unstable when it is at a small value (e.g. 1 or 5). However, it is natural to find that $T_{lazy}$ should not be too large as it could slow the training procedure. In the extreme case, if $T_{lazy}$ is infinite, we can see that the previous layer’s output $\\hat{H}^{(l)}$ is never updated and the parameters of the whole GCN, therefore, can never be properly optimized. It would be meaningful if the authors could discuss how to set $T_{lazy}$ for the balance of stableness and time cost which is an important point the paper has argued for the approach.\n\n3. Exact training time comparison should be stated. It would be necessary to state clearly how the time is computed in Table 2, e.g. fixed epochs or same validation loss, the latter one of which is a more proper choice as the authors are wishing to show the framework is efficient with similar accuracy. With this stated, it is also strange to find that the accuracy of LGCN is higher than LU-DGL-GCN in Table 2 while it is not the case in Figure 3 where Sequential_test is larger than any lazy-update.\n\n### Clarifications\nPlease address and clarify the cons above.\n\n\n[1] L2-GCN: Layer-Wise and Learned Efficient Training of Graph Convolutional Networks\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZYePZHk32L0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studies the problem of online or incremental learning in temporal graphs (dynamic networks), and more precisely, whether past data can be discarded/ignored without losing predictive accuracy under the assumption that there is the presence of a distribution shift. This question has been essentially investigated over the years in various contexts, e.g., relational learning and classification in dynamic or time-evolving networks. It is also completely obvious that forgetting older data, especially under the assumption of a distribution shift, makes sense and is the correct thing to do. This is exactly what has been done in time-series forecasting for decades. The problem formulation is unclear and can be more precisely defined and motivated appropriately. This needs to be fixed. Are the class labels of a node changing over time, so if a node has label A at time t, then at time t+1 it could have label B, etc. This doesn’t seem true, as it seems the class labels of the nodes are “static”, which is unrealistic in many cases. How are the graph snapshots created? How was the timespan selected? What does every time step represent (1 hour, 5 minutes, etc.)?  Also, are the node features changing over time? This doesn’t seem true, but if this is the case, then it is unclear why this would be the case in practice (it would be great to provide some motivation for this, or an example application or problem where this may be true). There are many assumptions that make this problem unrealistic. Furthermore, there have even been works that study the dynamic node classification problem previously, see [1-2] below. \n\nThe contribution and novelty of this work is unclear. Many important related works are missing. There have been countless works that have studied the impact of the temporal window and its size, as well as discarding past data, and using different amounts, as well as the representation of that past data (exponentially weighting links). This work also studies the impact of ignoring past data on node classification. Furthermore, many of the standard papers on this topic are seemingly missing such as CTDNE [10] and JODIE [6]. There are many other important works on incremental/online learning in dynamic and streaming graphs that are missing in the paper, see [4]-[13], which need to be referenced and appropriately discussed, mentioning the differences, and so on. The real contribution seems to be a new dataset with a controlled distribution shift. But putting this work into perspective with the related work, and explicitly stating the differences would help clarify the contribution and better position this work with respect to the existing literature.\n\n\nPros\n  + Paper is well-written for the most part and easy to understand\n  + New dataset with controlled distribution shift\n\nCons\n  - Limited technical novelty and contribution\n  - Important related work is missing and should be discussed appropriately to better position the work \n  - Problem formulation is unclear and can be more precisely defined, and motivated. \n  - Previous work has studied essentially the same research question and findings are obvious\n\nThe results and findings are in terms of time steps, however, the notion of a time step is not the same for every graph, nor is it ever discussed how the time steps are actually derived. Does every time step represent 30 seconds, 5 minutes, 1 hour, 1 day, etc. Furthermore, the results only make sense for the specific time step chosen for each graph. For instance, it is mentioned that “GNNs achieve 95% accuracy with a small window size of 3 or 4 time steps”. However, if the time step is extremely large then the result/findings change. And so all the findings in this paper and the discussion depend precisely on the data and the authors choice of how to create the time steps, and what granularity to use, which isn't discussed. This issue was discussed extensively in previous work. Minor comment: the labels in nearly all the figures are too small to read.\n\n\n\n\n1. Time-evolving relational classification and ensemble methods\n2. Deep dynamic relational classifiers: Exploiting dynamic neighborhoods in complex networks\n3. A task-driven approach to time scale detection in dynamic networks\n4. Dynamic Node Embeddings From Edge Streams\n5. Afraid: fraud detection via active inference in time-evolving social networks\n6. Learning Dynamic Embeddings from Temporal Interactions\n7. Node Embedding over Temporal Graphs\n8. Representation Learning in Continuous Entity-Set Associations\n9. Efficient representation learning using random walks for dynamic graphs\n10. Continuous-Time Dynamic Network Embeddings\n11. Dyn2Vec: Exploiting dynamic behavior using difference networks-based node embeddings for classification\n12. Real-Time Streaming Graph Embedding Through Local Actions\n13. Temporal Graph Offset Reconstruction: Towards Temporally Robust Graph Representation Learning\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MNWzxywcNVF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work empirically evaluates the sliding-window strategy for training GNNs with temporal graphs. One may cast the temporal nature of the graph data in an online setting, under which the change of the graph structure as well as the variation of the classes cause distribution shift. The authors conduct a series of experiments to show that the sliding-window strategy is as effective as using the entire historical data for training.\n\nPluses:\n\n+ For different temporal graphs, the duration of a time step and the number of time steps (window size) are often ad-hocly defined and are not comparable. The authors introduce a measure of temporal difference that facilitates a more principled definition of the time step and the window size so that they are comparable across datasets.\n\n+ The authors pose four important questions and conclude clear answers based on experimentation. The findings are: (1) incremental training is necessary to account for distribution shift, compared to a once-trained, static model; (2) incremental training with warm start does not always yield better performance than cold start; (3) the window size needs be large enough for incremental training to catch up with the performance of full-data training (e.g., covering at least 50% receptive field); and (4) these findings extend to several GNN models.\n\n+ The authors compile three temporal graphs, which enrich the availability of benchmark datasets.\n\nMinuses:\n\n- The empirical findings are very much expected, which means that they are not exciting. From the methodological point of view, using sliding windows to train temporal GNNs is a no brainer choice if certain RNN modeling is involved. Since most of the presented results are naturally expected and there lacks theory/method contribution, the reader is unsure about the value of the paper.\n\n- A common pattern of the contributed datasets is that nodes and edges are inserted but never deleted. While the empirical findings are quite natural in this simple scenario, there will be a lot more uncertainty when the scenario becomes increasingly complex. For example, in social networks, accounts represented by nodes may be deleted and relationships represented by edges may dynamically change.\n\n  For another example, in communication networks where an edge denotes communication between two entities, the edges are instant and time stamped. The challenge in this case is less about distribution shift, but more about how to handle edges and what are the consequences. The online learning of this kind of data necessarily goes beyond a simple GNN such as the ones experimented in this paper, but the findings will be more valuable.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BhmdttMeRJJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Temporal graphs can naturally model many real-world networks, and many graph neural network (GNN)-based methods have been proposed recently. Existing temporal GNNs can handle vertices and edges appearing / disappearing over time, but not vertex classes. This paper precisely considers this problem, and\n1) compiles three vertex classification datasets for future research,\n2) proposes an experimental procedure for evaluating performance under this setting,\n3) explores 5 existing GNNs, and concludes that incremental training for limited periods is as good as that over full timelines.\n\n\n\n## Pros\n1) (Motivation) It is reasonable to assume that new classes can appear over time in real-world networks. It is also worth investigating whether the full temporal graph (seen so far) is actually required for GNN neighbourhood aggregation in the current timestep.\n2) (Relevance) Learning representations on temporal graphs is a challenging, fast-growing topic, and relevant to the ICLR community.\n\n\n\n## Cons\n1) (Soundness) Tables 2, 3, and 4 compare accuracies of different static GNNs with varying window sizes (proposed idea) and with full graph (existing idea) which is informative. However, to increase the impact of the paper, the proposed idea (with static GNNs) should also be compared against state-of-the-art temporal GNNs on full graphs (in all these tables). As already cited by the authors, recent temporal GNNs include (but are not limited to)\n(a) EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs, In AAAI'20,\n(b) Inductive Representation Learning on Temporal Graphs, In ICLR'20.\n2) (Significance) The experiments in the paper are restricted to multi-class vertex classification with new classes appearing over time (in just one dataset domain based on scientific publications). The authors should clarify what challenges one would face for multi-label classification commonly seen with some datasets (e.g. social networks). It would be more convincing if experiments were also conducted on link prediction (e.g. social network link prediction with new classes i.e. communities appearing over time).  \n3) (Originality) Although the assumptions (classes appearing/disappearing over time), evaluation procedure, and datasets have not been considered / proposed before, the novelty of the paper is quite limited. As also acknowledged by the authors, the paper explores well-known existing static GNNs for temporal graphs. From this point of view, the paper is of limited originality since it explores well-known algorithms in an unexplored setting.\n\n\n\nTo summarise, the paper has strong arguments along the axis of motivation but the major weaknesses outweigh the strengths.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an empirical paper that proposed a few different settings for applying GNNs on temporal data, including what context window to use, code-start vs warm-start, incremental training vs static.  This paper also proposed and released a few more temporal graph datasets, which could be useful.

The consensus assessment of the reviewers is that the contributions of this paper are incremental, and the results are expected and not exciting enough.

I want to in particular point out that the results highlighted in the paper, that a GNN with window size 1 is sufficient to recover 90% of the performance of the model on full graph, is probably not the correct message to communicate.  This either indicates that the data and task used in the benchmarks do not require sophisticated long-horizon temporal information (which makes the comparison between any methods uninteresting), or it indicates that the metric is not sensitive enough to sufficiently distinguish models trained with different settings.

I would recommend rejection and encourage the authors to improve this paper.
Decision: Reject