Towards Better Dynamic Graph Learning: New Architecture and Unified Library
Authors: Le Yu
Leilei Sun
Bowen Du
Weifeng Lv
Venue: NeurIPS 2023 poster (2023)
Keywords: dynamic graph learning
Transformer-based architecture
dynamic graph library
TL;DR: This paper proposes a new Transformer-based architecture and a unified library for dynamic graph learning.

Abstract: We propose DyGFormer, a new Transformer-based architecture for dynamic graph learning. DyGFormer is conceptually simple and only needs to learn from nodes' historical first-hop interactions by: (1) a neighbor co-occurrence encoding scheme that explores the correlations of the source node and destination node based on their historical sequences; (2) a patching technique that divides each sequence into multiple patches and feeds them to Transformer, allowing the model to effectively and efficiently benefit from longer histories. We also introduce DyGLib, a unified library with standard training pipelines, extensible coding interfaces, and comprehensive evaluating protocols to promote reproducible, scalable, and credible dynamic graph learning research. By performing exhaustive experiments on thirteen datasets for dynamic link prediction and dynamic node classification tasks, we find that DyGFormer achieves state-of-the-art performance on most of the datasets, demonstrating its effectiveness in capturing nodes' correlations and long-term temporal dependencies. Moreover, some results of baselines are inconsistent with previous reports, which may be caused by their diverse but less rigorous implementations, showing the importance of DyGLib. All the used resources are publicly available at https://github.com/yule-BUAA/DyGLib.

Towards Better Dynamic Graph Learning: New Architecture and Unified Library
Le Yu, Leilei Sun; Bowen Du, Weifeng Lv
State Key Laboratory of Software Development Environment School of Computer Science and Engineering Beihang University
{yule, lei weileisuu,duuoreeeee.. 1wf @buua. edu cn
Abstract
We propose DyGFormer, Transformer-base architecture for dynamic graph learning. DyGFormer is conceptually simple and only needs to learn from nodes' historical first-hop interactions by: (i) neighbor co-oceurrence encoding scheme that explores the correlations of the source node and destination node based on their historical sequences; (ii) patching technique that divides each sequence into multiple patches and feeds them to Transformer, allowing the model to ef- fectively and efficiently benefit from longer histories. We also introduce DyGLib, a unified library with standard training pipelines, extensible coding interfaces, and comprehensive evaluating protocols promote reproducible, scalable, and credible dynamic graph learning research. By performing exhaustive experiments on thirteen datasets for dynamic link prediction and dynamic node classification tasks, we find that DyGFormer achieves state-of-the-an arr performance the datasets, demonstrating its effectiveness in capturing nodes' correlations and long-term temporal dependencies. Moreover, some results of baselines are inconsis- tent with previous reports, which may be caused by their diverse but less rigorous implementations, showing the importance of DyGLib. All the used resources are publicly available at https: //ggttb.tt//t///////// mon/yule-aie walyluulll//aaae BUAA/DyGLix
most of
Introduction
Dynamic graphs denote entities as nodes and represent their interactions links with timestamps [26], which can model many real-world scenarios such as social networks [28, 50, 2), user-item interaction systems [31, 15, 71. 72, 70], traffic networks [67, 63, 19, 4, 691, and physical systems [24, 47, 43]. In recent years, representation learning on dynamic graphs has become trending research topic [26, 49, 65). There are two main categories of existing methods: discrete-time [39, 17, 48, 11, 66] and continuous-time [64, 45, 60, 51, 12]. In this paper, we focus the latter approaches because they are more flexible and effective than the formers and are being increasingly investigated.
Despite the rapid development of dynamic graph learning methods, they still suffer from two lim- itations. Firstly, most of them independently compute the temporal representations of nodes in an interaction without exploiting nodes' correlations, which are often indicative of future interactions. Moreover, existing methods learn the interaction level and thus only work for nodes with fewer interactions. When nodes have longer histories, they require sampling strategies to truncate the inter- actions for feasible calculations of the computationally expensive modules like graph convolutions [55, 64, 45, 36, 9, 59], temporal random walks [60, 25] and sequential models [57, 12]. Though some approaches use memory networks [61, 53] sequentially process interactions with affordable computational costs [28, 55, 45, 36, 59, 35], they are faced with the vanishing/exploding gradients
"Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

due to the usage of recurrent neural networks [40, 57]. Therefore, we conclude that previous methods lack the ability to capture either nodes' correlations or long-term temporal dependencies.
Secondly, the training pipelines of different methods are inconsistent and often lead to poor re- producibility. Also, existing methods are implemented by diverse frameworks (e.g., Pytorch [41], Tensorflow [1]. DGL [58]. PyG [16], C++), making it time-consuming and difficult for researchers to quickly understand the algorithms and further dive into the core of dynamic graph learning. Although there exist some libraries for dynamic graph learning [18, 46, 74], they mainly focus on dynamic net- work embedding methods [18], discrete- tetrete--imeeeee graph learning methods [46], or engineering techniques for training on large-scale scale dynamic graphs [74] (elaborated in Section 2). Currently, we find that there are still no standard tools for continuous- dynamic graph learning.
In this paper, we aim to address the above drawbacks with two key technical contributions.
We propose a new Transformer-based dynamic graph learning architecture (DyGFormer). DyGFormer is conceptually simple by solely learning from the sequences of nodes' historical first- hop interactions. To be specific, DyGFormer is designed with a neighbor co-occurrence encoding scheme, which encodes the appearing frequencies of each neighbor in the sequences of the source and destination nodes to explicitly explore their correlations. In order to capture long-term temporal dependencies, DyGFormer splits each node's ' sequence into multiple patches and feeds them to Transformer [56]. This patching technique not only makes the model effectively benefit from longer histories via preserving local temporal proximities, but also efficiently reduces the computational complexity to constant level that is irrelevant the input sequence length.
dynamic graph learning library (DyGLib). DyGLib an open-source toolkit with standard training pipelines, extensible coding interfaces, and comprehensive evaluating strategies, aiming to foster standard, scalable, and reproducible dynamic graph learning research. DyGLib has integrated a variety of continuous-time dynamic graph learning methods as well as benchmark datasets from various domains. trains all the methods via the same pipeline to eliminate the influence of different implementations and adopts modularized design for developers to conveniently incorporate new datasets and algorithms based on their specific requirements. Moreover, DyGLib supports both dynamic link prediction and dynamic node classification tasks with exhaustive evaluating strategies to provide comprehensive comparisons of existing methods.
We present a unified continuous-
To evaluate the model performance, we conduct extensive experiments based on DyGLib, including dynamic link prediction under transductive and inductive settings with three negative sampling strategies as well as dynamic node classification. From the results, we observe that: (i) DyGFormer outperforms existing methods on most datasets, demonstrating its superiority in capturing nodes' correlations and long-term temporal dependencies; (ii) some findings of baselines are not in line with previous reports because of their varied pipelines and problematic implementations, which illustrates the necessity of introducing DyGLib. We also provide in-depth amalysis of the neighbor co-occurrence encoding and patching technique for a better understanding of DyGFormer.
2 Related Work
Dynamic Graph Learning. Representation learning on dynamic graphs has been widely studied in recent years [26, 49, 65]. Discrete-time methods manually divide the dynamic graph into a sequence of snapshots and apply static graph learning methods on each snapshot, which ignore the temporal order of nodes in each snapshot [39, 17, 48, 11, 66). contrast, continuous-time methods directly learn on the whole dynamic graph with temporal graph neural networks [55, 64, 45, 36, 9, 59). memory networks [28, 55, 45, 36, 59, 35], temporal random walks [60, 25] or sequential models [57, 12]. Although insightful, most existing dynamic graph learning methods neglect the correlations between two nodes in an interaction. They also fail to handle nodes with longer interactions due to unaffordable computational costs of complex modules or issues in optimizing models (e.g., the vanishing/explodigg gradients). In this paper, we propose a new Transformer -aassorrer--baedd architecture to show the necessity of capturing nodes' correlations and long-tem term temporal dependencies, which is achieved by two designs: a neighbor co-occurrence encoding scheme and patching technique.
Transformer-based Applications in Various Fields. Transformer [56] is an innovative model that employs the self-attention mechanism to handle sequential data, which has been successfully applied in variety of domains, such natural language processing [13, 33, 6], computer vision [7, 14, 34]

---

and time series forecasting [30, 73, 62]. The idea of dividing the original data into patches as inputs of the Transformer has been attempted in some studies. ViT [14] splits an image into multiple patches and feeds the sequence of patches' linear embeddings into a Transformer, which achieves surprisingly good performance on image classification. PatchT'ST [38] divides a time series into subseries- patches and calculates the patches by channel-independens Transformer for long-term multivariate time series forecasting, In this work, we propose a patching technique to learn on dynamic graphs, which can provide our approach with the ability to handle nodes with longer histories.
Graph Learning Library. Currently, there exist many libraries for static graphs [5, 58, 16, 22, 29, 32], but few for dynamic graph learning [18, 46, 74]. DynamicGEM [18] focuses on dynamic graph embedding methods, which just consider the graph topology and cannot leverage node features. PyTorch Geometric Temporal [46] implements discrete-ting algorithms for spatiotemporal signal processing and is mainly applicable for nodes with aligned historical observations. TGL. [74] trains on large-scale dynamic graphs with some engineering tricks. Though TGL. has integrated some continuous-timg
methods, they are somewhat out-of-date, resulting in the lack of state-of- the-art models. Moreover, TGL is implemented by both PyTorch and C++, which needs additional compilation and increases the usage difficulty. In this paper, we present a unified continuous-time dynamic graph learning library with thorough baselines, diverse datasets, extensible implementations, and comprehensive evaluations to facilitate dynamic graph learning research.
3 Preliminaries
Definition 1. Dynamic Graph. We represent dynamic graph as a sequence of non-decreasing
chronological interactions G = {(u1, (1,t1,, (u2, U2 ...}. with 0 S t S f2 S ..., where us, U, E N denote the source node and destination node of the i-th link timestamp 4.. N is the set of all the nodes. Each node u E E can be associated with node feature Tu, E IRdx, and each interaction (u, v. has link feature e'u, E Rds dy and dg denote the dimensions of the node feature and link feature. If the graph is attributd, we simply set the node feature and link feature to zero vectors. i.e., Fy and ey, O.
Definition Problem Formalization. Given the source node u, destination node v, timestamp I, and historical interactions before t, i.e., {(u', d', {t' It' t). representation learning on dynamic graph aims to design model to learn time-aware representations h², E Bd and h² E Rd for and with d as the dimension. We validate the effectiveness of the learned representations via two common tasks in dynamic graph learning: (i) dynamic link prediction, which predicts whether u and are connected at I; (ii) dynamic node classification, which infers the state of u or v at l.
4
New Architecture and Unified Library
4.1 DyGFormer: Transformer-based Architecture for Dynamic Graph Learning
The framework of our DyGFormer is shown in Figure I, which employs Transformer [56] as the backbone. Given an interaction (u, u. t), we first extract historical first-hop interactions of source node u and destination node e before timestamp t and obtain two interaction sequences St and St. Next, in addition to computing the encodings of neighbors, links, and time intervals for each sequence, we also encode the frequencies of every neighbor's appearances in both St, and St to exploit the correlations between u and v, resulting four encoding sequences for w/e in total. Then, we divide each encoding sequence into multiple patches and feed all the patches into a Transformer for capturing long-term temporal dependencies. Finally, the outputs of the Transformer are averaged to derive time-aware representations of u and v at timestamp t (i.c., ht, and h²). which can be applicd in various downstream tasks like dynamic link prediction and dynamic node classification.
Learning from Historical First-hop Interactions. Unlike most previous methods that require nodes' historical interactions from multiple hops (e.g.. DyRep [55], TGAT [64], TGN [45], CAWN [60]), we only learn from the sequences of nodes' historical first-hop interactions, turning the dynamic graph learning task into a simpler sequence learning problem. Mathematically, given an interaction (u, v, 1). for source node u and destination node we obtain the sequences that involve first-hop interactions of u and v before timestamp I, which are denoted by St = ((u, u', ,',(') 1') Ir' < <1) U (lu,, u (²) It' < <1) and St (v. (u.v',e) ('))) It' < t) U ((o', v, t') It' < t}. respectively.
3

---

N
Ensoding Neighhors, Tinks & Time Intervals
Extract hop Intereetions
X
Patching Aligiment & Concatenation
E.
Feed Forward Network
Average Pooling
X
Encoding Neighboe Co Co-necumencen
I' 12 251
Normalizttico
Xhe
A
ha
Dynamic Graph
Patching & Aligmment Concalenation
Multi-head Self-Attentin
Extract First- hop Internctions
Encoding Neighhors, Intervals
-2
Downstream Tisks
Layer Nommalzztion
Figure 1: Framework of the proposed model.
Encoding Neighbors, Links, and Time Intervals. For source node u, we retrieve the fea- tures of involved neighbors and links in sequence S based the given features represent their encodings, which are denoted by X E RlSt|xdy and X E RlS|Ixde, Following [64], we learn the periodic temporal patterns by encoding the time interval At' = t' via V dr [cos (wrAt'). sin (wi (tt) , cos (wa, At") sin (Wa, At')], where W1," ... Way are train- able parameters. dr is the encoding dimension. The time interval encodings of interactions St is denoted by X E pglSt|xdx We use the same process to get the corresponding encodings for destination node v, i.e., X E S. xdx
X X Ixd, and X St
Neighbor Co-occurrence Encoding Scheme. Existing methods separately compute representations of node u and v without modeling their correlations. We present a neighbor co-occurrence encoding scheme tackle this issue, which assumes the appearing frequency of a neighbor in sequence indicates its importance, and the occurrences a neighbor in sequences of u and v (i.e., co- occurrence) could reflect the correlations between and That is to say, if and have more common historical neighbors in their sequences, they are more likely to interact in the future.
Formally, for each neighbor in the interaction sequence Sy and S we count its occurrences in both S and S and derive two-dimensional vector. By packing the vectors of all the neighbors together, we can get the neighbor co-occurrence features for and which are represented by x2 For example, suppose the historical neighbors of and are {a, b,a) a) and {b, b, a, c}. The appearing frequencies of a, b, and e in we's historical interactions are 2/1, 1/2, and 0/1, respectively. Therefore, the neighbor co-occurrence features of and e denoted
C E xxx2 and C E I st
by C [[2. 1]. [1, 2]. [2, I]] and C [[1. 2] [1, 2] [2. 1] [0, I Then, we apply a function
f (-) to encode the neighbor co-occurrence features by
= f + (C. (J) RlS
where * could be or v. The input and output dimensions of f (-) are and de. In this paper, we implement f (-) by two-layer perceptron with ReLU activation [37]. It is important to note that the neighbor co-occurrence encoding scheme general and can be easily integrated into some dynamic graph learning methods for better results. We will demonstrate its generalizability in Section 5.3.
Patching Technique. Instead of focusing on the interaction level, we divide the encoding sequence into multiple non-overlapping patches to break through the bottleneck of existing methods in capturing long-term temporal dependencies. Let P denote the patch size. Each patch composed of P temporally adjacent interactions with flattened encodings and can preserve local temporal proximities.
Take the patching X N.N E IxdN as an example. X will be divided into 1 [St] patches total (note that we will pad X X if its length (S)) cannot be divided by P), and the patched encoding is represented by M N x Similarly, we can also get the patched M
encodings M E R. M x. M x M M Xdv-
M E E xd M E R and M.c E x Note that when |S becomes

---

longer, we will correspondingly increase P. making the number of patches (i.e., ly and (2)) at constant level to reduce the computational cost.
Transformer Encoder. We first align the patched encodings to the same dimension d with trainable weight W. E pdd and b, E RR to obtain Z E R xd and z E B xd where could be N, E, T or C. To be specific, the alignments are realized by
z = M W. + b. E R xd Z = M W. + b. E RR xd
(2)
Then, we concatenate the aligned encodings of u and v, and get Z = Z Z E Z T Z RC x4d and z Zi Ei, Z R x4d
E
Next, we employ a Transformer encoder to capture the temporal dependencies, which is built by stacking I. Multi-head Self-Attention (MSA) and Ferward Network (FFN) blocks. The residual connection [20] is employed after every block. We follow [14] by using GELU [21] instead of ReLU [37] between the two-layer perception each FFN block and applying Layer Normalization (LN) [3] before each block rather than after. Instead of individually processing Z and Z our Transformer
encoder takes the stacked Z IZ Z E (CC+ x4d as inputs, aiming to learn the temporal dependencies within and across the sequences of u and v. The calculation process is
QQKT
Attention (Q, K, V) = Softmax
V.
(3)
vd
FFN (O, W1. by 22, b2) = GELU(OW)) + b1) W2 + b2.
(4)
0 Attention (LN(Z W
W K,s LN(Z v.))
(5)
O MSA (Z + z (1..1.... .I........... (..0)) W/ Z
(6)
- (LN (O.)), W1, b/, W2, bb) + O
(7)
WQ, E Radxd, W K,, E aadxd, Wi. E Ridxdd, W E ER/dd W E Rddx16d bf E R 16d
W2 gl and by Rd are trainable parameters the 1-th layer. We set dN d, 4d/1 with
I as the number of attention heads. The input of the first layer is Z Z' E ((. and the
output of the L-th layer is denoted by H' tt. E R xxx))
Time-aware Node Representation. The time-aware representations of node a and at timestamp t are derived by averaging their related representations in H' with an output layer,
h MEAN (H' to :l) Wout + bout E RRduu h MEAN (H' C2 + (,,:1) :I) Wout + bout E IRdour
(8)
where W Wout E 14dx and bout E Rdout are trainable weights with dout as the output dimension.
4.2 DyGLib: Unified Library for Continuous-Time Dynamic Graph Learning
We introduce unified library with standard training pipelines, extensible coding interfaces, and comprehensive evaluating strategies for reproducible, scalable, and credible continuous dynamic graph learning research. The overall procedure of DyGLib is shown in Figure 3 in Section A.3.
Standard Training Pipelines. To eliminate the influence of different training pipelines in previous studies, we unify the data format, create a customized data loader, and train all the methods with the same model trainers. Our standard training pipelines guarantee reproducible performance and enable users to quickly identify the key components of different models. Researchers only need to focus on designing the model architecture without considering other irrelevant implementation details.
Extensible Coding Interfaces. We provide extensible coding interfaces for the datasets and algo- rithms, which are all implemented by PyTorch. These scalable designs enable users incorporate new datasets and popular models based on their specific requirements, which can significantly reduce the usage difficulty for beginners and allow experts to conveniently validate new ideas. Cur- rently, DyGLib has integrated thirteen datasets from various domains and nine continuous-time dynamic graph learning methods. It is worth noticing that we also found some issues in previous implementations and have fixed them in DyGLib (see details in Section B.3).
5

---

Comprehensive Evaluating Protocols. DyGLib supports both transductiveeinductive dynamic link prediction and dynamic node classification tasks. Most previous works evaluate their methods on the dynamic link prediction task with the random negative sampling strategy but few models already reach saturation performance under such a strategy, making it hard to distinguish more advanced designs. For more reliable comparisons, we adopt three strategies (i.e., random, historical, and inductive negative sampling strategies) in [44] to comprehensively evaluate the model performance.
Experiments
In this section, we report the results of various approaches by using DyGLib, We show the superiority of DyGFormer over existing methods and also give an in-depth analysis of DyGFormer.
5.1 Experimental Settings
Datasets and Baselines. We experiment with thirteen datasets (Wikipedia, Reddit, MOOC, LastFM, Enron, Social Evo., UCI, Flights, Can. Parl., US Legis., UN Trade, UN Vote, and Contact), which are collected by [44] and cover diverse domains. Details of the datasets are shown in Section I. We compare DyGFormer with eight popular continuous-time dynamic graph learning baselines that based on graph convolutions, memory networks, random walks, and sequential models, including JODIE [28], DyRep [55]. TGAT [64], TGN [45], CAWN [60], EdgeBank [44], TCL [57], and GraphMixer [12]. We give the descriptions of baselines in Section B.2.
Evaluation Tasks and Metries. We follow [64, 45, 60, 44] to evaluate models for dynamic link prediction, which predicts the probability of a link occurring between two given nodes at a specific time. This task has two settings: the transductive setting aims to predict future links between nodes that are observed during training, and the inductive setting predicts future links between unseen nodes. We use multi-layer perceptron to take the concatenated representations of two nodes as inputs and return the probability of a link as the output. Average Precision (AP) and Area Under the Receiver Operating Characteristic Curve (AUC-ROC) are adopted as the evaluation metrics. We adopt random (rnd), historical (hist), and inductive (ind) negative sampling strategies in [44] for evaluation, where the latter two strategies are more challenging. Please refer to [44] for more details. We also follow [64, 45] to conduct dynamic node classification, which estimates the state of node in a given interaction at a specific time. A multi-layer perceptron is employed to map the node representations to the labels. We use AUC-ROC as the evaluation metric due to the label imbalance. For both tasks, we chronologically split each dataset with the ratio of 70%/15% % for rraining/validationneesss
Model Configurations. For baselines, in addition to following their official settings, we also perform an exhaustive grid search to find the optimal configurations of some critical hyperparameters for more reliable comparisons. As DyGFormer can access longer histories, we vary each node's 's input sequence length from 32 4096 by a factor of 2. To keep the computational complexity at constant level that is irrelevant to the input length, we correspondingly increase the patch size from to 128. Please see Section B.5 for the detailed configurations of different models.
Implementation Details. For both tasks, we optimize all models (i.e., excluding Edge which has no trainable parameters) by Adam [27] and use supervised binary cross-entropy loss as the objective function. We train the models for 100 epochs and use the early stopping strategy with patience of the validation set for testing. We set
20. We select the model that achieves the best performance the learning rate and batch size to 0.0001 and 200 for all the methods on all the datasets. We run the methods five times with seeds from 0 4 and report the average performance to eliminate deviations. Experiments are conducted on an Ubuntu machine equipped with one Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz with 16 physical cores. The GPU device NVIDIA Tesla T4 with 15 GB memory.
5.2 Performance Comparisons and Discussions
We report the performance of different methods on the AP metric for transductive dynamic link prediction with three negative sampling strategies Table 1. The best and second-best results are emphasized by bold and underlined fonts. Note that the results are multiplied by 100 for better display layout. Please refer to Section C. and Section C.2 for the results of AP for inductive dynamic link prediction as well as AUC-ROC for transductive and inductive dynamic link prediction tasks.

---

Table AP for transductive dynamic link prediction with random, historical, and inductive negative
sampling strategies. NSS is the abbreviation of Negative Sampling Strategies.
NSS
Datasets
JODIE
Dykep
TGAT
TGN
CAWN
EdgeBank
TCL
GraghMiser
DyGiormer
Wikspedia
96.50
0.14
0.06
0.06
95.76
0303
90.37 0.00
97.25
0.02
Reddit
98.31 = 0.14
98.22
+ 0.04
98.52 0.02
98.63
0.05
99.11
+ 0.01
94,86 4 0.00 97.53 0.02 97.31 + 0.00 99.22
0.01
MOOC
80.23
2.44
81.97
+ 0.49
85.84
89.15
1,60
81.15
+ 0.25
57.97
0.00 82.38
0.24
82.78
0.15 8752
0.49
LaxEM
70.85
2.13
73,42
0221 77.07
86.99
0.06
67.27
75.61
0.224 93,00
0.12
Enron
84.77
0.30
82.38
+ 3.36
71.12 = 0.97
86.53
1.11
89.56
+ 0.09
83.53
1: 0.00
79.70 0.71
82.25
0.16
92.47
0.12
Social Evo.
89.89 + 0.55
8887
+ 0.30
93.16
0.17 93.57
+ 0.17
84.9
+ 0.09
74.95
+ 0.00
93.13
+ 0.16
93.37
0.07 94.73
0.01
UCI
89.43
1,09
92.34
95.18
0.06
89.57
93.25
95.79
Flights
95.60 4: 1.73
95.29
1: 0.72
94.03
0.18 97.95
0.14
98.51
0.01
89.35
1 0.00
91.23
0.02
90.99
1. 0.05
98.91
0.01
Can. Part.
69.26
+ 0.31
66.54
+ 2.76
70.73
0.72
70.88
34
09.82
2.34
64.55
+ 0.00
68.67
2.67
72.04
+ 0.46
97.36
0.45
Legis.
75.05
1.52
0.58
70.58
70.74
1.02
71
0.59
UN Trade
64.94
1: 0.31
63.21
1: 0.93
61.47
0.18
65.03
1.37
65.39
0.12
60.41
1 0.00
62.21
0.03
62.61
1: 0.27
66.46
1.29
UN Vote
63.91
0.81 62.81
+ 0.80
52.21 0.98
65.72
2.17
$2.84
0.10
58,49
+ 0.00
$1,90 + 0,30
52.11
+ 0.16
55.55
0.42
Contact
1.33
89
98.29
0.01
Ave. Rank
5.08
5.85
3.69
4.31
7.54
6.92
5.46
Wikepedda
83.01
=: 0.06
79.93
+: 0.56
87.38 0.22
86.80
0.33
71.21
: 1.7
73,35
4 0.30 89.05 +: 0.39
901.0
:: 0.10
82:21
=: 2.54
Reddit
80.05
0.36 79.83
0.31
22
045
77.14
78.44
81.57 0.67
MOOC
78.94
=: 1,25
75.60
+ 1.12
82.19
0.62
87,06
1.93
74,05
= 0.95
60.71
+ 0.00
77,06
0.41
77.77
1: 0.92
85.85
0.66
Lasl'M
74.35 4: 3,81 74.92
1: 2.46
71.59
0.24
76.87
1 4.64
69.88
0.43
73.03
1 0.00
59.30
2.31
72.47
1: 0.49
81.57
0.48
69.85
71.19
73.11
64.73
70.66
77.98
0.92
75.63
0.73
Social Evo.
87,44 = 6.78
93.29
4: 0.43
95.01
0.4
94,45
0.56
85.53
= 0.38
50.57
1: 0.00
94,74
0.31
94.93
0.31
97.38
0.14
bist
Flights
UCI
66.48
75.24 4: 5.80
2.59
67.61
55.10
1: 3.14
0.99
72.38
68.27
1.37
66.70
80.43
1 2.12
64.72
65.30
1 0.43
0.7
65.50
1 0.00
80.25 2: 2.74
70.68
84,11
71.47
1: 1.35
0.26
66.59
82.17
0.82
0.49
Can. Parl.
$1.79 0.63 63.31
+ 1,23
67.13 0.84
68.42
3.07
66.53
2.77 63.84
+: 0.00
65.93 + 3,00
74
0.87
97.00
0.31
US Legis.
$1.71
1: 5.76
86.88
1: 2.25
62.14
5.60
74.00
1 7.57
68.82
8.23
63.22
1 0.00
80.53
4 3.95
81.65
1
1.02
85.30
3.88
UN Trade
61.39 1.83
59.19
1,07
S5.74
58.44
55,71
0.38
81.32
55,90
$7.05
1.22
64.41
1.40
UN Vote
70.02, 0.81 69.30
+ 1.12 52.96
14
69.37
199
51.26
0.04
84,89
+ 0.00
52,
2.35
51.20
1.60
60.84
1.58
Contact
95.31
2.13
96.39
+: 0,20
96.05
0.52
93.05
1 2.35
84.16
+
0.49
85.81
1 0.00
93.86
2: 0.21
93.36
2 0.41
97.57
0.06
Avg. Rank
5,46
5.08
3.85
5,46
4,00
Wikipedia
75.65
=: 0.79 70.21
1 1.38 87.00
0.16
85.62
0.44
74,16
1,6
80.63
1 0.00
86.76
0.72
88.59
0.17
78.29
5.35
Reddit
86.98 +: 0.16 86.30
+ 0.26
89.50
0.24
88.10
0.24
91.67
0.24
85.48
+ 0.00
87.45
0.29
85.26
0.11
91.11
-0.40
MOOC
65.23
0.95
27.50
73.51
0.94
74:27
81.24
0..9
LasFM
62.67 :: 4.19 64.41
1: 2.70
71.13
0.17
65.95
1 5.95
67.48
1 0.77
75,49
1 0.00
58.21
0.89
68.12
0.33
73.97
= 0.50
Enron
68.95
4: 0.98 67.79
+ 1,53 63.94
1,36
70.89
1 2.22
75.15
0.58
73,89
1 0.00
71.29
0.32
75.01
1: 0.79
77.41
0.89
Social
89.82
93.28
0.48
95.13
94.72
97.68
0.10
ind
Fighhs
UCI
65.99
69.07 4.02
1.40 $4.79
70.57
+ 1,76 68.67
+ 1.82
75.48
0.26
0.84
71.09
70.94
1 2.22
0.71 64.61
69.18
048
52 $1.08
57,43
1 0.00
0.00
74.62
76.01 : 1.11
1 0.18
74.87
80.10
1 0.21
0.51
72.25
70.92
= 1.71
1.78
Can. Part.
48.42 0 0.66 58.61
0,86
65.34
67.75
00
62,16
+
65.85
19.48
95.44
0.57
Legis.
50.27 = 5.13 83.44
+ 1,16 91
= 5.82
67
6.47
65.81
+ 8.52
64.74
0.00
78.15
3.34
79.63
2: 0.84
81,25
3.62
UN Trade
60.42 =: 1.48 60.19
4 1.24 60.61
1.24
61.04
6.01
62.54
1 0.67
72,97
1 0.00
61.06
1.74
60.15
1: 1.29
55.79
= 1.02
UN Vate
67.79 1,46 67.53
1,98
$2.89
1.11
63
267
$2,19
0.34
0.00
50.62
$1.60
0.73
$1.91
0.84
Contact
93.43
1.78 94.18
0.10
94
35
90.18
3.28
89.31
0.27
85,20
+ 0.00
91,35 0.21
90.87
4: 0.35
94.75
0.28
Arg. Bank
6.62
6.35
4.15
1.38
5:46
5.62
4.09
4.46
3,23
Since EdgeBank can be only evaluated for transductive dynamic link prediction, we do not show its performance under the inductive setting. From the results, we have two main observations.
Firstly, DyGFormer usually outperforms baselines and achieves an average rank of 2.49/2.69 on AP/AUC-ROC for transductive and 2.69/2.56 for inductive dynamic link prediction across three negative sampling strategies. This is because: (i) The neighbor co-occurrence encoding scheme helps DyGFormer exploit correlations between the source node and destination node, which are often predictive for future links (see Section 5.5). (ii) The patching technique allows DyGFormer to access longer histories and capture long-term temporal dependencies (see Section 5.4). In Table 9 in Section B.5, the input sequence lengths of DyGFormer are much longer than those of baselines on several datasets, indicating that it can utilize longer sequences better. We also observe the varying results of DyGFormer across different negative sampling strategies and give an analysis in Section 5.7.
Secondly, some of our findings of baselines differ from previous reports. For instance, the perfor- mance of some baselines can be significantly improved by properly setting some hyperparameters. Additionally, some methods would obtain worse results after we fix the problems or make adaptions in their implementations. More explanations can be found in Section B.4. These observations highlight the importance of rigorously evaluating different methods by unified library and verify the necessity of introducing DyGLib to facilitate the development of dynamic graph learning.
We also report the results of dynamic node classification in Table 15 in Section C.3, We observe that DyGFormer obtains better performance than most baselines and achieves an impressive average rank of 2.50 among them, demonstrating the superiority of DyGFormer once again.
5.3 Generalizability of Neighbor Co-occurrence Encoding Scheme
Our Neighbor Co-occurrence Encoding scheme (NCoF) versatile and can be easily integrated with dynamic graph learning methods based on sequential models. Hence, we incorporate NCoE with
7

---

Table 2: AP for TCL with NCoE.
Datasets
TCL
w/ NCoE
Improv.
DyRep
GraphMiier DyGG
Wikipedia
96.47
99.09
2.72%
Reddit
97.53
99.04
1.55%
0.90
MOOC
82.38
86.92
5.51%
0.90
0.85
LastFM
67.27
84.02
24.90%
0.80
Enron
79.70
90.18
13.15%
Social Evo.
93.13
94.06
1.00%
UCI 89.57
94.69
5.72%
0.60
Flights
91.23
97.71
7.10%
10
10
Can. Parl. 68.67
69.34
0.98%
logulength
(b) Perfommincc Parl..
logelength
US Legis. 69.59
69.47
-0.17%
UN Trade 62.21
63.46
2.01% %
Figure 2: Performance of different methods on LastFM and Can. Parl. with varying input lengths.
UN Vote 51.90
51.52
-0.73%
Contact 92.44
97.98
5.99%
TCL and GraphMixer and show their performance in Table 2 and Table 16 in Section C.4. We find TCL and GraphMixer usually yield better results with NCoE, achieving average improvement of 5.36% and 1.86% over all datasets. This verifies the effectiveness and versatility of the neighbor co-occurrence encoding, and highlights the importance of capturing correlations between nodes. Also, as TCL and DyGFormer are built upon Transformer, TCL w/ NCoE can achieve similar results with DyGFormer datasets that enjoy shorter input sequences (in which cases the patching technique in DyGFormer contributes little). However, when datasets exhibit more obvious long-term temporal dependencies (e.g., LastFM, Can. Parl.), the performance gaps become more significant.
5.4 Advantages of Patching Technique
validate the advantages of our patching technique in preserving the local temporal proximities and reducing the computational complexity, which helps DyGFormer effectively and efficiently utilize longer histories. We conduct experiments on LastFM and Can. Parl. since they can benefit from longer historical records. For baselines, we sample more neighbors perform more causal anonymous walks (starting from 32) to make them access longer histories. The results are depicted in Figure 2, where the x-axis is represented by a logarithmic scale with base 2. We also plot the performance of baselines with the optimal length by unconnected points based on Table 9 in Section B.5. Note that the results of some baselines are incomplete since they raise the out-of-memory error when the lengths are longer. For example, TGAT is only computationally feasible when extending the input length 32, resulting in discrete points with lengths 20 (the optimal length) and 32.
From Figure 2, we conclude that: (i) most of the baselines perform worse when the input lengths become longer, indicating they lack the ability to capture long-tem temporal dependencies; (ii) the baselines usually encounter expensive computational costs when computing on longer historics. Although memory network-based methods (i.e., DyRep and TGN) handle longer histories with affordable computational costs, they cannot benefit from longer histories due to the potential issues of vanishing exploding gradients; (iii) DyGFormer consistently achieves gains from longer sequences, demonstrating the advantages of the patching technique in leveraging longer histories.
We also compare the running time and memory usage of DyGFormer with and without the patching technique during the training process. The results are shown in Table 17 in Section C.5. We could observe that the patching technique efficiently reduces model training costs in both time and space, allowing DyGFormer to access longer histories. As the input sequence length increases, the reductions become significant. Morcover, with the patching technique, we find that DyGFormer achieves an average improvement of 0.31 %% and 0.74% in performance on LastFM and Can. Parl. than DyGFormer without patching. This observation further demonstrates the advantage of our patching technique in leveraging the local temporal proximities for better results.
5.5 Verification of the Motivation Neighbor Co-occurrence Encoding Scheme
To verify the motivation of NCoE (i.e., nodes with more common historical neighbors tend to interact in the future), we compare the performance of DyGFormer and DyGFormer without NCOE. We choose 0.5 as the threshold and use TP, TN, FN, and FP to denote True/False Positive/Negative.

---

Common Neighbor Ratio (CNR) is defined as the ratio of common neighbors in source node u's sequence S, and destination node e's sequence S,, i.e. SS,, S U S. We focus on links whose predictions of DyGFormer w/o NCoF are changed by DyGFormer (i.e., FN-TP, FP-TN, TP-FN, and TN-FP). We define Changed Link Ratio (CLR) as the ratio of the changed links to their original set, which is respectively computed by F |FN], | /PI, /TP->FN// and /TN-FP/ T NCoE is helpful, DyGFormer will revise more wrong predictions (more FN-TP and FP) and make fewer incorrect changes (fewer TP-FN and TN-FP). We report CLR and average CNR links in the above sets on five typical datasets in Table 3.
Table 3: CLR and CNR of changes made by DyGFormer.
CLR (%)
CNR (%)
Datasets
FN-TP
FP-TNN
TP-FN
TN-FFP
FN-TTP
FP-+TN
TP-FFN TN-
Wikipedia
68.36
72.73
1.68
1.69
18.16
0.01
0.10
2.49
UCI
71.45
94.11
7.29
1.82
19.08
2.49
3.35
13.02
Flights
83.66
83.83
1.73
2.11
37.09
2.28
7.06
20.28
US Legis.
31.63
23.67
6.63
1.59
69.92
62.13
61.14
63.80
UN Vote
44.02
36.46
28.95
30.53
78.57
81.39
80.86
77.02
We find NCoE effectively helps DyGFormer rectify wrong predictions of DyGFormer w/o NCoE on datasets with significantly higher CNR of positive links than negative ones, which happens with most datasets. Concretely, for Wikipedia, UCI, and Flights, their CNRs of FN-TP are much higher than FP-TN (e.g.. 37.09% vs. 2.28% on Flights) and DyGFormer revises most wrong predictions of DyGFormer w/o NCoE (e.g.. 83.66% for positive links in FN and 83.83 33.3% for negative links in FP on Flights). Corrections made by our encoding scheme are less obvious between positive and negative links are similar, which occurs in only 2 of 13 datasets. For US Legis. and UN Vote, their CNRs between FN and FP analogous (e.g., 69.92% vs. 62. 13% on US Legis.), weakening the advantage of our neighbor co-occurrence encoding scheme (e.g., only 31.63%/23.6 62%.6 of positive/negative links are corrected in FN/FP on US Legis.). Therefore, we conclude that the neighbor co-occurrence encoding scheme helps DyGForme capture common historical neighbors in Su and S,, and bring better results in most cases.
datasets whose CNRs
5.6
When Will DyGFormer Be a Good Choice?
Note that DyGFormer is superior to baselines by 1) exploring the source and destination nodes' correlations from their historical sequences by neighbor co-occurrence encoding scheme; 2) using the patching technique to attend longer histories. Thus, DyGFormer tends to perform better datasets that favor these two designs. We define Link Ratio (LR) as the ratio links in their corresponding positive negative set, which can be computed by TP/TTPFFN) TN / (TN+FP), FN/(TFFNN, and FP FP/TTN+FP) As a method with more TP and TN (i.e., fewer FN and FP) is better, we report the results of LR and average CNR of links in TP and TN on five typical datasets in Table 4.
Table 4: LR and CNR of TP and TN with Table 5: LR and CNR of FP under random, historical, random negative sampling strategy.
and inductive negative sampling strategy.
IR (%)
CNR
(%)
LR(%)
CNR(S) (%)
Datasets
Datasets
TP
TN
TP
TN
rnd
hist
ind
md
hist
ind
Wikipedia
92.74
97.19
59.09
0.01
Wikipedia
2.81
89.28
94.53
0.02
14.00
11.66
UCI
82.70
96.77
28.03
1.45
UCI
3.23
64.93
76.42
9.98
12.22
13.81
Flights
96.13
95.33
47.58
1.40
Flights
4.67
94.52
92.94
0.01
35.62
30.29
US Legis.
78.95
56.83
75.18
53.98
US Legis.
43.17
17.31
21.21
79.40
87.51
75.51
UN Vote
65.18 45.43
56.24
76.02
UN Vote
54.57
39.90
52.92
79.60
75.53
79.15
We observe when CNR of TP is significantly higher than CNR of TN in the datasets, DyGFormer often outperforms baselines (most datasets satisfy this property). For Wikipedia, UCI, and Flights, their CNRs of are much higher than those of TN (e-g.. 59.09% vs. 0.01% % on Wikipedia). Such a characteristic matches the motivation of our neighbor co-occurrence encoding scheme, enabling DyGFormer to correctly predict most links (e.g., 92.74% of positive links and 97. 97% of negative links are properly predicted on Wikipedia). Moreover, LastFM and Can. Parl. can gain from

---

longer histories (see Figure 2 and Table 9 in Section B.5), DyGFormer is significantly better than baselines on these two datasets. When CNRs of TP and TN are less distinguishable in the datasets, DyGFormer may perform worse (only 2 out of 13 datasets show this property). For US Legis., the CNRs between TP and TN are close (i.e., 75.18 18% vs. 53.9 53,98%) making DyGFormer worse than memory-based baselines (i.e., JODIE, DyRep, and TGN). For UN Vote, its CNR of TP is even lower than that of TN (i.e., 56.24% vs. 76.02%), which is opposite to our motivation, leading to poor results of DyGFormer than few baselines. Since these two datasets cannot obviously gain from longer sequences either (see Table 9 in Section B.5), DyGFormer obtains worse results on them. Thus, we conclude that for datasets with much higher CNR of TP than CNR of TN or datasets that can benefit from longer histories, DyGFormer is a good choice. Otherwise, we may need to try other methods.
5.7 Why do DyGFormer's Performance Vary across Different Negative Sampling Strategies?
Compared with the random (rnd) strategy, historical (hist) and inductive (ind) strategies will sample previous links as negative ones. This makes previous positive links negative, which may hurt the performance DyGFormer since the motivation of our neighbor co-occurrence encoding scheme is violated. As positive links are identical among rn,, hist, and ind, we compute LR and the average CNR of links in FP and show results in Table 5.
We find when hist or ind causes several magnitudes higher CNR of FP than rnd in the datasets, DyGFormer drops sharply. For Wikipedia, UCI, and Flights, the CNRs of FP with hist/ind much higher than rnd (e.g.. 14.00% 1atp/116666 166%% vs. 0.02% on Wikipedia). This misleads DyGFormer to predict negative links as positive and causes drops (e.g.. 89.28% of negative links are incorrectly predicted with hist/ind on Wikipedia, while only 2.81% % are wrong with rd). We also note the drops in UCI are milder since the changes in CNR caused by hist ind vs. rnd are less obvious than changes in Wikipedia and Flights. When changes in CNR of FP caused by hist or ind are not obvious in the datasets, DyGFormer is less affected. Since hist/ind makes little changes in CNRs of FP on US Legis., we find it ranks second with hist/ind, which may indicate DyGFormer is less influenced by the neighbor co-occurrence encoding scheme and generalizes well various negative sampling strategies. For UN Vote, although its CNRs of FP are not affected by hist and ind either, DyGFormer still performs worse due to its inferior performance with rnd. Hence, we deduce that our neighbor co-occurrence encoding may be sometimes fragile to various negative sampling strategies if its motivation is violated, leading to the varying performance of DyGForeer
5.8 Ablation Study
Finally, we validate the effectiveness of the neighbor co-occurrence encoding, time encoding, and mixing of the sequence of source node and destination node in DyGFormer. From Figure 4 in Section C.6. we observe that DyGFormer obtains the best performance when using all the components, and the results would be worse when any component is removed. This illustrates the necessity of each design in DyGFormer. Please refer Section C.6 for detailed implementations and discussions.
Conclusion
In this paper, we proposed a new Transformer-basee architecture (DyGFormer) and a unified library (DyGLib) to foster the development of dynamic graph learning. DyGFormer differs from previous methods in (i) a neighbor co-occurrence encoding scheme to exploit the correlations of nodes in each interaction; and (ii) patching technique to help the model capture long-term temporal dependencies. DyGLib served as toolkit for reproducible, scalable, and credible continuous ttime dynamic graph learning with standard training pipelines, extensible coding interfaces, and comprehensive evaluating protocols. We hope our work can provide new perspectives on designing new dynamic graph learning frameworks and encourage more researchers to dive into this field. In the future, we will continue to enrich DyGLib by incorporating the recently released datasets and state-of-the-ar models.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (No. 62272023 and 5 51991395) and the Fundamental Research Funds for the Central Universities (No. YWF-23-L-1200
10

---

References
[1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg. Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation, pages 265 283. USENIX Association, 2016.
[2] Unai Divare--Rodriuuee Federico Battiston, Guilherme Ferraz de Arruda, Yamir Moreno, Matjaž Perc, and Vito Latora. Evolutionary dynamics of higher-order interactions in social networks. Nature Human Behaviour, 5(5):586-55)) 2021.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1rXvv:1677006400 2016.
[4] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting, In Advances Neural Information Processing Systems 33, 2020.
[5] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez,, Vinícius Flo- res Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew M. Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, abs/1806.0113 2018.
[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33, 2020.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-ent to--to-ed object detection with transformers. In Computer Vision ECCV 2020 16th European Conference, volume 12346 of Lecture Notes in Computer Science, pages 213-229 Springer, 2020.
[8] Yukuo Cen, Zhenyu Hou, Yan Wang, Qibin Chen, Yizhen Luo, Xingcheng Yao, Aohan Zeng. Shiguang Guo, Peng Zhang, Guohao Dai, Yu Wang, Chang Zhou, Hongxia Yang. and Jie Tang. Cogdl: An extensive toolkit for deep learning on graphs. CoRR, abs/2103.009955 2021.
[9] Xiaofu Chang, Xuqin Liu, Jianfeng Wen, Shuang Li, Yanming Fang. Le Song, and Yuan Qi. Continuous-time dynamic graph learning via neural interaction processes. In The 29th ACM International Conference on Information and Knowledge Management, pages 145-154. ACM, 2020.
[10] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST@ EMNLP 2014, pages 103-111. Association for Computational Linguistics, 2014.
[11] Weilin Cong, Yanhong Wu, Yuandong Tian, Mengting Gu, Yinglong Xia, Mehrdad Mahdavi, and Chun-cheng Jason Chen. Dynamic graph representation learning via graph transformer networks. CoRR, abs/21112 0ts/2111.10447,, 2021.
[12] Weilin Cong. Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, and Mehrdad Mahdavi. Do we really need complicated model architectures for temporal networks? In International Conference on Learning Representations, 2023.
11

---

[13] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2019.
Association for Computational Linguisties,
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations. OpenReview.net, 2021.
[15] Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, and Philip S. Yu. Continuous-time sequential recommendation with temporal graph collaborative transformer. In The 30th ACM International Conference on Information and Knowledge Management, pages 433-442, ACM, 2021.
[16] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. CoRR, 1es/1993,,2428, 2019.
[17] Palash Goyal, Sujit Rokka Chhetri, and Arquimedes Canedo. dyngraph2vec: Capturing network dynamics using dynamic graph representation learning. Knowl. Based Syst. 187, 2020.
[18] Palash Goyal, Sujit Rokka Chhetri, Ninareh Mehrabi, Emilio Ferrara, and Arquimedes Canedo. Dynamicgem: A library for dynamic graph embedding methods. CoRR, abs/18 11.118811100734 2018.
[19] Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In The Thirty-Third AAAI Conference on Artificial Intelligence, pages 922-929. AAAI Press, 2019.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778. IEEE Computer Society, 2016.
[21] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 08015, 2016.
[22] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems 33, 2020.
[23] Shenyang Huang, Farimah Poursafaei, Jacob Danovitch, Matthias Fey, Weihua Hu, Emanuele Rossi, Jure Leskovec, Michael Bronstein, Guillaume Rabusseau, and Reihaneh Rabbany. Tempo- ral graph benchmark for machine learning on temporal graphs. arXiv preprint ar 277702230..01026 2023.
[24] Zijie Huang, Yizhou Sun, and Wei Wang. Learning continuous system dynamics from trregularl-ssaplle partial observations. In Advances in Neural Information Processing Systems 33, 2020.
[25] Ming Jin, Yuan-Fang Li, and Shirui Pan. Neural temporal walks: Motif-aware representation learning on continuous-time dynamic graphs. In NewrlPS, 2022.
[26] Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, and Pascal Poupart. Representation learning for dynamic graphs: survey. J. Mach. Learn. Res., 177:03, 2020.
[27] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, 2015.
[28] Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1279-1278. ACM, 2019.
12

---

[29] Jintang Li, Kun Xu, Liang Chen, Zibin Zheng, and Xiao Liu. Graphgallery: A platform for fast benchmarking and easy development of graph neural networks based intelligent software. In 43rd IEEF/ACM International Conference on Software Engineering: Companion Proceedings, pages 13-16. IEEE, 2021.
[30] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in Neural Information Processing Systems 32, pages 5244-5254, 2019.
[31] Xiaohan Li, Mengqi Zhang, Shu Wu, Zheng Liu, Liang Wang, and Philip Yu. Dynamic graph collaborative filtering. In 20th IEEE International Conference on Data Mining, pages 322-33222223 -313.. IEEE, 2020.
[32] Meng Liu, Youzhi Luo, Limei Wang, Yaochen Xie, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang, Yi Liu, Keqiang Yan, Haoran Liu, Cong Fu, Bora Oztekin, Xuan Zhang, and Shuiwang Ji. DIG: A turnkey library for diving into graph deep learning research. J. Mach. 2021.
Learn. Res., 22:240:1-2444
[33] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/ Noss/190.111 1692, 2019.
[34] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2027 IEEE/CVF International Conference on Computer Vision, pages 9992-10002. IEEE, 2021.
[35] Yuhong Luo and Pan Li. Neighborhood-aware scalable temporal network representation learning.
In The First Learning on Graphs Conference, 2022.
[36] Yao Ma, Ziyi Guo, Zhaochun Ren, Jiliang Tang, and Dawei Yin. Streaming graph neural networks. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 719-728. ACM, 2020.
[37] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, pages 807-814. Omnipress, 2010.
[38] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.
[39] Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kaneza- shi, Tim Kaler, Tao B. Schardl, and Charles E. Leiserson. Evolvegen: Evolving graph con- volutional networks for dynamic graphs. In The Thirty-Fourde AAAI Conference Artificial Intelligence, pages
AAAI Press, 2020.
[40] Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of JMLR Workshop and Conference Proceedings, pages 1310-1318. JMLR.org, 2013.
[41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf,, Edward Z. Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 6024-8035, 2019.
[42] James W Pennebaker, Martha E Francis, and Roger Booth. Linguistic inquiry and word count: Liwe 2001. Mahway: Lawrence Erlbaum Associates, 7 7102001):2001, 2001.
[43] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W. Battaglia. Learning mesh-based simulation with graph networks. In 9th International Conference on Learning Representations. OpenReview.nnt, 2021.
13

---

[44] Farimah Poursafaci, Andy Huang. Kellin Pelrine, and Reihanch Rabbany. Towards better evaluation for dynamic link prediction. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarky Track, 2022.
[45] Emanuele Rossi. Ben Chamberlain, Fabrizio Frasca, Davide Eynard. Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. In ICML 2020 Workshop on Graph Representation Learning, 2020.
[46] Benedek Rozemberezki, Paul Scherer, Yixuan He, George Panagopoulos, Alexander Riedel, Maria Sinziana Astefanoaei, Oliver Kiss, Ferenc Béres, Guzmán López, Nicolas Collignon, and Rik Sarkar. Pytorch geometric temporal: Spatiotemporal signal processing with neural machine learning models. In The 30th ACM International Conference on Information and Knowledge Management, pages 4564 4573. ACM, 2021.
[47] Alvaro Sanchez-Gonzalez Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter W. Battaglia. Learning to simulate complex physics with graph networks. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8459-8468 PMLR, 2020.
[48] Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang, and Hao Yang. Dysat: Deep neural representation learning on dynamic graphs via self-attention networks. In The Thirteenth ACM International Conference on Web Search and Data Mining, pages 519-527. ACM, 2020.
[49] Joakim Skarding, Bogdan Gabrys, and Katarzyna Musial. Foundations and modeling of dynamic networks using dynamic graph neural networks: A survey. IEEE Access, 9779143-79168, 2021.
[50] Weiping Song, Zhiping Xiao, Yifan Wang, Laurent Charlin, Ming Zhang, and Jian Tang. Session-based social recommendation via dynamic graph attention networks. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 555-563. ACM, 2019.
[51] Amauri H. Souza, Diego Mesquita, Samuel Kaski, and Vikas Garg. Provably expressive temporal graph networks. In NeurlPS, 2022.
[52] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: simple way prevent neural networks from overfitting. J. Mach. Learn. Res., 15()):129-1998, 2014.
[53] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Advances in Neural Information Processing Systems 28, pages 1440-2448, 2015.
[54] Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mip-mixer: An all-mlp architecture for vision. In Advances in Neural Information Processing Systems 34, pages 14261-24227 2021.
[55] Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha. Dyrep: Learning representations over dynamic graphs. In 7th International Conference on Learning Representa- tions. OpenReview.net, 2019.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, pages 5998-6008, 2017.
[57] Lu Wang. Xiaofu Chang. Shuang Li, Yunfei Chu, Hui Li, Wei Zhang, Xiaofeng He, Le Song, Jingren Zhou, and Hongxia Yang. TCL: transformer-based dynamic graph modelling via contrastive learning. CoRR, abs/2105.07944 2021.
[58] Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, Ziyue Huang, Qipeng Guo, Hao Zhang, Haibin Lin, Junbo Zhao, Jinyang Li, Alexander J. Smola, and Zheng Zhang. Deep graph library: Towards efficient and scalable deep learning graphs. CoRR, abs/1909.01315, 2019.
14

---

[59] Xuhong Wang, Ding Lyu, Mengjian Li, Yang Xia. Qi Yang, Xinwen Wang, Xinguang Wang, Ping Cui, Yupu Yang. Bowen Sun, and Zhenyu Guo. APAN: asynchronous propagation attention network for real-time temporal graph embedding. In International Conference Management ACM, 2021.
of Data, pages 2628-2636
[60] Yanbang Wang, Yen-Yo Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. In 9th International Conference Learning Representations. OpenReview.net, 2021.
[61] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In 3rd International Conference on Leurning Representations, 2015.
[62] Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems 34, pages 22419-22430, 2021.
[63] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporall graph modeling. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 1907-1913, ijcai.org, 2019.
[64] Da Xu, Chuanwei Ruan, Evren Körpeoglu, Sushant Kumar, and Kannan Achan. Inductive representation learning temporal graphs. In 8th International Conference Learning Representations. OpenReview.net, 2020.
[65] Guotong Xue, Ming Zhong, Jianxin Li, Jia Chen, Chengshuai Zhai, and Ruochen Kong. Dynamic network embedding survey, Nenrocomputing, 472:212-223, 2022.
[66] Jiaxuan You, Tianyu Du, and Jure Leskovec. ROLAND: graph learning framework for dynamic graphs. In The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2358-2366. ACM, 2022.
[67] Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pages 3634-3640. ijcai.org. 2018.
[68] Le Yu. An empirical evaluation of temporal graph benchmark. arXiv preprint arXiv:2307.12510 2023.
[69] Le Yu, Bowen Du, Xiao Hu, Leilei Sun, Liangzhe Han, and Weifeng Lv. Deep spatio-temporal graph convolutional network for traffic accident prediction. Neurocomputing, 423:135-147, 2021.
[70] Le Yu, Zihang Liu, Leilei Sun, Bowen Du, Chuanren Liu, and Weifeng Lv. Continuous-timm user preference modelling for temporal sets prediction. IEEE Transactions on Knowledge and Data Engineering, 2023.
[71] Le Yu, Guanghui Wu, Leilei Sun, Bowen Du, and Weifeng Lv. Element-guided temporal graph representation learning for temporal sets prediction. In The ACM Web Conference 2022, pages 1902-1913. ACM, 2022.
[72] Mengqi Zhang, Shu Wu, Xueli Yu, Qiang Liu, and Liang Wang. Dynamic graph neural networks for sequential recommendation. IEEE Transactions on Knowledge and Data Engineering, 2022.
[73] Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Thirty-Fifth AAAI Conference on Artificial Intelligence, pages 11106-11115. AAAI Press, 2021.
[74] Hongkuan Zhou, Da Zheng, Israt Nisa, Vasileios loannidis, Xiang Song, and George Karypis. Tgl: general framework for temporal gnn training on billion-scale graphs. Proceedings of the VLDB Endowment, 15(8):1522-1588 2022.
15


Reviews:
[{"review_id": "QAhGXipibT", "reviewer": "Reviewer_Ea5H", "summary": "This paper presents a transformer-based architecture (DyGFormer) for dynamic graph learning, based on a node co-occurrence encoding scheme and patching. Further, they present DyGLib a library for uniform evaluation of dynamic graph learning techniques. Extensive experimental evaluations over diverse datasets show that DyGFormer performs well.", "questions": "Is it possible to integrate DyGLib / DyGFormer into TGL ([71] in paper)? If so, that would prevent a fragmentation of frameworks, and also allow dynamic graph learning research on billion-scale graphs.\n\nAlso, see the weaknesses section.", "limitations": "Limitations are discussed but in the appendix.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "**Originality.** While the transformer architecture is well-known, it's application to dynamic graph learning, and the proposed co-occurrence encoding and patching schemes are novel. Various libraries / frameworks (as discussed in 108) exist for dynamic graph learning, so DyGLib is not novel in that regard, but a fresh rigorous and extensible evaluation is appreciated.\n\n**Quality / Clarity.** The paper is well-written and easy to follow. The DyGLib codebase is high-quality and well-documented. At a glance, it looks easy to use. Also, the appendix is thorough and well-put-together.\n\n**Significance.** Dynamic graph learning is an important research problem. This work not only presents a good solution but also paves the way for rigorous future work.", "weaknesses": "While DyGFormer outperforms the baselines in avg. rank, there is a problematic trend that on some datasets DyGFormer can be much worse than the best or second best baseline in terms of absolute performance points (Table 1). This needs some analysis. What characteristics of these datasets make DyGFormer a bad choice against the baselines? What aspects of DyGFormer and the baselines could be causing their poor and better performance respectively? Why does DyGFormer's superiority falter when going from \"rnd\" to \"hist\" to \"ind\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the helpful reviews. We have analyzed how the properties of datasets affect DyGFormer and explained the reasons for its varying performance under various negative sampling strategies. We have also discussed the possibility of integrating DyGLib and TGL. We hope our answers have well addressed your concerns.\n\n**W1(Part 1/2): What characteristics of datasets make DyGFormer a bad choice against baselines? What aspects of DyGFormer and baselines cause their poor and better performance?** \n\nDyGFormer is superior to baselines by 1) exploring the source and destination nodes’ correlations from their historical sequences by neighbor co-occurrence encoding, which assumes more common historical neighbors imply more future interactions(Lines 170-171); 2) using the patching technique to attend longer histories. Thus, DyGFormer tends to perform better on datasets that favor these two designs.\n\nFor analysis, we give some symbols first. $TP$, $TN$, $FN$ and $FP$ are True/False Positive/Negative. Link Ratio($LR$) is the ratio of links in its corresponding set, which is computed by $TP/(TP+FN)$, $TN/(TN+FP)$, $FN/(TP+FN)$ and $FP/(TN+FP)$. Common Neighbor Ratio($CNR$) is the ratio of common neighbors in source node $u$’s sequence $S_u$ and destination node $v$’s sequence $S_v$, i.e., $|S_u \\cap S_v|/|S_u \\cup S_v|$. As a method with more $TP$ and $TN$(fewer $FN$ and $FP$) is better, we compute $LR$ and averaged $CNR$ of links in $TP$ and $TN$. We show results in the $LR$($CNR$) format on five typical datasets due to space limit. \n\n||$TP$|$TN$|\n|---|---|---|\n|Wikipedia|92.74(59.09)|97.19(0.01)|\n|Flights|96.13(47.58)|95.33(1.40)|\n|UCI|82.70(28.03)|96.77(1.45)|\n|US Legis.|78.95(75.18)|56.83(53.98)|\n|UN Vote|65.18(56.24)|45.43(76.02)|\n\nWe observe **when** $CNR$ **of**  $TP$ **is several times higher than** $CNR$ **of $TN$ in the datasets, DyGFormer often outperforms baselines (most datasets have this property).** For Wikipedia, Flights and UCI, their $CNR$s of $TP$ are much higher than those of $TN$(e.g., 59.09% vs. 0.01% on Wikipedia). This property matches the neighbor co-occurrence encoding’s assumption, enabling DyGFormer to correctly predict most links (92.74% of positive links and 97.19% of negative links on Wikipedia). Moreover, as LastFM and Can. Parl. can gain from longer histories (see Figure 3, Table 7), DyGFormer is significantly better than baselines on them. **When $CNR$s of $TP$ and $TN$ are less distinguishable in the datasets, DyGFormer may do worse (only 2 out of 13 datasets show this property).** For US Legis., the $CNR$ between $TP$ and $TN$ is close (75.18% vs. 53.98%), which makes DyGFormer worse than memory-based baselines (JODIE, DyRep, and TGN). For UN Vote, its $CNR$ of $TP$ is even lower than that of $TN$(56.24% vs. 76.02%), which is opposite to our assumption, making DyGFormer perform worse than a few baselines. Since these two datasets don’t obviously gain from longer sequences either (see Table 7), DyGFormer cannot achieve satisfactory results on them.\n\n**W1(Part 2/2): Why does DyGFormer's superiority falter when going from \"rnd\" to \"hist\" to \"ind\"?**\n\nCompared with rnd, hist and ind will sample previous links as negative ones. This makes previous positive links negative, which tends to hurt DyGFormer since the assumption of neighbor co-occurrence encoding scheme may be violated. As positive links are the same among rnd, hist, and ind, we compute $LR$ and averaged $CNR$ of links in $FP$ and show results in the $LR$($CNR$) format.\n\n|||$FP$|\n|---|---|---|\n|Wikipedia|rnd|2.81(0.02)|\n||hist|89.28(14.00)|\n||ind|94.53(11.66)|\n|Flights|rnd|4.67(0.01)|\n||hist|94.52(35.62)|\n||ind|92.94(30.29)|\n|UCI|rnd|3.23(9.98)|\n||hist|64.93(12.22)|\n||ind|76.42(13.81)|\n|US Legis.|rnd|43.17(79.40)|\n||hist|17.31(87.51)|\n||ind|21.21(75.51)|\n|UN Vote|rnd|54.57(79.60)|\n||hist|39.90(75.53)|\n||ind|52.92(79.15)|\n\nWe find **when hist or ind causes several times higher** $CNR$ **of** $FP$ **than rnd in the datasets, DyGFormer drops sharply.** For Wikipedia, Flights and UCI, the $CNR$s of $FP$ with hist/ind are much higher than rnd (14.00%/11.66% vs. 0.02% on Wikipedia). This misleads DyGFormer to predict negative links as positive and causes drops (89.28%/94.53% of negative links are incorrectly predicted with hist/ind on Wikipedia, while only 2.81% are wrong with rnd). We also note the drops on UCI are milder since the changes in $CNR$ caused by hist or ind vs. rnd are less obvious than changes in Wikipedia or Flights. **When changes in** $CNR$ **of** $FP$ **caused by hist or ind are not obvious in the datasets, DyGFormer is less affected.** As hist/ind makes little changes in $CNR$s of $FP$ on US Legis., we find it ranks second with hist/ind, which may indicate DyGFormer is less influenced by neighbor co-occurrence encoding on US Legis. and generalizes well to various negative sampling strategies. For UN Vote, though its $CNR$s of $FP$ are not affected by hist and ind either, DyGFormer still performs badly due to its inferior performance with rnd. \n\nTo this end, we conclude that for datasets with much higher $CNR$ of positive links than negative ones or datasets that can benefit from longer histories, DyGFormer is a good choice. Otherwise, we may need to try other methods. Also, our neighbor co-occurrence encoding may be sometimes fragile to various negative sampling strategies if its assumption is violated. It’s promising to tackle this as future work.\n\n**Q1: Is it possible to integrate DyGLib/DyGFormer into TGL?**\n\nYes. Technically, TGL trains on large-scale dynamic graphs with some engineering implementations (e.g., realizing temporal neighbor sampler by C++ for efficient sampling, using multiple GPUs for parallel training). Our DyGLib contains more popular dynamic graph learning models and supports comprehensive evaluations. Hence, DyGLib and TGL are complementary and basically compatible as most of their modules are developed by PyTorch. It is promising to combine them for a better framework."}, {"review_id": "6ADlrMKwz3", "reviewer": "Reviewer_QwYr", "summary": "This paper propose DyGFormer, a new Transformer-based architecture for dynamic graph learning, whose novelty mainly includes a neighbor co-occurrence encoding scheme and a patching technique. Moreover, it introduce DyGLib, a unified library to promote reproducible, scalable, and credible dynamic graph learning research. In experiments ,the DyGFormer achieves sota performance on most of the datasets.", "questions": "1. Can you add the introduction and experiment of PINT in this paper?", "limitations": "The authors adequately addressed the limitations. I have no ethical concerns with this paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper is clearly written and easy to follow.\n2. I think this unified library is high-quality, meaningful and urgently needed, which can well promote the development of dynamic graphs\n", "weaknesses": "1. In DyGFormer , I think the introduction of transformer is not so novel. which has similarity with the attention mechanism of TGAT and TGN.\n2. PINT[1], a work in NeurIPS 2022, is a new and sota model in the area of dynamic graph. Why this paper has not mention it?\n\n[1] Souza A, Mesquita D, Kaski S, et al. Provably expressive temporal graph networks[J]. Advances in Neural Information Processing Systems, 2022, 35: 32257-32269.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Following the reviews, we have emphasized the novelty of DyGFormer in the designs of the neighbor co-occurrence encoding scheme and patching technique. We have also added introductions and empirical comparisons with the PINT baseline. We hope our answers have sufficiently addressed your questions and we are glad to provide more answers if there are any further issues. \n\n**W1: In DyGFormer, I think the introduction of transformer is not so novel, which has similarity with the attention mechanism of TGAT and TGN.**\n\nPlease kindly note that Transformer is the backbone of our DyGFormer, but it is not the essential part of our technical contributions. Instead, the novelties of DyGFormer come from the original designs of the neighbor co-occurrence encoding scheme and patching technique.\n- The neighbor co-occurrence encoding scheme explores the correlations of the source node and destination node based on their historical sequences. This point is less investigated in the existing dynamic graph learning methods because they separately compute the temporal representations of the source and destination node, and thus fail to model their correlations (e.g., JODIE, DyRep, TGAT, TGN, CAWN, …). \n- The patching technique helps DyGFormer to effectively and efficiently leverage longer histories, which is also not well addressed by previous methods.\n\nExtensive experimental results demonstrate the effectiveness of these two designs. Last but not least, we want to clarify that compared with the mentioned TGAT and TGN, our DyGFormer only needs to learn from the sequences of historical first-hop interactions of nodes, which is conceptually simple but effective. Therefore, we conclude that DyGFormer differs from existing dynamic graph learning methods with original technical contributions.\n\n**W2&Q1: PINT[1], a work in NeurIPS 2022, is a new and sota model in the area of dynamic graph learning. Why this paper has not mentioned it? Can you add the introduction and experiment of PINT in this paper?**\n\nThanks for providing this insightful work and we apologize for not mentioning this reference. PINT [1] studies the representational power and limits of two categories of temporal graph networks, and proposes a novel architecture that is provably more expressive than both categories. Here, we provide experimental comparisons between our DyGFormer and PINT. Due to limited time during the rebuttal period, we directly compare with the official results of average precision reported in the PINT paper [1]. Please note that the comparisons are fair because the statistics and preprocessing of datasets in PINT are identical to our work.\n\n|  |  | Wikipedia | Reddit | LastFM | Enron | UCI |\n| --- | --- | --- | --- | --- | --- | --- |\n| transductive setting | PINT | 98.78 $\\pm$ 0.10 | 99.03 $\\pm$ 0.01 | 88.06 $\\pm$ 0.70 | 88.71 $\\pm$ 1.30 | **96.01 $\\pm$ 0.10** |\n|  | DyGFormer  | **99.03 $\\pm$ 0.02** | **99.22 $\\pm$  0.01** | **93.00 $\\pm$ 0.12** | **92.47 $\\pm$ 0.12** | 95.79 $\\pm$ 0.17 |\n| inductive setting | PINT | 98.38 $\\pm$ 0.04 | 98.25 $\\pm$ 0.04 | 91.76 $\\pm$ 0.70 | 81.05 $\\pm$ 2.40 | 93.97 $\\pm$ 0.10 |\n|  | DyGFormer  | **98.59 $\\pm$ 0.03** | **98.84 $\\pm$ 0.02** | **94.23 $\\pm$ 0.09** | **89.76 $\\pm$ 0.34** | **94.54 $\\pm$ 0.12** |\n\nFrom the results, we find that DyGFormer performs better than PINT in all the cases except for the transductive setting on UCI, which demonstrates the superiority of DyGFormer. We will include the above introductions and experiments of PINT in the revised version of our paper.\n\n[1] Provably expressive temporal graph networks, NeurIPS, 2022."}, {"review_id": "KddTCNyyQl", "reviewer": "Reviewer_eUNf", "summary": "This paper proposes a new dynamic graph learning architecture and implemented it as part of a new unified graph library with extensive experiment results to verify the effectiveness of the proposed algorithms.\n \nThe author proposes a new Transformer-based architecture DyGFormer for dynamic graph learning. The architecture is designed to overcome some of the limitations of previous methods by:\n- Leveraging a neighbor co-occurrence encoding scheme, which captures the frequency of appearances of each neighbor in the sequences of source and destination nodes. This explicitly explores the correlations between nodes.\n- Utilizing a patching technique, which allows splitting each source/destination node’s sequence into multiple patches, enabling the capture of long-term temporal dependencies and reducing the computational complexity to a level that doesn't depend on the input sequence length.\n\nThe author also developed a library called DyGLib as a unified library specifically tailored to continuous-time dynamic graph learning. Its key features include:\n- Standardized training pipelines to facilitate reproducibility and consistency across different methods.\n- Extensible coding interfaces for better adaptability and scalability and integrates with 13 graph datasets and 9 graph algorithms", "questions": "1) Why does explicitly modeling the neighbor co-occurrence encoding help? What does the model eventually learn in the encoding module? What is the motivation of this function design which computes f(count_in_Su) + f(count_in_Sv) for each neighbor node?\n2) The dynamic graph models consider the depth of graph searching but the proposed only considered first-layer interaction. Does it imply that high-level interaction modeling is not needed for dynamic graphs?", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper presents an extensive and technically rigorous evaluation. The authors benchmark their model on 13 dynamic graph datasets against nine state-of-the-art dynamic graph learning models, providing thorough evidence for their conclusions. The comprehensive ablation study further supports the effectiveness of the proposed components.\n\nThe proposed DyGFormer architecture performs favorably in comparison to other dynamic graph learning models, with only a few exceptions.\n\nDyGFormer handles long-term node interactions efficiently, thanks to the novel integration of neighborhood co-occurrence encoding and the patching technique.", "weaknesses": "The paper lacks a comprehensive technical discussion motivating the design of the proposed model. The co-occurrence matrix is said to model node correlations better than other models, but it's not entirely clear why explicit modeling of correlations leads to superior performance. Node proximity and similar neighbors are features that even basic models like the Graph Convolutional Network (GCN) can capture without explicit modeling.\n\nThe paper's presentation could be improved. The title, for instance, may overpromise what the paper actually delivers - I was expecting a unified learning algorithm along with a software tool. However, on closer reading, it appears that the main contribution is from DyGFormer, while DyGLib, though useful, doesn't seem to bring much novelty or be absolutely essential to the paper's contributions. The preliminaries section seems unfinished, and the section on DyGLib includes several unsupported claims and lacks novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the helpful feedback. We have explained the motivation for our neighbor co-occurrence encoding. We have also clarified some presentations and our opinions on high-order interactions for dynamic graph learning. We are glad to answer more if there are any further issues.\n\n**W1: The paper lacks a comprehensive technical discussion …(GCN) can capture without explicit modeling.**\n\nOur neighbor co-occurrence encoding can exploit the correlations between source node $u$ and destination node $v$ by jointly learning from their sequences. However, the GCNs (involved in DyRep, TGAT and TGN) only capture relationships inside $u$/$v$’s own interactions as they compute for $u$/$v$ separately, and fail to explore the correlations between $u$ and $v$. Results of ablation study(orange bar) in Figure 4 show the need of learning correlations between $u$ and $v$. See **Q1** for an analysis of neighbor co-occurrence encoding’s motivation.\n\n**W2: The paper's presentation could be improved …unsupported claims and lacks novelty.**\n\n- **Title**. “New Architecture” means DyGFormer is a new dynamic graph learning architecture with original designs(neighbor co-occurrence encoding and patching technique), which is different from current methods. “Unified Library” means DyGLib is a unified library with standard training pipelines, extensible coding interfaces and comprehensive evaluating protocols, which is needed for dynamic graph learning. \n- **Claims of DyGLib**. DyGLib aims to solve different training pipelines(Lines 218-222), various and problematic implementations(Lines 223-229, Section B.3), and insufficient evaluation protocols(Lines 230-237) in previous studies. Empirical findings(Lines 295-300, Section B.4) well support DyGLib’s claims. Thus, DyGLib contributes to reproducible, scalable and credible dynamic graph learning research. \n\n**Q1(Part 1/2): Why does explicitly modeling neighbor co-occurrence encoding help? What does the model learn?**\n\nOur neighbor co-occurrence encoding’s motivation is more common historical neighbors imply more future interactions(Lines 170-171). To analyze its effect, we compare DyGFormer and DyGFormer without Neighbor Co-occurrence Encoding(i.e., DyGFormer w/o NCoE). We first give some symbols. $TP$, $TN$, $FN$ and $FP$ are True/False Positive/Negative. We focus on links whose predictions of DyGFormer w/o NCoE are changed by DyGFormer($FN$→$TP$, $FP$→$TN$, $TP$→$FN$ and $TN$→$FP$) and define Changed Link Ratio($CLR$) as the ratio of the changed links to their original set, which is computed by $|FN$→$TP|/|FN|$, $|FP$→$TN|/|FP|$, $|TP$→$FN|/|TP|$ and $|TN$→$FP|/|TN|$. Common Neighbor Ratio($CNR$) is the ratio of common neighbors in source node $u$’s sequence $S_u$ and destination node $v$’s sequence $S_v$, i.e., $|S_u \\cap S_v|/|S_u \\cup S_v|$. If our encoding is helpful, DyGFormer will fix wrong predictions of DyGFormer w/o NCoE(more $FN$→$TP$,$FP$→$TN$) and make fewer incorrect changes(fewer $TP$→$FN$,$TN$→$FP$). We compute $CLR$ and averaged $CNR$ of links in the above sets, and show results in $CLR$($CNR$) format on five typical datasets due to space limit.\n\n||$FN$→$TP$|$FP$→$TN$|$TP$→$FN$|$TN$→$FP$|\n|---|---|---|---|---|\n|Wikipedia|68.36(18.16)|72.73(0.01)|1.68(0.10)|1.69(2.49)|\n|Flights|83.66(37.09)|83.83(2.28)|1.73(7.06)|2.11(20.28)|\n|UCI|71.45(19.08)|94.11(2.49)|7.29(3.35)|1.82(13.02)|\n|US Legis.|31.63(69.92)|23.67(62.13)|6.63(61.14)|1.59(63.80)|\n|UN Vote|44.02(78.57)|36.46(81.39)|28.95(80.86)|30.53(77.02)|\n\nWe find **our neighbor co-occurrence encoding helps DyGFormer correct wrong predictions of DyGFormer w/o NCoE on datasets with several times higher $CNR$ of positive links than negative ones (this occurs on most datasets).** For Wikipedia, Flights and UCI, their $CNR$s of $FN$ are much higher than $FP$(37.09% vs. 2.28% on Flights) and DyGFormer corrects most wrong precitions of DyGFormer w/o NCoE(83.66% for positive links $FN$→$TP$, 83.83% for negative links $FP$→$TN$ on Flights). **Corrections made by our encoding are less obvious when $CNR$s between positive and negative links are similar, which only occur in 2 out of 13 datasets.** For US Legis. and UN Vote, their $CNR$s between $FN$ and $FP$ are analogous(69.92% vs. 62.13% on US Legis.), weakening our encoding’s advantage(only 31.63%/23.67% of positive/negative links are corrected in $FN$→$TP$/$FP$→$TN$ on US Legis.).\n\nThus, we conclude neighbor co-occurrence encoding helps DyGFormer capture common historical neighbors in $S_u$ and $S_v$, and bring better results in most cases. We further compare DyGFormer with DyGFormer w/ SepNO variant that learns from $S_u$ and $S_v$ separately(orange bar in Figure 4) and get similar observations, which supports our motivation again.\n\n**Q1(Part 2/2): What’s the motivation of function f(count_in_Su) + f(count_in_Sv) for each neighbor node?**\n\nThis function is designed to encode the co-occurrence features of each neighbor by combining the appearing frequencies in $S_u$ and $S_v$. Take source node $u$ as an example. Its neighbor co-occurrence features is $C_u^t\\in\\mathbb{R}^{|S_u^t|\\times 2}$, where $C_u^t[:,0]$ is the appearing frequency of nodes $u^\\prime\\in S_u$ in $S_u$ and $C_u^t[:,1]$ is the frequency of nodes $u^\\prime\\in S_u$ in $S_v$. We first apply $f()$ on $C_*^t[:,0]$ and $C_*^t[:,1]$, and then add them to get the encoded co-occurrence features. \n\n**Q2: The dynamic graph models … Does it imply high-level interaction modeling is not needed for dynamic graphs?**\n\nOur experiments show that though DyGFormer only learns from first-hop interactions, it usually beats baselines that use high-order interactions(DyRep, TGAT, TGN…). This indicates dynamic graph learning can be well down just with first-hop interactions when we use effective designs. But we cannot say high-order interactions are not needed. Instead, it is better to infer the benefits of using high-order information in baselines are not obvious and better designs are desired to utilize such information."}, {"review_id": "1isRXKb5f3", "reviewer": "Reviewer_ErNY", "summary": "In this paper, the authors considered the dynamic graph representation learning (a.k.a. dynamic network embedding) problem and proposed a novel transformer-based architecture - DyGFormer, with several original designs (e.g., a neighbor co-occurrence encoding scheme, a patching technique, etc.) Moreover, the authors also implemented a unified continuous-time dynamic graph learning library, which include several baselines, widely-used datasets, and a common evaluation pipeline. Exhaustive experiments with various settings of dynamic link prediction and dynamic node classification were also conducted to validate the effectiveness of the proposed method.", "questions": "See W1-W7.\n\nAs claimed before Eq. (3)-(7), the row dimensionality of ${\\bf{Z}}^t$ (i.e., the input of transformer encoder) is $(l_u^t + l_v^t)$, which is related to the length of $S_u^t$ and $S_v^t$. Since different nodes $u$ may have $S_u^t$ with different length, it seems that one can only feed the input ${\\bf{Z}}^t$ w.r.t. one link $(u, v, t)$ to the transformer encoder every time due to the non-fixed dimensionality of input. Can we simultaneously feed ${\\bf{Z}}^t$ w.r.t. multiple (i.e., more than one) links $(u, v, t)$ into the transformer encoder and derive multiple embedding pairs $({\\bf{h}}_u^t, {\\bf{h}}_v^t)$? If yes, how to derive these embedding paris $({\\bf{h}}_u^t, {\\bf{h}}_v^t)$? If not, there seems to be an efficiency issue for the proposed method.\n  \nConsider an extreme case. For some new (i.e., previously unobserved) nodes without any available attributes and historical neighbors, can the proposed method still be able to support the inductive inference (e.g., dynamic link prediction and node classification) w.r.t. these nodes?", "limitations": "See W1-W7.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(S1) The overall presentation of this paper is clear, which makes it easy to grasp the key ideas.\n\n(S2) There are some original designs (e.g., neighbor co-occurrence encoding scheme and patching of historical neighbor sequence) in the proposed method.\n\n(S3) The authors conducted many experiments on 13 datasets for both dynamic link prediction and dynamic node classification with various settings.\n\n(S4) The authors also implemented a unified library including some typical baselines and a common evaluation pipeline for various task settings (e.g., transductive and inductive dynamic link prediction and node classification), with the code provided for review.", "weaknesses": "(W1) Some statements are with too many citations (e.g., 'continuous-time methods [27, 53, 62, 44, 35, 9, 55, 58, 57, 24, 34, 12]', 'graph convolutions [53, 62, 44, 35, 9, 57]', etc,) which are hard to check their sources. Some references are also repeated in successive sentences (e.g., 'only a few libraries are specifically designed for dynamic graphs [18, 45, 71]. DynamicGEM [18] is a library for dynamic graph embedding methods'). It is better to ensure that there are at most 5 references for each statement. Some of the references can also be replaced with concrete examples or methods introduced in this paper. Furthermore, references of some important statements are also missing, e.g., 'unlike most previous methods that need to retrieve nodes' historical interactions from multiple hops' but what do 'previous methods' refer to?\n\n(W2) Some problem statements regarding model optimization are unclear. Concretely, the formal definition of training loss is not given. In general, we can divided existing dynamic graph representation techniques into the task-dependent and task-independent methods, which are respectively trained via (i) supervised losses related to and (ii) unsupervised losses regardless of the downstream task. It is unclear that the proposed method is task-dependent or task-independent. In the baselines, DyRep and TGAT are task-independent while CAW is task-dependent (with a loss designed for dynamic link prediction), as I can check. From my perspective, it is unfair to compare both types of method in a common experiment setting, where task-dependent methods are expected to have better performance than task-independent approaches, due to the incorporation of supervised information related to the downstream task. For task-independent methods, the authors should also clarify the downstream module (e.g., logistic regression, SVM, MLP, etc.) to support a concrete task. More importantly, the evaluation pipeline of a unified library should also cover both the settings.\n  \n(W3) Although the authors gave toy examples for some procedures of the proposed method (e.g., the computation of neighbor co-occurrence encoding), several details still need further clarification (e.g., how to pad a derived patch, etc.) It is recommended summarizing the overall procedure to derive all the encoding and patches in terms of pseudo-code (even in supplementary material).\n  \n(W4) In related research, inductive dynamic link prediction includes the prediction of links between (i) one old (i.e., previously observed) node and one new (i.e., previously unobserved) node as well as between (ii) two new nodes. It is unclear that the inductive setting in this paper refers to both cases or just the latter case.\n  \n(W5) Although the authors included 7 baselines in experiments, most of them are based on time-encoded deep sequential models (e.g., with RNN and attention). In addition, there are some other related approaches based on temporal point process (e.g., HTNE [1] and TREND [2]) and neural ordinary differential equation (e.g., GSNOP [3]) that are not considered in experiments. A unified library should cover various types of method.\n- [1] Zuo, Yuan, et al. Embedding temporal network via neighborhood formation. KDD 2018.\n- [2] Wen, Zhihao, and Yuan Fang. Trend: Temporal event and node dynamics for graph representation learning. Web Conference 2022.\n- [3] Luo, Linhao, Gholamreza Haffari, and Shirui Pan. Graph sequential neural ode process for link prediction on dynamic and sparse graphs. WSDM 2023.\n  \n(W6) In supplementary material, although the authors gave some details regarding the datasets, the source information (i.e., where can we download these datasets) is missing. In Table 4 of supplementary material, what does 'duration' means? Does it means the time granularity? Consistent with the statistics of 'duration', what is the total number of time steps for each dataset? From my perspective, the scale of a dynamic graph is related to the (i) number of nodes $N$ and (ii) number of time steps $T$. As I can check, all the datasets are with $N<20,000$, which cannot be considered as large datasets. Can the proposed method be scaled up to larger datasets? Or are there any possible solutions to addressing the scalability issue (perhaps as one future research direction)?\n  \n(W7) It seems that the authors upload the paper to ArXiv before the formal submission to NeurIPS. Several month ago, google scholar and github recommended the ArXiv version of this paper to me. As a result, I have already known the names and institutions of the anonymous authors, which breaks the double-blind policy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed comments. We have clarified some presentations and provided detailed settings. We have also discussed the mentioned references and tested methods on larger datasets. We hope our answers can sufficiently address your concerns, and if that is the case, we kindly ask you to consider increasing the score.\n\n**W1: Too many or repeated citations. References of some important statements are missing.**\n\n1) We have listed some typical methods and changed the sentences to “continuous-time methods like JODIE[27], DyRep[53], TGAT[62], TGN[44], CAWN[58], TCL[55], GraphMixer[12] and others[35, 9, 57, 24, 34]” and “graph convolutions like DyRep[53], TGAT[62], TGN[44] and others[35, 9, 57]”.\n\n2) We clarify the former citations “…for dynamic graphs[18,45,71]” are used to list some libraries, while the latter “DynamicGEM [18]” is cited to discuss a specific one. So they don't seem to repeat.\n\n3) We have changed the presentation to “Unlike most previous methods … hops (e.g., DyRep[53], TGAT[62], TGN[44], CAWN[58])”.\n\n**W2: Unclear training loss and downstream modules.**\n\nFor fair comparisons, all models are optimized by supervised binary cross-entropy loss for dynamic link prediction and node classification tasks. In our initial attempt, we also tried to train some baselines with their own unsupervised loss(e.g., negative log-likelihood loss for temporal point process in DyRep), but they didn’t work well. For both tasks, we use MLPs as downstream modules(Lines 254-255, 261-262).\n\n**W3: Some details of the overall procedure need clarification.**\n\nDue to space limit, we elaborate on how to perform padding in a batch and we'll add an algorithm to describe the overall process in the revised paper. For a batch of multiple links, we first obtain sequences of source and destination nodes in the batch (denoted by $S_U$ and $S_V$). Then, we get the maximal sequence length in $S_U$ and $S_V$ (denoted by $l_U^{max}$ and $l_V^{max}$). If $l_U^{max}$ or $l_V^{max}$ cannot be divided by patch size $P$, we’ll increase them until they are divisible. Next, we apply zero-padding (as we use node with index 0 as padding marker) to sequences shorter than  $l_U^{max}$ and $l_V^{max}$ in $S_U$ and $S_V$. Finally, we extract encodings for the padded sequences in $S_U$ and $S_V$, divide them into  $l_U^{max}/P$ and $l_V^{max}/P$ patches, and combine them as the inputs of Transformer. Note that we'll store the original lengths of source and destination nodes, and use them to derive their final embeddings, see Equation 8.\n\n**W4: Unclear inductive setting.**\n\nA link is treated as inductive if it contains at least one new node. So the inductive setting covers both cases.\n\n**W5: Some related methods are not considered.**\n\nThanks for providing these works! HTNE and TREND are built on Temporal Point Process(TPP), which can solve the same tasks in our work. But currently, we cannot integrate them into DyGLib since their codes are unavailable. Similar to HTNE and TREND, DyRep is also a TPP-based method that uses deep neural networks to parameterize intensity function in TPP. Our DyGFormer outperforms DyRep in the experiments, which verifies its superiority. We focus on general dynamic graphs while GSNOP is designed for sparse dynamic graphs, which can be viewed as another line of work parallel to ours. These contents will be added to the revised paper.\n\n**W6(Part 1/2): Some details of datasets are missing.**\n\nWe gave the URL to download all datasets in README.md file in our codes. But we cannot show it here as NeurIPS 2023 forbids external links in rebuttal. “duration” is the time span of the dataset. For Wikipedia, Reddit, MOOC, LastFM, Enron, Social Evo. and UCI, the numbers of time steps are 152,757, 669,065, 345,600, 1,283,614, 22,632, 565,932 and 58,911. Their time granularities are Unix timestamps. For Flights, Can. Parl., US Legis., UN Trade, UN Trade and Contact, the time steps/time granularity are 122/days, 14/years, 12/congresses, 32/years, 72/years and 8,064/5 minutes.\n\n**W6(Part 2/2): Can the method be scaled up to larger datasets?**\n\nYes. We have tested DyGFormer on two larger datasets(tgbl-review with 352,637 nodes and 4,873,540 links, tgbl-coin with 638,486 nodes and 22,809,486 links) in [1]. Due to limited time, we use hyperparameters on Wikipedia and the searches of hyperparameters are desired in future. The datasets use Mean Reciprocal Rank(MRR) for dynamic link prediction. We show results on the test set, and blank spaces mean the methods were too low or out of memory on GPU. We find DyGFormer scales well to larger datasets, while some baselines(JODIE, CAWN) fail. It does medium on tgbl-review and best on tgbl-coin, which shows its effectiveness.\n\n||tgbl-review|tgbl-coin|\n|---|---|---|\n|JODIE|**54.97±0.91**||\n|DyRep|36.70±1.30|43.40±3.80|\n|TGAT|45.50±2.57|60.88±1.25|\n|TGN|53.20±2.00|58.30±5.00|\n|CAWN|34.43±1.39||\n|EdgeBank|9.99±0.00|58.00±0.00|\n|TCL|36.13±0.54|63.95±0.62|\n|GraphMixer|53.12±0.22|73.25±1.75|\n|DyGFormer|39.10±2.24|**74.88±0.04**|\n\n**W7: Break of double-blind policy.**\n\nOur submitted paper, appendix and codes are anonymized without identifying information. Also, NeurIPS 2023 allows authors to submit anonymized work of preprints. So we obey the double-blind policy.\n\n**Q1: Can we simultaneously feed multiple links into transformer encoder and derive multiple embedding pairs?**\n\nYes. We use mini-batch training for DyGFormer by applying zero padding on multiple links in a batch. See **W3** for more details.\n\n**Q2: Can the method support inductive inference for nodes without attributes and historical neighbors?**\n\nDyGFormer can handle the latter case (nodes without historical neighbors) and the inductive setting already covers it (when DyGFormer predicts a new node in evaluation data). But if nodes have no attributes either, the task is hard for most existing models, which is an interesting future direction.\n\n[1]Temporal Graph Benchmark for Machine Learning on Temporal Graphs, arXiv, 2023."}, {"review_id": "qC0MDbfHRp", "reviewer": "Reviewer_zX7A", "summary": "This paper studies the problem of continuous-time dynamic graph learning. The authors proposed a Transformer-based architecture DyGFormer to learn the dynamic edge representation, which mainly consists of a neighbor co-occurrence encoding scheme to count the co-occurrence of nodes, a patching technique to split the node’s sequence into multiple patches, and a Transformer as the backbone.  The authors also propose a dynamic graph learning library DyGLib, and re-report the important baselines’ performance based on DyGLib. Experimental results conducted on 13 benchmark datasets verify the effectiveness of the proposed method. ", "questions": "Please refer to the weaknesses part.", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe authors propose a co-occurrence encoding scheme and a patching technique to capture the correlation between sequences and the long histories of sequences respectively.  As far as I know, it is less studied than learning the dynamic link representations direct from the sequences of nodes’ first-hop interactions under the dynamic graph learning problem, especially how to model the long histories of the interaction sequence.\n2.\tThe authors conduct extensive experiments on 13 benchmark datasets to verify the effectiveness of the proposed two-component. Besides the proposed method, the authors also re-implemented the important baselines in the area, and point out that some findings of baselines are not in line with previous reports because of their varied pipelines and problematic implementations, which may benefit future research in the continuous-time dynamic graph learning community.\n3.\tThe authors propose a continuous-time dynamic graph learning problem. This library provides a unified pipeline to train and evaluate different baselines. I think this library can provide a tool for researchers in the community to conveniently evaluate different methods in a unified setting.\n", "weaknesses": "1.\tIn the problem formalization of section 3, the authors define the target of the dynamic graph learning problem as learning the time-aware representations for each node. But the DyGFormer actually learns the dynamic link representations but not the node representations. Or more accurately, the DyGFormer learns the contextual node representation $h^t_{u|(u,v)}$ but not $h^t_u$, where the node representation $h^t_{u|(u,v)}$ relies on the other node v. It will be better to add some discussion for this in the paper.\n2.\tIn section 5.2, the authors say that “some of our findings of baselines differ from previous reports”, but the authors do not point out which baselines’ performance improves. I suggest the authors explicitly list which baselines’ performance improves.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive reviews. In response to your comments, we have clarified the relationships between the learned representations of links and nodes. We have also specified the baselines whose observations are different from previous reports. We would be pleased to explain more if further discussions are required.\n\n**W1: It will be better to add some discussions for the learning of dynamic link representations but not node representations.**\n\nFor dynamic graph learning, the status of each node is usually affected by its interactions with other nodes. Therefore, it is reasonable to learn the dynamic node representations based on their links. More concretely, for the dynamic link prediction task, future links of nodes are naturally influenced by their historical interactions. This allows us to learn from links and derive time-aware representations of source and destination nodes based on the link. For the node classification task, interactions may lead to the state change of nodes (e.g., on Wikipedia and Reddit datasets, the state of a user may change from not banned to banned because of an interaction [1]). Such a phenomenon also indicates that the dynamics of nodes can be well captured when learning from links. It is also worth noticing that many previous works (e.g., JODIE, DyRep, TGAT, TGN, …) learn in the same way as well.\n\n**W2: In section 5.2, the authors say that “some of our findings of baselines differ from previous reports”, but the authors do not point out which baselines’ performance improves. I suggest the authors explicitly list which baselines’ performance improves.**\n\nThe differences between our observations and previous reports on baselines are in two main aspects. \n\n- **The performance of some baselines can be significantly improved by properly setting certain hyperparameters.** Let’s take TGAT for transductive dynamic link prediction with the random negative sampling strategy as an example (see Table 1 in our paper). Compared with the performance in [2], the results of AP are significantly improved on datasets like MOOC (from 0.61 to 0.86), LastFM (from 0.50 to 0.73), Enron (from 0.59 to 0.71), Social Evo. (from 0.76 to 0.93), Flights (from 0.89 to 0.94), and Contact (from 0.58 to 0.96). Similar improvements can also be found on JODIE, DrRep, and TGN on several datasets. This is because we perform the grid search to find the best settings of hyperparameters, which effectively improves the performance of baselines;\n- **Some methods would obtain worse results after we fix the problems or make adaptions in their implementations.** Let’s take CAWN for transductive dynamic link prediction with the random negative sampling strategy as an example (see Table 1 in our paper). Compared with the results in [2], the performance on the AP metric of CAWN drops sharply on datasets like LastFM (from 0.98 to 0.87), Can. Parl. (from 0.94 to 0.70), US Legis. (from 0.97 to 0.71), UN Trade (from 0.97 to 0.65), and UN Vote (from 0.82 to 0.53). This is because [2] uses the problematic implementations of CAWN to conduct experiments and the results will sometimes become worse after fixing the issues. You can also see Sections B.3 and B.4 in the Appendix for more detailed explanations.\n\n[1] Predicting dynamic embedding trajectory in temporal interaction networks, KDD, 2019.\n\n[2] Towards better evaluation for dynamic link prediction, NeurIPS, 2022."}]

Author rebuttal:
We would like to thank all the reviewers for their valuable feedback and helpful comments on our work. We are delighted by the reviewers’ acknowledgments that the proposed DyGFormer is novel with original designs (Reviewers Ea5H, eUNf, ErNY, and zX7A), the presented DyGLib is of high quality and advances the dynamic graph learning field (Reviewers Ea5H, QwYr, and zX7A), the experimental evaluations are rigorous and extensive (Reviewers Ea5H, eUNf, ErNY, and zX7A), and the paper is well-organized and easy to follow (Reviewers Ea5H, QwYr, ErNY, and zX7A).

To the best of our efforts, we have provided thorough responses to address the concerns raised by each reviewer. In particular, the responses mainly consist of:

- Analysis of why the performance of DyGFormer varies across different datasets and various negative sampling strategies;
- Additional introductions and empirical comparisons with the PINT baseline;
- Explanations of the motivation of the neighbor co-occurrence encoding scheme;
- Discussions of several related dynamic graph learning methods based on temporal point process and neural ordinary differential equation;
- Additional evaluations of DyGFormer and baselines on larger datasets;
- Specifications of some details in experimental settings;
- Elaborations of certain experimental analysis.

In summary, all the comments are very valuable for us to improve the quality of this work and we will incorporate them in the revised version of our paper. Once again, we thank the reviewers for their constructive feedback.

Meta-review: This paper introduces an innovative approach to dynamic graph learning through the DyGFormer architecture and DyGLib library. Reviewers appreciate the originality, clear presentation, and significance of the work in addressing an important research problem. DyGFormer's strengths lie in its novel features, such as co-occurrence encoding and patching, which contribute to its efficient handling of long-term node interactions.

Reviewers raise concerns about technical discussions motivating the proposed model's design, as well as the need for clearer explanations and addressing technical issues. However, the value of the contributions, coupled with the extensive evaluation, outweighs these concerns. Acceptance is recommended, and it is encouraged to integrate the reviewers' feedback during the preparation of the final version of the paper.
Decision: Accept (poster)