SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Le Yu, Leilei Sun\* Bowen Du, Weifeng Lv
State Key Laboratory of Software Development Environment
{yule,leileisun,dubowen,lwf}@buaa.edu.cn
We propose DyGFormer, a new Transformer-based architecture for dynamic graph learning. DyGFormer is conceptually simple and only needs to learn from nodes' historical first-hop interactions by: (i) a neighbor co-occurrence encoding scheme that explores the correlations of the source node and destination node based on their historical sequences; (ii) a patching technique that divides each sequence into multiple patches and feeds them to Transformer, allowing the model to effectively and efficiently benefit from longer histories. We also introduce DyGLib, a unified library with standard training pipelines, extensible coding interfaces, and comprehensive evaluating protocols to promote reproducible, scalable, and credible dynamic graph learning research. By performing exhaustive experiments on thirteen datasets for dynamic link prediction and dynamic node classification tasks, we find that DyGFormer achieves state-of-the-art performance on most of the datasets, demonstrating its effectiveness in capturing nodes' correlations and long-term temporal dependencies. Moreover, some results of baselines are inconsistent with previous reports, which may be caused by their diverse but less rigorous implementations, showing the importance of DyGLib. All the used resources are publicly available at https://github.com/yule-BUAA/DyGLib.
Dynamic graphs denote entities as nodes and represent their interactions as links with timestamps [26], which can model many real-world scenarios such as social networks [28, 50, 2], user-item interaction systems [31, 15, 71, 72, 70], traffic networks [67, 63, 19, 4, 69], and physical systems [24, 47, 43]. In recent years, representation learning on dynamic graphs has become a trending research topic [26, 49, 65]. There are two main categories of existing methods: discrete-time [39, 17, 48, 11, 66] and continuous-time [64, 45, 60, 51, 12]. In this paper, we focus on the latter approaches because they are more flexible and effective than the formers and are being increasingly investigated.
Despite the rapid development of dynamic graph learning methods, they still suffer from two limitations. Firstly, most of them independently compute the temporal representations of nodes in an interaction without exploiting nodes' correlations, which are often indicative of future interactions. Moreover, existing methods learn at the interaction level and thus only work for nodes with fewer interactions. When nodes have longer histories, they require sampling strategies to truncate the interactions for feasible calculations of the computationally expensive modules like graph convolutions [55, 64, 45, 36, 9, 59], temporal random walks [60, 25] and sequential models [57, 12]. Though some approaches use memory networks [61, 53] to sequentially process interactions with affordable computational costs [28, 55, 45, 36, 59, 35], they are faced with the vanishing/exploding gradients
due to the usage of recurrent neural networks [40, 57]. Therefore, we conclude that previous methods lack the ability to capture either nodes' correlations or long-term temporal dependencies.
Secondly, the training pipelines of different methods are inconsistent and often lead to poor reproducibility. Also, existing methods are implemented by diverse frameworks (e.g., Pytorch [41], Tensorflow [1], DGL [58], PyG [16], C++, making it time-consuming and difficult for researchers to quickly understand the algorithms and further dive into the core of dynamic graph learning. Although there exist some libraries for dynamic graph learning [18, 46, 74], they mainly focus on dynamic network embedding methods [18], discrete-time graph learning methods [46], or engineering techniques for training on large-scale dynamic graphs [74] (elaborated in Section 2). Currently, we find that there are still no standard tools for continuous-time dynamic graph learning.
In this paper, we aim to address the above drawbacks with two key technical contributions.
We propose a new Transformer-based dynamic graph learning architecture (DyGFormer). DyGFormer is conceptually simple by solely learning from the sequences of nodes' historical first-hop interactions. To be specific, DyGFormer is designed with a neighbor co-occurrence encoding scheme, which encodes the appearing frequencies of each neighbor in the sequences of the source and destination nodes to explicitly explore their correlations. In order to capture long-term temporal dependencies, DyGFormer splits each node's sequence into multiple patches and feeds them to Transformer [56]. This patching technique not only makes the model effectively benefit from longer histories via preserving local temporal proximities, but also efficiently reduces the computational complexity to a constant level that is irrelevant to the input sequence length.
We present a unified continuous-time dynamic graph learning library (DyGLib). DyGLib is an open-source toolkit with standard training pipelines, extensible coding interfaces, and comprehensive evaluating strategies, aiming to foster standard, scalable, and reproducible dynamic graph learning research. DyGLib has integrated a variety of continuous-time dynamic graph learning methods as well as benchmark datasets from various domains. It trains all the methods via the same pipeline to eliminate the influence of different implementations and adopts a modularized design for developers to conveniently incorporate new datasets and algorithms based on their specific requirements. Moreover, DyGLib supports both dynamic link prediction and dynamic node classification tasks with exhaustive evaluating strategies to provide comprehensive comparisons of existing methods.
To evaluate the model performance, we conduct extensive experiments based on DyGLib, including dynamic link prediction under transductive and inductive settings with three negative sampling strategies as well as dynamic node classification. From the results, we observe that: (i) DyGFormer outperforms existing methods on most datasets, demonstrating its superiority in capturing nodes' correlations and long-term temporal dependencies; (ii) some findings of baselines are not in line with previous reports because of their varied pipelines and problematic implementations, which illustrates the necessity of introducing DyGLib. We also provide an in-depth analysis of the neighbor co-occurrence encoding and patching technique for a better understanding of DyGFormer.
Dynamic Graph Learning. Representation learning on dynamic graphs has been widely studied in recent years [26, 49, 65]. Discrete-time methods manually divide the dynamic graph into a sequence of snapshots and apply static graph learning methods on each snapshot, which ignore the temporal order of nodes in each snapshot [39, 17, 48, 11, 66]. In contrast, continuous-time methods directly learn on the whole dynamic graph with temporal graph neural networks [55, 64, 45, 36, 9, 59], memory networks [28, 55, 45, 36, 59, 35], temporal random walks [60, 25] or sequential models [57, 12]. Although insightful, most existing dynamic graph learning methods neglect the correlations between two nodes in an interaction. They also fail to handle nodes with longer interactions due to unaffordable computational costs of complex modules or issues in optimizing models (e.g., the vanishing/exploding gradients). In this paper, we propose a new Transformer-based architecture to show the necessity of capturing nodes' correlations and long-term temporal dependencies, which is achieved by two designs: a neighbor co-occurrence encoding scheme and a patching technique.
Transformer-based Applications in Various Fields. Transformer [56] is an innovative model that employs the self-attention mechanism to handle sequential data, which has been successfully applied in a variety of domains, such as natural language processing [13, 33, 6], computer vision [7, 14, 34]
and time series forecasting [30, 73, 62]. The idea of dividing the original data into patches as inputs of the Transformer has been attempted in some studies. ViT [14] splits an image into multiple patches and feeds the sequence of patches' linear embeddings into a Transformer, which achieves surprisingly good performance on image classification. PatchTST [38] divides a time series into subseries-level patches and calculates the patches by a channel-independent Transformer for long-term multivariate time series forecasting. In this work, we propose a patching technique to learn on dynamic graphs, which can provide our approach with the ability to handle nodes with longer histories.
Graph Learning Library. Currently, there exist many libraries for static graphs [5, 58, 16, 22, 8, 29, 32], but few for dynamic graph learning [18, 46, 74]. DynamicGEM [18] focuses on dynamic graph embedding methods, which just consider the graph topology and cannot leverage node features. PyTorch Geometric Temporal [46] implements discrete-time algorithms for spatiotemporal signal processing and is mainly applicable for nodes with aligned historical observations. TGL [74] trains on large-scale dynamic graphs with some engineering tricks. Though TGL has integrated some continuous-time methods, they are somewhat out-of-date, resulting in the lack of state-of-the-art models. Moreover, TGL is implemented by both PyTorch and  $\mathrm{C + + }$ , which needs additional compilation and increases the usage difficulty. In this paper, we present a unified continuous-time dynamic graph learning library with thorough baselines, diverse datasets, extensible implementations, and comprehensive evaluations to facilitate dynamic graph learning research.
Definition 1. Dynamic Graph. We represent a dynamic graph as a sequence of non-decreasing chronological interactions  $\mathcal{G} = \{(u_1,v_1,t_1),(u_2,v_2,t_2),\dots \}$  with  $0\leq t_{1}\leq t_{2}\leq \dots$  , where  $u_{i},v_{i}\in \mathcal{N}$  denote the source node and destination node of the i-th link at timestamp  $t_i$  .  $\mathcal{N}$  is the set of all the nodes. Each node  $u\in \mathcal{N}$  can be associated with node feature  $\pmb {x}_u\in \mathbb{R}^{d_N}$  , and each interaction  $(u,v,t)$  has link feature  $\pmb{e}_{u,v}^{t}\in \mathbb{R}^{d_{E}}$ $d_N$  and  $d_{E}$  denote the dimensions of the node feature and link feature. If the graph is non-attributed, we simply set the node feature and link feature to zero vectors, i.e.,  $\pmb{x}_u = \mathbf{0}$  and  $\pmb{e}_{u,v}^{t} = \mathbf{0}$
Definition 2. Problem Formalization. Given the source node  $u$ , destination node  $v$ , timestamp  $t$ , and historical interactions before  $t$ , i.e.,  $\{(u', v', t') | t' < t\}$ , representation learning on dynamic graph aims to design a model to learn time-aware representations  $\mathbf{h}_u^t \in \mathbb{R}^d$  and  $\mathbf{h}_v^t \in \mathbb{R}^d$  for  $u$  and  $v$  with  $d$  as the dimension. We validate the effectiveness of the learned representations via two common tasks in dynamic graph learning: (i) dynamic link prediction, which predicts whether  $u$  and  $v$  are connected at  $t$ ; (ii) dynamic node classification, which infers the state of  $u$  or  $v$  at  $t$ .
4 New Architecture and Unified Library
4.1 DyGFormer: Transformer-based Architecture for Dynamic Graph Learning
The framework of our DyGFormer is shown in Figure 1, which employs Transformer [56] as the backbone. Given an interaction  $(u,v,t)$ , we first extract historical first-hop interactions of source node  $u$  and destination node  $v$  before timestamp  $t$  and obtain two interaction sequences  $S_u^t$  and  $S_v^t$ . Next, in addition to computing the encodings of neighbors, links, and time intervals for each sequence, we also encode the frequencies of every neighbor's appearances in both  $S_u^t$  and  $S_v^t$  to exploit the correlations between  $u$  and  $v$ , resulting in four encoding sequences for  $u / v$  in total. Then, we divide each encoding sequence into multiple patches and feed all the patches into a Transformer for capturing long-term temporal dependencies. Finally, the outputs of the Transformer are averaged to derive time-aware representations of  $u$  and  $v$  at timestamp  $t$  (i.e.,  $\pmb{h}_u^t$  and  $\pmb{h}_v^t$ ), which can be applied in various downstream tasks like dynamic link prediction and dynamic node classification.
Learning from Historical First-hop Interactions. Unlike most previous methods that require nodes' historical interactions from multiple hops (e.g., DyRep [55], TGAT [64], TGN [45], CAWN [60]), we only learn from the sequences of nodes' historical first-hop interactions, turning the dynamic graph learning task into a simpler sequence learning problem. Mathematically, given an interaction  $(u,v,t)$ , for source node  $u$  and destination node  $v$ , we obtain the sequences that involve first-hop interactions of  $u$  and  $v$  before timestamp  $t$ , which are denoted by  $\mathcal{S}_u^t = \{(u,u',t') | t' < t\} \cup \{(u',u,t') | t' < t\}$  and  $\mathcal{S}_v^t = \{(v,v',t') | t' < t\} \cup \{(v',v,t') | t' < t\}$ , respectively.
Figure 1: Framework of the proposed model.
Encoding Neighbors, Links, and Time Intervals. For source node  $u$ , we retrieve the features of involved neighbors and links in sequence  $\mathcal{S}_u^t$  based on the given features to represent their encodings, which are denoted by  $\pmb{X}_{u,N}^{t}\in \mathbb{R}^{|\mathcal{S}_{u}^{t}|\times d_{N}}$  and  $\pmb{X}_{u,E}^{t}\in \mathbb{R}^{|\mathcal{S}_{u}^{t}|\times d_{E}}$ . Following [64], we learn the periodic temporal patterns by encoding the time interval  $\Delta t' = t - t'$  via  $\sqrt{\frac{1}{d_T}}$  [cos  $(w_{1}\Delta t')$ , sin  $(w_{1}\Delta t')$ , ..., cos  $(w_{d_T}\Delta t')$ , sin  $(w_{d_T}\Delta t')$ ], where  $w_{1},\dots ,w_{d_{T}}$  are trainable parameters.  $d_{T}$  is the encoding dimension. The time interval encodings of interactions in  $\mathcal{S}_u^t$  is denoted by  $\pmb{X}_{u,T}^{t}\in \mathbb{R}^{|\mathcal{S}_{u}^{t}|\times d_{T}}$ . We use the same process to get the corresponding encodings for destination node  $v$ , i.e.,  $\pmb{X}_{v,N}^{t}\in \mathbb{R}^{|\mathcal{S}_{v}^{t}|\times d_{N}}$ ,  $\pmb{X}_{v,E}^{t}\in \mathbb{R}^{|\mathcal{S}_{v}^{t}|\times d_{E}}$ , and  $\pmb{X}_{v,T}^{t}\in \mathbb{R}^{|\mathcal{S}_{v}^{t}|\times d_{T}}$ .
Neighbor Co-occurrence Encoding Scheme. Existing methods separately compute representations of node  $u$  and  $v$  without modeling their correlations. We present a neighbor co-occurrence encoding scheme to tackle this issue, which assumes the appearing frequency of a neighbor in a sequence indicates its importance, and the occurrences of a neighbor in sequences of  $u$  and  $v$  (i.e., co-occurrence) could reflect the correlations between  $u$  and  $v$ . That is to say, if  $u$  and  $v$  have more common historical neighbors in their sequences, they are more likely to interact in the future.
Formally, for each neighbor in the interaction sequence  $S_{u}^{t}$  and  $S_{v}^{t}$ , we count its occurrences in both  $S_{u}^{t}$  and  $S_{v}^{t}$ , and derive a two-dimensional vector. By packing the vectors of all the neighbors together, we can get the neighbor co-occurrence features for  $u$  and  $v$ , which are represented by  $C_{u}^{t} \in \mathbb{R}^{|S_{u}^{t}| \times 2}$  and  $C_{v}^{t} \in \mathbb{R}^{|S_{v}^{t}| \times 2}$ . For example, suppose the historical neighbors of  $u$  and  $v$  are  $\{a, b, a\}$  and  $\{b, b, a, c\}$ . The appearing frequencies of  $a, b$ , and  $c$  in  $u / v$ 's historical interactions are 2/1, 1/2, and 0/1, respectively. Therefore, the neighbor co-occurrence features of  $u$  and  $v$  are denoted by  $C_{u}^{t} = [[2, 1], [1, 2], [2, 1]]^{\top}$  and  $C_{v}^{t} = [[1, 2], [1, 2], [2, 1], [0, 1]]^{\top}$ . Then, we apply a function  $f(\cdot)$  to encode the neighbor co-occurrence features by
$$
\boldsymbol {X} _ {* , C} ^ {t} = f \left(\boldsymbol {C} _ {*} ^ {t} [:, 0 ]\right) + f \left(\boldsymbol {C} _ {*} ^ {t} [:, 1 ]\right) \in \mathbb {R} ^ {\left| \mathcal {S} _ {*} ^ {t} \right| \times d _ {C}}, \tag {1}
$$
where  $*$  could be  $u$  or  $v$ . The input and output dimensions of  $f(\cdot)$  are 1 and  $d_{C}$ . In this paper, we implement  $f(\cdot)$  by a two-layer perceptron with ReLU activation [37]. It is important to note that the neighbor co-occurrence encoding scheme is general and can be easily integrated into some dynamic graph learning methods for better results. We will demonstrate its generalizability in Section 5.3.
Patching Technique. Instead of focusing on the interaction level, we divide the encoding sequence into multiple non-overlapping patches to break through the bottleneck of existing methods in capturing long-term temporal dependencies. Let  $P$  denote the patch size. Each patch is composed of  $P$  temporally adjacent interactions with flattened encodings and can preserve local temporal proximities. Take the patching of  $X_{u,N}^{t} \in \mathbb{R}^{|S_{u}^{t}| \times d_{N}}$  as an example.  $X_{u,N}^{t}$  will be divided into  $l_{u}^{t} = \lceil \frac{|S_{u}^{t}|}{P} \rceil$  patches in total (note that we will pad  $X_{u,N}^{t}$  if its length  $|S_{u}^{t}|$  cannot be divided by  $P$ ), and the patched encoding is represented by  $M_{u,N}^{t} \in \mathbb{R}^{l_{u}^{t} \times d_{N} \cdot P}$ . Similarly, we can also get the patched encodings  $M_{u,E}^{t} \in \mathbb{R}^{l_{u}^{t} \times d_{E} \cdot P}$ ,  $M_{u,T}^{t} \in \mathbb{R}^{l_{u}^{t} \times d_{T} \cdot P}$ ,  $M_{u,C}^{t} \in \mathbb{R}^{l_{u}^{t} \times d_{C} \cdot P}$ ,  $M_{v,N}^{t} \in \mathbb{R}^{l_{v}^{t} \times d_{N} \cdot P}$ ,  $M_{v,E}^{t} \in \mathbb{R}^{l_{v}^{t} \times d_{E} \cdot P}$ ,  $M_{v,T}^{t} \in \mathbb{R}^{l_{v}^{t} \times d_{T} \cdot P}$ , and  $M_{v,C}^{t} \in \mathbb{R}^{l_{v}^{t} \times d_{C} \cdot P}$ . Note that when  $|S_{u}^{t}|$  becomes
longer, we will correspondingly increase  $P$ , making the number of patches (i.e.,  $l_u^t$  and  $l_v^t$ ) at a constant level to reduce the computational cost.
Transformer Encoder. We first align the patched encodings to the same dimension  $d$  with trainable weight  $\mathbf{W}_{*} \in \mathbb{R}^{d_{*} \cdot P \times d}$  and  $\mathbf{b}_{*} \in \mathbb{R}^{d}$  to obtain  $\mathbf{Z}_{u,*}^{t} \in \mathbb{R}_{u}^{l_{u}^{t} \times d}$  and  $\mathbf{Z}_{v,*}^{t} \in \mathbb{R}_{v}^{l_{v}^{t} \times d}$ , where * could be  $N, E, T$  or  $C$ . To be specific, the alignments are realized by
$$
\boldsymbol {Z} _ {u, *} ^ {t} = \boldsymbol {M} _ {u, *} ^ {t} \boldsymbol {W} _ {*} + \boldsymbol {b} _ {*} \in \mathbb {R} ^ {l _ {u} ^ {t} \times d}, \boldsymbol {Z} _ {v, *} ^ {t} = \boldsymbol {M} _ {v, *} ^ {t} \boldsymbol {W} _ {*} + \boldsymbol {b} _ {*} \in \mathbb {R} ^ {l _ {v} ^ {t} \times d}. \tag {2}
$$
Then, we concatenate the aligned encodings of  $u$  and  $v$ , and get  $\mathbf{Z}_u^t = \mathbf{Z}_{u,N}^t\| \mathbf{Z}_{u,E}^t\| \mathbf{Z}_{u,T}^t\| \mathbf{Z}_{u,C}^t\in \mathbb{R}^{l_u^t\times 4d}$  and  $\mathbf{Z}_v^t = \mathbf{Z}_{v,N}^t\| \mathbf{Z}_{v,E}^t\| \mathbf{Z}_{v,T}^t\| \mathbf{Z}_{v,C}^t\in \mathbb{R}^{l_v^t\times 4d}$ .
Next, we employ a Transformer encoder to capture the temporal dependencies, which is built by stacking  $L$  Multi-head Self-Attention (MSA) and Feed-Forward Network (FFN) blocks. The residual connection [20] is employed after every block. We follow [14] by using GELU [21] instead of ReLU [37] between the two-layer perception in each FFN block and applying Layer Normalization (LN) [3] before each block rather than after. Instead of individually processing  $Z_u^t$  and  $Z_v^t$ , our Transformer encoder takes the stacked  $Z^t = [Z_u^t; Z_v^t] \in \mathbb{R}^{(l_u^t + l_v^t) \times 4d}$  as inputs, aiming to learn the temporal dependencies within and across the sequences of  $u$  and  $v$ . The calculation process is
$$
\text {A t t e n t i o n} \left(\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {V}\right) = \operatorname {S o f t m a x} \left(\frac {\boldsymbol {Q} \boldsymbol {K} ^ {\top}}{\sqrt {d _ {k}}}\right) \boldsymbol {V}, \tag {3}
$$
$$
\operatorname {F F N} \left(\boldsymbol {O}, \boldsymbol {W} _ {1}, \boldsymbol {b} _ {1}, \boldsymbol {W} _ {2}, \boldsymbol {b} _ {2}\right) = \operatorname {G E L U} \left(\boldsymbol {O} \boldsymbol {W} _ {1} + \boldsymbol {b} _ {1}\right) \boldsymbol {W} _ {2} + \boldsymbol {b} _ {2}, \tag {4}
$$
$$
\boldsymbol {O} _ {i} ^ {t, l} = \text {A t t e n t i o n} \left(\ln \left(\boldsymbol {Z} ^ {t, l - 1}\right) \boldsymbol {W} _ {Q, i} ^ {l}, \ln \left(\boldsymbol {Z} ^ {t, l - 1}\right) \boldsymbol {W} _ {K, i} ^ {l}, \ln \left(\boldsymbol {Z} ^ {t, l - 1}\right) \boldsymbol {W} _ {V, i} ^ {l}\right), \tag {5}
$$
$$
\boldsymbol {O} ^ {t, l} = \operatorname {M S A} \left(\boldsymbol {Z} ^ {t, l - 1}\right) + \boldsymbol {Z} ^ {t, l - 1} = \left(\boldsymbol {O} _ {1} ^ {t, l} \| \dots \| \boldsymbol {O} _ {I} ^ {t, l}\right) \boldsymbol {W} _ {O} ^ {l} + \boldsymbol {Z} ^ {t, l - 1}, \tag {6}
$$
$$
\boldsymbol {Z} ^ {t, l} = \operatorname {F F N} \left(\ln \left(\boldsymbol {O} ^ {t, l}\right), \boldsymbol {W} _ {1} ^ {l}, \boldsymbol {b} _ {1} ^ {l}, \boldsymbol {W} _ {2} ^ {l}, \boldsymbol {b} _ {2} ^ {l}\right) + \boldsymbol {O} ^ {t, l}. \tag {7}
$$
$\pmb{W}_{Q,i}^{l}\in \mathbb{R}^{4d\times d_{k}}$ $\pmb{W}_{K,i}^{l}\in \mathbb{R}^{4d\times d_{k}}$ $\pmb{W}_{V,i}^{l}\in \mathbb{R}^{4d\times d_{v}}$ $\pmb{W}_0^l\in \mathbb{R}^{I\cdot d_v\times 4d}$ $W_{1}^{l}\in \mathbb{R}^{4d\times 16d}$ $b_{1}^{l}\in \mathbb{R}^{16d}$ $W_{2}^{l}\in \mathbb{R}^{16d\times 4d}$  and  $b_{2}^{l}\in \mathbb{R}^{4d}$  are trainable parameters at the  $l$  -th layer. We set  $d_{k} = d_{v} = 4d / I$  with  $I$  as the number of attention heads. The input of the first layer is  $\pmb {Z}^{t,0} = \pmb {Z}^t\in \mathbb{R}^{(l_u^t +l_v^t)\times 4d}$  , and the output of the  $L$  -th layer is denoted by  $\pmb {H}^t = \pmb {Z}^{t,L}\in \mathbb{R}^{(l_u^t +l_v^t)\times 4d}$
Time-aware Node Representation. The time-aware representations of node  $u$  and  $v$  at timestamp  $t$  are derived by averaging their related representations in  $H^t$  with an output layer,
$$
\boldsymbol {h} _ {u} ^ {t} = \operatorname {M E A N} \left(\boldsymbol {H} ^ {t} [ l _ {u} ^ {t},: ]\right) \boldsymbol {W} _ {\text {o u t}} + \boldsymbol {b} _ {\text {o u t}} \in \mathbb {R} ^ {d _ {\text {o u t}}}, \tag {8}
$$
$$
\pmb {h} _ {v} ^ {t} = \mathrm {M E A N} \left(\pmb {H} ^ {t} [ l _ {u} ^ {t}: l _ {u} ^ {t} + l _ {v} ^ {t},: ]\right) \pmb {W} _ {o u t} + \pmb {b} _ {o u t} \in \mathbb {R} ^ {d _ {o u t}},
$$
where  $\pmb{W}_{out} \in \mathbb{R}^{4d \times d_{out}}$  and  $\pmb{b}_{out} \in \mathbb{R}^{d_{out}}$  are trainable weights with  $d_{out}$  as the output dimension.
4.2 DyGLib: Unified Library for Continuous-Time Dynamic Graph Learning
We introduce a unified library with standard training pipelines, extensible coding interfaces, and comprehensive evaluating strategies for reproducible, scalable, and credible continuous-time dynamic graph learning research. The overall procedure of DyGLib is shown in Figure 3 in Section A.3.
Standard Training Pipelines. To eliminate the influence of different training pipelines in previous studies, we unify the data format, create a customized data loader, and train all the methods with the same model trainers. Our standard training pipelines guarantee reproducible performance and enable users to quickly identify the key components of different models. Researchers only need to focus on designing the model architecture without considering other irrelevant implementation details.