Todyformer: Towards Holistic Dynamic Graph Transformers with Structure-Aware Tokenization
Authors: Mahdi Biparva
Raika Karimi
Faezeh Faez
Yingxue Zhang
Venue: Accepted by TMLR ()

Abstract: Temporal Graph Neural Networks have garnered substantial attention for their capacity to model evolving structural and temporal patterns while exhibiting impressive performance. However, it is known that these architectures are encumbered by issues that constrain their performance, such as over-squashing and over-smoothing. Meanwhile, Transformers have demonstrated exceptional computational capacity to effectively address challenges related to long-range dependencies. Consequently, we introduce Todyformer—a novel Transformer-based neural network tailored for dynamic graphs. It unifies the local encoding capacity of Message-Passing Neural Networks (MPNNs) with the global encoding of Transformers through i) a novel patchifying paradigm for dynamic graphs to improve over-squashing, ii) a structure-aware parametric tokenization strategy leveraging MPNNs, iii) a Transformer with temporal positional-encoding to capture long-range dependencies, and iv) an encoding architecture that alternates between local and global contextualization, mitigating over-smoothing in MPNNs. Experimental evaluations on public benchmark datasets demonstrate that Todyformer consistently outperforms the state-of-the-art methods for downstream tasks. Furthermore, we illustrate the underlying aspects of the proposed model in effectively capturing extensive temporal dependencies in dynamic graphs.

Published in Transactions on Machine Learning Research (06/2024)
Todyformer: Towards Holistic Dynamic Graph Transformers with Structure-Aware Tokenization
Mahdi Biparval
mahdi mabdi.biprra@hhauee..com
Huowei Noah's s Ark Lab
Raika Karimi
makuukariim@@hhuuusss.coom
Huawei Noah's 's Ark Lab
Faezeh Faez
facach.facz2hhuweek.com
Huawei Noah's Ark
Yingxue Zhang
Huawei Noah's Ark
Reviewed on OpenReview: https: // openreview. net/ forum? id- nAQSUqEspb
Abstract
Temporal Graph Neural Networks have garnered substantial attention for their capacity to model evolving structural and temporal patterns while exhibiting impressive performance. However, it is known that these architectures encumbered by issues that constrain their performance, such as over-squashing and over-smoothing. Meanwhile, Transformers have demonstrated exceptional computational capacity to effectively address challenges related to long-range dependencies. Consequently, we introduce Todyformer a novel Transformer- based neural network tailored for dynamic graphs. unifies the local encoding capacity of Message- paassagggasinng Neural Networks (MPNNs) with the global encoding of Transformers through a novel patchifying paradigm for dynamic graphs to improve over-squashing, ii) structure-aware parametric tokenization strategy leveraging MPNNs, iíi) Transformer with temporal positional-encodiig to capture longerange dependencies, and iv) an encod- ing architecture that alternates between local and global contextualization, mitigating over- smoothing in MPNNs. Experimental evaluations on public benchmark datasets demonstrate that Todyformer consistently outperforms the state-of-thc-art methods for downstream lasks. Furthermore, we illustrate the underlying aspects of the proposed model in effectively capturing extensive temporal dependencies dynamic graphs. The code is publicly avail- atlla
able at hhttps://githnn http:///gggtuuu..cmmaaaaaa..iiie http:/hwwweinoohhhhee noah
1 Introduction
Dynamic graphs, driven by the surge of large-scale structured data on the internet, have become pivotal in graph representation learning. Dynamic graphs are simply static graphs where edges have time attributes Kazemi et al. 2020). Representation learning approaches for dynamic graphs aim to learn how to effectively encode recurring structural and temporal patterns for node-level downstream tasks. For instance, Future Link Prediction (FLP) uses past interactions to predict future links, while Dynamic Node Classification (DNC) focuses on predicting labels of upcoming nodes based on impending interactions {Xu et al. 2019a
While models based on
Neural Networks (MPNN) (Gilmer et al. 2017 Luo et al. 2021 have
demonst. impressive performance on encoding dy danii graphs (Rossi el al. 2020 Wang et al. 202 Jin
et al. 2022 Luo & Li 2022), many approaches have notable limitations. Primarll, these methods often rely
"equal contribution.

---

Published in Transactions on Machine Learning Research (06/2024)
heavily chronological training or use complex memory modules for predictions (Kumar et al.
2019 Xu
et al. 2020: Rossi et al. 2020 Wang et al. (001), leading to significant computational overhead, especially for dynamic graphs with many edges. Additionally, the use of inefficient message- message-passing procedures can be problematic, and some methods depend on computationally expensive random- saoooo-wwallbaee basek algorithms (Wang
et al. 2021 Jin al. 2022]. These methods often require heuristic feature engineering, which is specifically tailored for edgo-level tasks.
Moreover, there is growing consensus within the community that the message-passingg paradigm is in- herently constrained by the hard inductive biases imposed by the graph structure (Kreuzer et al. 2021 A central concern with conventional MPNNs revolves around the ovcr-smoothing problem stemmed from the exponential growth of the model's computation graph (Dwivedi & Bresson 2020). This issue becomes pronouncdd when the model attempts to capture the higher-order long-range aspects of the graph structure. Over-smoothing hurts model expressiveness in MPNNs where the network depth grows in an attempt to increase expressiveness. However, the node embeddings tend to converge towards constant uninformative representation. This serves as a reminder of the lack of flexibility observed in early recurrent neural networks used in Natural Language Processing (NLP), especially when encoding lengthy sentences or attempting to capture long-range dependencies within soquences (Hochreiter & Schmidhhber
1997). However, Transform-
ers have mitigated these limitations in various data modalities (Vaswani et al. 2017 Devlin et al. 2018 Liu
et al. 2021 Dosovitskiy et al. 2020; Dwivedi & Bresson 2020). Over-squashing is another problem that
message-passingg networks suffer from since the amount of local information aggregated repeatedly increases proportionally with the number of edges and nodes (Hamilton 2020: Topping et al. 2021).
MPNN-based models for learning dynamic graphs do not deviate from aforementioned drawbacks. To address these challenges, propose Todyforme novel Graph Transformer model on dynamic graphs that unifies the local and global message-passing paradigms by introducing patchifying, tokenization, and encoding modules that collectively aim to improve model expressiveness through alleviating over-squashing and over- smoothing in a systematic manner. To mitigate the neighborhood explosion (i.c, over-squashing), we employ lemporal-order-prest preserving patch generation, a mechanism that divides large dynamic graphs into smaller dynamic subgraphs. This approach breaks the larger context into smaller subgraphs suitable for local cessage-passingg instead of relying on the model to directly analyze the granular and abundant features of large dynamic graphs.
Moreover, we adopt a hybrid approach to successfully encode the long-ten term- contextual information, where we use MPNNs for tasks they excel in, encoding local information, while transformers handle distant contextual dependencies. In other words, our proposed architecture adopts the concept of learnable structure-aware tokenization, reminiscent of the Vision Transformer (ViT) paradigm (Dosovitskiy al. 2020), to mitigate computational overhead. Considering the various contributions of this architecture, Todyformer dynami- cally alternates between encoding local and global contexts, particularly when capturing information for anchor nodes. This balances between the local and global computational workload and augments the model expressiveness through the successive stacking of the MPNN and Transformer modules.
2 Related Work
Representation lcarning for dynamic graphs: Recently the application of machine learning to
Continuous- Time Dynamic Graphs (CTDG) has drawn the altention of the graph community (Kazemi et al. 2020p. RNN-baseed methods such as JODIE (Divakaran & Mohan 2020), Know-E (Trivedi et al. 2017], and DyRep (Trivedi et al. 2019) typically update node embeddings scquentially as new edges ar- rive. STAR (Xu et al. (019b) is one of the early works that uses attentive approach for temporal graphs. TGAT (Xu et al. 2020). akin to GraphSAGE (Hamilton et al. 2017 and GAT (Veličković et al. 20118). uses attention-basee message- paagggg-asinnggggee to aggregate messages from historical neighbors of an anchor node. TGN [Rossi et al. 2020) augments the message- paggge-ppsnnggggg with an RNN- memory module that stores the history of all nodes with a memory overhead. CAW (Wang et al. 2021 and NeurTWs (Jin et al., 2022) abandon the common messago-passing paradigm by extracting temporal features from temporally-samphe causal walks.
TRRN (Xu et al. 2021) is another memory-based model that benefit from self-attention to reason over set of
We are going open-source the cade upon acceptance.
2

---

Published in Transactions on Machine Learning Research (06/2024)
memories. CAW operates directly within link streams and mandates the retention of the most recent links, eliminating the need for extensive memory storage. Moreover, /Sozaa et al. (2022) investigates the theoreti- cal underpiunings regarding the representational power of dynamic encoders based on message- message-passing and temporal random walks. DyG2Voc (Alomrani et al. 2022) proposes an efficient attention- basod encoder- decoder MPNN that leverages temporal edge encoding and window-based subgraph sampling to regularize the representation learning for task-agnostic node embeddings. GraphMixer (Cong et al., (023) simplifies the design of dynamic GNNs by employing fixed-time encoding functions and leveraging the MLP-Mixer architecture [Tossiikhin et al. 2021
Graph Transformers: Transformers have been demonstrating remarkable efficacy across diverse data modalities (Vaswani et al. 2017 Dosovitskiy et al. 2020j. The graph community has recently started to embrace them in various ways (Dwivedi & Bresson 2020). Graph-BERT (Zhang et al.) 2020) avoids message-
passing by mixing up global and relative scales of positional encoding. KKeezer et al. (2021) proposes a refined inductive bias for Graph Transformers by introducing a soft and learnable positional encoding (PE) rooted in the graph Laplacian domain, signifying a substantiive stride in encoding low-level graph structural intricacies. Ying et al.] (2021) is provably more powerful than 1-WL; it abandons Laplacian PE in favor of spatial and node centrality PEs. Subsequently, SAT (Chen et al. 2022) argues that Transformers with PE do not necessarily capture structural properties. Therefore, the paper proposes applying GNNs to obtain initial node representations. Graph GPS (Rampášek et al. 2022) provides a recipe to build scalable Graph Transformers, leveraging structural and positional encoding where MPNNs and Transformers are jointly utilized to address over-smoot,ing, similar to SAT. TokenGT (Kim et al. 2022) demonstrates that standard Transformers, without graph-specific modifications, can yield promising results in graph learning. It treats nodes and edges independent tokens and augments them with token-wisc embeddings to capture the graph structure. He et al. (2023) adapts MLP-Mixer (Tolstikhin et al. 2021] architectures to graphs, partitioning the input graph into patches, applying GNNs to each patch, and fusing their information while considering both node and patch PEs.
While the literature adapts Transformers to static graphs, a lack of atlention is eminent on dynamic graphs. In this work, we strive to shed light on such adaptation in a principled manner and reveal how dynamic graphs can naturally benefit from a unified local and global encoding paradigm.
3 Todyformer: Tokenized Dynamic Graph Transformer
We begin this section by presenting the problem formulation of this work. Next, we provide the method- ological details of the Todyformer architecture along with its different modules.
Problem Formulation: Continuous- Time Dynamic Graph (CTDG) 9 = (V, x²) with N = | nodes and E' |8| edges can be represented as a sequence of interactions E = {er, e2.., egp, where X" E N and X E Ex are the node and edge features, respectively. DV and DE are the dimensions of the node and edge feature space, respectively. An edge e, = (u,, Ui, t4, m.) links two nodes u4 Us E V at a continuous timestamp & E R. where m, E X an edge feature vector. Without loss of generality, we assume that the edges are undirected and ordered by time (i.e., 44 S 441)). A temporal sub-graph Gi is defined a set consisting of all the edges in the interval [t;,4y] such that Egg (ex 14 S l < 4,)- Any two nodes can interact multiple times throughout the time horizon; therefore, G is a multi-graph. Following DyG2Vec (Alomrani et al. 2022), we adopt a window-based encoding paradigm for dynamic graphs to control representation learning and balance the trade-off bet efficiency and accuracy according to the charactcristics of the input data domain. The parameter W controls the size of the window for the input graph Gij, where j = i + W. For notation brevity, we assume the window mechanism is implicit from the context. Hence, we use G as the input graph unless explicit clarification is needed.
Based on the downstream task, the objective is to learn the weight parameters 0 and y of dynamic graph encoder ENCODER, and decoder DECODER, respectively. ENCODER, projects the input graph G DECODER to the node embeddings H E N capturing temporal and structural dynamics for the nodes. Mcanwhile, a decoder DECODER, outputs the predictions given the node embeddings for the downstream task, enabling accurate future predictions based on past. interactions. More specifically:
3

---

Published in Transactions on Machine Learning Research (06/2024)
Encoder
Psr
Fic-2
Local Encoder
Global Encoder
Loss Function
Patchifying
Decoder
Pa
Positional Encoding
Figure 1: Illustration of Todyformer encoding-decoding architecture, Before being fed into the Encoder, the temporal graph is segmented into M non-overlapping subgraphs (patches) based on the timestamps of the edges, with each subgraph containing an equal number of edges. This partitioning process divides the overall time interval into M segments such that W Ex=r At,. Here, W denotes the window size, P, represents the j-th patch, n, denotes the i-th node the graph, and L represents the number of encoder blocks.
H = ENCODER&(
Z DECODER, (H),
(1)
Here, Z represents predictions for the ground-truth labels. In this work, we focus on common downsst ream tasks defined similarly to Alomrani (2022) for training and evaluation: i) Future Link Prediction (FLP) and ii) Dynamic Node Classification (DNC).
3.1 Encoder Architecture
Todyformer consists of blocks of encoding ENCODER, [LLOcaLENCODER' GlObaLLENCDDrR")) Ho where LoCALLNCODDRR = [LooCALExCODER'] and GL.OBALENCODER = (GI.OBAL.NCODDR) H are the sets of local and global encoding modules, respectively. As illustrated in Figure | the encoding network of Todyformer benefits from alternating architecture that alternates between local and global message- passing. The local encoding is structural and temporal, according to the learnable local encoder, and the global encoding in this work is defined to be temporal. We leave the structural and temporal global encoding for future work. In the following, we define each encoding module in more detail.
3.2 Patch Generation
Inspired by Dosovitskiy et al. 2020). Todyformer begins by partitioning a graph into M subgraphs, each con- laining an equal number of edges. This partitioning performed based on the timestamp associated with each edge. Specifically, the patchifier PATCHIFIER evenly segments the input graph G with & = e2.,, odges into M non-overlapping subgraphs of equal size, referred as patches. More concretely:
P PATCHIFIERGG:AM
(2)
where (9m E f1,2.... }1 and the m-th graph, denoted Gm, consists of edges with indices in the E range {mm - 1) M + 1....... m M Partitioning the input graph into M disjoint subgraphs helps message- passing to be completely scparated within each patch. Additionally, M manages the trado-off between alleviating wvurssqush and maintaining the local encoder's expressiveness. Through ablation studies, we empirically reveal how different datasets react to various M values.
4

---

Published in Transactions on Machine Learning Research (06/2024)
Local Encoder
Rg
Ry
Global Encoder
Rag
R2
11
Tokenleer
Node Packing
71
Transfomerr
11:
0.0.0.
Node Unpacking
Ry
Ry
Figure 2: Schematic depiction of the computation flow in the local and global encoding modules, particularly highlighting node packing and unpacking modules in Todyformer.
3.3 Local Encoding: Structure-Awan Tokenization
Local encoding LOCALENCODEER (ToKENIZER', PACKER') contains two modules: the tokenization TOKENIZER' and the packing PACKER modules. The former handles local tokenization, and the latter packs tokens into a sequential data structure that will be consumed by the global encoder.
Atrecture-Aware Tokenization: Following the recent trend in Graph Transformers, where tokenization is structure-aware, local encoding in Todyformer utilizes a dynamic GNN to map the input node embeddings the latent embeddings that Transformer will process later on. It should be noted that the local encoder has learnable parameters encode both structural and temporal patterns in the patches. Without loss of generality, we use DyG2Vec (Alomrani et al. 2022) a powerful attentive messago-passing model to locally encode input features into semantically meaninglul node tokens.
H TOKENIZER (FC
where H = (H is the set of node embeddings H; for M different patches, and Hi- is the set of node embeddings computed by the previous block. As illustrated in Figure [ the output of one block from the global encoder is transferred as the input into the local encoder of the subsequent block. should noted that 7. = X for the first layer, where = / M-1 the set of node features for all patches.
Once the node features are locally encoded into node tokens, the next step is pack the set of node embeddings H' into the standard format required by Transformers. Since a node may appear in multiple patches, to collect all the node embeddings for particular node across the patches, node node-packing module PACKER' is utilized. This module collects the embeddings of all nodes across the patches and arranges them in a sequential data format as follows:
Packing:
H' PACKETI P)
where H' E N such that N is the number of nodes in the input graph G, M the total number of patches, and DH is the dimension of the embedding space. The module PACKER' uses P to figure out which patches a node belongs to. Consequently, the output of the local encoding module is structured in tensor that can be easily consumed by a Transformer. The computation flow in the local encoder is shown in Figure Since nodes may have interactions for a variable number of times in the input graph, it necessary to pad the short sequences with the [MASK] lokens at the end. Then, the mini-batch of sequences can be easily packed into dense tensor and fed as input Transformers.
2
3.4 Global Encoding
The packed node tokens will be fed into the global encoding module perform long-range message- passing beyond the local context of the input patches. Therefore, Todyformer not only maxinizes
5

---

Published in Transactions on Machine Learning Research (06/2024)
the parametric capacity of MPNNs to encode local context but also leverages the long-range capaci- ties of Transformers to improve the model expressiveness. The global encoder GIOOBA..NNCDDER [POSTTIONALENCOEE) TRANS', UNPACKER') consists of the positional encoder POSITIONALENCODER Transformer TRANS', and unpacking module UNPACKER' according to the details provided in the following.
Positional Encoding: Transformers are aware of the ordering in the input sequences through positional encoding. Various systematic approaches have been investigated in the literature for the sake of improved expressiveness (Dwivedi & Bresson 2020 Kreuzer et al. 2021). Once the structural and temporal features are locally mapped into node tokens, and the sequential imput H' is packed at layer I, positional encoding is needed to inform the Transformer of the temporal ordering of the node tokens on global scale. The positional encoding in Todyformer is defined based on the notion of the position and the encoding function. The position can be explicitly defined as the global edge index of a node upon an interaction at a timestamp or implicitly defined as the patch or occurrence indices. The encoding function can be linear or sinusoidal mapping. The positional encoding P fused into the packed node emboddings through the addition modulation, as follows:
H' H
== POSITIONALENCODER''O E NXMxD
(5)
The global encoding updates node embeddings through a dot-product Multi-head Self- Attention (MSA) Transformer architecture follows: The global encoder computes contextualized em- bedding for each node based on the represenlation learned for that node across different patches. This is accomplished by utilizing Transformer architecture as follows:
Transformer:
QKT VTg
H' TRANS' TRANS' Transformer (Q, = softmax
)
(6)
where Q = H'W, E RNNMXD K- H' E E and V= H'W E R-NMXDD are the query, key, and value, respoctively, and Wg, Wa E xD and W, E RD are learnable matricos, which are shared among all the graph nodes. We apply an attention mask enforce directed connectivity between node tokens through time, where node token from the past is connected to all others in the future. The Transformer module is expected to learn temporal inductive biases from the context how to deploy attention on recent interactions versus carly ones.
For intermediate blocks, the unpacking module UNPACKER is necessary to transform the packed, unstructured sequential node embeddings back into the structured counterparts that can be processed alternately by the local encoder of the next block. It is worth mentioning that the last block does not require an unpacking module. Instead, readout function READOUT is defined to return the final node embeddings consumed by the task-specific decoding head:
Unpacking:
H = UNNAACER'(EA'
= READOUT(H" NXD" ENDD
(7)
where 76' (- is the set of node embeddings H, for M different patches, READOUT is the readout function, and " is the dimension of the output node embeddings. The readout function is defined to be MAX, MEAN, or LAST pooling layer.
3.5 Improving Over SwortSmoothing by Alternating Architecture
Over-smoothing is critical problem in graph representation learning, where MPNNs fall short in encoding long-range dependencies beyond a few layers of message-passing. This issue is magnified in dynamic graphs when temporal long-range dependencies intersect with structural patterns. MPNNs typically fall into the wver-smoothing regime beyond a few layers (e.g., 3), which may not be sufficient to capture long-range temporal dynamics. We propose to address this problem by letting the Transformer widen up the temporal contextual node-wise scope beyond a few hops an alternating manner. For instance, a 3-layer MPNN encoder can reach patterns up to 9 hops away in Todyformer.

---

Published in Transactions on Machine Learning Research (06/2024)
Table 1: Future Link Prediction performance in AP (Mean + Std) on the test set.
Setting
Model
MOOC
LastFM
Enron
UCI
SocialEvol.
JODIE
0.797 1 0.01
0.691 + 0.010
0.785 1 0.020
0.869 1: 0.010
0.847 4: 0.014
Transductive
DyRep
0.840 + 0.004
0.683 + 0.033
0.795 + 0.042
0.524 + 0.076
0.885 0.004
TGAT
0.793 + 0.006
0.633 + 0.002
0.637 + 0.002
0.835 + 0.003
0.631 + 0.001
TGN
0.911 + 0.010
0.743 + 0.030
0.866 + 0.006
0.843 + 0.090
0.966 + 0.00
CaW
0.940 0.014
0.903 1e-4
0.970 + 0.001
0.939 + 0.008
0.947 + le-4
NAT
0.874 0.004
0.859 :: 1c-4
0.924 + 0.001
0.944 + 0.002
0.944 0.010
GraphMixer
0,835 = 0,001
0.862 + 0.003
0.824 + 0.001
0.932 + 0.006
0.935 4 3e-4
Dygformer
0.892 = 0.005
0.901 + 0.003
0.926 + 0.001
0.959 + 0.001
0.952 = 2c-4
DyG2Vec
0.980 = 0.002
0.960 1c-4
0.991 + 0.001
+: 0.007
0.987 2e4
Todyformer
0.992 - 7e-4
0.976 + 3e-4
0.995 + 6e-4
0.994 4e-4
0.992 + le-4
JODIE
0.707 1 0.029
0.865 1 0.03
0.747 1 0.041
0.753 1 0.011
0.791 1 0.031
Inductive
DyRep
0.723 = 0.009
0.869 + 0.015
0.666 + 0.059
0.437 + 0.021
0.901 + 3e-1
TGAT
0.805 = 0.006
0.644 = 0.002
0.693 + 0.004
0.820 + 0.005
0.632 0.005
TGN
0.855 + 0.014
0.789 = 0.050
0.746 + 0.013
0.791 + 0.057
0.904 + 0.003
CaW
0.933 + 0.014
0.890 + 0.001
0.962 + 0.001
0.931 + 0.002
0.950 + 1e-4
NAT
0.832 1 1c-4
0.878 + 0.003
0.949 + 0.010
0.926 + 0.010
0.952 0.006
GraphMixer
0.814 = 0.002
0.821 + 0.004
0.758 + 0.004
0.911 + 0.004
0.918 - 6c-4
Dygformer
0,869 + 0,004
0.942 + 9c-4
0.897 + 0.003
0.945 + 0.001
0.931 + 4c-4
DyG2Vee
0.938 + 0.010
0.979 + 0.006
0.987 + 0.004
0.976 +: 0.002
0.978 0.010
Todyformer
0.948 + 0.009
0.981 + 0.005
0.989 1 8
0.983 1 0.002
0.9821 1 0.005
4 Experimental Evaluation
In this section, we evaluate the generalization performance of Todyformer through a rigorous empirical assessment spanning wide range of benchmark datasets across downstream tasks. First, the experimental sctup is explained, and comparison with the stato-of-tho-art (SoTA) on dynamic graphs is provided. Next, the quantitative results are presented. Later, in-depth comparative analysis and ablation studies are provided to further highlight the role of the design choices in this work.
4.1 Experimental Setup
The performance of Todyformer is compared with a wide spectrum of dynamic graph encoders, ranging from mandom-walle based to attentive memory-basee approaches: DyRep (Trivedi et al., 2019
Baselines:
JODIE (Kumar et al. 2019). TGAT (Xu et al. 2020). , TGN (Rossi et al. 2020), CaW (Wang et al. 2021
NAT [Luo & Li 2022), and DyG2Vec (Alomrani et al. 2022). CAW samples temporal random walks and learns temporal motifs by counting node occurrences in each walk. NAT constructs temporal node represen- tations using a cache that stores limited set of historical interactions for each node. DyG2Vec introduces window-based MPNN that attentively aggregates messages in a window of recent interactions. Recently, GraphMixer (Cong et al. 2023) has been presented simple yet effective MIL--Mixe-baseed dynamic graph encoder. Dygformer (Yu et al. 2023) also presents Transformer architecture that encodes the one-hop node neighborhoods. In the Dygformer framework, patchifying entails the application of historical interaction analysis to a specific node's context. In other words, the interaction history of a node within the one-hop context is retrieved and encoded using Transformers. proach, our methodology involves the extraction of subgraphs and subscquent higher-order messago-passing within a patch, without strict emphasis on individual one-hop node neighborhoods. Consequently, our ap- proach to patchifying and tokenization operates at the graph level where the input to Transformers is the encoded node embedding returned from the local encoding. This process ensures that the embeddings gener- ated from MPNN are updated utilizing message-passing information from all nodes within the corresponding subgraphs.
contrast to Dygformer's node-level ap-
Downstream Tasks: We evaluate all models on both FLP and DNC. In FLP, the goal is to predict the probability of future edges occurring given the source and destination nodes, and the timestamp. For each positive edge, we sample a negative edge that. the model is trained to predict as negative. The DNC task
7

---

Published in Transactions on Machine Learning Research (06/2024)
Table 2: Future Link Prediction performance on the test set of TGBL datasets measured using Mean
Reciprocal Rank (MRR). The baseline results are directly taken from Huang et al. (2023)
Model
Wiki
Review
Coin
Comment
Flight
Avg. Rank
Dyrep
0.366 = 0.014
0.367 + 0.013
0.452 + 0.046
0.289 + 0.033
0.556 + 0.014
44
TGN
0.721 + 0.004
0.532 + 0.020
0.586 + 0.037
0.379 + 0.021
0.705 + 0.020
2
CAW
0.791 1: 0.015
0.194 1: 0.004
00
00
00M
4.4
TCL,
0.712 + 0.007
0.200
+ 0.010
00M
00M
00M
4.8
GraphMixer
0.701 1 0.014
0.514 + 0,020
OOM
OOM
00M
4.4
EdgeBank Todyformer
0.641
0.0836
0.1494
0.364
0.580
4,6
0.7738 + 0.004
0.5104 + 86e-4
0.689 + 18e-4
0.762 + 98e-4
0.777 + 0.014
1.6
involves predicting the ground-truth label of the source node of a future interaction. Both tasks are trained using the binary cross-entropy loss function. For FLP, we evaluate all models in both transductive and inductive scttings. The latter is more challenging setting where model makes predictions nodes. The Average Precision (AP) and the Area Under the Curve (AUC) metrics are reported for the FLP and DNC tasks, respectively. DNC is evaluated using AUC due to the class imbalance issue.
unseen
Datasets: In the first set of experiments, we use 5 real-world datasets for FLP: MOOC, and LastFM (Kumar et al.) 2019); SocialEvolution, Enron, and UCI (Wang et al. 2021). Three real-world datasets including Wikipedia, Reddit, MOOC (Kumar et al., 2019) are used for DNC as well. These datasets span a wide range of the number of nodes and interactions, timestamp ranges, and repetition ratios. The dataset statistics are presented in Appendix
A.2 We employ the same 70%-15%--5%% chronological split for all datasets, 2021 The datasets split differently under two settings: Transductive and
outlined in Waag et al.
Inductive. All benchmark datascts are publicly available. We follow similar experimental setups to Alomrani et al. )022);; Wang et al. 2021) on these datasets to split them into training, validation, and lest sets under the transductive and inductive settings.
In the second set of experiments, we evaluate Todyformer on the Temporal Graph Benchmark for link pre- diction datasets (TGBL) (Huang al. 2023). The goal is to target large-scale and real-world experimental setups with a higher number of negative samples generated based on two policies: random and historical. The deliberate inclusion of such negative edges aims to address the substantial bias inherent in negative sampling techniques, which can significantly affect model performance. Among the five datasets, three are extra-large-scale, where model training on a regular setup may take weeks of processing. We follow the experimental setups similar to Huang et al. (0023 to evaluate our model on TGBL (e.g., number of trials or negative sampling).
Todyformer has large number of hyperparameters to investigate. There are common design choices, such as activation layers, normalization layers, and skip connections that we assumed the results are less sensitive to in order to reduce the total number of trials. We chose L = 3 for the number of blocks in the encoder. The GNN and Transformers have three and two layers, respectively. The neighbor sampler in the local encoder uniformly samples (64, 1) number of neighbors for 3 hops. The model employs uniform sampling within the window instead of selecting the latest. N neighbors of a node (Xu el. al. 22000] Rossi al. 2020). For the DNC task, following prior work by Rossi et al. (0200) the decoder is trained on top of the frozen encoder pre-trained on FLP.
Model Hyperparameters:
4.2 Experimental Results
We present a comparative analysis of AP scores on the test set for future link prediction (both transductive and inductive) across several baselines in Table ] Notably, a substantial performance gap is evident in the transductive setting, with Todyformer outperforming the second-best model by margins exceeding 1.2%, 1.6%, 0.6%, and 0.5% on the MOOC, LastFM, UCI, and SvoialEvolve datasets, respectively. Despite the large scale of the SocialEvolve dataset with around 2 million edges, our model achieves SoTA performance on this dataset. This observation reinforces the conclusions drawn in Xu ct al. (2020), emphasizing the pivotal role played by recent temporal links in the future link prediction task. Within the inductive setting, Todyformer continues to exhibit superior performance across all datasets. The challenge posed by predicting links over unseen nodes impacts the overall performance of most methods. However, Todyformer consistently outperforms the basclines' results on all datasets in Table [ These empirical results support the hypothesis that model expressiveness has significantly improved while enhancing the generalization under the two experimental settings. Additionally, Todyformer outperforms the two latest
Future Link Prediction:
8

---

Published in Transactions on Machine Learning Research (06/2024)
Table 3: Dynamic Node Classification performance in AUC (Mean + Std) on the test set. Avg. Rank reports the mean rank of a method across all datasets.
Model
Wikiiedia
Reddit
MOOC
Arg. Rank
TGAT
0.800 1 0.010
0.664 1: 0.009
0.673 1 0.006
3.6
JODIE
0.843 0.003
0.566 1 0.016
0.672 1 0.002
4.6
Dyrep
0.873 0.002
0.633 1: 0.008
0.661 1 0.012
TGN
0.828 0.004
0.655 1 0.009
0.674 0.007
3.3
VeG2
0.824 + 0.050
0.649 + 0.020
0.785 + 0.005
3.3
Todyformer
0.861 + 0.017
0.656 + 0.005
0.743 + 0.009
SoTA methods, namely GraphMixer (Cong et al. 2023) and Dygformer Yu al.) 2023). The results further validate that dynamic graphs require encoding of long-range dependencies that cannot be simply represented by short-range onc-hop neighborhoods. This verifies that multi-scale encoders like Todyformer are capable of learning inductive biases across various domains.
Additionally, the performance of Todyformer on two small and three large TGBL datasets is presented in Table 2 On extra-large TGBL datasets (Coin, Comment, and Flight), Todyformer outperforms the SoTA with significant margins, exceeding 11%, 39%, and 7%, respectively. This interestingly supports the hypothesis behind the expressive power of the proposed model scale up the data domains with extensive long-range interactions. In the case of smaller datasets like TGBL-Wiki and TGBL-Review, our approach attains the second and third positions in the ranking, respectively. It should be noted that the hyperparameter search has not been exhausted during experimental evaluation. The average ranking reveals that Todyformer is ranked first, followed by TGN in the second place in this challenging experimental setup. Notably, datasets like TGBLFlight, TGBLCoin, and TGBLCommenl, characterized by imbalances labels, represent particularly challenging tasks for models, making the achievements of todyformer even more remarkable. Additionally, Figure 7 depicts experiments conducted with a large number of layers. Notably, the figure illustrates that the alternating mode results in a slight performance increase as the number of layers grows. Conversely, stacking layers of LIPNN-based models such dyg2vec leads to drop in model performance.
Dynamic Node classification: Tody has undergone extensive evaluation across three datasets dedicated to node classification. In these datasets, dynamic sparse labels are associated with nodes within defined time horizon after interactions. This particular task grapples with substantial im- balanced classification challenge. Table presents the AUC metric, known for its robustness toward class imbalance, across various methods on the three datasets. Notably, Todyformer demonstrates re- markable performance, trailing the best by only 4% on the MOOC dataset and 1% on both the Red- dit and Wikipedia datascts. Across all datasots, Tody former consistently secures the second-best, po- sition. However, it is important to acknowledge that no model exhibits consistent improvement across all datasets, primarily due to the presence of data im- balance issues inherent anomaly detection tasks (Ranshous et al. 2015). To establish the ultimate best model, we have computed the average ranks of various methods. Todyformer emerges as the top performer with an impressive rank of validating the overall performance improvement.
0-985
0.985
Averag Preciiio
0.975
0.975
0.9
0.970
0.965
0.000
0.55
LustM
0.950
MOOC
15
2/5
S0k
100k:
159K
200K 2r0k
Numberr of Poncher
Windoww Sizee
Figure Sensitivity analysis on the number of patches and input window size values MOOC and LastFM. The plot the left has fixed input window size of 262144, while the one on the right has 32 patches.
9

---

Published in Transactions on Machine Learning Research (06/2024)
Speed vs Performance: LaKEM
Speed Pefformance: SocilErol
Speet vs Perforaacc: MODC
DyC220c
150
Dyrove
CAR
0.5
NAC
0.50
Jocks
TGX
0.50
204
030
0.1
832
70AT
257
300
730
1108
1226
1390
1138
2006
300
1990
1390
200)
2200
3000
3550
-000
100
200
200
200
Infirass Tame (Sraad))
Inficmua (Soaadli
Infinnnc Tima (Seoaad))
Figure 4: The performance versus inference time across LastFM, SocialEvol, and MOOC datascts.
4.3 Ablation Studies and sensitivity analysis
We conducted an evaluation of the model's performance across various parameters and datasets to assess the sensitivity of the major hyperparameters. Figure 3 illustrates the sensitivity analysis regarding the win- dow size and the number of patchee, with one parameter remaining constant while the other changes. As highlighted in Xu et al. (2020), recent and frequent interactions display enhanced predictability of future interactions. This predictability is particularly advantagcous for datascts with extensive long-range depen- dencies, lavoring the utilization of larger window size values to capture recurrent patterns. Conversely, in datasets where recent critical interactions reflect importance, excessive emphasis on irrelevant information becomes prominent when employing larger window sizes. Our model, complemented by uniform neighbor sampling, achieves balanced equilibrium between these contrasting sides of the spectrum. As an example, the right plot in Figure 3 demonstrates that with a fixed number of patches (i.e., 32), an increase in window size leads to a corresponding increase in the validation AP metric on the LastFM dataset. This trend is particularly notable in LastFM, which exhibits pronounced long-range dependencies, in contrast to datasets like MOOC and UCI with medium- to short-range dependencies.
In contrast, in Figure 8 on the left side, with window size of 262k, vary the mumber of patches. Specifically, for the MOOC dataset, performance exhibits upward trajectory with increase in the number of patches from 8 to 16; however, it experiences a pronounced decline when the number of patches reaches 32. This observation aligns with the inherent nature of MOOC datasets, characterized by their relatively high density and reduced prevalence of long-range dependencies. Conversely, when considering LastFM data, the model maintains consistently high performance even with 32 patches. Tn essence, this phenomenon underscores the model's resilience on datasets featuring extensive long-range dependencies, illustrating trade-off between encoding local and contextual features by tweaking the number of patches.
In Table 4 we conducted ablation studies on the major design choices of the encoding network to assess the roles of the three hyperparameters separately: a) Global encoder, b) Alternating mode, and c) Positional Encoding. Across the four datasets, the alternating approach exhibits significant performance variation compared to others, ensuring the mitigation of sversshht and the capturing of long-range dependencies. The outcomes of the singlo-layer vanilla transformer as global encoder attain the sccond-best position, affirming the efficacy of our global encoder in enhancing expressiveness. Finally, the global encoder without PE closely resembles the model with only local encoder (i..,, DyG2Vec MPNN model).
4.4 Training/Inference Speed
In this section, an analysis of Figure 4 is provided, depicting the performance versus inference time across three sizable datasets. Considering the delicate trade-off between performance and complexity, our model surpasses all others in terms of Average Precision (AP) while concurrently positioning in the left segment
10

---

Published in Transactions on Machine Learning Research (06/2024)
Table 4: Ablation studies on three major components: Global Encoder (G. E.), Positional Encoder (P. E.), and number of alternating blocks (Alt. 3).
Dataset
G. E.
P. E.
Alt. 3
AP
0.950
MOOC
0.981
0.987
0.960
LastEM
0. 961
0.976
D. 0.981
UCI
0.988
987
0.993
0.987
SocialEvolution
989
0.991
of the diagrams, denoting the lowest inference time. Notably, as depicted in Figure 1
Todyformer remains
lighter and less complex than state-of-the-art (SOTA) models like CAW across all datasets.
5 Conclusion
We propose Todyformer, tokenized graph Transformer for dynamic graphs, where over-smoothing and over- squashing are empirically improved through local and global encoding architecture. We present how to adapt the best practices of Transformers in various data domains (e.g., Computer Vision) to dynamic graphs in principled manner. The primary novel components are patch generation, structure-aware tokenization using typical MPNNs that locally encode neighborhoods, and the utilization of Transformers to aggregate global context in an alternating fashion. The consistent experimental gains across different experimental settings empirically support the hypothesis that the SoTA dynamic graph encoders severely suffer from over-squashing and over-smoothing phenomena, especially the real-world large-scale datasets introduced in TGBL. We hope Todyformer sheds light the underlying aspects of dynamic graphs and opens up the door for further principled investigations on dynamic graph transformers.
References
Mohammad Ali Alomrani, Mahdi Biparva, Yingxue Zhang, and Mark Coates. Dyg2vec: Representation learning for dynamic graphs with self-supervision. arXiv preprint arXiv:2210.16906, 2022.
Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In International Conference on Machine Learning, pp. 3469 3489. PMLR, 2022.
Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, and Mchrdad Mahdavi. Do we really need complicated model architectures for temporal networks? arXiv preprint arXiv:2302.11638 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pro-training of deop bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Aswathy Divakaran and Anuraj Mohan. Temporal link prediction: A survey. New Generation Computing, 38(1):213 258, 2020.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 19929, 2020.
11

---

Published in Transactions on Machine Learning Research (06/2024)
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.0969) 2020.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message
passing for quantum chemistry. In International conference on machine learning. PMLR, 2017.
William L Hamilton. Graph representation learning. Morgan Claypool Publishers, 2020.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation lcarning on large graphs.
Proc. Adv. Neural Inf. Proc. Systems, 2017.
Xiaoxin He, Bryan Hooi, Thomas Laurent, Adam Perold, Yann LeCun, and Xavier Bresson. A generalization of vit/ mit/mlp-mxxer to graphs. In International Conference on Machine Learning, pp. 12724-12745. PMLR, 2023.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, (8)):1335-1788) 1997.
Shenyang Huang, Farimah Poursafaei, Jacob Danovitch, Matthias Fey, Weihua Hu, Emanuele Rossi, Jure Leskovec, Michael Bronstein, Guillaume Rabusscau, and Reihanch Rabbany. Temporal graph benchmark for machine learning temporal graphs. arXiv preprint arXiv:2307.01026 2023.
Ming Jin, Yuan-Fang Li, and Shirui Pan. Neural temporal walks: Motif-aware representation learning on continuous-timg dynamic graphs. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning vector represen- tation of time. arXiv preprint arXiv:1907.05321, 2019.
Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, and Pascal Poupart. Representation learning for dynamic graphs: A survey. Journal of Machine Learning Research, 2020.
Jinwoo Kim, Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure transformers are powerful graph learners. Advances in Neural Information Processing Systems, 35: 14582-14595, 2022.
Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34: 21618-21629, 2021.
Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. Proc. Int. Conf. on Knowledge Discovery & Data Mining, 2019.
Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Dongsheng Luo, Wei Cheng, Wenchao Yu, Bo Zong, Jingchao Ni, Haifeng Chen, and Xiang Zhang, Learn- ing to drop: Robust graph neural network via topological denoising. In Proceedings of the 14th ACM international conference on web search and data mining, pp. 779-787, 2021.
Yuhong Luo and Pan Li. Neighborhood-aware scalable temporal network representation learning. In The First Learning on Graphs Conference, 2022.
Farimah Poursafaci, Shenyang Huang, Kellin Pelrine, and Reihanch Rabbany. Towards better evaluation for dynamic link prediction. Advances in Neural Information Processing Systems, 35:32928 022292--2294,, 2022.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35:14501- -4551,, 2022.
Stephen Ranshous, Shitian Shen, Danai Koutra, Steve Harenberg, Christos Faloutsos, and Nagiza F. Sama- tova. Anomaly detection in dynamic networks: a survey. WTREs Computational Statistics, 7(3):223-244 2015.
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bron- stcin. Temporal graph networks for deep lcarning on dynamic graphs. In ICML Workshop on Graph Representation Learning, 2020.
Amauri Souza, Diego Mesquita, Samuel Kaski, and Vikas Garg. Provably expressive temporal graph net- works. Advances in Neural Information Processing Systems, 35:32257-32288 2022.
Ily O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 34:24261 24222, 2021.
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. arXiv preprint arXiv:211..14522, 2021.
Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In Proc. Int. Conf. on Machine Learning, pp. 3462-3471. PMLR, 2017.
Rakshit Trivedi, Mchrdad Farajtabar, Prasenject Biswal, and Hongyuan Zha. Dyrep: Learning representa- tions over dynamic graphs. In Proc. Int. Conf. Learning Representations, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph Attention Networks. In ICLR, 2018.
Yanbang Wang, Yen-Yu Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. In Proc. Int. Conf. on Learning Representations, 2021.
Da Xu, Chuanwei Ruan, Evron Korpeoglu, Sushant Kumar, and Kannan Achan. Inductive representation learning on temporal graphs. Proc. Int. Conf. on Representation Learning, 2020.
Dongkuan Xu, Wei Cheng, Dongsheng Luo, Yameng Gu, Xiao Liu, Jingchao Ni, Bo Zong, Haifeng Chen, and Xiang Zhang. Adaptive neural network for node classification in dynamic networks. In 2019 IEEE International Conference Data Mining (ICDM), pp. 1402 1407. TEEE, 2019a.
Dongkuan Xu, Wei Cheng, Dongsheng Luo, Xiao Liu, and Xiang Zhang. Seatio-tempora attentive rnn for node classification in temporal attributed graphs. In LJCAL, pp. 3947-3953, 2019b.
Dongkuan Xu, Junjie Liang, Wei Cheng, Hua Wei, Haifeng Chen, and Xiang Zhang. Transformer-style relational reasoning with dynamic memory updating for temporal network modeling. In Proceedings of 2021.
the AAAI conference on artificial intelligence, volume 35, pp.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, #4:28877-28885 2021.
Le Yu, Leilei Sun, Bowen Du, and Weifeng Lv. Towards better dynamic graph learning: New architecture and unified library. arXiv preprint arXiv:2303.13047, 2023.
Jiawei Zhang, Haopeng Zhang, Congying Xia, and Li Sun. Graph-bert: Only attention needed for learning graph representations. arXiv preprint arXiv:2001.0514 2020.
13

---

Published in Transactions on Machine Learning Research (06/2024)
A Appendix
A.1 Table of Notations
Table 5: Overall notation table of the main symbols in the paper.
bodd
The
The seroher sigre
The direreke of the enrhedlliy quan
The sitdie eac.
The yember Novks
Solo Matrions
A Guzilazoan Tine Dysantt Geaple (CTDS)
The segaetoo intreattioss
Yae sct vestao.
The
The puichs..
X
The sct of ank fnstun all patcles Eas query.
The
The rods endedillgg
the local eodde.
The uods eaboldinp after qubrin the glsta ausdss. The ert of tods anbeddiing abs parting
The prstitione the ground treth bork
The askk snichllupe elee upedins
Tescurlie Paraucterss
Lae wash penuratios the ussdee.
The wegle panaeeters the dooder
The loszall; usisis of tae geagay
kasselis ustcis
Functions Monllcss
The dynami grogh encoder
Decroness
The dicoder
LocaLEscoman
cacoaling undako.
The porthiiuiin modtte.
TTee todenizatio modile.
The pucking mottle
UstweRER!
The sqaakng mndiie..
Posrnose: Thexomed
The poctional modde
The Trnedomer modtle.
READOST
The malinn lustan.
A.2 Dataset Statistics
In this section, we provide an overview of the statistics pertaining to two distinct sets of datasets utilized for the tasks of Future Link Prediction (FLP) and Dynamic Node Classification (DNC). The initial set, detailed in Table G presents information regarding the number of nodes, edges, and unique edges across seven datasets featured in Table D and Table 3 For these three datasets, namely Reddit, Wikipedia, and MOOC, all edge features have been incorporated, where applicable. Furthermore, within this table, the last column represents the percentage of Repetitive Edges, which signifies the proportion of edges that occur more than once within the dynamic graph.
Table 6: Dynamic Graph Datasets. % Repetitive Edges: % of edges which appear more than once in the dynamic graph.
Dataset
Nodes
Edges
# Unique Edges
Edge Features
Node Labels
Hipartite
% Repetitive Edgees
Reddit
11,000
672,447
78,516
54%
Wikipedia
9,227
157,474
18,257
48%
MOOC
7,144
411,749
178,443
53%
LastEM
1980
1,293,103
154,993
68%
UCI
1899
59,835
13838
62%
Enron
184
125,235
2215
92%
SocialEvolution
2,099,519
2506
97%
A.2.1 Temporal Graph Benchmark (TGB) dataset
In this section, we present the characteristics of datasets as proposed by the Dynamic Graph Encoder Leaderboard (Huang et al., 2023). Similar to previous benchmark datasets, we have conducted comparisons regarding the number of nodes, edges, and types of graphs. Additionally, we report the Number of Steps
14

---

Published in Transactions on Machine Learning Research (06/2024)
and the Surprise Index, as defined in Poursafaei al. 2022 which illustrates the ratio of test edges that were not observed during the training phase.
Table 7: Statistics of TGBL Dynamic Graph Datasels
Dataset
# Nodes
# Edges
# Steps
Edge Features
Bipartite Surprise Index Poursafaci et al. (002
Wiki
9,227
157,474
152,757
0.108
Review
352,637
4,873,540
6,865
0.987
Coin
638,486
22,809.186
1, 7,295,720
0.120
Comment
994,790
44,314,507
30,998,030 030
0.823
Flight
18143
67,169,570
1,385
0.024
A.3 Implementation details
In this section, we elucidate the intricacies of our implementation, providing a comprehensive overview of the specific parameters our model accommodates during hyperparameter optimization. Subsequently, we delve into a discussion the optimal configurations and setups that yield the best performance for our proposed architecture.
Furthermore, in addition to in-depth discussion of the baselines incorporated into our paper, we also offer comprehensive overview of the respective hyperparameter configurations in this section. We are confident that with the open-sourcing of our code upon acceptance and the thorough descriptions of our model and baseline methodologies presented in the paper, our work is fully reproducible.
A.3.1 Supplementary Methodology
Patch1 Patch2
Patch1
Patch2
Patch M
UH
Patch
Time
Local Enc (MPNN)
Uapacking
Packing
Globol Enc Chrnnsorme)
Embeddings
81
8
Positional Encoding
M
Figure In-depth illustration of model architecture, encompassing packing and unpacking mechanisms, positional encoding, and encoder components.
15

---

Published in Transactions on Machine Learning Research (06/2024)
A.3.2 Evaluation Protocol
Transductive Setup: Under the transductive setting, a dataset is split normally by time, i.e., the model is trained on the first 70% of links, validated on %15 and tested on the rest.
Inductive Sctup: In the inductive setting, strive to test the model's prediction performance edges with unseen nodes. Therefore, following (Wang et al.) 2021), randomly assign 10% of the nodes to the validation and test sets and remove any interactions involving them in the training set. Additionally, to ensure an inductive setting, we remove any interactions not involving these nodes from the test set.
A.3.3 Loss Function
As previously discussed in the main body of this paper, focus on two specific downstream tasks: FLP, and DNC. For the former, we employ the binary cross-entropy loss, while for the latter, our model is trained through the minimization of the cross-entropy loss function. The formula for the binary cross-entropy loss
is presented below:
Hyy 1) (y log(ū) (1 y) log(1 9))
where y E {0, }} is the true label, and ú is the predicted probability that the instance belongs to class 1. Moreover, the formulation of the cross-entropy loss is as follows:
Hy, = - Σ yi log(Us)
where i represents the index over all classes, Yi is the true probability of the sample belonging to class i, encoded as a one-hot vector. It is 1 for the true class and 0 for all other classes. Finally, di is the predicted probability of the sample belonging to class i.
A.3.4 Best Hyperparameters for Benchmark datasets.
Table 8 displays the hyperparameters that have been subjected to experimentation and tuning for each dataset. For each parameter, range of values has been tested as follows:
Window Size (W): This parameter signifies the window length chosen for selecting the input subgraph based on edge timestamps. It falls within the range E { 16384, 32768 ,65536, 262144 }.
Number of Patches: This parameter indicates the count of equal and non-overlapping chunks for each imput subgraph. is the range of E {8, 16,,22).
#Local Encoders: This parameter represents the number of local encoder layers within cach block, and its value falls within the range of E (1,2).
Neighbor Sampling (NS) mode: E {uniform. last). In the case of uniform Neighbor Sampler (NS), it uniformly selects samples from the 1-hop interactions of a given node. Conversely, in last mode, it samples from the most recent interactions.
Anchor Node Mode: E [GlobalTarget, Incal LocalTarget. depending on the mechanism of neighbor sampling we can sample from nodes within all patches (LocalInput), nodes within the next patch (LocalfTarget), or global target nodes (GlobalTarrett).
Batch Size: E (8. 16, 16,32, 64]
Positional Encoding: E (SineCosine, Time2Ves ec, Identily, Linear]
SineCosine is utilized as the Positional Encoding (PE) method following the experiments conducted in Appendix A.4.3
Selecting Best Checkpoint: Throughout all experiments, the models undergo training for duration of 100 epochs, with the best checkpoints selected for testing based on their validation Average Precision (AP) performance.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Table Best parameters the model pipeline after hyperparameter search.
Dataset
Window Size (W)
Number Patches
# LLocal Encoders
NS Mode
Anchor Node Mode
Batch Size
Reddit
262144
32
uniform
GlobalTargee
200
Wikipedia
65536
uniform
Global GlobaTTarget
MOOC
65536
8
uniform
GlobalTarget
200
LastFM
262144
32
uniform
GlobalTarger
200
UCI
65536
uniform
GlobalTarget
200
Faron
65536
8
uniform
Global Target
200
SocialEvvlution
65536
8
2
uniform
Global Targgt
200
A.3.5 Best Hyperparameters for TGB dataset
In this section, we present the optimal hyperparameters used in our architecture design for each TGBL dataset. We conducted hyperparameter tuning for all TGBL datasets; however, due to time constraints, we explored a more limited set of parameters for the large-scale dataset. Despite Todyformer outperforming its counterparts on these datasets, there remains potential for further improvement through an extensive hyperparameter search.
Table 9: Optimal window size W for downstream training.
Dataset
Window Size (W)
Number of Patches
Finst-hop NS size
NS Mode
Anchor Node Mode
Batch Size
Wiki
262144
32
256
uniform
GlobalTarget
32
Review
262144
32
64
uniform
GlobalTarget
64
Comment
65536
uniform
GlobalTarget
256
Coin 65536
8
64
uniform
GlobalTarret
96
Flight
65536
uniform
GlobalTarget
128
A.4
More Experimental Result
In this section, present the additional experiments conducted and provide analysis of the derived results and conclusions.
A.4.1 FLP result on Benchmark Datasets
Table 10 an extension of Table now incorporating the Wikipedia and Reddit datasets. Notably, for these two datascts, Todyformer attains the highest test Average Procision (AP) score in the Transductive sctup. However, it secures the second-best and third-best, positions in the Induetive setup for these Wikipedia and Reddit respectively. While the model does not attain the top position these two datasets for inductive setup, its performance is only marginally below that of state-of-the-art (SOTA) models, which have previously achieved accuracy levels exceeding 99% Average Precision (AP).
A.4.2 FLP validation result on TGBL dataset
As discussod in the paper, Todyformer has boen compared bascline methods using the TGBL datasct. Table I1 represents extension of Table specifically for validation (MRR). The results presented in both tables in line with counterpart methods outlined by (Hung et al. 2023). Tt is important to note that for the larger datasets, TCL, GraphMIxer, and EdgeBank were found to be impractical due to memory constraints, mentioned in the paper.
A.4.3 Complementary Sensitivity Analysis and Ablation Study
In this section, we have presented the specifies of sensitivity and ablation experiments, which, while of lesser significance in our hyper-tuning mechanism, contribute valuable insights. In all tables, the Average Precision scores reported in the table extracted from the same epoch on the validation set. Table T2 showcases
17

---

Published in Transactions on Machine Learning Research (06/2024)
Table 10: Future Link Prediction performance in AP (Mean Std). Bold font and ul font represent first-best and recond-best performance respectively.
Setling
Model
Wikipedia
Reddit
MOOC
LasPM
Euron
UCI
SucialEvol.
Transductive
JODIE
0.956 0.002
0.979 + 0.001
0.797 + 0.01
0.691 0.010
0.785 0.020
0.869
0.010
0.847 0.014
DyRep
0.955
0.004
0,981 + le-4
0.840 + 0.004
0.683 + 0.033
0.795 005::00000
0.524 + 0.076
0.076
0.885 0.004
TGAT
0.968 0.001
0,986 + 3e-1
0.793 = 0,006
0.633 + 0.002
0.637 +
0.835 + 0.003
0.631 = 0.001
TGN
0.986 0.001
0.955 4 0.001
0.911 = 0.010
0.713 2 0.030
0.806 1: 0.006
0.843 0.090
0.960 = 0.001
CaW
2e-4
0.940
0.903 =:
0.947 1c-4
NAT
0.987 0.001
0.991 1: 0.001
0.874 1 0.004
0.859 1: 1o.4
0,924 1.:4:100.00 0.001
0.944 0.002
0.944 0.010
GraphMixeer
0.974 + 0.001
0.975 + 0.001
0.835 + 0.001
0.862 0.003
0.824 + 0.001
0.932 + 0.006
0.935 + 3e-4
Dygformer
0.991 + 0.0001
0.992 + 0.0001
0.892 0.005
0.901 + 0.003
0.926 + 0.001
0.959 + 0.001
0.952 + 2e-4
DyG2Vee
0.995 0.003
0.996 2--1
0.980 0.002 002
0. 0.960 = 1e-1
0.991 0.001
0.988 0.007
0.987 + 26-1
Todyformer
+ 2e-1
0.992 7e-4
0.976 + 3e-4
0.995 + 6e-1
0.994 4e-1
1e-4
JODIE
0.891 = 0.014
0.865 + 0.021
0.707 = 0.029
0.865 = 0.03
0.717 + 0.011
0.753 - 0.011
0.791 = 0.031
Induetive
DyRep
0.890 = 0.002
0,921 + 0.003
0.723 1: 0.009
0,869 0.015
0.666 1: 0.059
0.437 0.021
0.904 : 3e-4
TGAT
0.979 0.001
0.006
0.004
0.005
TGN
0.971 0.001
0,951 = 0.002
0.855 0.014
0.789 0.050
0.716:0.000
0.791 0.057
0,901 0.023
CaW
0.977 0,006
0,984 + 2e-1
0,933 = 0.014
0,890 + 0.001
0,962 + 0.001
0,931 0.002
0.950 = 1c-4
NAT
0.986 0.001
0.986 1 0.002
0.832 1: 1e-4
0.878 1 0.003
0.949 1 0.010
0.926 0.010
0.932 0.006
GraphMixer
0.966 +: 2c-4
0.952 2c-4
0.814 + 0.002
0.821 0.004
0.758 0.004
0.911 0.004
0.918 6r-4
Dygformer
0.985 +
0 2n-4
0.869 0.004
0.942 + 9e-4
0.897 + 0.003
0.945 0.001
0.901 4c-4
DyG2Vee
0.992 001
0.991 + 0.002
0.938 0.010
0.979 + 0.006
0.987 + 0.004
0.976 + 0.002
0.978 0,010
Todyformer
0 0.989 = Ge-1
0.963 = 0.002
0. 9.488 = 0.009
0. 9..81 0.005
0.989 : Se-1
0.983 + 0.002
0.9821 0.005
the influence of varying input window sizes and patch sizes on two datasets. Table 13 illustrates the effects of various PEs, including SineCosine, Time2Vec (Kazemi et al. 2019). Identity, Linear, and a configuration utilizing Local Input as the Anchor Node Mode. The table presents a comparison of results for these different PEs. Notably, the architecture appears to be relatively insensitive to the type of PE used, as the results all fall within similar range. However, it is worth mentioning that SineCosine PE slightly outperforms the others. Consequently, SineCosine PE will be selected as the primary module for all subsequent experiments.
In 14 additional ablation study has been conducted to elucidate the influence of positions tagged to each node before being input to the Positional Encoder module. Varions mechanisms for adding positions are delineated as follows:
Without PE: No position is utilized or tagged the nodes.
Random Index: An index randomly generated and added to the embeddings of a given node.
Patch Index: The index of the patch from which the embedding of the given node originates is used as a position.
Edge Time: The most recent edge time within its patch is employed as a position.
Edge Index: The index the most recent interaction within the corresponding patch is utilized as a position.
As evident from the findings in this table, the validation performance exhibits high sensitivity to the positional encoder's outcomes. Specifically, the model without positional encoder (PF) and the model with random
Table 11: (Validation) Future Link Prediction performance in Validation MRR on TGB Leaderboard datasets.
Model
TGBWiki
L'GBReview
TGGCCoin
TG ComComeentt
T'GBFlight
Avg. Rank
Dyrep TGN
0.411 0.015
0.356 0.016
0.512 + 0.014
0.291 0.028
0.573 0.013
4.2
0.737 = 0.004
0.465 + 0.010
0.607 + 0.014
0.356 + 0.019
0.731 + 0..110
2.2
CAWN
0.014
0.201 - 0.002
00M
00M
0OM
3
TCL
0.001
0.194 0.012
00M
00M
OOM
5
GraphMixer
0.707 0.014
0.411 +
0.025 0.025
00M
0OM
QOM
EdgeBank
0.641
0.0894
0,1444
0.388
0.492
4.6
Tody former
0.799 0.0092
0.4321 0.0040
0.6852 + 0.0021
0.7402
0.0037
0.7932 0.014
1.2
18

---

Published in Transactions on Machine Learning Research (06/2024)
indices manifest the lowest performance among all available options. Consistent with expectations from previous experiments, the patch index yields the highest, performance, providing compelling rationale for its incorporation into the architecture.
In Table 4 which presents the ablation study, we meticulously assess the impact various components of the model architecture, particularly those associated with the alternating mode, such as packing and umpacking. This section has been comprehensively updated in the paper's appendix. The table delineates different scenarios:
If the model lacks any of the three components (G.E, P.E. Alt.3), it signifies that the model solely relies on an MPNN-based encoder (local encoder).
If it includes only G.E, it indicates the presence of packing alongside one block of Local and Global encoders without any unpacking.
If G.E and P.E. present without Alt.3, this suggests the inclusion of one global and local encoder block coupled with patch positional encoding.
Finally, if all components present, it signifies the incorporation of three blocks of local and global encoders with positional encoding, well as packing and unpacking modules within cach block.
Table 12: Sensitivity analysis on the number of patches and the target window size.
dataset
Input Window size
Number of Patches
Average Procision f
LastEM
262144 144
0.9772
LastFM
2621
0.9791
LastFM
2621 44
32
0.9758
MOOC
2621 44
0.9811
MOOC
2621 44
0.9864
MOOC
2621
32
0.9696
LastFM
16384
32
0.9476
LastFM
32768
32
0.9508
LastFM
0.9591
LastFM
262144
32
0.9764
MOOC
16384
32
0.9798
MOOC
32768
32
0.9693
MOOC
65536
0.9685
MOOC
262144
32
0.9726
Table 13: Ablation study positional encoding options on the MOOC Dataset. This table compares the validation performance at the same epoch across various setups.
Positional Encoding
Anchor Node Mode Average Precision
SincCosinePos
global target
0.9901
Time2VecPos
global target
0.989
IdentityPos
global target
0.99
LinearPos
global target
0.9886
SineCosinePos
local input
0.9448
19

---

Published in Transactions on Machine Learning Research (06/2024)
Table 14: Ablation study on the input of positional encoding on the MOOC Dataset. This table compares the validation performance the same epoch across various types of positions tagged to nodes before PE layer.
Positional Encoding (PE) Input
Average Precision T
without PE
0.9872
random index
0.9873
patch index
0.9889
edge time
0.9886
edge index
0.9877
A.5 Computational Complexity
A.5.1
Qualitative Analysis for Time and Memory Complexities
In this section, we delve into the detailed measurement and discussion of the computational complexity of Todyformer. Initially, we adopt the assumption that the time complexity of Transformers is O(x2) for an input sequence of length X. The primary complexity of Todyformer encompasses both the complexity the Mcssage Passing Neural Network (MPNN) component and the complexity of the Transformer. To claborate further, assuming we have sparse dynamic graph with temporal attributes, we can replace the complexity of MPNNs with O x (N +)), where N and E represent the number of nodes and edges within the temporal input subgraph, and I is the number of MPNN layers for the Graph Neural Network (GNN) local encoder. In the transformer part, N unique nodes are fed into the AlutthheadAAttentinn module. If the maximum length of the sequence fed to the Transformer is Na, then the complexity of the Mulcihead-Attention module is O(N2). Notably, Na is at most equal to M, the number of patches. This scenario occurs when a node appears in all M patches and has interactions in all patches. Consequently, if L is the number of blocks, the final complexity given by:
xlLxxx + (V M²) O(N+
(10)
The LHS part of Equation 10 can be simplified to RHS we assume that L, I, and M are negligible compared to N and The RHS of this equation is the time complexity of GNNs for sparse graphs.
A.5.2 Training/Infereen Speed
In this section, an analysis of Figure A is provided, depicting the performance versus inference time across three sizable datasets. Considering the delicate trade-off between performance and complexity, our model surpasses all others in terms of Average Precision (AP) while concurrently positioning the left segment of the diagrams, denoting the lowest inference time. Notably, as depicted in Figure 4 Todyformer remains lighter and less complex than state-of-the-ar (SOTA) models like CAW across all datasets.
A.6 Discussion on Over-Smoo and Over-Squashing
In Figure D the blue curve illustrates the Average Precision performance of dynamic graph Message Passing Neural Networks (MPNNs) across varying numbers of layers. Notably, an observed trend indicates that as the number of layers increases, the performance experiences decline characteristic manifestation of oversmoothing and oversquashing phenomena.
Within the same figure, the red circle dots represent the performance of MPNNs augmented with trans- formers, specifically Todyformer with a single block. It is noteworthy that the increase in the number of MPNN layers from 3 to 9 in this configuration results in a comparatively minor performance drop compared to traditional MPNNs.
Furthermore, the green stars denote the performance of Todyformer with an alternating mode, where the total number of MPNNs is 9, and three blocks are incorporated. In this setup, transformer is introduced
20

---

Published in Transactions on Machine Learning Research (06/2024)
after every 3 MPNN layers. Strikingly, this configuration outperforms all others, especially those that stack similar number of MPNN layers without the insertion of a transformer layer in the middle of the architecture. This empirical observation serves as a significant study, highlighting the efficacy of our architecture in addressing oversmoothing and oversquashing challenges.
Sonsitivity Analysis on the Number Layers and Blocks
Blocks
Block
0,9925
No Block
Average Precision
0.9000
0.9875
0,9850
0,9800
0.9775
0.9750
Number of Layers
Figure 6: Sensitivity analysis on the number of layers and blocks conducted on the MOOC dataset.
To demonstrate the resilience of our model against oversmoothing, have devised novel experiment under distinct setup. Due to constraints in computing resources and to accommodate models with a considerable number of layers on GPUs, we have opted for a smaller dataset. Furthermore, considering the significant stacking of layers (approximately 50), we have adjusted the neighbor sampling size to one for layers beyond the initial layer, primarily to mitigate memory consumption and ensure compatibility with GPU memory constraints.
In Figure present comparative analysis of Average Precision (AP) scores across different models, including MPNN-based model such dyg2vec, and todyformer, varying in the number of layers. Notably, the performance of MPNN exhibits decline with an increase in the number of layers. Despite employing an approximation technique during neighbor sampling to accommodate larger number of layers, the decline in performance is relatively modest. However, the observed decreasing trend suggests a hypothetical association with the issuc of oversmoothing. Conversely, for todyformer, we observe reverse pattern, with minor improvement in performance as the number of layers increases. This empirical observation further strengthens the argument that todyformer is resilient to the effects of oversmoothing.
21

---

Published in Transactions on Machine Learning Research (06/2024)
Sensitivity Analysis on the Number of Layers
99.5
MPNN
Todyformer
Averago Precision
99.0
90.5
98.0
97.5
30 Number of Layers
Figure 7: Sensitivity analysis on the number of layers conducted on the Enron dataset. The neighbor sampling size configured to be one for subsequent layers following the initial layer.
22


Reviews:
[{"review_id": "4Fry0HTBGY", "reviewer": "Reviewer_jrhC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. the overall performance of Todyformer looks quite good. \n\nWeaknesses:\n\n1. the notation is not clear, especially for eq(6). H^l is a tensor of shape N×M×D, why the shape of Q, K, V is NxD? How M is eliminated?\n2. the figure 1 is also not clear, the shape is NxM or NxMxD? \n3. it seems the computational cost is very large, it could be (NxM)^2.\n4. the patch partitioning is not clear, how the time is considered here? just evenly split them? Besides, how the Window size W is considered here?\n5. The motivation of this paper is to \"improve model expressiveness through alleviating over-squashing and over- smoothing in a systematic manner.\" However, the experiment results seems cannot support this motivation. There are not clear evidence that the proposed method can alleviate over-squashing and over-smoothing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Todyformer, a Transformer-based neural network tailored for dynamic graphs, used for Temporal Graph Neural Networks. Todyformer unifies local encoding capabilities of Message-Passing Neural Networks (MPNNs) with the global encoding prowess of Transformers.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Refer to Weaknesses", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4KIcaKNUVB", "reviewer": "Reviewer_eUmK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- A primary novelty of Todyformer is its alternating use of local and global encoders within each layer. This approach addresses the challenge of handling dynamic graphs, which typically feature nodes with extensive neighboring connections compared to static graphs. This approach represents a key contribution in the field of dynamic graph representation.\n- Experimental results demonstrate Todyformer's superior performance over existing baselines. The significance of the model's design choices is further validated through comprehensive ablation studies presented in Section 4.3.\n\nWeaknesses:\n\n- **The paper's presentation requires refinement.** The current use of terminologies and notations is often confusing, detracting from the overall readability. Additionally, the omission of crucial details undermines the paper's comprehensiveness. Refer to the \"Requested Changes\" section for more detailed feedback.\n- **The novelty of the approach may be overstated.** While the paper asserts novel contributions in patch generation, structure-aware tokenization, and the utilization of Transformers to aggregate global context in an alternating fashion, some of these elements resemble concepts previously explored in works such as Dygformer. As such, the truly novel aspect appears to be the specific implementation of altering architecture with local and global layers, casting some doubt on the claimed extent of novelty.\n- **The marginal performance gains observed in the Todyformer, relative to state-of-the-art architectures, seem to primarily result from the addition of more layers over DyG2Vec.** This raises questions about the actual efficacy of the proposed idea of altering local and global encoders and its substantial role in enhancing performance.\n    - Although Todyformer marginally surpasses DyG2Vec across several datasets, as indicated in Table 1 and Figure 4, it incurs a significant increase in inference time—between two to four times longer than DyG2Vec.\n    - This trade-off is somewhat justified by Todyformer's use of DyG2Vec for structure-aware tokenization within its processing pipeline.  Nonetheless, it suggests that the performance improvements attributed to the model's additional components might be marginal, achieved at the expense of higher computational resources.\n    - The extent to which these improvements represent a meaningful advancement in the field hinges on a careful consideration of their practical impact versus the additional resources required.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the Todyformer, a novel approach to modeling dynamic graphs through a three-stage process: patchifying, tokenization, and encoding. Unlike prior efforts that mostly address static graph structures, Todyformer uniquely targets dynamic graphs, where each edge includes a temporal dimension. The proposed method addresses the over-squashing issue prevalent in dynamic graphs by segmenting large graphs into smaller subgraphs during the patchifying stage, akin to techniques used in image processing. Additionally, Todyformer employs a dynamic graph neural network (DyG2Vec) in its tokenization process to effectively capture temporal patterns. The novel encoding stage combines both local and global encoders within a single layer, processing them sequentially to enhance model performance significantly.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. The methodology behind the ablation study on the alternating mode is unclear. Clarification is needed on whether the study compares architectures with sequential local and global encoders without node packing and unpacking processes. Detailed explanations are essential for understanding the impact of alternating modes on performance enhancement.\n2. The term \"window sizes\" introduced in Section 4.3 requires a definition, as it appears without prior explanation in the methodology section.\n3. Enhance readability by standardizing citation formatting. Enclose all citations within parentheses, for example, converting \"Divakaran & Mohan (2020)\" to \"(Divakaran & Mohan, 2020)\".\n4. Figures 1 and 2 could benefit from improvements for better clarity:\n    - Consider rotating textual elements in Figures 1 and 2 where feasible, such as $P_k$ and $n_k$, to facilitate easier reading.\n    - Clarify the symbols used in the figures, specifying the representations of $n$, $P$, and $c$, and ensuring each symbol's meaning is easily accessible without extensive manuscript review.\n    - Expand on the patchifying process within Figure 1, explicitly indicating whether each temporal subgraph corresponds to a patch and providing more detailed context.\n5. Suggestions for notation improvements:\n    - Reevaluate the use of \"Tokenizer\" for a module that functions similarly to an MPNN, focusing on message passing between nodes. A more intuitive naming convention might be to rename \"Tokenizer\" to simply \"MPNN\".\n    - Replace \"Transformer\" with \"Self-Attention\" in Equation 6 to avoid confusion, as \"Transformer\" generally refers to a combination of self-attention and MLP layers.\n    - Provide explicit details and notations for the PACKER and UNPACKER components, as their descriptions are currently lacking.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JLdKGJFinV", "reviewer": "Reviewer_5WPT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros of Todyformer:\n1: Todyformer consistently outperforms state-of-the-art methods in experimental evaluations on public benchmark datasets, showcasing its effectiveness in capturing extensive temporal dependencies in dynamic graphs.\n2: The integration of Message-Passing Neural Networks and Transformers in Todyformer offers a unique approach to addressing issues like over-squashing and over-smoothing in Temporal Graph Neural Networks.\n3: Todyformer introduces novel strategies such as patchifying paradigm, structure-aware parametric tokenization, and temporal positional encoding, enhancing its ability to model evolving structural and temporal patterns.\n4: The encoding architecture of Todyformer dynamically alternates between local and global contextualization, striking a balance that mitigates over-smoothing and improves model expressiveness.\n5: Todyformer demonstrates significant performance gains on extra-large datasets, indicating its capability to scale up to data domains with extensive long-range interactions.\n\nCons of Todyformer:\n1: The paper mentions that the hyperparameter search has not been exhausted during experimental evaluation, indicating that further optimization may be required to fine-tune the performance of Todyformer.\n\n2: While Todyformer outperforms the two latest state-of-the-art methods, GraphMixer and Dygformer, the paper does not provide a detailed comparison with a broader range of existing models, potentially limiting the scope of the evaluation. A thorough literature review is required for a high quality paper. For example, the following two papers are highly relevant and using similar strategies, thus should be discussed and compared:\nTransformer-Style Relational Reasoning with Dynamic Memory Updating for Temporal Network Modeling. AAAI'21.\nSpatio-Temporal Attentive RNN for Node Classification in Temporal Attributed Graphs. IJCAI'19.\nBesides, the following paper is also relevant, should be discussed:\nAdaptive Neural Network for Node Classification in Dynamic Networks. ICDM'19.\nRefer to the \"3.5 Improving Over-Smoothing by Alternating Architectur\". The following paper is highly relevant, thus should be discussed:\nLearning to Drop: Robust Graph Neural Network via Topological Denoising. WSDM'21.\n\n3: The paper acknowledges that no model exhibits consistent improvement across all datasets due to data imbalance issues inherent in anomaly detection tasks, suggesting that Todyformer may face challenges in handling imbalanced datasets.\n\nOverall, Todyformer presents a promising approach to dynamic graph analysis, offering innovative solutions to common challenges in Temporal Graph Neural Networks and demonstrating strong performance in experimental evaluations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The PDF paper introduces Todyformer, a novel Transformer-based neural network tailored for dynamic graphs. The paper addresses common issues in Temporal Graph Neural Networks, such as over-squashing and over-smoothing, by combining the local encoding capacity of Message-Passing Neural Networks with the global encoding of Transformers. Todyformer introduces innovative strategies like a patchifying paradigm for dynamic graphs, structure-aware parametric tokenization, temporal positional encoding in Transformers, and an encoding architecture that balances local and global contextualization to mitigate over-smoothing.\nExperimental evaluations on public benchmark datasets demonstrate that Todyformer consistently outperforms state-of-the-art methods in downstream tasks, showcasing its effectiveness in capturing extensive temporal dependencies in dynamic graphs. The paper provides valuable insights into the design and performance of Todyformer, highlighting its potential to advance the field of dynamic graph analysis. However, important references are missing, raising novelty issue without discussion these references.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. While Todyformer outperforms the two latest state-of-the-art methods, GraphMixer and Dygformer, the paper does not provide a detailed comparison with a broader range of existing models, potentially limiting the scope of the evaluation. A thorough literature review is required for a high quality paper. For example, the following two papers are highly relevant and using similar strategies, thus should be discussed and compared:\nTransformer-Style Relational Reasoning with Dynamic Memory Updating for Temporal Network Modeling. AAAI'21.\nSpatio-Temporal Attentive RNN for Node Classification in Temporal Attributed Graphs. IJCAI'19.\nBesides, the following paper is also relevant, should be discussed:\nAdaptive Neural Network for Node Classification in Dynamic Networks. ICDM'19.\nRefer to the \"3.5 Improving Over-Smoothing by Alternating Architectur\". The following paper is highly relevant, thus should be discussed:\nLearning to Drop: Robust Graph Neural Network via Topological Denoising. WSDM'21.\n\n2.  The paper acknowledges that no model exhibits consistent improvement across all datasets due to data imbalance issues inherent in anomaly detection tasks, suggesting that Todyformer may face challenges in handling imbalanced datasets. Thus, it is highly suggested to include an experiment to test the performance on imbalance dataset.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has proposed a new method of Todyformer for modeling dynamic graphs, and provided experiments to demonstrate its superior performance. The contribution is solid and will be beneficial to the TMLR community. The reviewers have provided further suggestions in their recommendations, based on which the authors could further revise the paper. Below, I summarize those comments.

1. Please further polish the paper to improve the presentation.
2. Some of the proposed methods resemble concepts previously explored in works such as Dygformer. Please further clarify the differences of the proposed method from the previous approaches.
3. The performance gains observed in the Todyformer, relative to state-of-the-art architectures, seem to primarily result from adding more layers over DyG2Vec. Please clarify if this is the case.
Decision: Accept (poster)