Between Linear and Sinusoidal: Rethinking the Time Encoder in Dynamic Graph Learning
Authors: Hsing-Huan Chung
Shravan S Chaudhari
Xing Han
Yoav Wald
Suchi Saria
Joydeep Ghosh
Venue: Accepted by TMLR ()

Abstract: Dynamic graph learning is essential for applications involving temporal networks and requires effective modeling of temporal relationships.
Seminal attention-based models like TGAT and DyGFormer rely on sinusoidal time encoders to capture temporal dependencies between edge events. Prior work justified sinusoidal encodings because their inner products depend on the time spans between events, which are crucial features for modeling inter-event relations. However, sinusoidal encodings inherently lose temporal information due to their many-to-one nature and therefore require high dimensions. In this paper, we rigorously study a simpler alternative: the linear time encoder, which avoids temporal information loss caused by sinusoidal functions and reduces the need for high-dimensional time encoders. We show that the self-attention mechanism can effectively learn to compute time spans between events from linear time encodings and extract relevant temporal patterns. Through extensive experiments on six dynamic graph datasets, we demonstrate that the linear time encoder improves the performance of TGAT and DyGFormer in most cases. Moreover, the linear time encoder can lead to significant savings in model parameters with minimal performance loss. For example, compared to a 100-dimensional sinusoidal time encoder, TGAT with a 2-dimensional linear time encoder saves 43% of parameters and achieves higher average precision on five datasets. While both encoders can be used simultaneously, our study highlights the often-overlooked advantages of linear time features in modern dynamic graph models. These findings can positively impact the design choices of various dynamic graph learning architectures and eventually benefit temporal network applications such as recommender systems, communication networks, and traffic forecasting. The experimental code is available at: https://github.com/hsinghuan/dg-linear-time.git.

Published in Transactions on Machine Learning Research (07 (07/2025)
Between Linear and Sinusoidal: Rethinking the Time Encoder in Dynamic Graph Learning
Hsing-Huaa Chung
Michung@uteemss.ddi
Deportment of Electrical and Computer Engineering University of Teras Austin
Shravan Chaudhari
schaud35 Eigh. edu
Deportment of Computer Science Johns Hopkins University
Xing Han
shan5660jhu..o edu
Department of Computer Science Johns Hopkins University
Yoav Wald
your. wal.walddyyne edu
Center for Data Science New York Universily
Suchi Saria
ssaria jhu edu
Department of Computer Science Johns Hopkins University
Joydeep Ghosh
jghosh Guteeaas edu
Department of Electrical and Computer Engineering University of Texas Austón
Reviewed on OpenReview: https: openreview. net/ forum? id= W6GQud0CHg
Abstract
Dynamic graph learning is essential for applications involving temporal networks and re- quires effective modeling of temporal relationships. Seminal attention-based models like TGAT and DyGFormer rely on sinusoidal time encoders capture temporal dependencies between edge events. Prior work justified sinusoidal encodings because their inner products depend on the time spans between events, which are crucial features for modeling inter-event relations. However, sinusoidal encodings inherently lose temporal information due to their many-to-one nature and therefore require high dimensions. In this paper, rigorously study a simpler alternative: the linear time encoder, which avoids temporal information loss caused by sinusoidal functions and reduces the need for high-dimensional time encoders. We show that the self-attention mechanism can effectively learn to compute time spans be- tween events from linear time encodings and extract relevant temporal patterns. Through extensive experiments on six dynamic graph datasets, we demonstrate that the linear time encoder improves the performance of TGAT and DyGFormer in most cases. Moreover, the linear time encoder lead to significant savings in model parameters with minimal per- formance loss. For example, compared to 100-dimensionall sinusoidal time encoder, TGAT with 2-dimensional linear time encoder saves 43% of parameters and achieves higher av- erage precision on five datasets. While both encoders can be used simultaneously, our study highlights the often-overlooked advantages of linear time features modern dynamic graph models. These findings can positively impact the design choices of various dynamic graph learning architectures and eventually benefit temporal network applications such recom-

---

Published in Transactions on Machine Learning Research (07/2025)
mender systems, communication networks, and traffic forecasting. The experimental code is available at: http:://ghhiii.omm//higgnnnnngee
1 Introduction
Dynamic graph learning (Longa et al., 2023: Yu et al. 2023 Huang et al.) 2024) has gainod significant allention in recent years due to its wide range of applications, inc including recommender systems (Zhang et al. 2022), traffic forecasting (Yu et al.) 2017), and anomaly detection (Wang al. 2021a). A key task in dynamic graph learning is future link prediction (Poursafaci et al., 2022 Qin & Young 2023), where the goal is to predict future connections or interactions between nodes in a temporal graph based on historical data. Temporal features such as the interaction time play an important role in capturing the dynamic nature of these graphs. For instance, in a temporal person-to-person communication network, imagine an individual who exchanged several emails with Alice this morning but last contacted Bob weeks ago. The next message is far more likely to go to Alice than to Bob, underscoring how the timing of interactions must be captured by dynamic graph models when predicting future links.
One popular approach for encoding temporal information in dynamic graph learning is the sinusoidal time encoder. Initially introduced in the event sequence modeling literature (Xu et al. 2019) to make time encodings compatible with the self-attention mechanism, the sinusoidal time encoder maps timestamps or time differences as periodic functions of their values, represented as multi-dimenssonall sinusoidal time encodings. Temporal Graph Attention (TGAT) (da Xu et al.) 2020) introduced the sinusoidal time encoder to dynamic graph learning, and subsoquent works (Wang et al. 2021c Cong ct al., 2023] Yu et al. 2023) have widely adopted it as the default approach. Motivated by the importance of temporal information
dynamic graphs, we revisit the choice of time encoders in this study.
We examine a simpler alternative: the linear time encoder. Intuitively, sinusoidal functions are many-to- one mappings, which lead to inevitable temporal information loss, while the linear time encoder avoids it by representing time through one-to-one mappings. Previous work (Xu et al. 2019 da Xu et al. 2020 motivated sinusoidal encodings by showing that their inner products depend on the time spans between events, which are essential features for modcling sensitvns relationships. However, we prove that self-attention can also compute time spans effectively from linear time encodings. We demonstrate this through experiments on a synthetic task where such features are required for prediction. These obscrvations motivate us to evaluate the linear time encoder in dynamic graph learning. We conduct extensive experiments on future link prediction tasks across six dynamic graph datasets. Our results reveal that, out of 24 model-dataset combinations, the linear time encoder outperforms sinusoidal time encoder variants in 19 and 18 cases under random and historical negative sampling, respectively. The largest performance gains caused by switching from the sinusoidal to linear lime encoder on TGAT and DyGFormer (Yu et al. 2023) are 22.48 and 7.28 increases in average precision (AP) scores under historical negative sampling, respectively. Additionally, we show that we can reduce the dimensionality of linear time encodings in TGAT and save 43% of model parameters, with only maximum decrease of 2.44 in AP. In contrast, sinusoidal time encoding suffers maximum drop of 8.08 in AP under the same model size reduction. We perform similar experiments on reducing the temporal feature channel dimension in DyGFormer (Yu et al.) (023).. While both sinusoidal and linear time encoders experience similar performance drops on two datasets, the performance of linear time encoders surprisingly improves on two heh datasets, with 38% and 34% savings in model parameters, whereas sinusoidal time encoders continue to experience performance decline. These results confirm that linear time encoders are not only better performing but also more cost-effective than the widely used sinusoidal time encoders.
In summary, contribution is threefold. First, we examine the linear time encoder an alternative to the widely used sinusoidal time encoder, offering simpler and more efficient method for encoding temporal information in dynamic graph learning. Second, we prove that sell-atentionn can compute time spans ween events from linear time encodings and demonstrate it can learn such capabilities through experiments on synthetic datascts. Finally, we present extensive experimental results across six dynamic graph datasets, showing that the linear time encoder outperforms sinusoidal variants in most cases, while also being more cost-effective by reducing model parameters without significantly compromising performance.
2

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Sc fo): Hip
SCC
st), Hid
12 65
14
Figure 1: Future link prediction on contimuous-time dynamic graph. At time is, the link prodictor ranks how likely edges will form given the historical edge events happening at l,,..., ,4.
2 Preliminaries
We revisit the future link prediction problem on continuous-time dynamic graphs and review two seminal allention-based models: Temporal Graph Attention (TGAT) (da Xu et. al. 2020) and DyGFormer Yu et al. 2023). Then, we describe the widely-used sinusoidal time encoder. We focus on attention attention-based models because self-attention is prevalent in mainstream neural architectures for not only text (Waswani et al. 2017
and vision Dosovitskiy et al. 2021 but also graphs (Veličković et al. 2018 Dwivedi & Bresson 2021 K im
et al. 2022) We investigate the two models in specific because TGAT introduced the sinusoidal time encoder to work with graph attention network without additional heuristics and DyGFormer is the state-of- the-art transformer-bast dynamic graph model. In Section 3 we motivate the use of lincar encodings by proving that self-attention can compute time spans between events from these encodings, making this applicable to both models. Additionally, as TGAT and DyGFormer are representative graph and sequence- baseed models, our results apply to both paradigms.
2.1 Future Link Prediction
Future link prediction the problem of predicting whether an edge will appear in a continuous-time dynamic graph.
Definition 2.1 (Conlinuous- Time Dynamic Graph). A continuous-lime dynamic graph over time interval I = [0, is an ordered pair G = (V,E) where == (1....,N) is a set of nodes and & C ((i,j,t) E V2 x Ir} is a set of edges representing the interaction events between nodes where i is the source node, is the destination node, and is the time of interaction. Depending on individual datasets, each node i V may be associated with node attributes x; TRyv and each edge (i,j.t) & may also be associated with edge attributes x Rdd.
involves assigning a score that estimates how likely it is that target edge (i,j,t) will form at time given the history of interactions up that point. We call an edge (i,j,() positive (i, t) and negative otherwise. Evaluation for future link prediction typically treats the task as binary classification problem between positive and negative edges, using ranking metries such as Area Under the Curve (AUC) or Average Precision (AP). We expand on different ways of
Future link prediction (Poursafaei et al. 2022 Qin & Yeung 2023
sampling negative edges in Section 4.1.2
2.2 Attention- BreettionBBased Dynamic Graph Learning Models
To estimate how likely target edge (i.j.t) is occur, typical dynamic graph learning model computes the temporal representations of node i and according to their first-order or higher-order historical edge interactions. Using node i as an example, the first-order historical edge interactions involving node i are denoted as S = {(i, u, t')tt' < t, (i, vi,t E E] i(u,i,t')l < t, (,, e E}. The first-order temporal neighbors of node i are denoted as No = {(v, (')(t,) v,tt) E E (v,, E S The temporal representations of node and are later concatenated and forwarded through multi-layer perceptron (MLP) obtain final score how likely (i,j,) will happen. Below, briefly introduce two representative attention-
3

---

Published in Transactions on Machine Learning Research (07 (025))
based dynamic graph learning models, TGAT and DyGFormer. For more detailed review, please refer to Appendix A
TGAT is graph-based model that recursively aggregates higher-order temporal neighbor information through multiple layers. The initial temporal representation of node h (t), is its raw attributes x,, To compute the I'th layer temporal representation of node i, h(((t), (t), from the I - 1'th layer, TGAT uses self-attention where node acts as the query and its temporal neighbors act as the keys and values. Tet @ I gdr denote a time encoder that maps time values to di-dimensa time encodings. For each node v and interaction time l' involved, the input features lo self-attention consists of the -- 1'th layer temporal node representation h (t') and a time encoding that encodes the difference between t' and the target time t, i.c. ((t- After aggregating information from temporal neighbors into a vector, concatenatod with and forwarded through an MLP to get h (t).
(t')
DyGFormer is a sequence-basee based model that considers the first-order temporal neighbors of a node when computing its representation. To compute the temporal representation of node at time t, the target edge and the historical odge interactions involving i, St are retrieved and sorted by the timestamps in ascending order. The features corresponding to each edge are collected, including neighbor node attributes, edge attributes, neighbor co-occureence features, and time encodings Φ(t t') where t' is the edge timestamp, The target edge attributes are filled with a zero vector since the target edge has not occurred. Eventually, the four types of features are each linearly transformed into channel dimension of deh and concatenated together as the input sequence matrix to the transformer X Later, the sequence matrices X and X corresponding to the two nodes of the target edge (i. are concatenated in the sequence dimension and forwarded through an L-layer transformer to output (t), which contains edge interaction representations of both nodes. Finally, DyGFormer applies mean pooling and a linear transformation to the positions corresponding to i and in (t) to get their respective temporal representations.
2.3 Sinusoidal Time Encoder
Time encodings are essential for capturing the temporal relationships between events. Importantly, the time spans between events matter more than their absolute timestamps. For example, the influence of edge in- teraction at time & to another one at to should depend on 1 -t2. Motivated by this intuition, Xu et al. (2019) propose a time encoder that ensures the inner product between encodings reflects the time span. They achieve
this with the sinusoidal encoder (tt)= V žे [cos(wit)) sin(wit) ... cos(way t) sin(wa,, ()] with learnable pa-
rameters Wi....,Wd, Wdr R, which satisfies the desired property: (I(t1), ((t2)) = 1 Eee cos(wi(t: -t2)
The sinusoidal time encoder is then introduced to dynamic graph learning in TGAT (da Xu et al. 2020).
Several subsequent models such as GraphMixer (Cong et al. 2023) and DyGFormer (Yu et al. 2023) follow this choice. Their implementations use all cosine functions with phases to encode the difference bet ween the target time and an edge event time for the edge: (t--t') = (cos(wi(t-t')) This is valid design choice since cosines are as expressive as sines, and since only the time span between events matters, the time reference point can be shifted without affecting the span: |f1-t2| I(t-tz)- (t- While GraphMixer uses fixed frequency and phase parameters, TGAT and DyGFormer make them learnable, we focus learnable time encoders in this study.
... cos(wa, (t-t') ('--t)++ + )
3
Do We Need Sinusoidal Time Encodings?
3.1 Linear Time Encoder
Although the inner products of sinusoidal encodings can be expressed as function of the time differences, both the encodings and the inner products many-to-one mappings from time values a fixed range [],, which inevitably loses information. We consider a simple linear time encoder tb Ir - Rdd, to encode the difference between the target time t and an edge event time t', At' =t t::
((At') = At' At' +b, ... Wdr At' + bdr
(1)
4

---

Published in Transactions on Machine Learning Research (07 (07/2025)
where Wi:..., Wdy E R and Di,...,,ds ber R are learnable weights and biases parameters. In practice, we encode the standardized time difference value (ÁI' Mar)//asr where Pay and ose are the empirical mean and standard deviation of observed time differences for numerical stability and optimization efficiency purposes. Encoding standardized time differences will still result in time encodings lincar to the original time differences. Unlike sinusoidal time encodings, each dimension i of a linear time encoding is one-to-one mapping from At' as long as wi # On the other hand, linear time encodings do not satisfy the inner product property that the sinusoidal time encodings do: (.(t1), o((2)) (011 12). Nevertheless, we show that self-attention can compute the time spans between events from linear time encodings in the following proposition.
Proposition 3.1. Let x (xx1, (XM, (t)} a set of events where each event m contains a feature veclor Xm Rd and limeslamp In E Ir- Let be the largel lime. There exisls linear lime encoder & IT Rd,, a query weight matriz We Rddrtd)xdn, and a key weight matria W K R s(dr+d)xd, that factor the time spon I'm in between any two events m,n into the attention score:
qm W [t(t= - tm))
@t tn)
kn wk kn) = hilm in) + g( kn)
Xm
Xn
where h RR - R models the patterns relevant to the time span and g Rdr-d x Rdv+d + models other patterns between the events.
Proof. Our proof provides a construction of d, WQ. and Wx that only requires specifying two dimensions of the parameters. Please refer to Appendix for the complete proof.
Proposition 3.1 shows that there exists a particular combination of a linear time encoder and sell-attention weights that computes the time spans. To understand whether such a combination can be learned from data, we design simple but representative synthetic task of event sequence classification and test whether linear time encoder and transformer can successfully learn the task.
3.2 Synthetic Task: Event Sequence Classification
3.2.1
Data-Generating Process
Consider sequence-label pairs with the form of (z( t.),... (er, v) where and tm are the feature value and timestamp of event m, t is the target time, and y is the sequence label. The data aata-generating process is as follows:
1,
with probability 0.5
Fm
11, otherwise
{ ~ Exp(A)
Am
Exp(A), tm 1m-m1 + Alm for m = 2,...,
...,M M
Al
Exp(A), 1= L.M + Al
E~ N00,00
M = 1 Σ
exp(-a - tm 2m +c>
(2)
M==1
where the intensity rate À and exponential decay rate a are the parameters. The labeling function resonates with the assumption that the more recent historical edge event takes time, the more influence has on the current state of node.
5

---

Published in Transactions on Machine Learning Research (07/2025)
I-layer Sefalttattonn
Layer Autoregressive
Layer
Layer Autoregressve
6.6
0.00
sine+cosine
0.5
Irierr
0.4
0.00
d;
dr
di
d:
Figure 2: Test accuracies of transformers vs. dr on the synthetic task. The left two figures are the results of full self-attention and autoregressive attention of 1-layer transformers. The right two are the results of 2-layer transformers.
1-Layer Autoregressive, =
1-Layer Autoregressive, de
Leamndd
Learned
Attersion Score
822
1.20
820
1.15
1/5
Event Index:
Event Index: m
Figure 3: Average attention scores for each event index m of I-layer autoregressive transformers with linear time encodings. We compare the attention scores when using dr = and 8 with the true attention scores derived from the labeling function in Eq.
3.2.2 Event Sequence Classification With Transformers
We encode the event sequence with an L-layer transformer into a sequence representation and classify it with a linear head. The input X (MM+1)((d++1) is organized as:
Φ(1 41)
x1
:
X =
Φ(t tM) ((0)
IM
We follow the transformer layer implementation in DyGFormer as reviewed in Appendix A Let d, denote the attention dimension and ( + denote the l'th layer sequence representation. We experiment with full self-attention and autoregressive attention. In full self-attention, we follow DyGFormer by aver- aging (L) over the sequence dimension to obtain the sequence representation: = M+I autoregressive attention, we apply a causal mask that prevents an event from aggregating information about any future events. Following the convention of autoregressive transformers (Radford et al.) 2018). we take
x (L) 1M41. For
the last. "token" representation as the sequence represenlation: z ((L) eM+1- After getting the sequence representation, we apply a linear head for binary classification. We train the transformer and linear head by minimizing the binary cross entropy loss between the prediction and y.
3.2.3 Experiments
We conduct experiments see transformers with linear time encoders can learn the event sequence classifi- cation task. We use À 0.01, a 0.003, and M 7 to generate 2000 sequences total where 70%/15% /15% of the sequences are for train/valiin /at/iaaaaaaaaanessl..... We experiment with 1-layer and 2-layer transformers and vary dr in between {2, 4, 8, 16). We compare linear time encodings, sinusoidal time encodings with all cosines, and sinusoidal time encodings with sine and cosine pairs. We do 10 runs for each configuration and present the results in Figure 2
The results show that transformers with linear time encoders are capable of learning the event sequence classification task even when dr is small. In Figure 3 we also visualize the attention scores of 1-layer autoregressive transformers with linear time encodings. which essentially uses (X)M+1. as the query and (X)::;; as the keys and values. We know the true altention scores based the labeling function in Eq.

---

Published in Transactions on Machine Learning Research (07 (025)
DyGFormer
DyGFormerseparate
DyGDecoder
Transformer
Transformer (Causal Masking)
Neighbor Edge foature
(X))
(X))
(xt)
(x))
[x]]
Neighbor co-ocaurence Temporal festure
sequence the source the targel edge
sequence the
target edge
beginning of sequence embedding
Figure 4: An illustration of the DyGFormer variants that we experimented with on dynamic link prediction. DyGFormer-scpardd and DyGDecoder forward the source and destination node feature sequences separately through the transformers while DyGFormer concalenates them and forwards jointly. In addition, DyGDe- coder applies causal masking in its self-attention blocks and prepends a learnable "beginning of sequence" embedding to the input feature sequence matrices.
2 We average the Icarned and true attention scores over all sequences per event index and compare the two in the plots. According to the plots, the 1-layer transformer can learn the exponential decaying pattern when dr = 2 and becomes closer the true labeling function when dr 8. Our experiments confirm that self-attention can learn compute time spans from linear time encodings and extract relevant patterns. Motivated by this result, extend our empirical study to future link prediction dynamic graphs.
3.3 Modifying TGAT and DyGFormer
We study how different time encoders perform with TGAT and DyGFormer in the context of future link prediction. For TGAT, we replace its sinusoidal time encoder with linear time encoder and compare their performances. For DyGFormer, we consider 2 variants in addition to the original version and illustrate them in Figure The first variant, DyGFormer-separate computes the temporal representations of the source and destination nodes separately. The goal of this variant second variant, DyGDecodor, computes the source and destination node representations separately and in an autoregressive manner described in Section 222 In DyGecedrr, sequence" embedding the input feature sequence matrices of the source and destination nodes, signaling the start of the sequences and enabling the transformer to count the sequence lengths (Kazemnejad et al. 2024). Similar to our approach for TGAT, we modily the time encoders of all three DyGFormer versions.
rule out the effect of cross-sequence attention. The
prepend learnable "beginning of
To verify that any differences between the sinusoidal and linear time encoders are not solely due to standard- ization, we also introduce a sinusoidal-scale time encoder. This encoder employs the same scaling strategy as the linear time encoder the time differences but encodes them with sinusoidal functions, allowing us isolate the impact of scaling from the underlying encoding technique.
4 Experiments
4.1
Setup
4.1.1
Data
For our main experiments, we use six standard dynamic graph learning benchmark datasets: UCI [Panzaraaa ct al. 2009), Wikipedia (Kumar et al. 2019). Enron (Shetty Adibi (2004). Reddit (Kumar al. 2211)). LastFM (Kumar et al. 2019) and US Legis (Fowler) 2006 Huang al.J 2020). We list the details of the datasets in Appendix Following the setup of the unified dynamic graph learning library, DyGLib YYu)
7

---

Published in Transactions on Machine Learning Research (07/)025)
Table 1: Test AP (random NS) across six datasets. The best performance for each model variant high- lighted in bold and the second best performance highlighted by underline. Counts of dataset wins are also boldfaced, and in every model variant the linear time encoder secures the most wins.
Test AP
Time Encoder
UCI
Wikipedia
Enron
Reddit
LastFM
US Legis
# Wins
(Random NS)
TGAT
Sinusoidal
80.27 40.42
97.00
72.07.1.10
98.55
75.8210.77
67.42.1.4m
1
Sinusoidal-sclle
77.50
95.58_0..0
78. 45
98. 08.43.00.2
82. 72.70
69.01
1
Lincer
95.41 10.06
98.21
82.31 20.0e
98.5240.02
83.10
68.22-1.79
4
DyGPormer
Sinusoidal
96.01 10.10
99.04
93.63
99.20gu.01
93.6820.04
70.124..00
2
Simusodda--selle
84. 76.
99.15
93. 19.1911021
90.25.10.01
94 a4.33
70.08.0..4
0
Lincer
96. 20.4442
99.19.2o.z
93. 200
99.28-.0.1
94.34
70.44
4
DyCrormer-separate
Sinusoidal
96.15 +0.02
99.04=0.02
93.49
99.244n.01
93. 93.69=n.07
68,03=1.11
2
Sinusoidal-scale
16.77
99.15=o.m
93.33+0.14
99.29
=4.30=0.11
69.36+1.84
1
Linear
95. 80.88
99.24=0.02
93.49=0020
99.30=00.0
94.30=0.14
70.98=1.22
5
DyGDecoder
Sinusnidal
20.00
99.03=0.02
93.5320.04
99.25::0.01
94.21:0000
69.39=1.00
Simusoidal-scale
91.25=11.2
98, 08.93=00.14
93. 63.62e0.16
99.30=a.04
94.32+0.1a
69.52=0.a6
Linear
97.38
99.21=0.01
93.86=0011
99.32=0.01
94.44=0.0s
70.12=1.11
6
ct al. 2023). we split the time span of an entire dataset into 70%/15%/15% for train/validaaton/teen We
denote the starting time points for the validation and test splits by Lvat and best where 0 < baa < lesst T.
4.1.2 Negative Edge Sampling
As in DyGLib, we sample an equal amount of negative edges as the positive edges for the training and evaluation of future link prediction. We adopt two negative sampling (NS) strategies for evaluation: random NS and historical NS (Poursafaei et al. 2022). Let E(ī, = ((i.j.0)(i E E, << f} denote the set of edges that occur within the time range of (i, and P(t, = ((..))((, E E(i,)) denote the node pairs that interact within the range. Given a batch of positive test edges E(T, where bust i < < T, random NS substitutes the node pairs of the positive edges by sampling from all possible node pairs uniformly: (i, f) {(u, v, t) u, t)l,, that do not interact in the given batch of positive edges but have interacted in the past for substitution: Epie (i, 1) {(u, (), v) E P(O, 7), (u,v) & P(i, (i, E E(ī, We train under random NS and do model selection under random and historical NS for evaluation under random and historical NS, respectively.
Unif(V), (i, E f)). Historical NS draws node pairs
4.1.3 Evaluation Metric and Parper-Paramttes
We follow DyGLib to use the average precision (AP) as the evaluation metric, set the batch size to 200, and use the Adam optimizer KKngma 2014) with a learning rate of 0.0001. The sinusoidal time encoding dimension dr is set to 100. We also set the linear time encoding dimension to 100 for TGAT but set it to for DyGFormer variants since the time encoder is followed by a linear projection to a temporal feature channel of dimension deh, effectively resulting in a da--dimensional linear time encoder no matter what dr is.
We use the same other syper-parameet for training the original DyGFormer under random NS as the ones in DyGLib since they are well-tuned. For the LastFM dataset, directly apply the saper-parameters specified in DyGLib as well due to the scale of the dataset. As for other cases, we search the prper-ppramcters of cach method on each dataset. The search space for TGAT is the dropout rate among {0.1, 0.3, 0.53. The search space for DyGFormer variants is the combination of channel dimension deh E (30, 50} and the dropout rate among (0.1. 0.3, 0.5). The best syper-parameters are determined based on the validation AP score. Following Poursafaci al. (2022) and DyGLib, we report the average results and standard deviation over five runs.
8

---

Published in Transactions on Machine Learning Research (07/2025)
Table 2: Test AP (historical NS) across six datasets, The best performance for each model variant is highlighted in bold and the second best performance highlighted by underline. Counts of dataset wins are also boldfaced, and in every model variant the linear time encoder secures the most wins.
Test AP
Time Encoder
UCI
Wikipedia
Enron
Reddit
LastFM
US Legis
# Wins
(Historical NS)
TGAT
Sinusoidal
68.94 59.94
88 02.02.....
64.82-2.31
79.0 29.62.ao22
77
84.59.4.30
Simusoidal-cele
41.84
87.4940.64
78.40
79.38
80.90.2..5
31.96.2.51
0
Lincar
91.42.0.27
92.02.0.14
20.14
20.70
81.5210..8
81.6414.21
5
DyGPormer
Sinusoidal
82.13
83.81 40..8
28.13
83.11 10.04
82 92.8330020
84.5811.24
Simusoddl-seele
80.29
a2.4
75.32=2211
82 02.4
83.79
87.07
1
Lincer
89. 29.41
86.23-10.e5
78.24
82 82.31
84.43-1.14
85.80&1.36
4
DyCrormer-separate
Sinusoidal
=2.42
86, 25
75.70=1.95
82.60
82.70=0.40
8G.33+2.58
0
Sinusoidal-scale
80.92
84.03+0.50 80
76.87=1.10
83.05+1.4
83 83.90+0.46
82.54+a.29
0
Linear
83.09
88.89:.0.99
80.52#0065
13.8
84.26#0.55
89.44#1778
6
DyGDecoder
Sinusnidal
84.49=0.01
87.01+1042
79.75=0.18
84.6520447
84.70=0.18
84.74:-477
2
Simusoidalsscale
82.27=1.91
86.27
81.38=00.0
85.09
84.57=0.30
81.75=a4.17
1
Linear
87.08=0.89
88.29:0044
82.00=0.50
85.03=0.55
84.41+0.49
83.79+2.15 12
3
4.2
Results and Discussions
4.2.1 Cosine vs. Sine-Cosine Pairs
TGAT and DyGFormer implement sinusoidal time encoders using only cosines, while their respective papers describe them as sine-cosine pairs. To investigate this, we first compare the performance of sinusoidal time encoders with all cosines and with sine-cosine pairs. The results are presented in Appendix [ Table 4 We find that there is no significant difference in performance between the two versions. Therefore, simply adopt the version with all cosines implemented by TGAT and DyGFormer for the remainder of our study,
4.2.2 Evaluation Under Random NS
We compare sinusoidal, simusoidal-scale, and linear time encoders under random NS and present the results in Table []] The linear time encoder outperforms the others in 19 out of 24 model-dataset combinations, while the sinusoidal and simusoidal-seale encoders perform best in 5 and 2 combinations, respectively. The largest improvements in test AP scores when switching from sinusoidal to linear are 15.14 and 10.24 for TGAT on UCI and Enron, respectively. The most notable performance drop is modest decrease of 0.34 in the AP score for DyGrormer on Enron. When comparing linear and sinusoidal-scale time encoders, which both encode the same scaled time difference values using different functions, the linear encoder outperforms the sinusoidal-seale encoder in out of 24 cascs. This suggests that allowing self-attention to learn patterns relevant to time spans directly from the linear encodings is more effective than relying on pre-encoded sinusoidal functions.
Regarding the comparison between sinusoidal and sinusoidal-scale encoders, we observe that scaling consis- tently degrades performances on UCI, which is not the case for the other datasets. To investigate further, we analyze the inter-event times in UCL. For the source and destination nodes of each edge, we calculate the waiting times since their last interaction (see Appendix D.2 Table 5p. We find that the waiting times increase significantly during the test split, which may hinder the sinusoidal-scale encoder's performance, as it relies on time difference statistics in the train split for scaling. Despite the negative impact of scaling under distribution shift, the lincar time encoder is able to recover its performance. This echoes findings in the literature relative positional encoding in language models (Press el al. 2022) which discovered that the linear function of positional differences shows strong length generalization.
4.2.3 Evaluation Under Historical NS
For evaluation under historical NS, we use the validation AP scores under historical NS for model selection. We present the results in Table Among all 24 model-dataset combinations, the linear time encoder is
9

---

Published in Transactions on Machine Learning Research (07/202) (225)
Test AP (Rardom NSI
600
UCI
Wikipedia
330
Enron
96
94
75
Sinusoidal
Linear
52
51% params
75% para
200% serams
57% panumu
75% de sanama
102% panamu
57% paramu
75% parumm
100% parama
LastFM
Reddit
US Legis
TestAP (Random NS1
500
90
73
76
00
d-
-dr=2 57% params
57% paroms
76% parains
100% sorams
7696 60 aon
100% paroms
57% paroms
26% parsims
100% params
Figure 5: Test AP (random NS) vs. dr and the percentage of the number of parameters relative to dr = 100 in TGAT.
the best in 18 instances while the sinusoidal and sinusoidal-scale encoders are the best in 4 and instances, respectively.
The linear time encoder consistently outperforms the sinusoidal and sinusoidal-scale time encoder on UCL, Wikipedia, and Enron. The most significant differences caused by switching the time encoder from sinusoidal to linear are the increases in average test AP scores on UCT and Wikipedia from 68.94 to 91.42 and from 64.82 to 80.14 with TGAT. Other notable performance enhancements with DyGFormer variants include the 7.28 and 4.82 increases with DyGFormer on UCI and DyGFormer-soparat on Enron.
We observe that the linear time encoder does not achieve the best performance on US Legis under historical NS evaluation, winning only with DyCFormer-separate. This is likely because sinusoidal time encoders are sulficient for this dalaset's 's coarse-grained temporal granularity. US Legis is discrete-time dynamic graph with only 12 evenly spaced time steps in the units of congressional sessions, resulting in just 12 possible time differences: {0, 1,..., 11). In such a sctting, a simple onc-dimensional sinusoidal encoding like cos т At) can map all time differences to unique values in [-1,1] without collisions. contrast, applying this approach to continuous-tine time datasets where time differences span up to 10 seconds would lead to collisions due to limited numerical precision (e.g. cos( x 0) cos( 85) ss 1 in float32). This example illustrates why encoding time differences is simpler in US Legis and why sinusoidal encodings are sufficient, while the advantage of the linear time encoder's injective property emerges in continuous-time scenarios.
11
4.3
Additional Analyses
4.3.1 Reducing Temporal Feature Dimension
In the synthetic task experiments in Section 3.2.3 we observe that transformers with a low-dimensional linear time encoder can effectively learn the synthetic task. We explore whether the same holds for realistic dynamic graph datasets.
In TGAT, the original time encoding dimension dr is 100. We reduce dr to be 50 and 2 for both sinusoidal and linear time encodings and plot the results in Figure 5 The most significant performance decline with the sinusoidal time encoder occurs LastFM, where the AP score decreases by 8.08. In comparison, the largest performance drop with the linear time encoder is only 2.44 on UCI. On all datasets except Reddit, linear time encoder with dr = 2 outperforms sinusoidal time encoder with d, 100, while saving 43% of parameters.
We conduct similar experiments with DyGFormer. In the original DyGFormer with sinusoidal time encoders, the dimension for each channel de, 50. For UCI, Enron, LastFM, and US Legis, we also set deh 50 for DyGFormer with linear time encoders determined in the hyper-parameter selection phase. On these
10

---

Published in Transactions on Machine Learning Research (07/2025)
Test AP (Random NS)
28
UCI
Enron
LastFM
72
US Legis
97
24
71
Gy-
70
de- 24
di
du 24
de
24
15
Sinusoidal Linear
2000 Par Parameters (k)
200
902
1102
803
1200 1220 Parameters (K)
2000
1000
1300 5400 # Parameters (K)
1120 1200
600
1200 :200 # Parameters (k)
500
1000
Figure 6: Test AP (random NS) under varying temporal feature channel dimension die and number of parameters in DyGFereer.
3.5
11
10
Eo 159
G2 150
E- Ds1
E- 0st
(a) Sinusoidal/
(b) Linear/ DyGroreer-separate
(c) Sinusoidal/ DyGDecoder
(d) Linear/ DyGDecoder
DyGFormer-spparat
Figure 7: Attention scores (expressed in colors) of DyGoormer-separree and DyGDecoder with the sinusoidal and linear time encoders vs. t t- tx vs. t t- to of the historical edge event sequences Reddit's source nodes.
datasets, experiment with reducing solely the dimension of the temporal feature channel to 24 and 2 while the dimensions for the other channels are fixed at 50. We separately denote the temporal feature channel dimension by due since it is different from the dimensions for the other channels da, in this auxiliary experiment. The results, shown Figure D reveal that while performances decrease on Enron and Fa with both encoders as die is reduced, DyGFormer with a linear time encoder improves on UCI and US Legis, whereas the sinusoidal encoder continues decline. Overall, these experiments confirm that lincar time encoders are more parameter-effiiintt
4.3.2 Attention Score Analysis
Attention scores are useful for understanding how edge event representations aggregate information. We choose to visualize the attention score patterns in the last layers of DyGFormer-soparate and DyGDecoder since they rule out the effect of cross-sequence allention.
We sample K target edges from the test set. For each edge (i.j.t), we retrieve the historical edge event sequences of node i and j. For each sequence, record the attention score between query vector and key vector alongside their encoded time differences, ( 1-1, lg and ( - 4.- We then plot the attention scores and time differences for all K historical edge event sequences of the source nodes and repeat the process for the destination nodes. Figure 7 shows the visualization results for the source nodes of Reddit. The other results are presented in Appendix D.7
Figure 7a and 7b reveal distinct attention patterns of DyGFormer-scpprraee on Reddit when different time encoders are used. With the linear encoder, the attention scores are negatively correlated with the elapsed time since the key edge events, so the colors smoothly transition along tg. In contrast, the sinusoidal encoder produces scattered points of high attention scores separated by low-score regions, likely due to the time encoder's periodicity. However, such stark differences are absent across the other datasets. Regardless of the time encoder paired with DyCGormer-separatee attention scores tend to decline as I - 4, increases, implying that the freshness of past interactions typically the dominant factor.
Regarding the differences between the visualization results of DyGFormer-separee and DyGDecoder, the most notable one is that no attention scores recorded in DyGDecoder for - lg > - lg- This arises
11

---

Published in Transactions on Machine Learning Research (07/2025)
from its autoregressive attention mechanism. Another interesting distinction is that DyGForrmrr-epprrate highest attention scores tend to be close to the line L - 4 0 while DyCDecoders are close to the line t = - tg. This difference likely arises from their distinct pooling strategies. DyGDecoder uses the final event representation (i.e., the target odge event) as the sequence reprosentation, which aggregates information from preceding events. Consequently, it learns to attend more strongly to key vectors where is is close to l.. In contrast, DyGFormer-separate applics mean pooling over all edge event representations to form a sequence representation, so even a query vector corresponding to an early event allocates most of its attention to the latest events (where ty is closer to t) because those events are more influential.
In summary, our experimental results demonstrate that linear time encoders mostly outperform sinusoidal encoders when paired with TGAT and DyGFormer. Moreover, self-attention can effectively learn temporal patterns from linear time encodings with far fewer dimensions. While both time encoders can be used together in practice, as in Time2vec (Kazemi et al., 2019) used in time series modeling (Shukla & Marling 2021), our findings highlight that the linear encoder offers advantages in reducing model complexity. For large-scale temporal network applications, adopting linear time encodings can significantly streamline the architecture by avoiding high dimensional sinusoidal encodings, used in TGAT and DyGFormer.
5 Related Work
2012 Longa ct al. 2023) are evolving graphs associated with temporal information. There are two classes of dynamic graph learning methods: discrete- time methods (Pareja et al. 2020 Sankar et al.) 2020 the temporal order within each snapshot, and contimous-time methods (Kumar et al. 2019 Trivedi et al.,
Dynamic graphs (Holme & Saramäki
You et al.
2022). which discretize dynamic graphs into static snapshots and lose
2019 Rossi et al.] 2020 da Xu al. 2020 Wang et al. 2021c Cong et al. 2023 Yu et al.) 2023), which
directly model temporal dynamics and more versatile. This work focuses the latter.
Continuous-time methods can be divided into memory-based and non-memory-based methods. Memory- based methods (Trieei et al. 2019 Kumar et al., 2019 Rossi et al. 2020 maintain a memory for each node, updating it whenever an event. related to the node occurs. In contrast, non-memory-based methods retrieve a node's historical information as needed, allowing for more expressive models to compute its temporal representation. Non- no--mmmory-based methods can be further subdivided into graph-based and sequence- based approaches. TGAT (da Xu et al., 2020) and TCL (Wang al., 2021b) belong the graph- based category, as they aggregate higher-order temporal neighbor information. On the other hand, DyGFormer (Yu et al. 2023) and GraphMixer (Cong et al.) 2023) are sequence-based approaches because they only consider first-order temporal neighbors. CAWN (Wang et al., 2021c) sits between the two approaches, as it utilizes a recurrent neural network (RNN) Rumelhart et al. 1986) to learn representations from causal anonymous random walks over the graph. It is worth noting that TCL uses linear functions to represent time values, which similar lo the linear time encoder that we present in Section 3 However, TCL does not provide a justification for the design choice nor compare it with sinusoidal encodings. Aside from methods, evaluation is also a crucial component of dynamic graph lcarning. [Poursafaci et al. (2022) introduce historical and inductive negative sampling to test if future link prediction models simply memorize past edges. TGB (Huang et al., 2024 Gastinger et al.] 2024 provides standardized package that includes datasets, data loaders, and evaluators. DyGLib Yu et al. 2023) is a unified library for dynamic graph learning that contains implementations of various methods, data processing, and evaluation.
The sinusoidal time encoder (Xu et al. 2019 da Xu et al., 2020) resembles the positional encoding in Trans- former (Waswani et al. 2017). which uses sinusoidal functions to encode discrete language token positions. Subsequently, there have been a scries of development in relative positional encodings (RPE) in language
modeling (Shaw et al. 2018 Raffel et al. 2020 Press et al. 2022 Chi et al. 2022 Su et al. 2024), which inject relative positional information to attention scores after computing the inner products between key and query vectors. We do not consider language model RPEs in this paper for two reasons. Firstly, language model RPEs have not been directly applied to dynamic graph learning to the best of our knowledge. Secondly, there is only a single position for cach word token in language modeling while DyGFormer uses a patching technique to merge multiple edge events into a single "token". Therefore, each "token" in DyGFormer may be associated with multiple time positions and it is unclear how to compute the relative positional infor-
12

---

Published in Transactions on Machine Learning Research (07/2025)
mation between two tokens. This would not be problem for sinusoidal and linear time encodings because they can be pre-computed and then patched into the same vector before sell-attention. Teme2Vec Kazemi et al. 2019) is a time encoding technique used in irregular time series modeling (Shukla & Marlin 2021 that combines a linear time feature with multiple periodic time features. The key difference between Time2Vec's linear time feature and our linear time encoder is that our encoder standardizes time values, which we found significantly benefits optimization efficiency and improves results in the carly stages of our experiments. This work focuses on the comparison between linear and sinusoidal time encoders to isolate the effects of their respective functional forms. We leave thorough exploration of their combination to future work.
6 Conclusion
In this study, we present the linear time encoder as an effective alternative to the widely-used sinusoidal time encoder for attention-based dynamic graph learning models. Our theoretical and experimental findings demonstrate that the linear time encoder not only allows the self-attention mechanism to capture tempo- ral relationships but also outperforms sinusoidal time encoders in most scenarios, particularly in terms of predictive accuracy and parameter savings. While both encoders can be used together simultancously, study sheds lights on the advantages of linear time features, which are overlooked in modern dynamic graph models. Looking ahcad, exploring the possibility of adapting language model RPEs to contimous-time dynamic graphs is an exciting direction, as they have shown better generalization performances in the lan- guage domain. Ultimately, these advancements will enable more effective dynamic graph models, driving improvements applications such as recommender systems and traffic forecasting.
References
Ta-Chung Chi, fing-Han Fan, Peter J Ramadge, and Alexander Rudnicky. Kerple: Kernelized relative positional embedding for length extrapolation. Advunces in Newral Information Processing Systems, 35: 8386 8399, 2022.
Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, and Mchrdad Mahdavi. Do we really need complicated model architectures for temporal networks? In The Eleventh International Conference on Learning Representations, 2023. URL ayPPc0SyLv1
fort/frrmm?dd
da Xu, chuanwei ruan, evren korpeoglu, sushant kumar, and kannan achan. Inductive representation learning on temporal graphs. In International Conference on Learning Representations, 2020. URL https://
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. AAAI Workshop on Deep Learning on Gruphs: Melhods and Applications, 2021.
James H Fowler. Legislative cosponsorship networks in the us house and senate. Social networks, 28(4): 454-465, 2006.
Julia Gastinger, Shenyang Huang, Mikhail Galkin, Erfan Loghmani, Ali Parviz, Farimah Poursafaei, Jacob Danovitch, Emanuele Rossi, Ioannis Koutis, Heiner Stuckenschmidt, Reihaneh Rabbany, and Guillaume Rabusseau. TGB 2.0: benchmark for learning on temporal knowledge graphs and heterogeneous graphs. In The Thirty-eight Conference on Neurol Information Processing Systems Datasets and Benchmarks Track, 2024. URL httt:s///ppparrrrrooeeeeooooocco
Petter Holme and Jari Saramäki. Temporal networks. Physics reports, 519(3):97-1255 2012.
13

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Shenyang Huang, Yasmeen Hitti, Guillaume Rabusseau, and Reihanch Rabbany. Laplacian change point detection for dynamic graphs. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 349 358, 2020.
Shenyang Huang, Farimah Poursafaci, Jacob Danovitch, Matthias Fcy, Weihua Hu, Emanuele Rossi, Jure Leskovec, Michael Bronstein, Guillaume Rabusseau, and Reihaneh Rabbany. Temporal graph benchmark for machine learning on temporal graphs. Advances in Neural Information Processing Systems, 36, 2024.
Scyed Mchran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jasprect Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning a vector represen- tation of time. arXiv preprint arXi:1997.05322 2019.
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Newral Information Processing Systems, 2024.
Jinwoo Kim, Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure transformers are powerful graph learners. Advances in Neural Information Processing Systems, 35: 14582-14595, 2022.
Diederik P Kingma. Adam: A method for stochastic optimization. arviv preprint orXiv:1412.6900 2014.
Srijan Kumar, Xikun Zhang, and Jure Teskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & dola mining, pp. 1269 1278, 2019.
Antonio Longa, Veronica Lachi, Gabriele Santin, Monica Bianchini, Bruno Lepri, Pietro Lio, franco scarselli, and Andrea Passerini. Graph neural networks for temporal graphs: State of the art, open challenges, and opportunities. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: //openreview wttp:/vvwiieeeetooreeeeeeeeeeeee het//toruniid-piccMMtttg"
Pietro Panzarasa, Tore Opsahl, and Kathleen M Carley. Patterns and dynamics of users' behavior and interaction: Network analysis of an online community, Journal of the American Society for Information Science and Technology, (055)991-9322 2009.
Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao Schardl, and Charles Leiserson. Evolvegen: Evolving graph convolutional networks for dynamic graphs. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5363-5370, 2020.
James W Pennebaker. Linguistic inquiry and word count: Liwe 2001, 2001.
Farimah Poursafaei, Shenyang Huang, Kellin Pelrine, and Reihaneh Rabbany. Towards better evaluation for dynamic link prediction. Advances in Neural Information Processing Systems, 35:32928 32941, 2022.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with lincar biases enables input length rxtollaanio In International Conference on Learning Representations, 2022. URL https:
//openreview wwwparrrviieene/fouuumuuuma
Meng Qin and Dit-Yan Young. Temporal link prediction: A unified framework, taxonomy, and review. ACM Computing Surveys, 56(4):1 40, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1 166)) 2020.
14

---

Published in Transactions on Machine Learning Research (07 (07/)025)
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning dynamic graphs. In ICML 2020 Workshop on Graph Representation Learning, 2020.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, (2088)))) 536, 1986.
Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang, and Hao Yang. Dysat: Deep neural representation learning on dynamic graphs via self-attention networks. In Proceedings of the 13th international conference on web search and data mining, pp. 519 527, 2020.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint Xrxi 100.00215, 2018.
Jitesh Shetty and Jafar Adibi. The enron email dataset database schema and brief statistical report. Infor- mation sciences institute technical report, University of Southern California, 4(1):120 128, 2004.
Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time scries. In International Conference on Learning Representations, 2021. URL https://openreviee net/
4orum21d=440JJ6ww04
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfong Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhancod transformer with rotary position embedding. Neurocompuliing, 168:127063, 2024.
Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha. Dyrep: Learning representa- tions over dynamic graphs. In International conference on learning represenlations, 2019.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liö, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018. URL https:
//openreview wwwpeeewwwwwiwweeeeeeeeeeeeeeeee
JXMp kCZ.
Andrew Z Wang, Rex Ying, Pan Li, Nikhil Rao, Karthik Subbian, and Jure Leskovee. Bipartite dynamic representations ions for abuse delection. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 3638 3648, 2021a.
Lu Wang, Xiaofu Chang, Shuang Li, Yunfei Chu, Hui Li, Wei Zhang, Xiaofeng He, Le Song, Jingren Zhou, and Hongxia Yang. Tel: Transformer-baseee dynamic graph modelling via contrastive learning. arxiv preprint Xix::2105.09444 2021b.
Yanbang Wang, Yen-Yo Yu Chang, Yunyu Liu, Jure Leskovee, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. In International Conference on Learning Representations, 2021c. URL httpp:://pperreyt
net/forrum2id=WPP44YPPP
A Waswani, N Shazeer, N Parmar, J Uszkoreit, Jones, A Gomez, L Kaiser, and 1 Polosukhin. Attention is all you need. In NIPS, 2017.
Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self-attention with func- tional time representation lcarning. Advances in newral information processing systems, 32, 2019.
Jiaxuan You, Tianyu Du, and Jure Leskovec. Roland: graph learning framework for dynamic graphs. In
Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp. 2358-2366, 2022.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. arXiv preprint ar arXio:1709.04 04875, 2017.
Le Yu, Leilei Sun, Bowen Du, and Weifeng Lv. Towards better dynamic graph learning: New architecture and unified library. Advances in Neural Information Processing Systems, 36:67686-6777 67700, 2023.
Mengqi Zhang, Shu Wu, Xucli Yu, Qiang Liu, and Liang Wang. Dynamic graph neural networks for sequential recommendation. IEEE Transactions Knowledge and Data Engineering. 35(5):47414477 2022.
15

---

Published in Transactions on Machine Learning Research (07/2025)
A Details of Ettention-Bassd Dynamic Graph Learning Models
A.1 TGAT
TGAT (a Xu et al. 2020) is graph-based dynamic graph learning model. We describe the operation of the I'th layer of TGAT when computing the representation of node i at target time denoted by h The imitial temporal representation of node is its raw attributes, i.e. (t) = X; To compute the next layer of node i's temporal representation, TGAT uses self-attention where node acts as the query and its temporal neighbors act as the keys and values. For simplicity, we label the temporal neighbors of node as N? = ((11,4) (12, 12).... (0W7)):4/)))) Let Ir + Rdr denote the time encoder and Wq. WK, Wv R(d+dr)xd,, be the query, key, and value weight matrices. For each node and time t' involved, the imput to the self-attention consists of the / I'th layer representation h 1 (t') and the encoded time difference ((t - t'). In specific, the query vector corresponding to node at time t, (t(,), is computed as:
i,/(()) (t) Rd.
h (1)) (i(1== W
E
((0))
The time difference input to & is 0 because the difference between the target time and itself is O. The input to the key and value weight matrices are the temporal neighbor features N,)t):
ii (t))
Φt - t1) Φ - 12) T
h (
N₂(1=
E TR TRIN)I
((ww)))
(lww31) T
Φ(1 4x:)) )
The remaining part of self-attention carried out as follows aggregate the neighbor information to a vector (t)::
kt(t) = WK) E RlAllxd
Vi(t) = (t) E Rl
(1(1) (ttn(q.(), K,(1), V.(I))
K (t)q, Vdn
= vtt Softmax
E Rd
Finally, TGAT concatenates h2(t) with node i's raw attributes and forwards through an MLP obtain its l'th-layer temporal representation 6i (t).
h;,(t] x; + bo) + b
6 (1) W.ReLU
where Wo E (ddyx(da-vv) W, E gdxd, bo ,d b R2d are the MLP parameters. TGAT can be extended to make use of edge features corresponding to each temporal neighbor by concatenating them to N,(t).
A.2 DyGFormer
DyGFormer (Yu et al. 2023 is sequence-based dynamic graph learning model. To compute the temporal representation of node the target edge (i,j,) and the historical edge interactions involving i, S
16

---

Published in Transactions on Machine Learning Research (07 (025))
retrieved and sorted by the timestamps in ascending order. We denote the edge collection by S = S U {(),,,)) Then, the features corresponding to those edges are collected and arranged in the same order. There are four channels of features: temporal features, neighbor node features, edge features, and neighbor co-occurrence features. The temporal feature matrix X E R xllxdr consists the time encoding ((t-t') E Rd of each edge that has a timestamp t'. Similarly, the neighbor node feature matrix X E RIS x and edge feature matrix X [lai|xde contain the raw neighbor node attributes and edge attributes of each edge. Note that since the target edge has not occurred, its edge altributes are filled with a zero vector.
The use of neighbor co-occurrence features is a key technique in DyGFormer to strengthen the modeling of the correlation between two nodes, and j. of the target edge ji. t) for better link prediction. Given S and Sj. the occurrence frequencies of each neighbor in the histories nodes i and j are encoded into the neighbor co- occurrence matrices C; E IR x| and C₁ E || In specific, the n'th row of C. contains the occurrence frequencies of the m temporal neighbor of node in node i and j's histories where can be either i or j. For example, if the temporal neighbors of and j (including the target. edge) are {(u, 41), ) (v, 12), (we, 13), (j, {)} and {(v, t4); (u, to), (v, te), (v, 17), (i, t)}, we can count the frequencies of u, v w, i and j's appearances in i/j's historics as 1/1, 1/3, 1/0, 0/1, 1/0. Then, the neighbor co-occurrence matrices C and C₁ can be written as:
1
1
3
C
Cj
3
0
3
O
1
Later, the neighbor co-oceurrence matrices are transformed by an MLP fo into neighbor co-oceurrence features of dimension de:
X = fc(C))( Sc((C)) (c((C))))))((((C)..) E S
where: * can be either i or j.
After collecting the four channels of feature matrices, DyGFormer applies a patching technique, which essentially reshapes the matrices, to allow the transformer to capture longer-range information without increasing the computational cost. Let P denote the patch size. Given sequence of channel features X E Rls]]xd,, the sequence is divided into & = [/P1 patches and each patch is reshaped into single Pd.-dimensional vector, resulting in a shorter sequence of patches P E R If we fix the resulting sequence length E and increase the patch size P. the transformer can see through a longer history. The tradeoff is that the information of historical edge interactions within the same patch will be aggregated via a simple concatenation instead of the attention mechanism. The patching technique applied to all channels of features and produces the patch sequences for node i P RExPdv Pt. P R Pf.r E g Pl.c RR Afterward, each channel of the patch sequences is linearly transformed into the same dimension of deh and concatenated, resulting in the final input to the transformer:
x fi (P
E 14xxdd
where f1, f2, fs, and fa are linear transformations.
The feature matrices X and X corresponding to the two nodes of the target edge (1, are concatenated in the sequence dimension:
X
E
x
and forwarded through L transformer layers to output Z E x((+())xd where dy is the attention dimension.
DyGFormer sets the allention dimension d, all layers to be the same as the input dimension, Le. da = 4d
Let (t) E +D)x denote the l'th-layer sequence representation that is initialized as the input, i... X w will will) XRda denote the Pth layer attention weight matrices, and w( E
Z o (1)
17

---

Published in Transactions on Machine Learning Research (07/2025)
(0) Rd ddd, 2 E R xdd bj E Ridh, bg !
E IRda denote the l'th-layer MLP parameters. The l'th-layer
transformer computation goes as:
Q0 LayerNorml 7 ())
K()) LayerNorm( ())
V LayerNorm Z (L)
O Attn(Q K(), v(0) Softmax Q OOO
VVdn)v
(1) 0 (t)
bi (1)
b (1)
B
[b] (1)
1 (1)
(1++))xdd.
(1+15)xxd
Y GELU(LayrrNorm
+ B
+B²
Z (t) H (1) Y
After layers of transformer computations, DyGFormer applies mean pooling followed by a linear transfor- mation to the positions corresponding and j in ((4) to get their respective temporal representations.
B Proof of Proposition 3.1
be set of events where each event contains a feature vector Xm E Rdd and timestamp tm Ir. Let t be the target time. There erists linear lime encoder $ Ir + Rdr, a query weight matris We E ((dr+d) and key weight matric WK K tddrt that can factor the time span tm tn between any two events m, n into the attention score:
Proposition 3.1 (Restated). Let a =
9m We [ - (mm)
T ln)
kn Wk (gm: kn) hilt - tn) + g( kn)
Xm
Xn
where 1R R models the pallerns relevant to the time span and g Bar-d d x Rdr+d R. models other patterns between the events.
Proof. We provide a construction of Φ Wq, and Wx that only requires specifying two dimensions of the parameters. We first construct the linear time encoder t as follows:
wi (t t') + b1
( (()-t)= wa(1 1') + ba
wdy (t -t- t²) + bar
where W1 # 0, w2 o, and by There are no restrictions on other time encoder parameters: by. by, ba.... bay and wa, w4,... ... Wdy- Then, construct the query matrix We and the key matrix W K
18

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
follow:
0
91,3
91 dd
O
1
k1,3
ki,du
0
Q2.3
q2,d,
1
0
k2,3
d2,da
We
0
0
93.3
93.d,
WK
0
0
K3,3
K3,da
+d,
(d)xdd
0
0
hid
(dr (d)x)))dd
where only the first two columns are specified. Following the constructions, the query vector qn representing event m and the key vector k, representing event n have the form of:
w/ (t (m b1
-1
wi - tn) + bi
W
((t tm) Xm
93 kn
W
D tn)
h3
4m
Xn
qd,
kan
Therefore, the inner product between the two vectors will compute the time span between two events and factor it into the attention score:
(qm, kn) w (fm t.)) Σ qik: = ht(t - tn + 9(mm: k,) 2=3
Note that this is just one way of constructing the linear time encoder and atlention weights. There may exist numerous other possible implementations that achieve the same outcome. This proof inspired by Kazem- nejad et al. (2024) where they show that a two-layer autoregressive transformer without using positional encodings can recover the absolute and relative positions of tokens. The key difference between our setting and theirs is that our notion of distance between events is the time interval between two events instead of the number of "tokens" between them.
]
C Dataset Details
We describe the datasets that we used for the experiments below and present the dataset statistics in Table
UCT (Panzarasa, et al. 2009 An unattributed online communication network among University of California Irvine students.
Wikipedia (Kuaar et al.) 2019 This dataset contains edits made to Wikipedia pages
month where editors and pages represented nodes and timestamped posting requests are edges. Edge features are LIWC feature vectors (Pennebaker 2001) of length 172 extracted from the edit texts.
Enron (Shetyy& & Adibi 2004):
An unattributed temporal network of approximately 50,000 emails exchanged among employees of the Enron energy company over three years.
Reddit Kumar et al. 2019 : This datasot consists of subreddit posts over one month with nodes
representing users or posts and edges representing timestamped posting requests. Edge features are LIWC feature vectors (Pennebaker 2001) of length 172 extracted from the edit texts.
LastFM (Kumar al. 2019 An unattributed interaction network where nodes represent users and songs, and edges denote instances of user listening to song. The dataset captures the listening behaviors of 1,000 users with the 1,000 most popular songs the course of one month.
US Legis (Fowler 2006 Huang et al. 2020 A senate co-sponsorship graph documenting social inter- actions between U.S. Senate legislators with edge weights indicating how many times two legislators have co-sponsored bill.
19

---

Published in Transactions on Machine Learning Research (07/2025)
Table Dataset statistics.
Dataset
# Nodes
Total # Edges
Unique # Edges
Unique # Steps
Time Granularity
Duration
UCI
1899
59835
20296
58911
Unix timestamp
196 days
Wikipedia
9227
157474
18257
152757
Unix timestamp
month
Enron
184
125235
3125
22632
Unix timestamp
years month
Reddit
10984
672447
78516
669065
Unix timestamp
LastEM
1980
1293103
154993
1283614
Unix timestamp
month
Legis
225
60396
26423
12
Congresses
congresses
Table 4: A comparison between the sinusoidal time encoder using all cosine functions and one using sine- cosine pairs over six datasets. The better performance for each model variant is highlighted in bold. Same performances are highlighted by underline.
Test (Random NS)
Time Encoder
UCI
Wikipedia
Enron
Reddit
LastFM
US Legis
# Wins
TGAT
Cosine
80.27
97. 17.00
72.07
98 00
75.82s0.27
67.42:1.4
Sinc-cosine
81.48+0.47
97.11+0.14
70. 10.9122.12
98.51=0.02
76.05
67.14s190
DyGFormer
Cosine
96.01
20
03.63
99.20.0.01
93.66 04
70.124u.00
Sine cusine
96.03
09.04:0.04
93.53 10
99.21=000:
98.7320.05
70.23
Cosine
86.1510.00
93.49.0.15
99.24
93.692..07
68.03
Sine-cusine
95.88
-9.04
93. 03.42.0.32
99.24+0.01
93.73=0.07
1.1
DyGoere
Cossine
1:0.00
20.00
93.53
09.25.2001
69.39.1.00
Sine-cosine
96.09
99.07
03.38
99.26=0.01
94.29
68.46*2.21
D Supplementary Materials For Experiments
D.1 Cosine Sine-Cosine Pairs
We compare sinusoidal encoders with all cosines implemented in TGAT and DyGFormer and sinusoidal encoders with sine-cosine pairs described in their corresponding papers to determine which to use in study on dynamic graph datascts. We train TGAT, DyGFormer, DyGFormer-spprrate and DyGDecoder using both versions of the time encoder and present. the results in Table 4 The version with all cosines outperforms the sine-cosine pair encoder in 9 out of 24 model-dataset combinations, while the sine-cosine pair version is better in 12 combinations. The remaining 3 combinations result in tied average AP scores. Our findings indicate that there is no significant difference in performance between the two time encoders. Therefore, we simply adopt the sinusoidal time encoder with all cosines implemented in TGAT and DyGFormer for the remaining of our empirical study.
D.2 Waiting Time Statistics in UCI
In the main experimental results presented in Table we observe that the simusoidal-scale encoder consis- tently underperforms compared to the sinusoidal encoder the UCI dataset. To investigate this, analyze the dataset statistics, focusing the waiting time between a node's 's current interaction and its previous one, with the results shown in Table 5 Our analysis reveals that the waiting time significantly increases during the test split. This increase may explain the underperformance the simusidaal-cale encoder, as its scaler relies on the empirical mean and standard deviation from the train split.
D.3 Sinusoidal Time Encoding vs. Sinusoidal Positional Encoding
To highlight the importance of temporal information beyond mere sequential order, we conduct a separate experiment comparing sinusoidal time encodings with sinusoidal positional encodings. The latter encodes the positional indices of events rather than their time values. Table presents the results. We observe that time encoding outperforms positional encoding in 10 out of 12 cases, indicating that temporal features are
20

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 5: Statistics of the waiting time since the last edge interaction of source and destination nodes of each odge in UCL. We ignore the first appearance of a node since there are no previous interactions. The statistics are aggregated by vain/validation// splits.
Source Node
Destination Node
Waiting Time (Minutes)
Average
Median
Average
Median
Train
266.90
3.97
415.55
9.95
Validation
414.49
4.60
717.26
11.16
Test
2129.37
26.25
4476.33
145.57
Table 6: comparison between sinusoidal time encoding and sinusoidal positional encoding. The better performance for each model variant highlighted in bold. The number in the parentheses the performance change after switching from encoding time values to position indices.
Test AP
Encoded
UCI
Wikipodia
Enron
Reddit
LastPM
US Legis
(Random NS)
Feature
TGAT
Time
80.2750..
97.0020.16
72.07=1.0
98.55g00o
75.82:007
67-42-1
Position
79.09e0.30 (-1.18
95.96e9.00 (-1.92)
65.01=277 (-6446)
96.63e00: (+0.08)
58. 78m021 (-17.04)
50.11aso (-17.31)
DyCFormer
Thme
96.01
99.04.00
93.63.001
99.201041
03.88
70.12.0006
Position
95.94as.0 (0.07)
98.87amss (.0.17)
92.97aa..4 (-0.68)
99.19aaa1 (+0.01)
94.26a0.1 (+0.58)
66.58 (-1.44)
generally beneficial. Among all datasets, Reddit appears to be the least affected by the switch, with changes in AP scores below 0.1. This suggests that temporal features are less critical in Reddit, which aligns with our observations in Section 1.3.1 Figure)
D.4 Training Time and Memory Usage
For the main experiments, we measure the training time per epoch and peak GPU memory usage across datasets and model variants and present them in Table 7 and [ In general, models using the simusoidal time encoder exhibit longer runtimes due to the additional element-wise cosine computation. Furthermore, the sinusoidal encoder tends incur higher peak GPU memory usage for two main reasons. First, the cosine activation introduces additional intermediate tensor during computation. Second, the linear time encoder dimension can be reduced to 1 for DyGFormer variants due the subsequent linear projection layer as mentioned in Scction 1.33 resulting in lower memory usage than the sinusoidal encoder.
D.5 Beyond Pure Self-Attention Architectures
We conduct smaller-scale experiments to test if the effectiveness of the linear time encoder generalizes to model architoctures beyond TGAT and DyGFormer. We experiment with a memory-based model, TGN [Rossi et al. 2020), and a random walk-based model, CAWN (Wang et al. 2021c). The results are shown in Table g The linear time encoder outperforms the default sinusoidal time encoder 7 out of 8 evaluated cases. These results are consistent with our earlier findings on TGAT and the DyGFormer variants.
D.6 Dynamic Node Classification
In this work, we focus primarily on future link prediction, it is the most widely studied task in dynamic graph learning and benefits from wealth of publicly available datasets. This task is also fundamental, it evaluates model's ability to capture the inherent temporal structure of dynamic graphs. To complement our main results, we conduct smaller-scale experiments on dynamic node classification using the two datasets provided in DyGLib (Yu et al. )023): Wikipedia and Reddit. Following DyGLib's 's protocol, we first train the temporal embedding backbone using the future link prediction objective, then train node classification head while keeping the backbone fixed. The results are presented in Table I0 On these datasets, TGAT performs better with the linear time encoder, while DyGFormer performs better with the sinusoidal time encoder,
21

---

Published in Transactions on Machine Learning Research (07/2025)
Table 7: Training time per epoch (in seconds) across datasets and model variants. for each model-dataset combination is highlightod in bold.
The shorter training time
Training time / epocc (s)
Time Encoder
UCT
Wikipedia
Enron
Reddit.
Last FM
USLagis
TGAT
Sinusoidal
30.19
75.18
65.85
1219.34
734.51
31.38
Linear
29.79
73.99
65.40
1215.03
732.40
31.18
DyGFormer
Sinusoidal
24.72
59.00
82.15
472.43
1607.48
40.39
Linear
23.12
53.30
70.45
274.52
1087.59
35.25
Sinusoidal
24.87
50.07
86.51
298.74
148
36.54
Linear
22.76
58.58
73.08
282.65
994.33
32.94
DroDer
Simusoidal
24.71
62.83
86.91
323.38
1486.37
39.82
Linear
24.84
58.80
64.99
309.53
1370.09
33.76
Table 8: Peak GPU memory usage (in GB) across datasets and model variants. The fewer GPU memory usage for each model-dataset combination is highlighted in bold.
GPU memory usage (GB)
Time Encoder
UCI
Wikipodia
Reddit
Last
USLegis
Sinusoidal
1.930
1.997
1.972
2.332
2.728
Linsar
1.793
1.861
1.836
2.196
1.793
DyGFormer
Simusnidal
1.024
1.094
1.581
1.509
1.544
Linear 1.0U2
0.729
1.505
1.121
DyGoormer-eeparate
Simusnidal
0.963
0.709
1.136
Linear
0.621
1.006
1.448
1.087
2.741
1.037
DyGDeooder
Simusoidal
0.646
1.671
1.467
3.141
Linear
0.622
1.025
1.083
1.421
2.753
1.039
Table 9: Test AP (random NS) of TGN and CAWN with sinusoidal and linear time encoders. performance for each model-dataset combination boldfaced.
The better
Test AP (random NS)
Time Encoder
UCI
Wikipedia
Enron
Reddit
Simnsoidal
87. 87.00=0.96
97.91+0.00
16.92+1.21
98.51+0.05
TCN
Linear
94.53_..0
98.36
90.03
98.6110003
Simusoidal
95. 1820.00
98. 2.
91.09 to
99.122t.01
CAWN
Linear
96.21 =0.22
98.99=0.08
91.54=0.2
99.10+0.01
Table 10: Test AU-ROC of TGAT and DyGFormer with sinusoidal and linear time encoders on dynamic node classification datascts. The better performance for each modcl-dataset combination is boldfaced.
Test AU-ROC
Time Encoder
Wikipedia
Reddit
Sinusoidal Lincar
84.0811.90
68.77.1.49
TGAT
86.35
70.18
Simusoidal
87.54
68.76=2.18
DyGForeer
Linear
86.33+00.7
67.51 #1.02
suggesting that the two encoders are comparable for this task. A more comprehensive evaluation across additional datasets is needed to better understand their relative effectiveness in dynamic node classification, which we leave for future work.
D.7 Attention Score Analysis
In Section 4.3.2 Figure 7 visualizes the attention scores between query and key vectors for the historical edge event sequences of source nodes in Reddit, along with the encoded time differences tt and tx respectively. We present similar visualizations for the dostination nodes of Reddit Figure and five other datascts in Figures with those described in Section 4.3.2 in DyGFormer-separatee the attention scores concentrated near
g
10 and 12
The attention patterns observed in the continuous-tim datasets align
22

---

Published in Transactions on Machine Learning Research (07/202) 2025)
(5)
(5)
(5)
(a) Sinusoidal/
(b) Linear/
(c) Sinusoidal/
(d) Linear/ DyGDecoder
DyCFormeerssparate
DyGDecoder
Figure 8: Attention scores (expressed in colors) of SyGGormer-seppardde and DyGDecoder with the sinusoidal and lincar time encoders vs. t 1-1 l vs. t- by of the historical edge event sequences of Reddit's destination nodes.
(a) UCT/source
(b) /skipedlia/souroe
(c) Enron/source
(d) LastFM/source
(e) US Legis/sones /source
30 CX
(-2
[-
[-
(f) UCI/dest
(g) Wikipedia//dst
(b) Enron/dest
(i) LastFM/dee
(j) US Legis/dest
Figure 9: Attention scores (expressed in colors) of DyGoormer-separatee with the sinusoidal time encoder vs. t- la vs. t- of the historical edge event sequences of nodes in UCI, Wikipedia, Enron, LastFM, and US Legis. The upper and bottom rows are the results the source and destination nodes.
6-4 = 0, and in DyGDecoder, they focus around t-4, = l- However, attention patterns can vary across datasets. For instance, in DyGFoorerr-epparlle, the altention scores for UCT are tightly concentrated, while for LastFM, the scores more spread out across the (t (t-t,, (t- tx) plane, reflecting distinct characteristics of the datascts.
Being a discrete-time dataset, US Legis has a limited number distinct time differences, as seen in Figures De 10e
11j 12e and [2i In this case, we average all attention scores corresponding to the same (t- tg, t fg)) pair and visualize the result. Interestingly, in DyGFormer-scparatt the highest attention scores are not located at. ( / 0, but. al. / 4 1. This is because the key vectors corresponding to the most recent events before the target time & are located at - 1, and these key vectors have the greatest influence the node's state at time f. In contrast, the key vectors corresponding to the target edges lack informative edge features, as the interaction has not yet occurred. Consequently, these key vectors are less important for computing node's temporal representation. This should also hold for contimouss-imme datasets, where the key vectors encoding t e should have the highest attention scores, with e being a small positive quantity. However, this effect is not easily discernible in the plots due to the continuous nature of the time differences being visualized.
10 Ile
23

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
C-G
[-4,0)
2-6
1-1,01
C-
(a) UCI/source
(b) Wikipedia/source
(c) Enron/source
(d) LastFM/source
(e) US Legis/source
4
5,00
(i) UCT/dest
(g) Wikipedia//est
(b) Enron/dest
(i) Last FM/est/
(1) US Legis/dest
Figure 10: Attention scores (expressed colors) of DyGDecoder with the sinusoidal time encoder vs. L- la vs. - by of the historical edge event sequences of nodes in UCL, Wikipedia, Euron, LastFM, and US Legis. The upper and bottom rows are the results of the source destination nodes.
C-4
L-1
C-
(a) UCI/source
(b) Wikipedia /source
(c) Enron/source
(d) LastF /source
(c) US Legis/source
1.
E-6
[-
2-
(1) UCI/dest
(g) Wikipedia/dess
(h) Enron/dest.
(1) LastFM/dest
(i) US Legis/dest
Figure 11: Attention scores (expressed colors) of DyGFormer-soparate with the linear time encoder vs. t-te vs. t --f ty of the historical edge event sequences of nodes in UCI, Wikipedia, Enron, LastFM, and US Legis. The upper and bottom rows are the results the source and destination nodes.
24

---

Published in Transactions on Machine Learning Research (07/2025)
2-0
E-1e0)
2-
C-
(a) UCI/source
(b) Wikipedia/soure
(c) Enron/source /source
(d) BastMM/source
(e) US Legis/ /ggu//
(-
[-1,99
E-
(0) UCT/dest
(g) Wikipedia/dest
(b) Euron/dest
(i) LastFM/dest
(i) US Legis/dest
Figure 12: Attention scores (expressed colors) of DyGDecoder with the linear time encoder vs. t- ta vs. -- by of the historical edge event sequences of nodes in UCI, Wikipedia, Euron, LastFM, and US Logis. The upper and bottom rows are the results of the source and destination nodes.
25


Reviews:
[{"review_id": "hCg4V85rnq", "reviewer": "Reviewer_nqHa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed idea is intuitive, and both the formulation and theoretical justification are clearly presented.\n2. The experimental results show consistent performance improvements. Although this method does not always achieve the highest score on all datasets, the performance remains within one standard deviation of the SoTA, which is acceptable.\n3. The authors evaluate their approach on both DyGFormer and DyGDecoder models, demonstrating the generalizability of the new encoder.\n\nWeakness:\nSee requested changes", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to use a linear time encoder instead of a sinusoidal time encoder for dynamic graph learning. The author demonstrates that this new approach can outperform the baseline in most cases while also saving memory through many experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In Sections 2.3 and 3.1, the variable w is used as a learnable weight, but it is only explained in Section 3.1. For clarity, this explanation should be included in Section 2.3. Additionally, in the first two formulas of Section 2.3, the use of d should be replaced with d_T.\n2. Please double-check the value reported in Table 1 for DyGFormer-separate with sinusoidal-scale on the UCI dataset: 87.77 +/- 15.75. The standard deviation of 15.75 seems unusually high, could this be a typo?\n3. The author provides many figures of attention scores from DyGFormer-separate and DyGDecoder in both the main text and the appendix, which help illustrate the model’s behavior. In Section 4.3.2, the paper discusses the role of pooling strategies and highlights the importance of edge event timing in dynamic graphs. While these insights are valid, they should serve as the foundation of the paper rather than the main focus. The central contribution of the paper is the imprvoement of the proposed linear encoder, and I expect to see visual comparisons between linear encoders and sinusoidal encoders.\n4. Minor point: It would be better to replace the e-commerce platform example in the introduction. When I first read the paper, this example caught my attention and led me to expect that the model would be evaluated on e-commerce-related data. However, none of the datasets used in the experiments are related to e-commerce, which ultimately felt disappointing. While the example is engaging, I recommend choosing an example that aligns more closely with the datasets used in the study.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3Zkl9WqAFg", "reviewer": "Reviewer_v7gf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.  The paper challenges a widely adopted component (sinusoidal time encoder) in dynamic graph learning and proposes a simpler, yet effective, alternative. The findings have direct practical implications for designing more efficient dynamic graph learning architectures.\n2.  The authors provide a proof (Proposition 3.1) that self-attention can indeed learn to factor time spans from linear time encodings, which is a key motivation for using sinusoidal encoders. This theoretical insight is valuable.\n3.  The paper presents a comprehensive experimental evaluation across six diverse datasets and two different negative sampling strategies. The inclusion of a synthetic task to validate the learnability of time spans from linear encodings (Section 3.2) strengthens the claims.\n\nWeaknesses:\n1.  While TGAT and DyGFormer are representative, the study is confined to these two attention-based architectures. It would be interesting to see if the conclusions generalize to other types of dynamic graph models, including those not heavily reliant on self-attention or those using different mechanisms for incorporating time.\n2.  While the linear encoder shows overall strong performance, there are instances where it does not outperform or even underperforms sinusoidal encoders, for example, on the US Legis dataset under historical negative sampling for most model variants (Table 2). The paper offers a plausible explanation regarding the discrete nature of US Legis, but further investigation into why linear encoders might struggle in certain data regimes could be beneficial.\n3.  The paper mentions that standardized time differences are used for \"numerical stability and optimization efficiency purposes\". While the sinusoidal-scale encoder is introduced to isolate the impact of scaling, the interaction between standardization and the performance of linear encoders, especially in scenarios with significant distribution shifts in time differences (as noted for UCI in Section 4.2.2), could be a point of deeper discussion. It's not entirely clear if the performance fluctuations are solely due to the scaling strategy or an inherent property of linear encoding in such scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the use of a linear time encoder as an alternative to the more commonly used sinusoidal time encoder in attention-based dynamic graph learning models, specifically TGAT and DyGFormer. The authors argue that linear time encoders can prevent temporal information loss associated with sinusoidal functions and potentially reduce the need for high-dimensional time encodings. The paper provides a theoretical justification showing that self-attention mechanisms can effectively learn to compute time spans from linear time encodings. This is further supported by experiments on a synthetic dataset. Extensive experiments on six real-world dynamic graph datasets for future link prediction demonstrate that the linear time encoder generally outperforms sinusoidal encoders in terms of average precision and can lead to significant reductions in model parameters with minimal performance degradation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition the the mentioned weaknesses:\n\n1.  Proposition 3.1 is interesting, showing existence. However, the construction in Appendix B seems specific. Could the authors elaborate if common initialization schemes or training dynamics naturally lead to such solutions, or if specific architectural choices beyond the linear encoder itself facilitate this? The synthetic task (Section 3.2) shows it *can* be learned, but further insight into *how* readily it's learned in complex models would be valuable.\n2.  The paper sets the linear time encoding dimension to 100 for TGAT but to 1 for DyGFormer variants (Section 4.1.3), explaining that the latter is followed by a linear projection. While the dimensionality reduction experiments (Section 4.3.1) are insightful, a brief discussion on the sensitivity to the *initial* dimension choice for linear encoders, especially if the subsequent projection was not present, might be useful. For example, how critical is $d_T=1$ for DyGFormer if the channel dimension $d_{ch}$ is also small?\n3.  The paper introduces \"DyGDecoder\" as a variant computing representations \"separately and in an autoregressive manner\" and prepending a \"beginning of sequence\" embedding (Section 3.3). While Figure 6 shows attention patterns, a more explicit (even if brief) architectural diagram or description of how the autoregressive computation and the BOS token are integrated specifically for the link prediction task would enhance clarity. How does the \"target edge event\" fit into this autoregressive formulation when its features are zeroed out (as mentioned for DyGFormer in Section 2.2)?\n6.  The paper notes that US Legis is a \"discrete-time dynamic graph with only 12 evenly spaced time steps\" (Section 4.2.3). Given this, are time differences always integer multiples of some base unit? If so, this is quite different from the continuous-time nature of other datasets. Clarifying this might further explain why sinusoidal encoders might be sufficient or even preferable here, as the \"discretization\" inherent in sinusoidal functions over a limited set of inputs might align well with the data's temporal structure.\n7.  The caption for Figure 5 refers to \"varying time channel dimension $d_{tc}$\". However, the plots themselves and the text (Section 4.3.1) seem to vary $d_{ch}$ (channel embedding dimension) or effectively $d_T$ (time encoding dimension, which is 1 and then projected to $d_{ch}$ for the linear case in DyGFormer). It would be good to ensure consistent terminology or clarify if $d_{tc}$ is a distinct hyperparameter being varied. My understanding from Section 2.2 and 4.1.3 is that $d_{ch}$ is the key dimension here for DyGFormer.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GthgDn7yp4", "reviewer": "Reviewer_Comy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper is well-written and easy to follow. The paper challenges a small but influential design choice that everyone inherits without much thought. \n- Six datasets, two transformer‑style backbones, twenty‑four runs… that’s a lot of GPU time spent on a simple idea. \n- Practical impact - The attention heatmaps and the ablation curves make it clear where the gains come from. \n\n## Weaknesses\n- We only see results on TGAT and DyGFormer. I kept wondering how a memory‑centric GNN would react?\n- The authors report parameter counts but not wall‑clock time.\n- Narrow task scope – Future link prediction is nice, but node classification, temporal motifs, or cold‑start tests could paint a very different picture.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper challenges the de‑facto use of high‑dimensional sinusoidal time encoders in dynamic‑graph Transformers. It proposes a linear time encoder that feeds the raw time gap Δt through a learnable linear projection", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add one memory‑based and one walk‑based architecture to the test bed; even a small‑scale run would show the idea generalises.\n- Add a non‑link task so we know the results aren’t benchmark‑specific.\n- Provide per‑epoch time and peak GPU usage.\n- In Tables 1–2, visually highlight the win counts to let readers grasp the advantage instantly.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)