Do We Really Need Complicated Model Architectures For Temporal Networks?
Authors: Weilin Cong
Si Zhang
Jian Kang
Baichuan Yuan
Hao Wu
Xin Zhou
Hanghang Tong
Mehrdad Mahdavi
Venue: ICLR 2023 notable top 5% (2023)
Keywords: temporal graph
link prediction

Abstract: Recurrent neural network (RNN) and self-attention mechanism (SAM) are the de facto methods to extract spatial-temporal information for temporal graph learning. Interestingly, we found that although both RNN and SAM could lead to a good performance, in practice neither of them is always necessary. In this paper, we propose GraphMixer, a conceptually and technically simple architecture that consists of three components: (1) a link-encoder that is only based on multi-layer perceptrons (MLP) to summarize the information from temporal links, (2) a node-encoder that is only based on neighbor mean-pooling to summarize node information, and (3) an MLP-based link classifier that performs link prediction based on the outputs of the encoders. Despite its simplicity, GraphMixer attains an outstanding performance on temporal link prediction benchmarks with faster convergence and better generalization performance. These results motivate us to rethink the importance of simpler model architecture.

Published as a conference paper at ICLR 2023
Do WE REALLY NEED COMPLICATED MODEL ARCHI- TECTURES FOR TEMPORAL NETWORKS?
Weilin Cong
Si Zhang
Jian Kang
Penn State
Meta
University of Illinois at Urbana- Chamaaaan jiank28 1 wlinnoi.eeee
wei 1 liiipiueuu... edu
si zwanggmeta..commmmmmmmmmmmm com
Baichuan Yuan Hao Wu & Xin Zhou
Hanghang Tong
Meta
University of Illinois at Arbana-Champaigge htongli i inniiii......... edu
(bcyuan, haowul, markzhou (emmm.....mmmmmmmm.. com
Mehrdad Mahdavi
Penn State
mzm61 1 S@psu.ede
ABSTRACT
Recurrent neural network (RNN) and self-attention mechanism (SAM) are the de facto methods to extract ppaporaltempora information for temporal graph learning. Interestingly, we found that although both RNN and SAM could lead to good per- formance, in practice neither of them is always necessary. In this paper, we propose GraphMixer, conceptually and technically simple architecture that consists three components: D link-encode that is only based multi- perceptrons (MLP) to summarize the information from temporal links, 2 a node-encoder that is only based neighbor mean-pooling to summarize node information, and (3) an Mi.P-based link classifier that performs link prediction based on the outputs of the encoders. Despite its simplicity, GraphMixer attains outstanding perfor- mance on temporal link prediction benchmarks with faster convergence and better generalization performance. These results motivate us rethink the importance of simpler model architecture. [Code].
INTRODUCTION
In recent years, temporal graph learning has been recognized as an important machine learning problem and has become the cornerstone behind wealth of high-impact applications Yu et al. (2018); Bui et al. (2021); Kazemi et al. (2020); Zhou et al. (2020); Cong et al. (2021b). Temporal link prediction is one of the classic downstream tasks which focuses on predicting the future interactions among nodes. For example, in an ads ranking system, the user-ad clicks can be modeled as a temporal bipartite graph whose nodes represent users and ads, and links indicating when users click ads. Link prediction between them can be used to predict whether user will click an ad. Designing graph learning models that can capture node evolutionary patterns and accurately predict future links is crucial direction for many real-world recommender systems.
associated with timestamps
In temporal graph learning, recurrent neural network (RNN) and self-attention mechanism (SAM) have become the de facto standard for temporal graph learning Kumar et al. (2019); Sankar et al. (2020); Xu et al. (2020); Rossi al. (2020); Wang et (2020), and the majority of the existing works focus designing neural architectures with one of them and additional components to learn representations from raw data. Although powerful, these methods are conceptually and technically complicated with advanced model architectures. It is non-trivial to understand which parts of the model design truly contribute to its success, and whether these components are indispensable. Thus, in this paper, aim at answering the following two questions:
Q1: Are RNN and SAM always indispensable for temporal graph learning? To answer this question, propose Mixph simple architecture based entirely on the multi-layer perceptrons (MLPs) and neighbor mean mean-pooling pooling, which does not utilize any RNN or SAM in its model architecture (Section 3). Despite its simplicity, GraphMixer could obtain outstanding results when comparing it.

---

Published as a conference paper at ICLR 2023
t
t
t3
4
t
(now)
(paat)
Node features
X1
1/
fs
Temporal graph
11
U2
U3
U4
t6
UG
Link features
2
U2
XL2 (()),
L
Figure I: (Left) Temporal graph with nodes U1,..., , U. per-link timestamps & ....... ie indicate when two nodes interact. For example, vi, 02 interact at & lg. (Right) Each node has its node features (e.g.. x- for v1) and each temporal link has its link features (e.g., x1122 (t1). x (ts) are link features between V1, U2 at &1, ts). For scenarios without node or link features, we use all-zero vectors instead.
against baselines that are equipped with the RNNs and SAM. In practice, it achieves state-of-the-arr performance in terms of different evaluation metrics (e.g.. average precision, AUC, Recall @ K, and MRR) on real-world temporal graph datasets, with the even smaller number of model parameters and hyper-parameters, and conceptually simpler input structure and model architecture (Section 4).
Q2: What are the key factors that lead to the success of GraphMixer? We identify three key factors that contribute to the success of GraphMixer: ! The simplicity of GraphMixer's input data and neural architecture. Different from most deep learning methods that focus on designing conceptually complicated data preparation techniques and technically complicated neural architectures, we choose to simplifying the neural architecture and utilize a conceptually simpler data as input. Both of which could lead to better model performance and better generalization (Section 4.4). ② A time-encoding function that encodes any timestamp as an easily distinguishable input vector for GraphMixer. Different from most of the existing methods that propose learn the time-encoding function from the raw input data, our time-encoding function utilizes conceptually simple features and is fixed during training. Interestingly. we show that our fixed time-encoding function is more preferred than the trainable version (used by most previous studies). and could lead to a smoother optimization landscape, a faster convergence speed, and a better generalization (Section 4.2); + A link-encoder that could better distinguish temporal sequences. Different from most existing methods that summarize sequences using SAM, our encoder module is entirely based on MLPs. Interestingly, our encoder can distinguish temporal sequences that cannot be distinguished by SAM, and it could generalize better due to its simpler neural architecture and lower model complexity (Section 4.3).
To this end, summarize contributions as follows: Q We propose conceptually and technically simple architecture GraphMixer, 2 Even without RNN and SAM, GraphMixer not only outperforms all baselines but also enjoys a faster convergence and better generalization ability; 3 Extensive study identifies three factors that contribute to the success of GraphMixer. ④ Our results could motivate future research to rethink the importance of the conceptually and technically simpler method.
2 PRELIMINARY AND EXISTING WORKS
Preliminary. Figure is an illustration on the temporal graph. Our goal is to predict whether two nodes are connected at a specific timestamp la based on all the available temporal graph information happened before that timestamp. For example, to predict whether 01, are connected at to. we only have access the graph structure, node features, and link features with timestamps from t to to.
Related works. Most of the temporal graph learning methods are conceptually and technically complicated with advanced neural architectures. It is non-trivial fully understand the algorithm details without looking into their implementations. Therefore, we select the four most representative and most closely-related methods to introduce and compare them more details.
JODIE Kumar et al. (2019) is RNN-based method. Let us denote (t) as the embedding of node v; at time f, x (t) as the link feature between 1; By at time f, and m; as the timestamp that ii latest interact with other node. JODIE pre-processes and updates the representation of each node via RNNs (is it just one RNN or multiple RNNs). More specifically, when an interaction between Us Vÿ happens at time t, JODIE updates the temporal embedding using RNN by (t) = RNN (x, (m,),x, (m,)). x link (t), m²). Then, the dynamic embedding of node e; at time to is computed by (to) (1 + (to - m))w) +x (xmm.. Finally, the prediction any node
pair at time to is computed by MLP (fo) || h, (to))). where [-]]-]] is the concatenate operation
and MI.P(x) is applying 2-layer MI.P
2

---

Published as a conference paper at ICLR 2023
DySAT Sankar et al. (2020) is SAM-based method. DySAT requires pre pre-processing the temporal graph into multiple snapshot graphs by first splitting all timestamps into multiple time-slots, then merging all edges in each time-slot. Let (V,V,, E,) denote the t-th snapshot graph. To capture spatial information, DySAT first applies Graph Attention Network (GAT) Veličković et al. (2018) each snapshot graph 9, independently by (t) = GAT( Then, to capture of temporal information for each node, Transformer is applied to (t) = [X(t)]; at different timestamps to capture the temporal information by (ttt)). (,() - Trans former (xx(t)), .... x; (f6)). Finally, the prediction on any node pair at time to is computed by (L, |I h (fo))).
TGAT Xu et al. (2020) is a SAM-based method that could capture the spatial and temporal information simultaneously. TGAT first generates the time augmented feature of node i at time by concatenating the raw feature x; with a trainable time encoding z(() of time I, i.e., x(()) = x (() and 2() cos(/w bb). Then, SAM ís applied to the time augmented features and produces node representation (to) = SAM (to), {x, (h.) u E NN., (i))), where Nr. (i) denotes the neighbors of node i at time to and ft, denotes the timestamp of the latest interaction of node u. Finally, the prediction on any node pair at time to is computed by MLP (h, (fo) || h (f))]) TGN Rossi et al. (2020) is mixture of RNN- and SAM- method. In practice, TGN first captures the temporal information using RNN (similarly to JODIE), and then applies graph attention convolution to capture the spatial and temporal information jointly (similarly to TGAT).
Besides, we also consider the following temporal graph learning methods as baselines. These methods could be thought of as an extension on top of the above four most representative methods, but with the underlying idea behind the model design much more conceptually complicated. CAWs Wang et al. (2020) is mixer of RNN- and SAM- based method that proposes to represent network dynamies by extracting temporal network motifs using temporal random walks. CAWs replaces node identities with the hitting counts of the nodes based on set of sampled walks to establish the correlation between motifs. Then, the extracted motifs are fed into RNNs to encode each walk as a representation, and use SAM to aggregate the representations of multi-walks into single vector for downstream tasks. TGSRec Fan et al. (2021) is a SAM-based method that proposes to unify sequential patterns and temporal collaborative signals to improve the quality of recommendation. To achieve this goal, they propose advance the SAM by adopting novel collaborative attention, such that SAM can simultaneously capture collaborative signals from both users and items, as well as consider temporal dynamics inside sequential patterns. APAN Wang et al. (2021b) is RNN-based method that proposes to decouple model inference and graph computation to alleviate the damage of the heavy graph query operation to the speed of model inference. More related works are deferred to Appendix B.
3 GRAPHMIXER: A CONCEPTUALLY AND TECHNICALLY SIMPLE METHOD
In this section, we first introduce the neural architecture of GraphMixer in Section 3.1 then explicitly highlight its difference to baseline methods in Section 3.2.
3.1 DETAILS ON GRAPHMIXER: NEURAL ARCHITECTURE AND INPUT DATA
GraphMixer has three modules: ① link-encoder is designed to summarize the information from temporal links (e.g., link timestamps and link features); 2 node-encoder is designed to summarize the information from nodes (e.g., node features and node identity); 3 link classifier predicts whether a link exists based on the output of the aforementioned two encoders.
Link-encoder. The link-encoder is designed to summarize the temporal link information asso- ciated with each node sorted by timestamps, where temporal link information referring to the timestamp and features of each link. For example in Figure 1, the temporal link informa- tion for node V2 is {(1,, x x (41)) (13, (ta). (L4, link (ta)), ) (ts, x (Is))} and for node vs is only keep the top K most recent temporal link infor- mation, where K is a dataset dependent hyper-parameter If multiple links have the same timestamps, we simply keep them the same order as the input raw data. To summarize temporal link informa- tion, our link-en -ncoder should have the ability to distinguish different timestamps (achieved by our time-encoding function) and different temporal link information (achieved by the Mixer module).
{(t2, xlinh (t2)), (tg, xtc. (tink((t)) (to))). }.
In practice,
Time-encodinn function. To distinuuish different timestamps, we introduce our time-encoding function cos( which utilizes features w {a- (1-1))) R=1 to encode each timestamps into
3

---

Published as a conference paper at ICLR 2023
11
Time encoded by costte) goto
6 200
cos(f]w)
[ooss( (to (1) x w)
x12
Jmnk (1a))
Zero- poooddaa
1-layer MLP-Mixes
Token-mixer
(oo(f2w)
[ozol (Ao - (2) x w)
x (Aa])
WHY
(oo(13w)
[ooa((to 14) x w)
X2.4 ( (ta))
Channel-mixer
t2 tt
84
(os()aw)
[ozo((ta (At - (s) x ())
(ta3)
Mean-pooling
(a)
100
(b)
Figure 2: (a) Time-ennoding function that pre-process, timestamp into vector (ts() The x-axis
is the vector dimension and the y-axis is the cosine value. (b) link-encoder takes the temporal link information of node U2 as inputs and outputs vector (2(()) that will be used for link prediction.
a d-dimensional vector. More specifically, we first map each to a vector with monotonically exponentially decreasing values lw E (0, (] among the feature dimension, then use cosine function to project all values to cos(tw) E [-1,+1] The selection of a, 3 is depending on the scale of the maximum timestamp fma we wish to encode. In order to distinguish all timestamps, have to make sure 'max X a (f-1))/8 - 0 as -> d to distinguish all timestamps. In practice, we found d = 100 and 3 = Vd works well for all datasets. Notice that is fixed and will not be updated during training. As shown in Figure 2a, the output of this time-encoding function has two main properties that could help GraphMixer distinguish different timestamps: similar timestamps have similar time-encodings (e.g.. the plot of (1, (2) and the larger the timestamp the later the values in time-encodings converge to +1 (c.g.. the plot of f1, t3 or t4).
Mixer for information sumuaarizing. We use a 1-layer MLP-mixe Tolstikhin et al. (2021) to summarize the temporal link information. Figure 2b an example on summarizing the temporal link information of node U2. Recall that the temporal link information of node 1/ is {(t1, link (t))), (f3, xiikk (ta)), ( (t4, x2.4 (t4)), (ts, x1,2 (tt))) }. We first encode timestamps by our time-encoding function then concatenate it with its corresponding link features. For example, we encode (41, x (11)) as [cos((4g - (1))) || x (42))] where to is the timestamp that we want to predict whether the link exists. Then, stack all the outputs into big matrix and zero-pad to the fixed length K denoted as (2(t)). Finally, we use an I-layer MLP-mixer with mean- mean-pooling pooling to compress (2( into single vector to (4o). Specifically, the MLP-mixer takes (1o))) as input
(to Hinptt Husee Hmput + W GeLU (()) LayerNorm (Hmpu)); token
loken
Hehaneel
Hoken + GeLU Hoken
(()) channel
(2))
channel*
and output the temporal encoding (to) Mean Mean(Hehamma)) (Hehameel)
Please notice that zero pero-padding
operator is important to capture how often a node interacts with other nodes. The node with more zero-padded dimensions has less temporal linked neighbors. This information is very important in practice according to our experimental observation.
Node-encoder. The node-encoder is designed to capture the node identity and node feature in- formation via neighbor muca--polling Let us define the I-hop neighbor of node v, with link timestamps from i to ((v;; t, to). For example in Figure 1, we have (122; la; fo) {vi, 44) and N ((5s; 14, fo) = {vz}- Then, the node-info feature is computed based on the 1-hop neighbor by (to) mndee + Mean [ide vy E AV to T, to)). where T' is a dataset-dependent hyper- parameter. In practice, we found I-hop neighbors are enough to achieve good performance, and we use one-hot node representations for datasets without node features.
Link classifier. Link classifier is designed to classify whether a link exists at time to using the output of link-encoder (to) and the output of node-encoder s; (tt)). Let us denote the node u, representation at time to as the concatenation of the above two encodings h,(to) = [s:()) || [fo)]. Then, the prediction on whether an interaction between node nt, By happens time tn computed by applying 2-layer MLP model on (to) || h, (fo)), i.e., Ptj = MLP (to) (tt))))
3.2 COMPARISON TO EXISTING METHODS
In the following, highlight some differences between GraphMixer and other methods, which will be explicitly ablation studied in the experiment section (Section 4.4).
Temporal graph undirected graph. Most of the existing works consider temporal graphs as directed graphs with information only flows from the source node (e.g., users the recommender system) to the destination nodes (e.g., g., ads in the recommender system). However, we consider the
4

---

Published as a conference paper at ICLR 2023
Table I: Comparison on the average precision score for link prediction. GraphMixer uses one-hot node encoding for datasets without node features (marked by 1). For each dataset, we indicate whether we have the corresponding feature ("L' link features, "N" node features, and "T" link timestamps). Red is the best score, Blue is the best score excluding GraphMixer and its variants.
Reddit L. T
Wiki L. T
MOOC T
LastM T
GDELT
GDELT-ne T
GDELT-e
N, T
N. N.T
JODIE
99.30 0.01
98.81
0.01
16 1:00
67.51
0.87
98.27 + 0.02
97.13 0.02
96.96 0.02
DySAT
98.52 + 0.01
96.71 + 0.02
98.82 + 0.002
76.40
+ 0.77
98.52 + 0.02
82.47 + 0.13
97.25 + 0.02
TGAT
99.66
97.75
0.02
54.77
1.01
98.25 +: 0.02
84.30
+:00::000 0.10
96.96
0.02
TGN
0.08
99.35
+
99.62
82.23 0.50
98.15
1: 0.02
97.13
+.00
06.04:+0.00
CAWs-mean
98.43 +.002 0.02
97.72
+-22++0.00 0.08
12.99+00.87
76.35 4-05 0.08
96.11 0.12
69.20
+ 0.10
91.72 01.72+0.111 0.19
CAWs-attn
98.51
63.07 0.82
76.31 4
95.06
09.54
0.119
91.54 0.22
TGSRec
0.08
0.12
0.34
97.03 0.87
97.03
0.61
97.03 +
APAN
99.24 0.02
0.01
69.39
96.96 + 0.10
97.38 + 0.23
96.77 + 0.188
GraphMixer-L
99.84 0.01
99.70 0.01
99.81 0.01
95.50 0.08
98.99 1 0.02
96.14 0.02
98.99 1 0.02
GraphMixer-N
99.21 0.01
90.33
+ 0.01
97.35
+ 0.02
63.80
+ 0.03
94.41 0.02
96. 00.00- + 0.02
81 + 0.02
GraphMixer
99.93 0.01
99.85 + 0.01
99.91 0.01
96.31 0.02
98.89
98. + 0.02
98. 22 + 0.02
temporal graph as an undirected graph. By doing so, if two nodes are frequently connected in the last few timestamps, the "most recent 1-hop neighbors" sampled for the two nodes on the "undirected" temporal graph would be similar. In other words, the similarity between the sampled neighbors provides information
whether two nodes frequently connected in the last few timestamps, which is essential for temporal graph link prediction. Intuitively, if two nodes are frequently connected in the last few timestamps, they are also likely to be connected in the recent future.
Selection on neighbors. Existing methods consider either "multi-hop hopp recent neighbors" or "multi- hop uniform sampled neighbors", whereas only consider the "1-hop most recent neighbors". For example, TGAT Xu et al. (2020), DySAT Sankar et al. (2020), and TGSRe Fan et al. (2021) consider multi-hop uniform sampled neighbors; JODIE Kumar al. (2019), TGN Rossi et al. (2020), and APAN Wang et al. (2021b) maintain the historical node interactions via RNN, which can be think of as multi-hop recent neighbors; CAWs Wang et al. (2020) samples neighbors by random walks, which can also be think of as multi-hop recent neighbors. Although sampling more neighbors could provide a sufficient amount of information for models to reason about, it could also carry much spurious or noisy information. As a result, more complicated model architectures (e.g., RNN or SAM) are required to extract useful information from the raw data, which could lead to a poor model trainability and potentially weaker generalization ability. Instead, we only take the "most recent 1-hop neighbors" into consideration, which is conceptually simpler and enjoys better performance.
4 EXPERIMENTS
Dataset. We conduct experiments on five real-world datasets, including the Reddit, Wiki, MOOC, LastFM datasets that are used in Kumar et al. (2019) and the GDELT dataset' which is introduced in Zhou et al. (2022). Besides, since GDELT is the only dataset with both node and link features, we create its two variants to understand the effect of training data model performance: GDELT-e removes the link feature from GDELT and keep the node feature and link timestamps. GDELT-ne removes both the link and edge features from GDELT and only keep the link timestamps. For each dataset, we use the same 70%/15% 15% chronological splits for the wrain/validationteet sets as existing works. The detailed dataset statistics summarized in Appendix A.2.
We compare baselines that are introduced in Section 2. Besides, we create two variants to better understand how node- and link- fin-infformation contribute to our results, where GraphMixer-LL is only using link-encoder and GraphMixer-N is only using node-encoder. We conduct experiments under the transductive learning setting and use average precision for evaluation. The detailed model configuration, training and evaluation process are summarized in Appendix A.3. Due to the space limit, more experiment results on using Recall@ K, MRR, and AUC as the evaluation metrics, comparison on wall-clock time and number of parameters are deferred to Appendix C
Baselines.
Outline. We first compare GraphMixer with baselines in Section 4.1 then highlight the three key factors that contribute to the success of GraphMixer in Section 4.2, Section 4.3, and Section 4.4.
'The GDELT dataset used in paper is sub-sampled version because the original dataset is too big to fit into memory for single-machine training. In practice, keep temporal link per 100 continuous temporal link.
5

---

Published as a conference paper at ICLR 2023
4.1 MAIN EMPIRICAL RESULTS.
GraphMixer achieves outstanding performance. We compare the average precision score with baselines in Table We have the following observations: D GraphMixer outperforms all baselines on all datasets. The experiment results provide sufficient support on our argument that neither RNN nor SAM is necessary for temporal graph link prediction. @ According to the performance of GraphMixer-L on datasets only have link timestamp information (MOOC, LastFM, and GDELT-ne), we know that our time-encoding function could successfully pre-process each timestamp into a meaningful vector. In fact, we will show later in Section 4.2 that our time-encoding function is more preferred than baselines' trainable version. ② By comparing the performance GraphMixer-N and GraphMixer
Wiki, MOOC, and LastEM datasets, we know that mode-encode alone is not enough to achieve a good performance. However, it provides useful information that could benefit the link-encoder. 4 By comparing the performance of GraphMixer-N observe that using one-hot encoding outperforms using node features. This also shows the importance of node identity information because one-hot encoding only captures such information. (5 More complicated methods (e.g., CAWs, TGSRec, and DDGCL) do not perform well when using the default hyper-panne parammearsssrrs which understandable because these methods have more components with an excessive amount of Sypem-tarameters to tune.
GDELT and GDELT-ne,
Contrabaatnn wenteetnnnnnnnn Thii Any treeaeon
Thun Ang Procalon
Thais Ang Procaken
ODBLT
DrEAT
TEAT
TGAT D-SAI JoDre
DSSAT
JoOEE
24
45
se Tri Tixe DySSKT
TGN
TGN
0,035
JoEns
Graphhlane
0000
Epocho
Epocht
Epocta
Figure 3: Comparison the training set average precision and generalization gap for the first 100 training epochs. Results on other datasets can be found in Figure 8.
GraphMixer enjoys better convergence and generalization ability. To better understand the model performance, we take a closer look at the dynamic of training accuracy and the generalization gap (the absolute difference between training and evaluation score). The results are reported in Figure 3 and Figure 8: ! The slope of training curves reflects the expressive power and convergence speed of an algorithm. From the first row figures, average precision score in just a few epochs, and the training curve is very smooth when compared to baselines. Interestingly, we can observe that the baseline methods cannot always fit the training data, and their training curves fluctuate a lot throughout the training process. @ The generalization gap reflects how well the model could generalize and how stable the model could perform on unseen data (the smaller the better). From the second row figures, the generalization gap curve of GraphMixer is lesser and smoother than baselines, which indicates the generalization power of GraphMixer.
observe that GraphMixer always converge
high
GraphMixes enjoys a smoother loss landscape. To understand why "GraphMixer converges faster and generalizes better, while baselines suffe training unstable issue and generalize poorly", we explore the loss landscape by using the visualization tools introduced in Li et al. (2018a), 8a). We illustrate the loss landscape in Figure by calculating and visualizing the loss surface along two random directions near the pre-trained optimal parameters. The x- and y-axis indicate how much the optimal solution is stretched along the two random directions, and the optimal point is when x- and y-axis are zero. ! From Figure 4a, 4d, we know GraphMixer enjoys smoother landscape with flatter surface at the optimal point, the slope becomes steeper when stretching along the two random directions. The steeper slope on the periphery explains why GraphMixer could converge fast, the flatter surface at the optimal point explains why it could generalize well. 2 Surprisingly, we find that baselines have a non-smooth landscape with many spikes on its surface from Figure 4b, 4c, 4e, 4f. This observation provides sufficient explanation on the training instability and poor generalization issue of baselines as shown in Figure 3, 8. Interestingly, as we will show later in Section 4.2, the trainable time- eecoding function in baselines is the key to this non-smooth landscape issue. Replacing it with our fixed time-encoding function could flatten the landscape and boost their model performance.
II fact, tried different hyper-parameters based on their default values, hut the results similar.
6

---

Published as a conference paper at ICLR 2023
3.5
225
1.0
26
(a) GraphMixer on Wiki
(b) TGAT on Wiki
(c) TGN on Wiki
25
2.0
1.5
1.0
1.5
L.
0.00
1.0
1.0
0.5
-1.0 02 GO
9.0
-1.0
0.0
-1.
0.0
(d) GraphMixer on GDELI-e
1.
(c) TGAT on GDELT-
0.0
0
(f) TGN on GDELT
0.0
055
10
Figure 4: Comparison on the training loss landscape. Results on other datasets and other baselines can be found in Appendix E.
4.2 ON THE IMPORTANCE OF OUR FIXED ENCODING FUNCTION.
Existing works (i.e., JODIE, TGAT, and TGN) leverage trainable time-encoding function z(1) cos(tw) + b) to represent timestamps". However, we argue that using trainable time-encoding function could cause instability during training because its gradient @cos(tw-b) ( sin(Iw b) scales proportional to the timestamps, which could lead to training instability issue and cause the baselines' the non-smooth landscape issue shown in Figure 4. As an alternative, we utilize the fixed time-encoding function (t) = cos(tw) with fixed features w that could capture the relative difference between two timestamps (introduced in Section 3.1). To verify this, we introduce simple experiment to test whether the time-encoding functions (both our fixed version and baselines' trainable version) are expressive enough, such that a simple linear classifier can distinguish the time-encodings of two different timestamps produced by the time-encoding functions. Specially, our goal is to classify if f >> t2 by learning a linear classifier on (z(tz) || 2(t2)]. During training, randomly generate two timestamps t 12 [0, 10 and ask fully connected layer classify whether a timestamp is greater than another. As shown in Figure 5a, using the trainable time-encoding function (orange curve) will suffer from the unstable exploding gradient issue (left upper figure) and its performance remains almost the same during the training process (left lower figure). However, using our fixed time-encoding function (blue curve) does not have the unstable exploding gradient issue and can quickly achieve high accuracy within several iterations. Meanwhile, we compare the parameter trajectories of the two models in Figure 5b. We observe that the change of parameters on the trainable Lime-encoding function is drastically larger than our fixed version. A huge change in weight parameters could deteriorate the model's performance. Most importantly, by replacing baselines' trainable time-encoding function with our fixed version, most baselines have a smoother optimization landscape (Figure 6) and a better model performance (in Table 2). which further verifies argument that our fixed time-encoding function is more preferred than the trainable version.
dw.
Table 2: Comparison on average precision score with fixed/trainable time encoding function (TEF). The results before "= is for trainable TEF (same as Table 1) and after "->" is for fixed TEF.
Reddit
Wiki
MOOC
LastFM
GDELT-ne
GDELT-c
JODIE
99.30 + 99.76
98.81
99.00
99.16
99.17
67.51
79.89
97.13
98.23
96.96 - 96.96
TGAT
98.66 - 99.48
96.71 + 98.55
98.43 - 99.33
54.77 + 76.26
84.30 + 92.31
96.96 + 96.28
TGN
99.80 + 99.83
99.55 + 99.54
99.62 - 99.62
82,23 + 87.58
98.15 + 98.25
96.04 - 97.34
4.3 ON THE IMPORTANCE OF MLP-MIXER IN GRAPIIMIXER'S LINK-ENCODER
In this section, we aim to achieve a deeper understanding on the expressive power of the link-encoder by answering the following two questions: "Can we replace the MLP-mixer link-encoder with self- atlention?" and "Why MI.P-iixe is a good alternative of self-atiention?"" To answer these questions,
In fact, other baselines (e.g., CAWs, TGSRee, APAN) also utilize this trainable time-encoding function. However, we focus our discussion on the selected methods for the ease of presentation.
7

---

Published as a conference paper at ICLR 2023
Comparison between Grad Params
Grad. Parrms
7500
Trainable time encoder
1307
30
2300
Fixed lime-nncader
Trainable Lime-encoder
180
0.0
0.4
0.6
0.8 1.0
Comparison between accuracy
Accuracy
0.8
Fixed ticoder
Fixeed Lime-nncoder
Trainable time encoder
180
Epochs
(b)
(a)
Figure 5: (a) Comparison on the gradient parameters norm and accuracy at each iteration. (b)
Comparison on the trajectories of parameter change, where the radius is rz | doll. the angle
0, arccos ( /|0: l / ////olll))) and ö, w is the difference between to optimal point
w The more the model parameters change during training, the larger the semicirele.
10
10
14
(a) TGAT GDELT- (b) TGN GDELT-c
(c) TGAT Wiki (d) TGN on Wiki
Figure 6: Comparison on the training loss landscape fixed time-encoding function. Results on other
datasets and baselines can be found in Appendix F.
let us first conduct experiments by replacing the MLP-mixer in link-encoder with full/I-hop self- attention and sum/mean sum/meaan-pooling pooling, where full self-attention is widely used in Transformers and 1-hop self-attention widely used in graph attention networks. As shown Table 3, GraphMixer suffers from performance degradation when using self-attention: the best performance is achieved when using MLP-mixer with aero-padding, while the model performance drop slightly when using self- attention with sum-pooling (row 2 and 4), and the performance drop significantly when using self-attention with mocn--oolingg (row 3 and 5). Self-attention with mean-pooling has a weaker model performance because it cannot distinguish "temporal sequences with identical link timestamps and features" (e.g., cannot distinguish [a1, a1] and [a1] and it cannot explicitly capture "the length of temporal sequences" (e.g., cannot distinguish if [a1, aq] is longer than (aa]), which are both very important for GraphMixer understand how frequent a node interacts with other nodes. We explicitly verify this in Figure by first generating two temporal sequences (with timestamps but without link features), then encoding the timestamps into vectors via time-encoding function, and asking full self-attention and MI.P-mixer to distinguish. As shown Figure 7, self-attention with mean-pooling cannot distinguish two temporal sequences with identical timestamps (because all the self-attention weights are equivalent if the features of the node on the two sides of link are identical) and cannot capture the sequence length (because of mean areannpoooing simply averages the inputs and does not take the input size into consideration). However, MLP-mixer GraphMixer can distinguish the above two sequences because of zero- aaro-padding. Fortunately, the aforementioned two weaknesses could be alleviated by replacing the mean wrea--poling in temporal self-attention with the sum sum-pooling, which explains why using sum-pooling brings better model performance than mean-pooling. However, since self-attention modules have more parameters and are harder to train, they could generalize poor when the downstream task is not too complicated.
4.4
KEY FACTORS TO THE BETTER PERFORMANCE
One of the major factors that contributes to GraphMixer's success the simplicity of GraphMixer's neural architecture and input data. Using conceptually simple input data that better aligned with their labels allows simple neural network model to capture the underlying mapping between the input to their labels, which could lead to a better generalization ability. In the following, this by comparing the performance of GraphMixer with different input data in Table 4: ① Recall from Section 3.2 that the "most recent h-hop neighbors" sampled for the two nodes on the "undirected" temporal graph could provide information on whether two nodes are frequently connected in the last few timestamps, which is essential for temporal graph link prediction. To verify this, conduct
explicitly verify
8

---

Published as a conference paper at ICLR 2023
Table Comparison on the average precision score. h use 20 neighbors due out of GPU memory
Link info encoder with
Reddit
Wiki
MOOC
LastFM
GDELT ne
(Detault) MLP-mixer
+ Zero- paro-ppdding
99.93
99.85
99.91
96.31
98.39
Sum pooling
99.81
98.19
99.55
93.97
98.28
Full self-attention
Mean pooling
99.00
98.05
99.31
89.13
97.13
Sum pooling
99.81
98.01
99.30
93.69
98.16
1 1-hop etffaattenion
Mean pooling
98.94
97.29
98.96
72.32
97.09
Accuracy
GNT pool
Accuracy
0.5
GNT munn pool
0.4
GAT sul pool
MLP F-iixer
Numbar epochs
Number of epochs
(a) c.g., classify if [01. 01 [a1]
(b) c.g., classify if sizela1, 02] >> size[aa]
Figure 7: (a) We generate identical timestamp sequences with different length, then ask MLP-mixer
and GAT distinguish whether the generated sequence are identical (b) We generate random sequence with different length, then ask MLP-mixer and GAT to classify which sequence is longer.
ablation study by comparing the model performance on direct and undirected temporal graphs. As shown in the 1st and 2nd row of Taable 4, changing from undirected to direct graph results in a significant performance drop because such information is missing. 2 Recall from Section 3.1 that instead of feeding the raw timestamp to GraphMixer and encoding each timestamp with trainable lime-encoding function, GraphMixer encodes the timestamps via our fixed timeennnoding function and feed the encoded representation to GraphMixer, which reduces the model complexity of learning time-encoding function from data. This could be verified by the 3rd and 4th rows of Table where using the pre-encoded time information could give us a better performance. 3 Selecting the input data that has similar distribution in training and evaluation evaluation error. For example, using relative timestamps (i.c., each neighbor's 's timestamp is subtracted by its root node's timestamp) is better than absolute timestamps (e.g., using Unix timestamp)becauss the absolute timestamps in the evaluation set and training set are from different range when using chronological splits, but they are very likely overlap if using relative timestamps. As shown in the 3rd to 6th rows of Table 4, using relative time information always gives better model performance than using absolute time information. ④ Selecting the most representative neighbors for each node. For example, we found 1-hop most recent interacted neighbors are the most representative for link prediction. Switching to either 2-hop neighbors or uniform sampled neighbors will hurt the model performance according to the 7th to the 10th row of Table 4.
could also potentially improve the
Table 4: Comparison the average precision score of GraphMixer with different input data. The highlighted rows are identical to our default setting.
Reddit
Wiki
MOOC
LastI-M
Direct vs undirect temporal graph
Directed temporal graph
99.69
88.37
97.87
78.31
Undirected temporal graph
99.93
99.85
99.91
96.31
Relative timestamp (f, ta)
99.79
99.80
99.81
95.32
Time information
Relative time-encone enneeenc cos((1, to )w)
99.93
99.85
99.91
96.31
Absolute timestamp
98.90
98.23 18.23
98.73
92.25
Absolute time-encoding cos(t.w)
99.52
99.13
99.74
95.28
2-hop
Most recent neighbors
99.39
98.05
99.11
89.36
Neighhor
I-hop
Most recent neighhors
99.93
99.85
99.91
96.31
selection
2-hop
Uniform sample neighbors
97.66
92.57
98.87
65.72
1-hop
Uniform sample neighbors
98.19
94.74
98.40
60.02
5 CONCLUSION
In this paper, we propose a conceptually and technically simple architecture GraphMixer for temporal link prediction. GraphMixer not only outperforms all baselines but also enjoys a faster convergence speed and better generalization ability. An extensive study identifies three key factors that contribute the success of GraphMixer and highlights the importance of simpler neural architecture and input data structure. An interesting future direction, not limited to temporal graph learning, is designing algorithms that could automatically select the best input data and data pre- prooressssn strategies for different downstream tasks.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work was supported in part by NSF grant 20083988. Majority of this work was completed during Weilin Cong's internship at Meta Al under the mentorship Si Zhang. We also extend our gratitude to Long Jin for his co-mentorship and for his contribution to the idea of using MLP-Mixer on graphs.
REFERENCES
Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. arXiv preprint urXiv:2006.05205, 2020.
Xhac-Hoai Nam Bui, Jiho Cho, and Hongsuk Yi. Spatial-temporal graph neural network for traffic forecasting: An overview and open research issues. Applied Intelligence, pp. 1-12, 2021.
Huixuan Chi, Hao Xu, Hao Fu, Mengya Liu, Mengdi Zhang, Yuji Yang, Qinfen Hao, and Wei Wu. Long short-term preference modeling for continuous-time sequential recommendation. arXiv preprint ar arXiv:2208.005: 2022.
Weilin Cong, Morteza Ramezani, and Mehrdad Mahdavi. On provable benefits of depth in training graph convolutional networks. Advances in Neural Information Processing Systems, 34:9936-9949, 2021a.
Weilin Cong. Yanhong Wu, Yuandong Tian, Mengting Gu, Yinglong Xia, Chun cheng Jason Chen, and Mehrdad Mahdavi. Dyformer: A scalable dynamic graph transformer with provable benefits on generalization ability. 2021b.
Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, and Philip Yu. Continuous time sequential recommendation with temporal graph collaborative transformer. In Proceedings the 30th ACM International Conference on Information Knowledge Management, pp. 433-442, 2021.
Jianfei Gao and Bruno Ribeiro. On the equivalence between temporal and static equivariant graph representations. In Proceedings of the 39th International Conference on Machine Learning. 2022.
Palash Goyal, Sujit Rokka Chhetri, and Arquimedes Canedo. dyngraph2vec: Capturing network dynamics using dynamic graph representation learning. arXiv preprint 180.vv1809.02653 2018.
Ehsan Hajiramezanali, Arman Hasanzadeh, Krishna Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. Variational graph recurrent neural networks. Advances in neural information processing systems, 32, 2019.
Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, and Pascal Poupart. Representation learning for dynamic graphs: A survey. J. Mach. Learn. Res., 21 (70):1-73, 2020.
Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016.
Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2019.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Neural Information Processing Systems, 2018a
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirly-Second AAAI conference artificial intelligence, 201 1bb.
Yuhong Luo and Pan Li. Neighborhood-awee aware scalable temporal network representation learning. arXiv preprint arXiv:2209.0108. 2022.
10

---

Published as a conference paper at ICLR 2023
Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao Schardl, and Charles Leiserson. Evolvegen: Evolving graph convolutional networks for dynamic graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5363-5370, 2020.
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. In ICML, 2020 Workshop Graph Representation Learning, 2020.
Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang, and Hao Yang. Dysat: Deep neural rep- resentation learning on dynamic graphs via self-attention networks. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 519-527, 2020.
Amauri H Souza, Diego Mesquita, Samuel Kaski, and Vikas Garg. Provably expressive temporal graph networks. arXiv preprint arXiv:2209. 15059, 2022.
Sheng Tian, Ruofan Wu, Leilei Shi, Liang Zhu, and Tao Xiong. supeeviid representation learning on dynamic graphs. CIKM '21, 2021. doi: 10. 0..14453455633.444433
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 34: 24261-24272, 2021.
Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha. Dyrep: Learning representations over dynamic graphs. In International conference on learning representations, 2019.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In International Conference Learning Representations, 2018.
Lu Wang, Xiaofu Chang, Shuang Li, Yunfei Chu, Hui Li, Wei Zhang, Xiaofeng He, Le Song, Jingren Zhou, and Hongxia Yang. Tel: Transformer based dynamic graph modelling via contrastive learning. arXiv preprint arXiv:2105.07944 2021a.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv: (tt.:::990003333 2019.
Xuhong Wang, Ding Lyu, Mengjian Li, Yang Xia, Qi Yang, Xinwen Wang, Xinguang Wang. Ping Cui, Yupu Yang, Bowen Sun, al. Apan: Asynchronous propagation attention network for real-time temporal graph embedding. In Proceedings of the 2021 International Conference on Management of Data, pp. 6628-2638, 2021b.
Yanbang Wang, Yen-Yu Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. In International Conference on Learning Representations, 2020.
Yiwei Wang, Yujun Cai, Yuxuan Liang, Henghui Ding, Changhu Wang, Siddharth Bhatia, and Bryan Hooi. Adaptive data augmentation on temporal graphs. Advances Neural Information Processing Systems, 34:1440-1452. 2021c.
Yiwei Wang, Yujun Cai, Yuxuan Liang, Henghui Ding, Changhu Wang, and Bryan Hooi. Time-aware neighbor sampling for temporal graph networks. arXiv preprint 122.9221 2021d.
Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Inductive representa- tion learning temporal graphs. arXiv preprint arXiv:2002.07962, 2020.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In IJCAI, 2018.
11

---

Published as a conference paper at ICLR 2023
Hongkuan Zhou, Da Zheng, Israt Nisa, Vasileios Ioannidis, Xiang Song, and George Karypis. Tgl: A general framework for temporal gnn training on billion-scale graphs. arXiv preprint arXiv:2203.14883, 2022.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020.
12

---

Published as a conference paper at ICLR 2023
A EXPERIMENT SETUP DETAILS
A.1 HARDWARE SPECIFICATION AND ENVIRONMENT
We run our experiments on a single machine with Intel i9-10850K, Nvidia RTX 3090 GPU, and 64GB RAM memory. The code is written in Python 3.8 and use PyTorch 1.12.1 CUDA 11.6 to train the model on the GPU. Implementation details could be found at
A.2 DETAILS ON DATASET
The dataset used in this paper could be automatically downloaded by this script. Reddit dataser consists of one month of posts made by users on subreddits. The link feature is extracted by converting the text of each post into feature vector. Wikipedia dataset' consists of one month of edits made by edits on Wikipedia pages. The link feature is extracted by converting the edit test into an LIWC- feature vector. LastFM " consists of one month of who listens-to-white MOOC dataset' consists of actions done by students on a MOOC online course. GDELT dataset is temporal knowledge graph dataset originated from the Event Database which records events happening in the world from news and articles.
song information.
Table 5: Dataset statistic.
V
[
(fmax Amin E
dim(x." (x.))k)
dim (an())))))
Node features
Link features Timestumps
Reddit
10,984
672,447
172
Yes
Yes
Wiki
9,227
157.474
17
0
172
Yes
Yes
MOOC
7,144
411,749
3.6
0
No
No
Yes
LastFM
1,980
1,293,103
106
0
0
No
Yes
GDELT
8,831
1,912,909
0.1
413
186
Yes
Yes
GDELTT-e
8,831
1,912,909
0.1
0
0
No
No
Yes
GDELT-n
8,831
1,912,909
0.1
0
186
No
Yes
Yes
A.3 MODEL CONFIGURATIONS, TRAINING AND EVALUATION PROCESS
Baseline implementations. The implementation on JODIE, DySAT, TGAT, TGN, and APPN follows the temporal graph learning framework Zhou et al. (2022)). Compared the original baselines" implementation, this framework's 's implementation could achieve a better overall score than its original implementation. The implementation of CAWs-mean and CAWs-attn follows their official implementation" 10 we choose the number of random walk steps from 8, 16, 32 to balance the training time. The implementation of TGSRec follows their official implementation DDGCL follows their official implementation" 2 We directly test using their official implementation by changing our data structure to their required structure and using their default paramerrrremeees.....
The implementation of
GraphMixer implementation. We implement GraphMixer under the TGL. framework Zhou et al. (2022) and use their default hyper-parameters (e-g., learning rate 0.0001, weight decay 10-6, batch size 600, hidden dimension 100, etc) to achieve fair comparison. In GraphMixer, there are only two hyper-parametert as introduced in Section 3: The number of I-hop most recent neighbors and the time-slot size T. In practice, Saper-parameter T is set the time-gap of the last 2, 000 interactions, which is fixed for all datasets; ppprrpprreettr K = 10 for Reddit and LastEM, K = 20 for MOOC, and K 30 for GDELT and Wiki.
Download from http:///////aa..attrtrrreaeaaaaa
DDwnloodd from hhaphp/p...aago..aaacrr.ceeeeeer
Download from hoool///ge/aaao.aaaattccocooooor
DDwwloodd from
*Download from hhan://gwr/.aaaaecceeeeeceeeaaaa
'The TGL framework can be found at attppp///tttuuocccorr/aaaeeeeeee
"CAW's official implementation be found http:////tthha..cco//eeeeaaaaaaa
"TGSRec's ss official implementation can he found at httppp/s//gtthbb.ccc//////////ss
PDDCL''s official implementation can he found httpp///ggtthii...om///aaa///2/0
13

---

Published as a conference paper at ICLR 2023
Training and evaluation. A unified training and evaluation process (e.g., mini-batch and data preparation) is used for GraphMixer and baselines. Specifically, each mini-batch is constructed by first sampling set of positive node pairs and equal amount of negative node pairs. Then, an algorihmm-dependern node sampler is used to sample the neighboring of each mini-batch node and computed their node representation based on the sampled neighborhood. Finally, concatenate each node pair and use the link prediction classifier (introduced in Section 3.1) for binary classification. We conduct experiments under the transduction learning setting and use average precision for evaluation.
B
MORE DISCUSSION ON EXISTING TEMPORAL GRAPH METHODS
B.1 RECENT METHODS THAT WE DO NOT COMPARE WITH
There are other temporal graph learning algorithms that are related to the temporal link prediction task but we did not compare GraphMixer with them because (1) the official implementation of some of the above works are not released by the authors and we could not reproduce their results as reported in the paper, and (2) we already compare many recent baselines that we believe it is enough to verify the success of GraphMixer.
For example, MeTA Wang et al. (2021c) proposes data augmentation to overcome the over-fitting issue temporal graph learning. More specifically, they generate a few graphs with different data augmentation magnitudes and perform the message passing between these graphs to provide adaptively augmented inputs for every prediction. TCL Wang et al. (2021a) proposes to use a transformer to separately extract the temporal neighborhoods representations associated with the two interaction nodes and then utilizes a co-attentional transformer to model inter-dependencies at semantic level. To boost model performance, contrastive learning is used to maximize mutual information between the predictive representations of two future interaction nodes. TNS Wang et al. (2021d) proposes a temporal-aware neighbor sampling strategy that can provide an adaptive receptive neighborhood for every node at any time. LSTSR Chi et al. (2022) propose Long Short- Preference Modeling for Continuous- Time Sequential Recommendation to capture the evolution of short-term preference under dynamic graph. DyRep Trivedi et al. (2019) uses RNNs to propagate messages interactions to update node representations. DyNAERNN Goyal et al. (2018) uses a fully connected layer to first encode the network representation, then pass the encoded features to the RNN, and use the fully connected network to decode the future network structure. VRGNN Hajiramczanali et al. (2019) generalizes variational GAE Kipf & Welling (2016) to temporal graphs, which makes priors dependent on historical dynamics and captures these dynamics using RNN. EvolyeGCN Pareja al. (2020) uses RNN to estimate GCN parameters for future snapshots. DDGCL Tian et al. (2021) is a SAM-based method that propose a debiased GAN-type contrastive loss as the learning objective correct the sampling bias that occurred in the negative sample construction process of temporal graph learning. NAT Luo & Li (2022) maintain two sets of representations for each node, i.e., node representations and link representations. For each node, NAT not only preserve a node representation. but also keep node pair representations for a subset of neighbors of the node. PINT Souza et al. (2022) using 1- test and proposes temporal encoding to boost the expressive power.
B.2 WHY EXISTING METHODS ARE CONCEPTUALLY AND TECHNICALLY COMPLICATED?
Please notice that we are not claiming conceptually and technically complicated is bad. Instead, we are simply suggesting that the conceptually and technically complicated simpler methods might be more preferred than the complicated one if they could achieve similar performance.
We say a method conceptually complicated if the underlying idea behind the method is non-trine noivtrivial. For example, CAWs represents network dynamics by "motifs extracted using temporal random walks"" represents node identity by "hitting counts of the nodes based on a set of sampled walks": TGSRec takes "temporal collaborative signals" into consideration. These concepts are non-trivial to understand in the first place and could potentially require much domain knowledge from other fields to understand the behavior of the method.
We say a method is technically complicated the method is non-trivial to implement due to many hyper-parameters and many details that need to be taken care of, which could potentially make the application to a real-world scenario challenge. For example, JODIE and TGN require maintaining a "memory" for each node by using RNN, and this "memory" needs be reset every time after
14

---

Published as a conference paper at ICLR 2023
evaluation because then it might carry information about the evaluation data. CAWs extracts features by using multiple temporal random walks, which makes implementing and hyper-paramen parameter fine- tuning more challenging. MeTA and TCL consider many data augmentation strategies, each of which is not trivial to implement and could affect the model's performance different ways.
B.3 THFORETICAL WORKS ON TEMPORAL GRAPH LEARNING
Recently, researchers have investigated the expressive power of temporal graph neural networks using graph isomorphism tests. For instance, Gao Ribeiro (2022) have categorized temporal graph learning methods into "time-and-grap" anddhh" and "time-then-gres gevenngraph" and compared their expressiveness. They have demonstrated that "time- time-then-graphe outperforms "time-and -aadd"rrph' terms of 1-WL test expressive power. This partially explains why GraphMixer has shown good performance, as it can be thought of time-then-graphe algorithm. Additionally, Souza et al. (2022) have shown that incorporating temporal encoding and using the 1-WL test can enhance the expressive power of temporal graph neural networks.
C MORE EXPERIMENT RESULTS
C.I COMPARISON ON CONVERGENCE SPEED AND GENERALIZATION
We include the missing figures of Section 4. Results on other datasets and the discussion on the experiment results could be found next Figure 3.
Roddit
BySAT
DySA: JoBiE
BySAT
JoDIE
Tounn
JocmI
675
GrapaMoocc
1.0
24
TG24
TCM
ToN
Dy5t
DySAT
PYSAT JOnIz
Jobss
JOp
Gosplitas
605
600
Figure 8: Comparioon of the link prediction training average precision and generalization gap for the first 100 training epochs. Results on other datasets can be found in Figure
Epyclo
Epocito
C.2 TRANSDUCTIVE LEARNING WITH RECALL @ K AND MRR AS EVALUATION METRIC
Recall @K K and MRR (mean reciprocal rank) popular evaluation metrics used in the real-world recommendation system. The larger the numbers, the better the model performance. Our Recall and MRR is implemented based on the Open Graph Benchmark's link prediction evaluation metrics' implementation More specifically, we first sample 100 negative destination nodes for the source node of each temporal link node pair, then goal is to rank the positive temporal link node pairs higher than 100 negative destination nodes. In the following, we compare the Recall @ and MRR score of GraphMixer with the selected four most representative baselines. Please notice that since these methods are implemented under the same framework, the model performance is evaluated by using the same model used in Table the comparison is guarantced to be fair.
Table 6: Comparison on the Recall @ K and MRR.
Reddit
Wiki
MOOC
LastFM
GDELT
R@5
MRR
R@5
MRR
R@5
MRR
R@5
MRR
R@5
MRR
JODIE
01
0.6271
0.9098
0.7752
0.9818
0.7551
0.2034
0.1206
0.8554
0.6048
DySAT
0.9189
0.7774
0,8889
0.7561
0.9989
0.7906
0.4159
0.3322
0.8302
0.4236
TGAT
0.9774
0.8709
0.8508
0.6132
0.9736
0.7425
0.1040
0.0769
0.3513
0.2366
TGN
0.9787
0.9093
0.8878
0.8016
0.9904
0.9904
0.1649
0.1153
0.9297
0.7295
GraphMixer
1.0
0.9965
0. 0.99772
0.9876
0.9999
0.9910
0. 0.9998
0.9649
0.9930
0. 0.8934
Phtpps:/ :/tpps://giie tht.///ttttt.tet/////// httpa//cco/mnaasttatoooooeo http://pg/.....o.ooaaao/////////
eva luate.py
15

---

Published as a conference paper at ICLR 2023
We have the following observations Table 1:
According to the results in Table 6, GraphMixer could achieve outstanding performance across all datasets. Especially on the LastFM and GDELT datasets (denser graphs than other datasets). This might implics GraphMixer is more suitable for denser graphs than other baseline methods.
The results of some baseline methods behave less better with Recall @ K and MRR evaluation metrics. For example, TGN on LastFM dataset, TGAT on LastFM and GDELT, etc. The above results also imply the limitation of only considering average precision and AUC score for temporal link evaluation.
C.3 COMPARISON ON WALL-CLOCK TIME
In the following, we compare the wall-clock time it takes for GraphMixer and baselines to finish a single epoch of training.
Table 7: Comparison on the wall-clock computation time for single-epoch of training.
Reddit
Wiki
MOOC
LastFM
GDELT
JODIE
5 sec
2 sec
4 sec
11 sec
16 sec
DySAT
33 sec
6 sec
16 sec
41 sec
83 sec
TGAT
15 sec
4 sec
8 sec
28
41 sec
TGN
8 sec
2 sec
5 sec
15 sec
32
CAWs-mean
893 see
277
641 sec
797
4, 4,544
CAWs-attn
.30 sec
282
653 sec
832
634
TGSRec
538 sec
157 sec
656 sec
810 see
707 sec
APAN
13 sec
4
sec
28
25
GraphMixer
12 sec
3 sec
sec
sec
32 sec
When comparing the computation time of GraphMixer with CAWs, TGSRec, and DDGCL, we found that GraphMixer takes significantly lesser time than these baselines, which indicates the effectiveness of GraphMixer. When comparing the computation time of GraphMixer with JODIE, DySAT, TGAT, APAN, and TGN, we found that GraphMixer very close to or even slightly faster than some baseline methods. Our computation time is slightly slower than other baseline methods (e.g.. JODIE and TGN) mainly because these baselines are using well-optimized computation functions from DGL Wang et al. (2019), while GraphMixer is just using a composition of basic Py PyTorch functions. In fact, according to the Table 8, GraphMixer has a similar/smaller amount parameters with these baselines.
Table 8: Comparison on the number of model parameters.
GraphMixer
GraphMixer-T GraphMixer-S
Jodie
DySAT
TGAT
TGN
# Parameters (x10*)
2.25
1.42
2.07
1.21
9.38
3.80
3.54
Besides, our current implementation also need to preprocess the input data for each epoch of training. e.g., sorting nodes in subgraph according to temporal order and removing duplicated edges. For example, the data preparation at each epoch takes 41 sec on Reddit, sec Wiki, 20 sec on MOOC, 48 sec on LastEM, and 71 sec on GDELT. However, by caching the pre- preceesee data in the memory, only need pre- pre-process the input data the first epoch of the training process because neighbor selection is deterministic and the input data does not change at each epoch.
C.4 TRANSDUCTIVE LEARNING WITH AUC AS EVALUATION METRIC
AUC (Under the ROC Curve) one of the most widely accepted evaluation metric for link prediction, which has been used in many existing works Xu et al. (2020); Rossi et al. (2020). In the following, we compare the AUC score of GraphMixer with baselines. We have the following observations: ! GraphMixer outperforms all basclines on all datasets. In particular, GraphMixer attains more than 1% gain over all baselines on the LastFM, GDELT-ne, and GDELT- datasets, attains around 2% gain over Ron-RNN methods DySAT' and TGAT on the Wiki dataset, and attains around 1 11% gain over non-RNN methods DYSAT and TGAT on the GDELT-ne dataset. The experiment results provide sufficient support our argument that neither RNN nor SAM is necessary for temporal graph link prediction. 2 According to the performance of GraphMixer-1 on datasets only have link timestamp information (MOOC, LastFM, and GDELT-ne), successfully pre-pre pre-process each timestamp into a meaningful vector. 2 By comparing the performance GraphMixer-N and Graph on Wiki, MOOC, and LastEM datasets, know that node-info
know that our time-encoding function could
16

---

Published as a conference paper at ICLR 2023
encoder alone is not enough to achieve good performance. However, it provides useful information that could benefit the link-info encoder. 4 By comparing the performance of GraphMixer-N on GDELT and GDELT-ne, we observe that using one-hot encoding outperforms using node features. This also shows the importance of node identity information because one-hot encoding only captures such information.
Table 9: Comparison on the AUC score for link prediction. GraphMixer uses one-hot node encoding for datasets without node features (marked by t). For each dataset we indicate whether we have the corresponding feature ("I." link features, "N" node features, and "T" link timestamps).
Reddi L. T
Wiki
MOOC
LastFM
GDELT
GDALT-ne
GDELT- N.
L.T
T
T
L.
T
JODIE
99.30 = 0.01
98.81
0.01
99.16 +
67.51
+ 1.99
98.55
= 0.01
97.13
0.02
96.96
0.02
DySAT
98.52 1 0.01
96.71
0.00
98.82
0.01
76.40 1..00.:00
0.81
98.52
1 0.01
82.47
1..47.0.00
97.25
0.02
TGAT
99.66 + 0.01
0.00
98.43 1.00
54.77
4 1.02
08.25:0000
84.30
0.03
96.96
= 0.02
TGN
99.80 = 0.01
99.55
82.23
0.50
98.15
0.01
97.13
0.02
96.04
0.02
CAWs-menn
98.18 1 0.01
0.003
0.99
72.92
0.33
95.19
1 0.09
71.82
0.08
91.40
0.19
CAWs-attn
98.30 = 0.01
0.08
0..1
72.93
95.13 0.54
71.82
0.08
91.64
0.24
TGSRec
94.74
0.20
91.32
2..70
76.66
1.54
96.72
+ 0.42
96.72
96.72
0.42
APAN
99.24 = 0.01
97.25
98.58
62.73
0.64
96.46
0.11
98.39
0.17
97.85 0.19
GraphMixer-L
99.84 1 0.01
99.70
97.04
0.02
98,99 1 0.02
96.54
0.02
98.99
0.02
GraphMixer-N
90.53
0.01
91.49 0.01
98.66
0.02
71.51
1 0.03*
94.44 1: 0.02
96,00
0.02
98.81
1 0.02
GraphMixer
99.94 0.01
99.82
0.01
99.93 1: 0.01
97.38 + 0.02
98.89
+ 0.02
98.50
0.02
98.48 + 0.02
C.5 BASELINES WITH UNDIRECTED TEMPORAL GRAPH
GraphMixer utilizes undirected temporal graph to capture whether two nodes are frequently connected in the last few timestamps. In the following, we test whether using undirected temporal graph could improve the performance of bascline methods. As we can see from Table 10, using undirected temporal graph cannot improve the performance of baseline methods much because such information are already implicitly captured via their neural architecture design or sampling methods.
Table 10: Comparison on baselines with undirected temporal graph (Average precision AUC score).
Reddit
Wiki
MOOC
LastFM
JODIE
99.24
99.40
98.91
99.07
99.18
99.53
73.25
80.24
DySAT
98.53
98.42 42.422
96.61
96.88
98.80
99.26
76.23
73.90
TGAT
99.70
99.73
97.35
97.68
98.41
98.85
54.58
57.03
TGN
99.84
99.87
99.59
99.61
99.44
99.66
91.96
93.20
D DISCUSSION ABOUT MODEL. PERFORMANCE ON LASTFM
Our results show that GraphMixer could outperform baselines on LastFM dataset with a large margin. which is due to a composite effect of multiple factors. In the following, we summarize several potential factors that lead to our observation on the model performance.
Larger average time-gap. LastEM has a larger average time-gap (4max - mma)/ than other datasets. As shown in the dataset statistic (Table 5), LastFM has an average time gap of 106, which is significantly larger than other datasets. For example, Reddit's average time gap is 4, Wiki's average time gap is 17, MOOC's average time gap is 3.6, and GDELT's average time gap is 0.1. Since baseline methods are relying on RNN and SAM process the historical temporal information, they implicitly assumes the temporal information is "smooth" and with smaller average time gap. Therefore, baseline methods could potentially work better on the dataset with a smaller average time gap but are less ideal on LastFM. GraphMixer is not relying on RNN or
SAM. therefore could be less affected by the aforementioned issue.
Larger average node degree. LastFM has larger average node degree | than other datasets, which potentially prone to over-smoothing (aggregating features from many neighbors make output representation less distinguishable Li et al. (2018b)), over-squashing (aggregating much information into a limited memory might compress too much useful information Alon & Yahav (2020)) and over-fitting Cong et al. (2021a) effect. For example, according to the dataset statistic in Table 5, LastFM has an average node degree of 653, which is larger than the Reddit's s average node degree 61, Wiki's 's' average node degree 17, MOOC's 's average node degree 57, and GDELT's s average node degree 216. Existing methods either use the memory cell in RNN to store temporal information or use SAM to aggregate temporal information from multi-hops,
17

---

Published as a conference paper at ICLR 2023
which could be less ideal dense graph due to over-smoothing and over-squashing However, GraphMixer is less relying on the aggregation schema, therefore its performance better than the baseline methods.
Larger maximum timestamp. GraphMixer using fixed time encoder but baselines are using trainable time-encoders. Since the largest timestamp max in LastFM is larger than other datasets, the trainable time-encoder is more affected by the unbounded gradient issue as discussed in Table and Section 4.2. For example, the max in LastFM is 137 millon, while max in GDELT 0.2 millon, max in Reddit 2.6 millon, max in Wiki 2.6 millon, and max in MOOC 2.6 millon.
18

---

Published as a conference paper at ICLR 2023
E
MISSING FIGURES ON LOSS LANDSCAPE
E.
LOSS LANDSCAPE ON WIKI
Train loss 2d contours
Train loss 2d contours
Train loss 2d contours
1.0
1.0
1.0
0.5
0.5
0.5
-0.4
-0.5
-0.5
-1.0
-1.0
-1.0
-1.
-1.0
-0.5
0.0
1.0
1.5
-1.0
-0.5
0.0
0.5
-1.5
-1.0
-0.5
0.0
1.0
(a) TGN
(b) TGAT
(c) DySAT
Train loss 2d contours
Train loss 2d contours
1.5
1.5
1.0
0.0
0.0
-0.5
-1.0
-1.0
-1.5
-0.5
(d) JODIE
(c) GraphMixer
Figure 9: Comparison on the training loss landscape (Contour) on Wiki Dataset.
Train loss surface
Train loss 3d surface
Train loss surface
3.5
30
2.25
2.5
2.00
1.75
20
50
1.25
L.5
1816 1818.05 a5.. 6.5
0.5
3.5
0.0
L.0
-1.0
05
0.5 1.0
-1.0
(a) TGN
(b) TGAT
(c) DySAT
Train
1.0
1.0
0.5
1.0 0..00.0
10
-0.5
-0.5
1.0
-1.0
-1.0
(d) JODIE
(c) GraphMixer
Figure 10: Comparison on the training loss landscape (Surface) on Wiki Dataset.
19

---

Published as a conference paper at ICLR 2023
E.2 LOss LANDSCAPE ON REDDIT
Train loss 2d contours
Train Joss 2d contours
Train loss 2d contaurs
-15
-15
-10
-6.5 0.0
0.5
1.0
1.5
-5
-16
-0.5
0.0
0.5
1.0
1.5
-55
-1.
-0.5
0.0
0.5
1.0
1.5
(a) TGN
(b) TGAT
(c) DySAT
Train loss 2d contours
Train loss 2d contours
1.0
1.0
1.5
1.0
0.5
00
05
1.0
1.5
1.5
1.0
6.5
0.0
0.5
1.0
1.5
JODIE
GraphMixer
Figure 11: Comparison on the training loss landscape (Contour) on Reddit Dataset.
Train surface
Train surface
Train loss 3d surface
2.0
1.0
1.0
3.0
0.5
0.5
(a) TGN
-1.5
-155 (b) TGAT
(c) DySAT
Train loss 3d surface
Train 3d surface
15
15
10
10
00 10---amm
-1
(d) JODIE
(e) GraphMixer
Figure 12: Comparison on the training loss landscape (Surface) on Reddit Dataset.
20

---

Published as a conference paper at ICLR 2023
E.3
LOss LANDSCAPE ON MOOC
Train loss 2d contours
Train loss 2d contaurs
Train loss 2d contours
15
-1.0
0.0
0.5
1.0
-5
-1.0
0.5
0.0
0.5
1.0
-55
-1.0
-0.5
0.0
0.5
1.0
(a) TGN
(b) TGAT
(c) DySAT
Train loss 2d contours
Train loss 2d contours
1.5
-1.0
-0.5
0.0
0.5
1.0
155
-1.0
-0
0.0
0.5
1.0
(d) JODIE
(c) GraphMixer
Figure 13: Comparison on the training loss landscape (Contour) on MOOC Dataset.
Train loss 3d surface
Train surface
Train loss 3d surface
26
15
2.0
1.0
0.
1.0
0.5
18 05
5. 1.
0.0
-15
1.
-1.5
60
03
-1.3
(a) TGN
(b) TGAT
(c) DySAT
Train surface
Train loss 3d surface
135
22.5
18a
-1.5
-1.55
-15 18..5 S
0.0
La
1.5
(e) GraphMixer
(d) JODIE
Figure 14: Comparison on the training loss landscape (Surface) on MOOC Dataset.
21

---

Published as a conference paper at ICLR 2023
E.4 LOss LANDSCAPE ON LASTFM
Train loss 2d contours
Train loss 2d contours
Train loss 2d contours
-15
-.5
-1.0
0.5
0.0
0.5
1.0
-5
-1.0
-0.5
0.0
0.5
1.0
-55
-1.0
00.5
0.0
0.5
1.0
(a) TGN
(b) TGAT
(c) DySAT
Train loss 2d contours
Train loss 2d contours
-1.0
15
-1.0
-03
0.0
0.5
1.0
-15
-1.0
-0
5
D.0
0.5
1.0
(d) JODIE
(c) GraphMixer
Figure 15: Comparison on the training loss landscape (Contour) on LastFM Dataset.
Train loss 3d surface
Train
Train loss 3d surface
2.0
1.0
2
18 05
5.
0.0
10 a5
1.
-1.5
1.6
(a) TGN
(b) TGAT
(c) DySAT
Train surface
Train loss 3d surface
20
10
-15 18.. 50
-05
0.0
96
1.5
(e) GraphMixer
(d) JODIE
Figure 16: Comparison on the training loss landscape (Surface) on LastFM Dataset.
22

---

Published as a conference paper at ICLR 2023
E.5
LOss LANDSCAPE ON GDELT-NE
Train loss 2d contours
Train loss 2d contours
Train loss 2d contours
0.75
0.75
0.75
0.50
0.50
0.50
0.00
0.00
0.00
0.50
0.50
0.50
-0.75
-075
-075
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
(a) TGN
(b) TGAT
(c) DySAT
Train loss 2d contours
Train loss 2d contours
0.50
6.50
0.25
0.25
-0.75
-0.75
-1.00
-1.00
-0.5
0.0
-0.5
0.0
1.0
JODIE
GraphMixer
Figure 17: Comparison the training loss landscape (Contour) on GDELT-ne Dataset.
surface
Train loss 3d surface
surface
1.75
1.5
1.20
1.50
1.00
-2
1.0
0.75
1.00
0.5
0.5
1a
-1.0
-1.0
-0.5
-1.0
0.5
0.0
1a
-1.
(a) TGN
(b) TGAT
(c) DySAT
Truin loss surface
Train loss 3d surface
-10
0.5
-0.5
1.5
1.0
La
(c) GraphMixer
(d) JODIE
Figure 18: Comparison on the training loss landscape (Surface) on GDELT-ne Dataset.
23

---

Published as a conference paper at ICLR 2023
E.6
LOss LANDSCAPE ON GDELT-E
Train loss 2d contours
Train loss 2d contours
Train loss 2d contours
0.75
0.75
0.75
0.50
0.50
0.50
0.00
0.00
0.00
0.50
0.50
0.50
-0.75
-075
-075
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
(a) TGN
(b) TGAT
(c) DySAT
Train loss 2d contours
Train loss 2d contours
0,50
6,50
0.00
6.00
20
0.25
0.25
-0.75
-0.75
-1.00
-1.00
-0.5
0.0
1.0
-0.5
0.0
JODIE
GraphMixer
Figure 19: Comparison the training loss landscape (Contour) on GDELT-e Dataset.
Train
Train
255
2.5
1.0
15
13
10
0.5
1.0
1.0 1.35
00
0.0
0.0
1.533
1.5
0.
0.5
0.0
1.0
-35
0.
00.5
0.0
-1.0
-1.0
-1.0
(a) TGN
(b) TGAT
(c) DySAT
surface
Train loss 3d surface
1.9
-10
0.5
-1.
n.
-1.0
-3.5
0,2
05
(d) JODIE
(e) GraphMixer
Figure 20: Comparison on the training loss landscape (Surface) on GDELT-e Dataset.
24

---

Published as a conference paper at ICLR 2023
F
MISSING FIGURES ON LOSS LANDSCAPE WITH OUR TIME-ENCODING FUNCTION
F.I
TGAT WITH FIXED TIME-ENCODING FUNCTION
Train
Train contours
2d
1.0
1.5
1.0
0.5
1.0
0.5
0.0
0.5
0.0
0.5
0.0
0.5
-100
0.5
1.0
-1.0
1.0
1.5
1.0
0.0
1.0
(a) Reddit
(b) Wiki
(c) MOOC
loss 2d
Train loss 2d contours
1.0
0.75
0.5
0.
0.00
-05
-0.50
-10
0.75
-1.5
1.0
-0.5
6.0
1.6
(d) LastFM
(e) GDELT-e
Figure 21: Training loss landscape (Contour) of TGAT with fixed time-encoding function.
Train loss 3d surface
Train loss 3d surface
Train loss 3d surface
1.0
0.5
0.0
5.0
11
-0.5
-1.5
1.0
05
1.0
0.9
05
1.0
(a) Reddit
(b) Wiki
(c) MOOC
Train surface
Train loss 3d surface
10
-00
-15
-0.5
-1.0
0.3
1.0
-1.5
05
(d) LastFM
(e) GDELT-e
Figure 22: Training loss landscape (Surface) of TGAT with fixed time-encoding function.
25

---

Published as a conference paper at ICLR 2023
F.2
TGN WITH FIXED EINO-INGO FUNCTION
Train loss 2d contours
Train loss 2d contours
Train loss 2d contours
83
-15
-15
-15
-10
065
0.0
0.5
1.0
1.5
-55
-1.0
0.55
0.0
0.5
1.0
-55
-1.0
0.0
0.5
1.0
(a) Reddit
(b) Wiki
(c) MOOC
Train loss 2d contours
contours
1.00
9.75
0.50
0.25
-0.25
0.50
-1.00
-15
-1.0
-0.5
0.0
0.5
1.0
(d) LastFM
(c) GDELT-c
Figure 23: Training loss landscape (Contour) of TGN with fixed time-encoding function.
Trai loss 3d surface
Train loss 3d surface
Train loss 3d surface
0
0.5
3.0
5.5
-0.5
-1.0
-1.5
-1.0
-1.5
(a) Reddit
(b) Wiki
MOOC
Train loss 3d surfhce
Train loss 3d surface
30
2.5
1.5
1.5
0.5
1.0
18.0 0..0
0.0
1.0
10.5
0.0
0.5
04
0.
-1.5
-1.0
(d) LastFM
(e) GDELT-e
Figure 24: Training loss landscape (Surface) of TGN with fixed time-encoding function.
26

---

Published as a conference paper at ICLR 2023
F.3 JODIE WITH FIXED TIME-ENCODING FUNCTION
Train loss 2d contaurs
Train loss 2d contours
Train loss 2d contours
55
-10
-65
0.0
6.5
1.0
1.5
-5
-1.0
0.5
0.0
0.5
1.0
-55
-1.0
0.0
0.5
1.0
(a) Reddit
(b) Wiki
(c) MOOC
Train loss 2d contours
contours
1.00
9.55
0.50
0.25
-0.2
0.50
-1.00
-15
-1.0
-0.5
0.0
0.5
1.0
(d) LastFM
(c) GDELT-c
Figure 25: Training loss landscape (Contour) of JODIE with fixed lime-encoding function.
Train loss 3d surface
Train loss surface
Train loss 3d surface
15
10
1.
0.5
0.0
50 0.5
1.8
-1.0
-1.5
-1.5
(a) Reddit
(b) Wiki
MOOC
Train loss 3d surface
Train loss 3d surface
15.6
12.5
10.0
2
0.5
0.5
10
-10 Ro
-1.5
-1.0
-05
0.0
65
-1.0
-0.5
(d) LastEM
(c) GDELT-c
Figure 26: Training loss landscape (Surface) of JODIE with fixed time-encoding function.
27


Reviews:
[{"review_id": "nA7TPc5fns1", "reviewer": "Reviewer_naoZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose GraphMixer, a conceptually and technically simple architecture that consists of three components: 1 a link-encoder that is only based on multi-layer perceptions (MLP) to summarize the information from temporal links, 2 a node-encoder that\nis only based on neighbor mean-pooling to summarize node information, and 3 an MLP-based link classifier that performs link prediction based on the outputs of the encoders. The authors show that GraphMixer attains an outstanding performance on temporal link prediction benchmarks with faster convergence and better generalization performance.", "summary_of_the_review": "Please refer to the above points.", "strength_and_weaknesses": "Pros: Focused on the temporal graph learning, the authors propose a conceptually and technically simple architecture GraphMixer for temporal link prediction. GraphMixer not only outperforms all baselines but also enjoys a faster convergence speed and better generalization ability. The author conduct empirical study to identify three key factors that contribute to the success of GraphMixer and highlights the importance of simpler neural architecture and input data structure.\n\nCons: There is not obvious weakness in the draft.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cl9QvYbt-r", "reviewer": "Reviewer_7yc7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple architecture named GraphMixer for temporal graph learning. The GraphMixer is based on simple MLP structure and achieves state-of-the-art performances on temporal link prediction benchmarks.", "summary_of_the_review": "I think this paper is a simple and effective paper, thus I want to give it an accept rating.", "strength_and_weaknesses": "First of all, I'm not the expert in the graph learning area and my research focus is computer vision. From my side, I would to say: A simple method along with state-of-the-art performances satisfies me. \n\nWhen I read this paper, I'm satisfied by the simple method, the detailed experiments. Thus I would like to give an accept rating.\n\nHowever, there is still a little bit concerns:\n\n1. I think the major novelty lies in the time-encoding part, also the proposed GraphMixer performs extremely better on LastFM dataset (c.f., Table 2). Could the author give explanations of why GraphMixer work well on LastFM? Since I only observe marginal improvements on other datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MxRCxR--YO6", "reviewer": "Reviewer_xY6D", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, they carefully design and propose the GraphMixer network for temporal link prediction in dynamic graphs. With a carefully designed and ablate architecture separated into three components, the link-encoder, node-encoder, and MLP-Mixer based link classifier, they are able to achieve high performance on this task that has only previously been attained with RNN and Self-Attention based architectures. They include extensive experiments to study the different architecture choices they made and their effect on performance and generalization.", "summary_of_the_review": "This work proposes a novel architecture for link prediction in temporal graph networks that achieves very good performance without RNN or self-attention mechanisms. They achieve this using a carefully constructed network with MLP-mixer. They do detailed experiments to analyze and ablate the different design choices they made. A large strength of this paper is the extensive experiments ablating the details of the design choices they made how it affects performance, generalization, and training stability. This greatly helps the community understand how these architecture choices affected their model and may extend to other models. The paper may have somewhat limited novelty since the different parts of the design have all been explored elsewhere and in terms of setting a new SOTA, the model still has some under-analyzed weaknesses such as the slow training speed.", "strength_and_weaknesses": "Strengths:\n- Proposes a novel architecture that achieves performance competitive with other networks without memory or attention mechanisms.\n- Strong analysis and ablation of the different design choices made.\n\nWeaknesses:\n- The model takes significantly longer to train and there is not significant analysis of the inference cost. This lack of discussion in the main paper also makes some conclusions about faster convergence somewhat misleading.\n\nQuestions:\nWhy are the train average precisions for the competing methods in Figure 3 so much lower than the final results in Table 1. For example, DySAT training AP is around 60-70% while the final eval is 96.71. Are all the results from reimplementations or from taken from other papers.\n\nIt would likely be helpful to bring appendix C.3 into the paper and analyze and discuss more about the model efficiency since the model does take significantly longer to train than some baselines like TGN. If some of this is due to preprocessing, it would be helpful to separate the time required for each and/or analyze the inference FLOPs for your model compared to the others.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%