Recent Link Classification on Temporal Graphs Using Graph Profiler
Authors: Muberra Ozmen
Thomas Markovich
Venue: Accepted by TMLR ()

Abstract: The performance of Temporal Graph Learning (TGL) methods are typically evaluated on the future link prediction task, i.e., whether two nodes will get connected and dynamic node classification task, i.e., whether a node's class will change. Comparatively, recent link classification, i.e., to what class an emerging edge belongs to, is investigated much less even though it exists in many industrial settings. In this work, we first formalize recent link classification on temporal graphs as a benchmark downstream task and introduce corresponding benchmark datasets. Secondly, we evaluate the performance of state-of-the-art methods with a statistically meaningful metric Matthews Correlation Coefficient, which is more robust to imbalanced datasets, in addition to the commonly used average precision and area under the curve. We propose several design principles for tailoring models to specific requirements of the task and the dataset including modifications on message aggregation schema, readout layer and time encoding strategy which obtain significant improvement on benchmark datasets. Finally, we propose an architecture that we call Graph Profiler, which is capable of encoding previous events' class information on source and destination nodes. The experiments show that our proposed model achieves an improved Matthews Correlation Coefficient on most cases under interest. We believe the introduction of recent link classification as a benchmark task for temporal graph learning will be useful for the evaluation of prospective methods within the field.

Published in Transactions on Machine Learning Research (06/2024)
Recent Link Classification on Temporal Graphs Using Graph Profiler
Muberra Ozmen
mmberna@squureupp.omm
CashApp
Montreal, QC, Canada
Thomas Markovich I
kmarkonnchsstsqurrcuu com
CashApp
Cambridge, MA, USA
Reviewed on OpenReview: https: // openreview. net/ forem? d
Abstract
The performance of Temporal Graph Learning (TGL) methods are typically evaluated on the future link prediction task, ie., whether two nodes will get connected and dynamic node classification task, i.e., whether a node's class will change. Comparatively, recent link classification, i.e., what class an emerging edge belongs to, is investigated much less even though it exists in many industrial settings. In this work, we first formalize recent link classification on temporal graphs benchmark downstream task and introduce corresponding benchmark datasets. Socondly, we evaluate the performance of state-of-the- art methods with statistically meaningful metric Malthews Correlation Coefficienl, which is more robust to imbalanced datasets, in addition to the commonly used average precision and area under the curve. We propose several design principles for tailoring models to specific requirements of the task and the dataset including modifications on message aggregation schema, readout layer and time encoding strategy which obtain significant improvement on benchmark datasets. Finally, propose architecture that call Graph Profiler, which is capable of encoding previous events' class information on source and destination nodes. The experiments show that our proposed model achieves an improved Matthews Correlation Coefficient on most cases under interest. We believe the introduction recent link classification as a benchmark task for temporal graph learning will useful for the evaluation of prospective methods within the field.
1 Introduction
Graphs provide convenient structures to represent interactions relationships between entities by modeling them as odges between verticos. Using this representation allows one to build models that capture the
interconnected nature of complex systems such as social networks Eil-Kishky et al. 2022 Wu et al., 2022;
Gao et al. 2021) or transaction graphs (Liu et al. 2020 Zhang et al. 2022). Graph Representation Learning (GRL) rose in popularity due to the desire to apply deep learning to graph structured problems (Zhou et al.
2020
Wu et al. 2020 Hamilton 2020). Indeed, GRL has provided significant advances in fraud detection (Liu
al.
2020 Zhang et al. 2022), recommendation systems (Wu et al. 2022 Gao et al. 2021), chemistry and
materials science (Pezzicoli et al. 2022 Reiser et al.] 2022 Bongini et al. 2021 Han et al. 2021 Xiong
et al.
2021), ) traffic modeling (Rusek et al.) 2019 Chen et al.
2022). and weather simulation (Keisler 2022
Ma et al. 2022). among other possible applications. Many of these graph machine learning tasks can be
understood as either link prediction (Chen et al. 2020 Cai ct al. 2019 Zeb ct al. 2022 Chamberlain et al.
* "Also attending McGill University during this research,
i "Corresponding author

---

Published in Transactions on Machine Learning Research (06/2024)
O
100
FLP
RLC What will be the node colour in the future? What will be the colour of a new edge?
DNC
Which nodes will be connected next?
Figure 1: Differences between TGL tasks. Future Link Prediction (FLP): Given a temporal graph and a pair of nodes observed at a timestamp, a function is learned which predicts the probability of these two nodes linking at a later timestamp. Dynamic Node Classification (DNC): Let blue and purple colours represent the node classes which change over time. Given the current node classes along with the set of edges at time point, the aim is prodicting the node classes at later timestamp. Recent Link Classification (RLC): Let the colours blue and purple represent the edge classes, which are observed some time after the emergence of a link. That is, at the time when a link is first detected, the link class is unknown. Black edge have been recently observed but the class is not yet observed. In this sctting, the aim is to predict recent link classes given a history of previous interactions.
2022 or node classification tasks Kipf & Welling 2016 Zhang et al. 2019). Much of the early work focused on the scenario where the graph is static. Acknowledging that many tasks in industrial settings involve graphs that evolve in time, rescarchers defined sub-problem of GRL called Temporal Graph Learning (TGL), with time dependent versions of the original static lasks, yielding Future Link Prediction (FLP) and Dynamic Node Classification (DNC) respectively (Kumar et al. 2019 Arnoux et al. 2017). The former task, FLP, seeks to predict whether two vertices will be connected at some specified future time; while the latter, DNC, seeks to predict the class of a vertex at a future time (See Figure D Both of these tasks have a variety of applications in real world, e.g., predicting the probability of two people forming a tie or of a person deactivating their account on social media platforms, corresponding to FLP and DNC tasks
respectively (Min al. 2021 Song et al., 2021 Frasca et al. 2020 Zhang et al.) 2021a).
This begs the question "In analogy to the dynamic node classification task, is there temporal link classification task classification (Wang et al., 2023). which we lermn recend link classification (RLC). RLC is present in industrial settings but has yet to be formalized in research setting. The task of RLC requires that one classify a link that has been observed to exist but does not as yct have a known label. This task is important for sottings where we wish to classify the interaction but labels may be delayed. In social media settings, this could involve classifying interactions as abusive, and a transaction network it could involve classifying transaction as potentially fraudulent. More formally, we define RLC as task in which the predictive algorithm is given the source and destination entities of a recent interaction and its features, c.g., textual content of the post: and it must predict the target class value. The task of classifying recent. interactions has typically been treated as tabular data task, arising in applications such as fraud detection (Sarkar 2022). This approach represents each interaction between two entities as a sample with associated features. The sample can thus be represented as a row of a table, where each column is a feature. There usually an (implicit) assumption that the samples are independent. This neglects the fact that entities may participate in multiple interactions and thus induce dependencies between the samples. In practice, one option to address this deficiency is feature engincering. For tabular tasks, it has been previously observed that features representing counts, such as how many times user has previously liked another user's posts, provide significant metric uplift (Wu et al, 2019 Shan et al.) 2016). While the incorporation of such features is not explicitly graph machine learning, we lake the view that these features are the result of manual feature engineering inspired by graph-based intuition (Zhang et al. 2021b Martínez et al., 2016 Chamberlain et al. 2002).. Therefore, we believe that explicitly formulating RLC as a graph learning task and highlighting suitable datasets for the temporal graph machine learning community will encourage progress towards better solutions for an industrially relevant problem.
define?" And indeed, there is third common task that is based static link
2

---

Published in Transactions on Machine Learning Research (06/2024)
With this motivation in hand, we formalize two research questions that we wish to answer in this work: "Q1: How does recent. link classification differ from future link prediction?" and -Q2: What are the most critical design principles of recent link classification?". We answer the first question through a comparative study of baseline mothods on both tasks. We answer the second question by exploring a variety of model building blocks, some published previously, and some novel. In answering these research questions, we contribute a new TGL task, new figure of merit, a measure of halg-hhompphll, and a non-exhaustive set of design principles that comprise a design space for this new machine learning task.
2 Related Work
There exist two main models for dynamic graphs: discrete-time dynamic graphs and continnous-time dynamic graphs (Zhou et al. 2022). Discrete-time dynamic graphs manifest as sequences of static graph snapshots captured at distinct time intervals. Contimuuss-tmme dynamic graphs, on the other hand, offer greater generality and can be expressed as timed sequences of events. These events may involve the addition or deletion of edges, addition or removal of nodes, and transformations of node or edge features. Traditional static graph representations fall short capturing the inherent temporal dynamics. TGL. seeks to address this limitation by extending the principles of graph-based models to time-varying structures, enabling a comprehensive understanding of how relationships unfold and transform over different time intervals. [Ca & Ribeiro (2022) develop framework to analyze TGL architectures, and categorize methods in the literature into two groups: "time-and-gree graph' and time-then-graph: Timo-and-graph based architectures learn the evolution of node representations by building a representation for each graph snapshot. DySAT (Sankar et al. 2020) and Evolve GCN (Pareja et al.) 2020) fall under this category. Time- Time-then-ae then-graph based architectures, on the other hand, construct a multi-graph using the memory of all past obscrvations and build a static graph to learn the node representations. Most well-known methods which fall under this category are TGN (Rossi eet al., 2020) and TGAT (da Xu et al. 2020). Our research aligns with the "time-then-graph approach. To that end, we provide brief review of the methods that we have found influential.
TGN is a message passing based encoder which learns graph node embeddings on a continuous-time dynamic multi-graph represented as a sequence of time-stamped events. Tt involves three main building blocks: (1) message function, (2) memory function and (3) embeddings module. At each event, a message function in the form of event-adopted MLPs, calculates aggregated messages to pass on partics involved (i.c., nodes). A memory function, such as an LSTM or GRU, updates the memory state of each node by the aggregated messages. An embedding module calculates the new state of node embeddings as function of memory states and events. TGAT incorporates the self-attention mechanism a foundational element and introduce time encoding approach which allows the network to treat node embeddings as temporal functions, thereby enabling it to predictively generate embeddings for both newly introduced and previously existing nodes as the graph undergoes changes over time. In their work, WWan et al. (2021) present CAWN, a method for the inductive representation of temporal works which utilize lemporal random walks. Temporal random walks enable to encapsulate network dynamics by implicitly extracting temporal network motifs, thereby circumventing the intensive process of motif identification and enumeration. The methodology incorporates distinctive anonymization technique wherein node identities obscured by their visitation frequencies within a series of sampled walks which preserve the inductive nature of the model. Yu et al. (2023) introduce a transformer-basee based framework designed for dynamic graph learning. The proposed architecture focuses on learning from the historical first-hop interactions between nodes. The learning process is facilitated by two key strategies: a neighbor co-occurrence encoding scheme that uncovers the relationships between the source and destination nodes through their historical interactions; and patching method that segments each interaction sequence into smaller patches. The Graph Mixer (Cong et al. (023)) takes a simpler view by constructing model that has fixed time encoding, alongside a node encoder and a link encoder; the encodings are used as inputs for a link classifier that is trained to predict the existence of a link. All state- of-the-art methods experiment on benchmark FLP and DNC datasets. Despite its simple infrastructure, Graph Mixer is able to achieve state-of-the-at performance on both FLP and DNC tasks.
In general, TGL models are developed to address FLP and DNC tasks. However, the performance is highly sensitive to conaarchitottuall hyperparaneters such as batch size and the negative sampling strategy. These hyperparameters are difficult to tune and often represent a trade-off between computational efficiency and
3

---

Published in Transactions on Machine Learning Research (06/2024)
accuracy, optimizing them can provide distorted view of expected performance in industrial settings. Additionally, to the best of our knowledge, there is no method that is capable of generating embeddings for the relatively common setting that event labels are obtained with delay, We address these issues by formulating RLC as a new benchmark task for TGL and developing Graph Profiler which enables the construction of dynamic node profiles by taking the feature and label information associated with previous interact ions into account. Graph Profiler is capable of maintaining a long-term view of an entity's profile that can capture long-term preferences.
3 Problem Statement
A graph, G, is composed of a set of vertices, V, and edges E, where cach edge, i ++ indicates an interaction between a pair of vertices i and j. In most cases, the graph is constructed with entities as the vertices and interactions between those entities as edges. In the case of social networks vertices might be users and their posts, and edges might involve intcractions cither between users, such as follows or blocks, or interactions between users and posts such as likes or comments. For a general RLC lask we are given set of source entities M and destination entities, Vie and Vast; and set of interactions between them & {(8;; di 44, Xi, y=))== such that the interaction from source entity 5; E Vare to destination entity d; Vast is realized at time t; and associated with a raw feature vector of X; E Ra where daee denotes the number of features. We consider that interactions are effectively instantaneous, or that the timestamp marks the completion of the interaction. We consider settings where there are m classes (or types) of interaction. Each interaction is thus associated with a ground-trut target class, represented as a binary vector y, = (yr.,....,, such that yi. = 1 if interaction belongs to the jhh class and Vi., otherwise. The aim is to learn a classifier that maps features, timing, source and destination entities of an interaction to class, given access to history of preceding interactions. Given new interaction from source E Vare to destination d E Vast with features x E Rd which is realized at time 4, let & {(*1, di, y:) E 44 < l) denote the preceding observations, and let & = (01....,9.. denote the predicted target class likelihoods by the classifier, i.e., f(x, (s. d. t), &ee) ÿ. Traditionally, the quality of cstimation is ovaluated by the cross entropy loss Lee (y. = -- -E yj (g)( during training. In case that G is not bipartite, i.e., Varc n Vast * 0, this formulation requires us to learn different representations for a vertex as the sender and the receiver of a message. In the basic problem formulation, we assume that vertices are not attributed, i.e, raw features are not observed for them. At the time of the event occurrence, we observe the identities of the source and destination, well as the raw edge features. None of these changes over time. The observation of the edge label is delayed.
We identify three other subfields of GRL that are potentially relevant to the problem setting formulated by RLC:
Dynamic Graph Anomaly Detection: Anomaly detection within dynamic graphs focuses on identi- fying trends that starkly contrast with the norm over time (Yu et al. 2018 Zheng et al. 2019 Behrouz & Seltzer 2022). Specifically, in scenarios where there are only two types of edge labels and a pronounced class imbalance exists, this approach to anomaly detection proves pertinent to RLC task. However, its applicability does not extend universally across all other instances.
Link Sign Prediction: Among static graph learning approaches, the research most pertinent to RLC predominantly focus on link sign prediction. Link sign prediction aims to determine the positive
negative nature of relationships in networks (Song & Meyer 2015 Aggarwal et al. 2016 Dang & Ignat,
2018 Chen et al. 2023). Methods in this domain often leverage structural balance theories and employ various machine learning techniques to predict the sign of unlabeled edges based on known relationships.
Dynamic Link Property Prediction: Huang et al. (2023) extends the definition of FLP to dynamic link property prediction by enabling to predict aspects of a link beyond mere existence. However, their work lacks an experimental framework that explores edge properties other than existence.
4 Graph Profiler
In this section, we introduce Graph Profiler, a simple architecture that is designed to learn entity profiles, time-aggregated representations, by processing previously observed interactions. Subsequently it uses the
4

---

Published in Transactions on Machine Learning Research (06/2024)
Overall Flow
z
Profile Encoder
Containers
Profile Encoder
SEC
Edges
Edges
Events
Readout
Neighbour Loader
Update
dst
//
Destination Embeddings
History Summary
Events
9
Z = | JEV
dst
Message Encoder
msg
Message Passing
Mini-batoh batch
msg
Loss
Time Encoder
3 = (V, 8,
e
src
Overview of Graph Builder. Left: Overall flow. Given mini-batch of interactions, where each interaction is from source node src to a destination node dst, observed at time associated with features msg and belongs to class y, Graph Profiler follows these steps: (1) The profile encoder calculates source node embeddings based on the history of src (i.e., previous edges and events that are associated with src). (2) The identity of dst is used to retrieve the embedding of the destination node from the table of destination embeddings. (3) The message encoder performs a linear transformation on the edge features. (4) The time encoder projects timestamp t into time embedding. (5) The readout layer combines source z destination ddst message mag and time z embeddings to predict the class of the interaction. Afterwards, the containers are updated with the information in the mini-batch. Right: Profile builder. Given the current set of edges, the neighbourhood of src is loaded. The past events used to summarize the history of all nodes that belong to the neighbourhood of src. The node embeddings on the ego graph of src are updated using message passing to obtain the final source node embedding zare
Figure 2:
constructed entity profiles, together with observed interaction features, to make classification decisions about recent interactions. Graph Profiler is composed of five main learnable modules; profile encoder fprofle (-), message encoder Smsg (.), destination encoder Jast (-), time encoder Stime (-), and readout. Sete (.), and two containers at time t; previous events H, and meta-path edges Mr. For a given set interactions (odges), & {(s,, d, &, x, y.), the event and meta-path containers at time are defined as follows:
H₂ = fie (1. (1l...,My M} :4 <t
M₁ = (u,,)): :i.j E H4: (s; = A =) [( dd)
(2)
The event container simply stores the indices of the interactions observed up to time t. The meta-path containcr stores the pairs of source nodes that have interacted with the same destination node at least once prior to time (See Figure 3). Graph Profiler proceeds in the following way to perform training. We denote by Ebatch the mini batch of interactions that being processed. Let teurrent = minjes t, denote the first interaction time the batch, and dmocel denote the dimensionality of the emboddings constructed by the model. Given an interaction (ss, di, t, xx) E Ebatch, the computation proceeds through each module follows:
5

---

Published in Transactions on Machine Learning Research (06/2024)
1. The profile encoder calculates the source node profile z E Rmoddl based observations until feurrent: i.e,, Spoofile (8i, Hi Ho z.
2. The message encoder encodes the interaction: 2, z,na E Rdmodel i.e., Smag (xi) - 2,nER
3. The destination encoder generates destination embeddingg: z!aa E Rdmodde, i.e, fast (di) 2,,
4. The time encoder converts the interaction timestamp into time embedding vector z E RRawdee i.e.
ftime (44) = 2.
5. The readout layer predicts the interaction class 91, Le.,, Sete (2iTe 2,
mag ddst 2 8..
Once the predictions are made on mini-batch Ebatch, the containers are updated so that they include the interactions in the mini-batch. The meta-paths are recalculated according to Equation 2 We note that in the procedure outlined above, it is only the construction of the source node profile that uses information from the historical interactions in the calculations. The destination node embedding is computed using only the identity of the node, and the message encoding is derived via a network operating on the message features. The destination embeddings are learnable, as are the weights the message encoder network, so the historical interactions influence these embeddings via the training procedure. The overall flow is illustrated in Figure Next we explain how individual modules are trained and describe the procodure used to update containers.
Profile Encoder Inspired by our previous experiences work- ing on webscale recommendation systems, we derive graphs that allow us to capture source-source correlations that might be obscured through traditional message passing schemes due to an over-smoothing effect. Similar to previous work, we de- fine a meta-path as an edge that is constructed from path through the graph (Chen & Lei 2022 Huai et al., 2023 Huang et al. 2022). In our specific instance, consider second-order meelappaths that connect vertex which acts as source to another which acts as source through a shared destination vertex. The set of meta-paths is time dependent because the edges are parameterized by time. Given the set meta-path edges Micureal observed up until the current time the profile encoder first builds the ego graph Grauren (s:) over the set of ver- tices V Venrrone (s:) s:Ufu: (u, 8i) E Mimuroee } with set of edges Misurruu (s:) (u, (u, MMeuurraui Vu, Vicuieaa (si)). Thus, the ego graph Gtarrame (s;) = (Vterremc (Ms),MMenn
Bipartite graph
Mcta-paih graph
sowre
destination
Figure 3: Mela-path construction: On the left, a bipartite graph is provided that con- sists of edges from set of source entities to a set of destination entities. On the right, the final meta-path graph composed of meta-path edges is shown.
(s:))
has a vertex set that consists of s; and all vertices that are con- nected via a meta-path to 8,, and an edge set consisting of all meta- paths connecting these vertices. For each node Vi Vieurem (8:), we define the set of relevant event indices as Hi (u) (j : 8 u, vj E H.
The node embeddings are initialized by aggregating the embeddings of previous events associated with the corresponding node, i.e. (u) = faggregate (iaggegateeHHuaarre (Arrrrrnn (u)). For example, using mean aggregation schema with single layer of linear transformation, the node embeddings (0) E TR dodnddaal are initialized as follows:
h (n) EiEH (u) Haauream (w)
msg dst |
WT event
(3)
where FII-1 denotes the concatenation operator, and W Vevnnt Rehmodei are learnable weights for di 3 x dmodel m.. Recall that nn adet 2 E dRddmdall and y; E Ihm Then, using the GCN (Kipf & Welling 2016 framework, at the ttth layer the node embeddings are updated by passing messages between neighbour nodes.
ie., (8) (u) Sgen(he (h((t))()) (u),M,.... Me Introducing normalization coefficients Gu,e =
/deg(u)-V/dg(or with deg(.) denoting node degree Geanees write the node embedding update at layer h for

---

Published in Transactions on Machine Learning Research (06/2024)
normalized sum aggregation schema
h (19) (u)= =
E
Gua (n(h-1) (n)-))
(4)
(1.v)E (86)
where W E Rdmodal Xtmonde are learnable weights, and h+() (u) E Rdmoods, The profile embedding of source node s set to the final layer node embedding, ie., 2: h (s:) where K denotes the total number of message passing layers.
Time Encoder For the time encoder, we employ either the fixed time encoding function proposed by Cong et al. (2023) or the learnable time projection introduced by Kumar et al. (011)). Given a weight vector
W E R gdddee in general the time encoding function follows:
feime (t) cos(wt) =z',
where z' E Rmudde denotes the vector represcntation of timestamp t. The two variants of time encoding we investigate in this work differ in the calculation of the weight vector. In the learnable version, w E Rd is simply learned during training, so the time encoding layer is a linear projection, without any bias, followed by cosine scaling. In the case fixed time encoding as proposed by Cong et al. (2023), each dimension of the weight vector is set to exponentially decreasing values. The hyperparameters a and / are selected according to the scale of the minimum and maximum timestamps in the data. In practice a = B = /dmodel is found to perform well, and we follow this setting in our experiments.
(6-1), where a and 3 are scalars, so that wt is vector with monotonically
Other Modules The message encoder uses single linear layer to compute Siag (x:) x; WT mag bmsg where W mag E iddmadet x xduss are learnable weights and bmsg E Rddmddel is a learnable bias. For destination encoding, we use an embodding look-up table of size equal to the number of destination nodes. This is initialized randomly, fast (d.)
1d, WT dst: where 1d, E R |aalll denotes a one-hot vector representation of node d, and W dst Thoddl x [Van are learnable weights. The predictions at the readout layer are computed as 8; = Th + 2.st zm + 2 WT where W Agdmodaxdmood are learnable weights.
Unlike static graph learning methods, Graph Profiler is capable of encoding temporal properties of network. Graph Profiler has two main advantages over existing TGL methods: (1) enables the construction of dynamic entity profiles that take into account feature and label information of previous interaction in the neighbourhood; and (2) it is capable of maintaining long-tew teg-term view of an entity's profile that can capture long-term preferences. In addition, the modular structure of Graph Profiler is flexible, so the model can casily be adapted to suit the contextual properties of individual datascts.
5 Experiments
In order to understand RLC as novel task within TGL, we begin by evaluating a two layer Multi-layer Perceptron (MLP), TGN (Rossi et al. 2020), TGAT (da Xu et al. 2020), CAWN (Wang et al. 2021| and Graph Mixer (Cong et al. 2023) on RLC by making the appropriate modifications to the algorithms. We have chosen these mothods because they are state-of-tho-art TGL basclines developed for the FLP task. Based on our observations concerning the performance of the methods, we outline a set design principles that comprise the design space for RLC, With these design principles in mind, we present Graph Profiler and benchmark on six different datasets. For each dataset, we locate the optimal portion of our design space and discuss the correspondence between that and the underlying dynamics of the dataset under investigation.
Datasets We evaluated our methods on four benchmark datascts that have previously been used by the TGL community YELPCHI (Dou et. al.) 2020), WIKIPEDIA, Mooc, and REDDIT (Kumar et al. 2019). These datasets are usually employed as benchmarks for future link prediction; we adopt suitable measures to convert them to the RLC setting. In YELPCHI, the source entities are platform users and the destination entities include hotels and restaurants. An interaction happens when a user reviews one of the hotels or restaurants. The reviews are labeled either as filtered (spam) or recommended (legitimate). For WIKIPEDIA,
7

---

Published in Transactions on Machine Learning Research (06/2024)
Table 1: Dataset statistics. /vrl and [Vatt denote the number of source entities and destination entities, respectively; ameg denotes the number of interaction features; |&] denotes the number of interactions; p denotes the fraction of edge labels that are positive; He denotes the average edge homophily; H and H. denote the edge homophily for positive and negative classes, respectively, and H. denotes the balanced edge homophily. Edge homophily metrics are calculated over all interactions (edges) in the dataset.
Pare
Dast
[8|
dmss
P
He
N2
He
EPIC GAMES
542
614
17584
400
0.6601
0.8330
0.9038
0.6955
0.7663
YELPCHI
38,063
201
67,395
25
0.1323
0.7781
0.1589
0.8725
0.2533
WIKIPEDIA
8,227
1,000
157,474
172
0,0014
0.9975
0.0130
0.9988
0.0144
Mooc
7,047
97
411,749
4
0.0099
0.9809
0.0212
0.9904
0,0308
REDDIT
10,000
984
672,447
172
0.0005
0.9989
0.0025
0.9995
0.0030
OPEN SEA
57,230
1,000
282,743
35
0.4601
0.5865
0.5505
0.6171
0.5812
the set of entities is composed of users and pages, and an interaction happens when a user edits page. For REDDIT, entitics are users and subreddits, and an interaction represents a post written by a user on a subreddit. Some page edits on Wikipedia and posts on Reddit. may be controversial causing the user to be banned. Thus, on both datasets we base the target class of interaction on whether it is controversial or not, i.e., whether the user was banned as result of posting it. The interaction features for these three datasets are extracted based on the textual content of actions performed by students on MOOC online course. The source entities are the students and the destination entities are the course contents that the students interact with, e.g,, recordings or lecture notes. The interaction features are the types of activities the student performed during an interaction, e.g., viewing the video or submitting an answer on the forum. Sometimes, the students drop out of the course after an activity. We use this as a label to idcntify the target class of an interaction. Thus, all four datascts are binary recent link classification datasets for which the class imbalance is salient.
The Mooc dataset consists of
In addition to adapting benchmark TGL datasets to RLC task, process two tabular datasets that are not conventionally investigated in TGL sctting: EPIC GAMES D and OPEN SEA (La Cava et al. 2023a by Costa et al. 2023) 2 The Epic Games Store is a digital video game storefront, operated by Epic Games. The dataset includes the critiques written by different authors about the games released on the platform. The source and destination nodes represent authors and games, respectively, and the critiques form the set of interactions. We construct the interaction features by vectorizing the textual content of critiques using TF-IDF. We include the overall rating the author provided of the interaction is determined based on whether it was selected as top critique or not. Once a critique is released all the information ion regarding the author, game and features of the critique is available. Its selection as a top-critique is a delayed observation, so the data naturally forms an RLC task. Open Sea is one of the loading trading platforms in the Web3 ecosystem. The datasct is a collection of Non-Fungible Token (NFT) transactions. Sourced from Open Sea, it is provided as a natural language processing dataset and is mainly used for multimodal learning classification tasks. To the best of our knowledge the dataset has not been previously investigated in a TGL framework. In order to make it amenable graph learning, we identify disjoint sets of sellers and buyers of unique NFTs (these are identified by collection memberships and token IDs) to serve as source and destination nodes. The transaction features are binary representation of categorical variable fields associated with the transaction, the cryptocurrency exchange rates at the time of the interaction and the associated monetary values. The label of a transaction is determined based on a future transaction of the unique NFT. It is tagged as 'profitable' if the revenue obtained through the final sale is higher than the price paid at the purchase. The labels are thus delayed, because whether it will be a profitable investment is not known at the time of purchase. The data thus aligns with the RLC setting, In Appendix B further details on pre-processing of OPEN SEA dataset is shared. The datasets statistics are provided in Table [ In our experiments, data is divided into training (70%), validation (10%) and testing (20%) sets chronologically.
an additional interaction feature. The label

---

Published in Transactions on Machine Learning Research (06/2024)
Edge Homophily We introduce measure of edge homophily to understand the importance of graph information to our edge classification task. Denote by (a) an edge- wise neighbourhood operator that constructs a set of all edges that are connected to a given edge, a = (i.j), where i and is the source and j the destination. This operator forms the union of two scts, i.c., (a((a) = [()) where I(t) is the set of incoming edges connected to the source and O(j) is the set of outgoing edges connected to the destination. Our edge homophily measure is then defined as:
Hg) =
1 [l
Σ DEE BEAL
Σ
[(a)=)(()) |
(6)
where N is the edge-wise neighbourhood operator and is the operator that returns the label of the edge. The edge-wise neighbourhood operator constructs a set of all edges that are connected to a given edge, a (i,j), where i and j are the source and destination respectively, by constructing the union of two sets (a) I(i) U O(), where I(-) and O(-) construct the set of incoming and outgoing edges respectively. For the simplicity of notation, we have neglected the time dimension but this definition is easy to gener- alize to temporal graphs through the neighbourhood operators. Edge-homophily mcasures the fraction of neighbouring edges that have the same class, in analogy to the way node-homophily measures the fraction of neighbouring nodes with the same class. Node-homophily is an important dataset property that can be highly indicative of the value that can be derived by encoding graph structure in node classification tasks (Pe et al., (220),, particularly for embedding procedures that rely on smoothing over a neighbour- hood. The edge homophily definition in Equation P treats different classes equally, which can be misleading for imbalanced datasets. Therefore, using P to denote the fraction of positive edge labels, we also define balanced edge homophily metric for binary classification as HG) = (1 (1- p)H, (9) + pHz (g), where (9) T Enee+ EBEAT 11 and H. (9) E EBENT") 11m)tt)) Table 1 presents the [V5")
edge homophily values for the datasets we study. In Appendix C we illustrate the dynamics of edge ho- mophily over time. In the case of WIKIPEDIA, Mooc, and REDDIT, the overall edge homophily reaches high as 99%, whereas the homophily among positive class edges falls below 5%, phenomenon attributed to significant class imbalance. Across these three platforms, there is a noticeable trend of decreasing positive edge homophily over time, a pattern that emerges as banned users or students who withdraw from courses exit the network. Conversely, on YELPCHI, there's an observable increase in positive class homophily and a decrease in negative class homophily. This trend could stem from spam attacks on businesses, typically carried out by similar groups of reviewers whose numbers grow over time. A similar pattern is detected on Open Sea, suggesting increasing ratio of sellers who profit over time. Nonetheless, those who profited in the system's 's early stages remain aclive participants.
Performance Evaluation Two of the most common metrics used for performance evaluation for both FLP and DNC are area under the receiver operating characteristic curve (AUC) and average precision score (APS). These metrics exhibit some undesirable behaviour. AUC is known to saturate, such that it become impossible to differentiate between candidate algorithms. Both metrics can provide skewed mcasures of quality when applied to inbalanced datasets (Chicco & Jurman 2020 2023 As a result, we turn to the Gorodkin 2004 which is defined as:
Matthews Corvelation Coefficient (MCC) (uule 1912
C8 - E.G
MCC =
3 p. - E.E
where i'i a vector, with each element being the number of times a class occurred, 5 is a vector of the number of times each class is predicted, c is the number of samples correctly predicted, and is the total number of samples. This correlation coefficient has maximum value of 1, and the minimum value ranges between -1 and 0, depending on the distribution of the underlying data. A score of 0 indicates that the predictor is perfectly random; a score of indicates that the predictor is perfectly accurale; and a score -1 -1 indicates that the predictor is perfectly inaccurate. As an illustrative example, we present use case A1 from Table 4 in Chicco& & Jurman (2020). In this example, we have 100 total data points with 91 in the positive class and in the negative. For a hypothetical classifier that that predicts all but one data point as a member of the positive class; we find TP = 90, FN TN = 0, FP 9. This yields respectable APS of 0.90 but near
9

---

Published in Transactions on Machine Learning Research (06/2024)
Table 2: Performance of TGN variants. The performance of TGN variants is presented as the average of 10 random seed runs for each configuration. The best time encoding and aggregator versions are highlighted in bold and the cells representing readout configurations are shaded based on their values, with darker shading indicating higher performance.
YELPCHI
WIKIPEDIA
Mooc
REDDIT
MCC
APS
AUC
MCC
APS
AUC
MCC
APS
AUC
MCC
APS
AUC
Time Encoding
lxx
0.2621
0,3118
0,7590
0.29413
0.1237
0.9086
0,1004
0.0486
0.7631
0.0012
0,0019
0.6608
learn
0.2866
0.3278
0.7723
0.1933
0.0989
0.8728
0.0973
0.0571
0.7730
0.0444
0.0093
0.6508
Aggregator
esp
0.2803
0,3262
0.7700
0.1018
0,0712
0.8653
0.0630
0.0415
0.7494
0.0158
0.0036
0.6608
last
0,2866
0.3278
0.7723
0.2943
0.1237
0.9086
0.0477
0.0325
0.7045
0.0444
0.0055
0.6599
mean
0.2741
0,3217
0,7666
0,2034
0,0896
0.8955
0,1001
0.0571
0,7730
0.01
0.0093
0.6235
Readou:
SrU
0,2286
0,2391
0.7096
0.1237
0.0828
0.8368
0.0530
0.0416
0.7199
0.0105
0.0045
0.64
dst
0.2288
0,2311
0.7015
0.0972
0.0161
0.7298
0,0132
0.0377
0.7195
0.0099
0
0.6188
sro-dst
0,2319
0.2411
0.7094
0,1018
0.0355
0.8908
0.0924
0.0462
0.7485
0.0142
0.0031
6,00
sto-t
0.2311
0.2426
0.7147
0,1308
0.0844
0.8401
0.0507
0.0386
0.7191
0.0104
0.0021
0.6472
dst-t:
0.2277
0,2381
0.7063
0.1057
0.0469
0.7379
0,0338
0.9411
0.7205
0.0159
0.0124
0.6211
sre-dst-t
0,2290
0,2542
0.7151
0.1442
0.0346
0:9086
0.0903
0.0G0G
0.7729
0.041
0.0092
6308
sre-mag
0.2733
0.3106
0.7627
0,1530
0,0835
0. 0,8808
0.0996
0.0603
0.7763
0.0074
0.0021
0
dst-mag
0.2744
0.3200
0.7641
0.1184
0.0564
0. 0.8332
0.0475
0.0289
0.7112
0.0171
0.014
0.5087
sre-dst- misg
0,2644
0.3147
0.7629
0:2040
0.0714
0.8537
0.0894
0.0499
0.7708
0.0158
0.0033
sre-mag-1
0 126666
0.3278
0.7723
0,1764
0,0858
0. 1010
0.0571
0.7727
0,0000
0,0014
0.6005
dst-mag-t
0. 0.2803
0.3230
0.7009
0.1300
0.0617
0.7489
0.0462
0.0325
0.704b
0:0113
0.0093
0.5971
aro-d-s-msg--
0.2734
0.3217
0.7666
0.2945
0.12
0.9020
0.7780
0.0077
0.0049
0.6089
random MCC of -0.03. While simple, this is just one example where metrics like APS can mask underlying poor performance for imbalanced datasets. Chicco & Jurman (2023) further presents similar failure modes for AUC. Because of this, we choose MCC as our figure of merit for the RLC tasks that we present.
5.1 Key factors to tailor model to specific needs of data
With the introduction of RLC as a benchmark task to evaluate TGL. methods, we explore the performance of a well-known state-of-the art model, TGN (Rossi et al.) 2020), together with variants. We create these variants by constructing different message aggregation schema, readout layers, and time encoding strategies to better discover the potential of the overall architecture. More specifically, we experiment with (1) fixed and learnable time encoding proposed by (Cong et al. 2023 and by (Kumar et al., 2019), respectively; (2) mean, last, and exponential decay message aggregators; and (3) six different configurations of the input to the readout layer based on different combinations of source, destination, time and message embeddings calculated for the most recent event. As described by (Rossi et al. 2020), the mean aggregator calculates the state of a node by averaging the interactions held in the memory, whereas the last aggregator uses only the most recent interaction. The exponential decay variant. calculates a weighted average over the interactions in the memory by setting weights such that they decrease exponentially with respect to the time that has clapsed. Table 2 summarizes the results. We present violin plots for cach variant in Appendix E
The readout variations appeared to play a significant role in the model's performance, can be seen in Figure 4 These results on the WIKIPEDIA dataset are demonstrated for different model dimensions, i.e. d {100, 200} and a varying number of neighbours. We observe that incorporating the edge features as a residual at the final layer of the update helps to improve the performance in terms of MCC, which makes intuitive sense given that the message features for this dataset correspond to the edit's content. Interestingly, while we observe this trend when looking at the MCC curves, which exhibit a dramatic stratification in performance, the AUC curves show the opposite trend. We attribute this to AUC being unsatisfactory metric for the evaluation of RLC tasks, particularly if there is class imbalance. The APS and Loss curves exhibit trends that are similar to those of MCC (See Appendix D. We conclude that for abusc-related datasets the most recent interaction mailers most, and therefore aggregation based on the last event is more useful. In the case of predicting course completion, averaging over multiple previous actions is valuable, which captured by the outperformance of the mean aggregator. In general, we observe that involving recent interaction features in the readout layer is very useful. The configurations with msg included perform significantly better.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Table 3: Model comparison results. The performance of models presented as the average of random seed runs. The best and second best results are highlighted in red and blue, respectively.
EPIC GAMES
YELPCHI
WIKIPEDIA
Mooc
REDDIT
OPEN SEA
MLP
0.1554
0.2763
0.2354
0.0673
0.0021
0.1106
TGN
0.8373
0.2372
0.1442
0.0991
0.0174
0.1071
TGAT
0.5546
0,1890
0.0000
0.0000
0.0000
0.2245
CAWN
0.4925
0,2300
0.0000
0.0103
0.0000
0.3223
MCC
Graph Mixer
0.2316
0.2830
0.1442
0.1174
0.0000
0.2647
Modified TGN
0.8713
0.2866
0.2943
0.1004
0.0444
0.2647
Graph Profiler
0.9355
0.3274
0.2498
0.1739
0.0115
0.2959
MLP
0.6976
0.3254
0.0759
0.0169
0.0012
0.7790
TGN
0.9850
0.2461
0.0320
0.0526
0.0018
0.7912
TGAT
0.8844
0.1789
0.0157
0.0190
0.0014
0.6582
CAWN
0.8441
0,2168
0.0269
0.0367
0.0014
0.7247
APS
Graph Mixer
0.7522
0.3252
0.0550
0.0711
0.0021
0,8304
Modified TGN
0.9892
0.3249
0.1148
0.0430
0.0055
0.8225
Graph Profiler
0.9988
0.4059
0.0955
0.0896
0.0092
0.8459
MLP
0.61117
0.7694
0.7731
0.6447
0.5486
0.5776
TGN
0.9734
0.7135
0.7135
0.7672
0.5671
0,5908
TGAT
0.8470
0.6314
0.8908
0.6383
0.5336
0.6871
CAWN
0.7931
0.6850
0.8880
0.7107
0.6100
0.6991
AUC
Graph Mixer
0.7132
0.7650
0.7500
0.7515
0,6413
0,6789
Modified TGN
0.9807
0.7723
0.7723
0.7439
0.6508
0.6544
Graph Profiler
0.9974
0.8058
0.7821
0.7886
0.6280
0.6740
5.2 Model Comparison
Using the results from the TGN modifications, we have identified multiple core design principles associated with improved performance on the RLC task, and we have incorporated these principles into Graph Profiler. Specifically, we observed that a learnable timo-encoding provided improved results for three of four data sets. The src-dat-mag-t readout variant provided strong results across all four data sets. Because of these results, we designed Graph Profiler with learnable time encoder and src-dst-msgs readout function. To validate these results, we perform benchmark experiments of baselines and Graph Profiler on RLC. For
Model Dimension 100 Model Dimension 200
Model Dimenslon 100 Model Dimension 200
0.90
0.20
0.85
0.15
AUC
0.80
MCC
0.75
0.05
0.70
0.00
10 15 20 25
2
5
10
15
20
25
20 25 Number of Neighhors
2
5
10
15
10 15 20 25
Number of Neighbors
Number of Neighbors
Number of Neighbors
sre-dst
dst
sre-dst-msg
sre-msg dst-msg
Figure Readout variations on Wikipodia. The blue glyphs correspond combinations of the vertex features, while the red glyphs correspond to combinations of the vertex and message features. The star, circle, and triangle glyphs correspond to the sre-dst, src, and dst embeddings respectively.
11

---

Published in Transactions on Machine Learning Research (06/2024)
FLP
FL.P
RLC
0.85
0.85-
0.80
0.80-
0.20
0.75
0.75-
0.15
0.70
MCC
0.70
MCC
0.65
0.55
0.60-
0.05
0.55
0.55-
0.50
0.50-
045
100
25.0
50.
1000
100
1000
2090
4000
sona
7000
1us00
1080
2090
3000
4000
50n0
7000
10000
Negative Sampling Ratio
Batch Size
Batch Size
Figure 5: TGN performance on FLP vs RLC on Wikipedia with varying levels of negative sampling ratio and batch size.
the existing benchmark datasets, Graph Profiler obtains the best results for YELPCHI and Mooc, but our modified version of the TGN architecture outperforms for WIKIEDIIA and REDDIT. The most probable reason is that the ratio of the number of source nodes to the number of destination nodes is much higher for YELPCHI and MooC compared to WIKIPEDIA and REDDIT. Graph Profiler constructs embeddings for the source nodes via calculations over neighbouring interactions, but uses Iearnable embeddings in a lookup table for the destination nodes. In the majority of bipartite graphs examined, there is a significantly larger number of source nodes compared to destination nodes. This discrepancy allows for the practicality of learning a unique embedding vector for each destination node.
Another observation we draw from these model comparison results is the usefulness of MCC in highlighting the relative capabilities of different models. For example, Mooc Graph Profiler improves MCC by 73% compared to the Modified TGN, while the observed change for the AUC is only 3% (See Table 3). On the EPIC GAMES, which is less imbalanced, Graph Profiler outperforms her baselines based on MCC and APS, but the performance of the techniques cannot be reliably distinguished using AUC. On OPEN SEA the Graph Profiler and CAWN per par in terms of MCC and APS. Revisiting the dataset statistics provided in Table [ we conclude that encoding node profiles based on their shared history, i.e., employing the Graph Profiler architecture, is more effective on datasets with higher balanced edge homophily and less class imbalance. For such datasets, tracking a graph-wide memory, i.e., employing the TGN framework, is less effective.
5.3
On the importance of hyperparameter sensitivity differences between FLP and RLC
In our reproduction study we explored the effect of negative sampling ratio, batch size, and the number of sampled neighbors on the performance of our TGN baseline for the FLP task. We term these as non- architectural parameters because they influence the training but do not influence the architecture of the model itself. We explore these parameters because they impact the trade-off between model accuracy versus computational performance and utilization. In the example of batch size, this is typically tuned to be as large as possible to maximize GPU utilization, but we see in Figure 5 a steady decline in MCC as the batch size is increased (See Appendix D for other metries). Indeed, we observe variation in model performance due to changes in batch size that are larger than the variations across different model architectures. Intuitively, the decay makes sense, because the gradient updates become less frequent, but this points to a relatively significant but under-discussed trade-off that has major ramifications for production use-cases. In the case of the negative sampling ratio, we observe slight decline in MCC and decrease in the consistency between individual training runs as the number of negative samples increases. Thus, it can be concluded that RLC does not have the same dependence on batch-size and does not require the generation of negative samples. These observations lead us to the conclusion that (1) the assumptions made during the development of models for FLP may not hold for RLC, and direct translation of existing TGL methods, which are generally benchmarked on FLP tasks, perform RLC in industrial scttings not advisable, and (2) RLC is an interesting general purpose benchmark task for the TGL community, and should be treated differently from the current common tasks.
12

---

Published in Transactions on Machine Learning Research (06/2024)
5.4 Ablation on Learnable Destination Embeddings
In order lo investigate the impact of learnable des- tination embeddings, we experiment with and with- out learning set of embeddings for the destination nodes. The results are presented Table ! For most of the datasets (with the exception of RED- DIT), incorporating learnable destination node rep- resentations improves the predictive performance of Graph Profiler. The destination node embeddings are then used for building the source node profiles. It can be inferred that destination encoding enriches the source profile embeddings by lemporally smoothing the interactions to build sense of history. By contrast, using the destination in the readout itself means that the history of destination nodes is ignored. We believe that the difference for the REDDIT dataset may arise from the fact that the labels (whether a sub-reddit is controversial) is less dependent on the main post, compared other Wikipedia. Said another way, there are many Wikipedia pages that are prone to abuse for political reasons, and thus, the page profile (destination node) matters. By contrast, abuse on Reddit is less dependent on the sub-reddit (destination node) than the author (source node).
Table 4: Impact of learnable destination embeddings
MCC
APS
AUC:
with
without
wiih
withonn
wilth
withon!
EPiC GAMES
0.9355
0.7695
0.9988
0.9575
0.9974
0.9054
YELPCHI WIKIPEDIA
0.3274
0.3071
0.4059
0.3892
0.8058
0.8000
0.2498
0.1324
0.0955
0.7821
0.6946
Mooe
0.1739
0.0000
0.0896
0.0011
0.7886
0.5906
REDDIT
0.0115
0.0701
0.0002
0.0203
0.6280
0.6833
5.5 Ablation Study on Graph Profiler Components
To assess the signifieance of different components of Graph Profiler, we conducted an ablation study. This involved systematically removing specific module from the model and comparing the outcomes with those of the intact version. For more in-depth comparative analysis, utilized datasets from YELPCHI and EPIC GAMES. These were chosen for their comprehensive content and manageable sizes, facilitating the intensive computations required for thorough statistical evaluation. The findings are presented in Table 5 indicating that while each component plays a critical role, their significance varies across contexts. For instance, the profile encoder proves to be exceptionally valuable for the EPIC GAMES datasct, whereas encoding features of recent interactions has more pronouneed effect on the YELPCHI dataset. This suggests that, in the context of the EPIC GAMES platform, the profile information of reviewers holds greater predictive power for identifying top critiques than the content of their reviews. Conversely, for YELPCII, the content of reviews is more crucial in detecting spam.
Table 5: Impact of individual components. Each row reports the results with the configuration without the indicated module. Results are averaged over 10 different random seeds.
EPIC GAMES
YELPCHI
EPIC GAMES
YELPCHI
EPIC GAMES
YELPCHI
MCC
APS
AUC
Graph Profiler
0.9355
0.3274
0.9988
0.4059
0.9974
0.8058
w.o.
Profile Encoder
0.0000
0.2849
0.6097
0.3316
0.5000
0.7725
w.o.
Learning Destination Embeddings
0.7695
0.3071
0.9575
0.3892
0.9054
0.8000
w.o. Recent Event Message Encoder
0.6311
0.1926
0.8940
0.2215
0.8593
0.6568
Recent Event Time Encoder
0.8082
0.3072
0.9766
0.3970
0.9590
0.7986
6
Conclusion
In this work, we introduce Recent Link Classification (RLC) temporal graphs as benchmark downstream task and evaluate the most competitive state-of-the-art method's performance using a statistically meaningful metric, namely Matthews Correlation Coefficient (MCC), which is more robust to imbalanced datascts in comparison to the commonly used metrics. We propose several design principles, which involve the choice of message aggregation schema, readout layer and time encoding strategy, for tailoring models to specific requirements of the task and the dataset. We show that appropriate selection can lead to significant improvement on benchmark datasets. We present, Graph Profiler, an RLC algorithm designed for bipartite graphs which are commonly encountered in industrial settings. We believe the introduction of RLC
13

---

Published in Transactions on Machine Learning Research (06/2024)
benchmark task for temporal graph learning is useful for the evaluation of prospective methods within the field.
14

---

Published in Transactions on Machine Learning Research (06/2024)
References
Charu Aggarwal, Gewen He, and Peixiang Zhao. Edge classification in networks. In Proc. IEEF Int. Conf. Data Engineering (ICDE), 2016.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In Proc. ACM SIGKDD Int. Conf. Knowledge Dis- covery and Dala Mining, pp. 2623 2631, 2019.
Thibaud Arnoux, Lionel Tabourier, and Matthieu Latapy. Combining structural and dynamic information to predict activity in link streams. In Proc. IEEE/ACM Int. Conf. Advances in Social Networks Analysis and Mining, pp. 935- 935-942, 2017.
Ali Behrouz and Margo Seltzer. Anomaly detection in multiplex dynamic networks: from blockchain security to brain disease prediction. In Proc. Int. Conf. Neural Information Processing Systems (NeurIPS) Temporal Graph Learning Workshop, 2022.
Pictro Bongini, Monica Bianchini, and Franco Scarselli. Molecular generative graph neural networks for drug discovery. Newrocomputing, 450:242 252, 2021.
Ling Cai, Bo Yan, Gengehen Mai, Krzysztof Janowicz, and Rui Zhu. Transgen: Coupling transformation assumptions with graph convolutional networks for link prediction. In Proc. Int. Conf. on Knowledge Caplure, pp. 131-138, 2019.
Benjamin Paul Chamberlain, Sergey Shirobokov, Emanuele Rossi, Fabrizio Frasca, Thomas Markovich, Nils Hammerla, Michael M Bronstein, and Max Hansmire. Graph neural networks for link prediction with subgraph sketching. arXiu preprint arXiv:2209. 15486, 2022.
Bo Chen, Di Zhu, Yuwei Wang, and Peng Zhang. An approach to combine the power of deep reinforcement learning with graph neural network for routing optimization. Electronics, 11(3):368, 2022.
Hongxu Chen, Hongzhi Yin, Xiangguo Sun, Tong Chen, Bogdan Gabrys, and Katarzyna Musial. Multi-level graph convolutional networks for cross-platform anchor link prediction. In Proc. ACM SIGKDD Int. Conf. Knowledge Discovery and Dala Mining, pp. 1503 1511, 2020.
Jushuo Chen, Feifei Dai, Xiaoyan Gu, Haihui Fan, Jiang Zhou, Bo Li, and Weiping Wang, Learning pair- centric representation for link sign prediction with subgraph. In Proc. ACM Int. Conf. Information and Knowledge Management, 2023.
Yali Chen and Xiujuan Lei. Metapath aggregated graph neural network and tripartite heterogeneous networks for microbe-disease prediction. Frontiers in Microbiology. 13:919380, 2022.
Davide Chicco and Giuseppe Jurman. The advantages of the matthews correlation coefficient (mec) over fl score and accuracy in binary classification evaluation. BMC genomics, 21(1):1-13, 2020.
Davide Chicco and Giuseppe Jurman. The matthews correlation coefficient (mcc) should replace the roc auc the standard metric for assessing binary classification. BioData Mining, 16(1):1 23, 2023.
Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, and Mehrdad Mahdavi. Do really need complicated model architectures for temporal networks? In Proc. Int. Conf. Learning Representations (ICLR), 2023.
Davide Costa, Lucio La Cava, and Andrea Tagarelli. Show me your nft and i tell you how it will per- form: Multimodal representation learning for nft selling price prediction. In Proc. ACM Web Conf., pp. 1875 1885, 2023.
da Xu, chuanwei ruan, evren korpeoglu, sushant kumar, and kannan achan. Inductive representation learning temporal graphs. In Proc. Int. Conf. Learning Representations (ICLR), 2020.
Quang- Vinh Dang and Claudia-Lavinie Ignat. Link-sign prediction in dynamic signed directed networks. In Proc. IEF. Ind. Conf. Collaboration and Internet Compuling (CIC), 2018.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Yingtong Dou, Zhiwei Liu, Li Sun, Yutong Deng, Hao Peng, and Philip S Yu. Enhancing graph neural network-based fraud detectors against camouflaged fraudsters. In Proc. ACM Inl. Conf. Information and Knowledge Management, 2020.
Ahmed El-Kishky, Thomas Markovich, Serim Park, Chetan Verma, Baekjin Kim, Ramy Eskander, Yury Malkov, Frank Portman, Sofía Samanicgo, Ying Xiao, et al. Twhin: Embedding the twitter heterogencous information network for personalized recommendation. In Proc. ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, pp. 2842- -2500, 2022.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprinl Xiv: 02428, 2019.
Fabrizio Frasca, Emanuele Rossi, Davide Eynard, Ben Chamberlain, Michael Bronstein, and Federico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXi::2004.11196 2020.
Chen Gao, Yu Zheng, Nian Li, Yinleng Li, Yingrong Qin, Jinghua Piao, Yuhan Quan, Jianxin Chang, Depeng Jin, Xiangnan He, et al. Graph neural networks for recommender systems: Challenges, methods, and directions. Xiv preprint arXiv:2109. 10..122800 12848, 2021.
Jianfei Gao and Bruno Ribeiro. On the equivalence between temporal and static equivariant, graph repre- sentations. In Proc. Int. Conf. Machine Learning Research (PMLR), 2022.
Jan Gorodkin. Comparing two k-category assignments by k-category correlation coefficient. Computational biology and chemistry, 28(3-6):367-374, 2004.
William L. Hamilton. Graph Representation Learning, volume 14. Morgan and Claypool, 2020.
Kehang Han, Balaji Lakshminarayanan, and Jeremiah Liu. Reliable graph neural networks for drug discovery under distributional shift. arXiv preprint arXiv:2111.12067 2021.
Zepeng Huai, Yuji Yang, Mengdi Zhang, Zhongyi Zhang, Yichun Li, and Wei Wu. M2gnn: Metapath and multi-interest aggregated graph neural network for tag-based cross-domain recommendation. arXiv preprint ar xrx::2304.07991 2023.
Mingyuan Huang, Pengpeng Zhao, Xuefeng Xian, Jianfeng Qu, Guanfeng Liu, Yanchi Liu, and Victor S Sheng. Help from meta-path: Node and meta-path contrastive learning for recommender systems. In Proc. Int. Joint Conf. on Neural Networks (IJCNN), pp. 01-08. IEEE, 2022.
Shenyang Huang, Farimah Poursafaei, Jacob Danovitch, Matthias Fey, Weihua Hu, Emanuele Rossi, Jure Leskovec, Michael M. Bronstein, Guillaume Rabusseau, and Reihanch Rabbany. Temporal graph bench- mark for machine learning on temporal graphs. In Proc. Int. Conf. Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023.
Ryan Keisler, Forecasting global weather with graph neural networks. arXiv preprint arXiv:2202.01575, 2022.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint ar Xiv: 02907, 2016.
Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proc. ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2019.
Lucio La Cava, Davide Costa, and Andrea Tagarelli. Sonar: Web-based tool for multimodal exploration of non-fungible token inspiration networks. In Proc. ACM SIGIR, 2023a.
Lucio La Cava, Davide Costa, and Andrea Tagarelli. Visually wired nfts: Exploring the role of inspiration in non-fungible tokens. CoRR abs/2303 17031, 2023b.
Zhiwei Liu, Yingtong Dou, Philip Yu, Yutong Deng, and Hao Peng. Alleviating the inconsistency problem of applying graph neural network to fraud detection. In Proc. Int. ACM SIGIR Conf. Research and Development in Information Rebrieval, pp. 1569- 1572, 2020.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Minbo Ma, Peng Xie, Fei Teng, Tianrui Li, Bin Wang, Shenggong Ji, and Junbo Zhang. Histgun: Hierarchical spatio-temporal graph neural networks for weather forecasting. ar preprint or arXiv:2201.000 .09101, 2022.
Victor Martínez, Fernando Berzal, and Juan-Carlos Cubero. A survey of link prediction in complex networks.
ACM Computing Surveys (CSUR), 49(4):1-33, 2016.
Shengjie Min, Zhan Gao, Jing Peng, Liang Wang, Ke Qin, and Bo Fang. Stgsn- spatial-temporee graph neural network framework for time-evolving social networks. ShowllddgeBBseeee Systems, 214:106746, 2021.
Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao B. Schardl, and Charles E. Leiserson. EvolveGCN: Evolving graph convolutional networks for dynamic graphs. In Proc. AAAI Conf. Artificial Intelligence, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. Tn Proc. Int. Conf. Neural Information Processing Systems (NeurIPS), pp. 8024 8035. Curran Associates, Inc., 2019.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. In Proc. Inl. Conf. Learning Representalions (ICLR), 2020.
Francesco Saverio Pezzicoli, Guillaume Charpiat, and François P Landes. Se (3u-vquvvria graph neural networks for learning glassy liquids representations. arXiv preprint arXiv:2211.0320 2022.
Patrick Reiser, Marlen Neubert, André Eberhard, Luca Torresi, Chen Zhou, Chen Shao, Houssam Metni, Clint van Hoesel, Henrik Schopmans, Timo Sommer, et al. Graph neural networks for materials science and chemistry. Communications Materials, 3(1):93, 2022.
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. In Proc. Inl. Conf. Learning Represenla- tions (ICLR) Graph Representation Learning Workshop, 2020.
Krzysztof Rusek, José Suároz-Varcla, Albert Mestres, Pere Barlet-Ros, and Albert Cabellos-Aparicio. Un- veiling the potential of graph neural networks for network modeling and optimization in sdn. In Proc. ACM Symposium SDN Research, pp. 140-151, 2019.
Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang, and Hao Yang. Dysat: Deep neural representation learning on dynamic graphs via self-attention networks. In Proc. Int. Conf. Web Search and Data Mining, 2020.
Tushar Sarkar. Xbnet: An extremely boosted neural network. Intelligent. Systems with Applications, 15: 200097, 2022.
Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. Deep crossing: Web-scale modeling without manually crafted combinatorial features. In Proc. ACM SIGKDD Inl. Conf. Knowledge Discovery and Data Mining, pp. 255-262, 2016.
Chenguang Song, Kai Shu, and Bin Wu. Temporally evolving graph noural network for fake news detection. Information Processing & Management, 58(6):102712, 2021.
Dongjin Song and David Meyer. Link sign prediction and ranking in signed directed social networks. Social Network Analysis and Mining, 5, 2015.
Hewen Wang, Renchi Yang, Keke Huang, and Xiaokui Xiao. Efficient and effective edge-wise graph represen- tation learning. In Proc. ACM SIGKDD Int. Conf. Knowledge Discovery and Dato Mining, pp. 1826-2333 2023.
17

---

Published in Transactions on Machine Learning Research (06/2024)
Yanbang Wang, Yen- Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. In Proc. Int. Conf. Learning Regresentations (TCLR), 2021.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: survey. ACM Computing Surveys (CSUR), 55(5):1-37, 2022.
Xuyang Wu, Xinyang Gao, Weinan Zhang, Rui Luo, and Jun Wang. Learning over categorical data using counting features: With an application on click through rate estimation. In Proc. Int. Workshop on Deep Learning Praclice for High- DimhlsiDnms Sporse Dala, pp. 1-9, 2019.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chenggi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Trans. Neural Networks and Learning Systems, 32(1):4-3 24, 2020,
Jiacheng Xiong, Zhaoping Xiong, Kaixian Chen, Hualiang Jiang, and Mingyue Zheng. Graph neural networks for automated de novo drug design. Drug Discovery Today, 26(6):1382 1393, 2021.
Le Yu, Leilei Sun, Bowen Du, and Weifeng Lv. Towards better dynamic graph learning: New architecture and unified library. In Proc. Int. Conf. Neural Information Processing Systems (NeurIPS), 2023.
Wenchao Yu, Wei Cheng, Charu C. Aggarwal, Kai Zhang, Haifeng Chen, and Wei Wang. Netwalk: A flexible deep embedding approach for anomaly detection in dynamic networks. In Proc. ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2018.
G Udny Yule. On the methods of measuring association between two attributes. J. Royal Stalistical Sociely, 75(6):577-6552 1912.
Adnan Zeb, Summaya Saif, Junde Chen, Anwar UI Haq, Zhiguo Gong, and Defu Zhang, Complex graph convolutional network for link prediction in knowledge graphs. Expert Systems with Applications, 200: 116796, 2022.
Ge Zhang, Zhao Li, Jiaming Huang, Jia Wu, Chuan Zhou, Jian Yang, and Jianliang Gao. efraudcom: An e-commerce fraud detection system via competitive graph neural networks. ACM Thans. Information Systems (TOIS), 40(3): 2022.
Liang Zhang, Jingquu Li, Bin Zhou, and Yan Jia. Rumor detection based on sagnn: Simplified aggregation graph neural networks. Machine Learning and Knowledge Entraction, 3(1):84-0) 94, 2021a.
Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. Advances in Neural Information Processing Systems, 34: 9061-9073, 2021b.
Si Zhang, Hanghang Tong, Jiejun Xu, and Ross Macicjewski. Graph convolutional networks: a comprehensive review. Computational Social Networks, 6(1):1-22 23, 2019.
Li Zheng, Zhenpeng Li, Jian Li, Zhao Li, and Jun Gao. Addgraph: Anomaly detection in dynamic graph using attention-based temporal gen. In Proc. Tnl. Joint Conf. on Artificial Intelligence (IJCAT), 2019.
Hongkuan Zhou, Da Zheng, Israt Nisa, Vasileios Ioannidis, Xiang Song, and George Karypis. TGL: A general framework for temporal gnn training on billion-scale graphs. Proc. VLDB Endow., 15(8), 2022.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review methods and applications. Al open, 1:57 81, 2020.
18

---

caapmmigg-o-tmmoaaa-grrraaa Published in Transactions on Machine Learning Research (06/2024)
Appendices
A Implementation Details
In an effort to present fair comparisons, we performed 100 steps of hyperparameters optimization to op- timize the hyperparameters of all models using the software package OPTUNA (Akiba el. al. 2019). Each experiment was run over the same 10 seeds. All tuning was performed on the validation set where we max imize the average accuracy across all 10 soeds, and we report the test-seed averaged results on the test set that are associated with those hyperparameter settings that maximiize the validation accuracy. All models were implemented using PYTORCH GROMETRIC 2.3.1 Fey & Lenssen 2019) and PyTORCH 1.13 (Paskke et al. 2019). We implemented TGN using the layers that are publicly available in PYTORCH GEOMETRIC, GraphMixer and TGAT D were implemented using the authors opensource implementation provided on their github repository. All computations were run on an Nvidia DGX A100 machine with 128 AMD Rome 7742 cores and 8 Nvidia A100 GPUs.
B Details of Pre-processing on Open Sea Dataset
The OPEN SEA dataset is collection of Non-Fungible Token (NFT) transactions conducted between 2021 and 2023, provided by La Cava et al. (2023a) : Costa et al. (0223 La Cava et al. (2023b). Originally, the data involves 70 million transactions chronologically divided into ten splits. In order to scale our experiments, we use the first split which is composed of 7,097,215 transactions spanned between 2022- and 2023-02-18. In Table the fields of the raw data is provided. The pre-processing steps as follows:
The unique NFT ids are set grouping the transaction by token_ id and collection_name _amooo.
2. The transactions are filtered for each unique NFT, and assigned order based on their position in the sequence and the last ones are flagged. A new field of next reveneeveue is created such that for 4thh transaction in the sequence is assigned with the usd_gain of (k: + 1)th one, except the flagged ones.
3. For each transaction without the flag, usd_pe pedceri and the flagged transactions are filtered.
field of benefit is calculated by next_ rext_reveuue
The number of transactions roduces 2.979,950 which is still high for the scale of datasot size we experiment with. Thus, we select. 1000 different buyers by winner winner_accoun account who conduced most transactions and filter transactions that involve them, which reduces the number of transactions to 282,743.
The source node identities are set by seller seller_account account, destination node identities by winner winner_accoume account, the transaction time is set by tx_ tx_timestamp.
6. Feature vectors of transactions are calculated by concatenating normalized quantitative feature fields [fees_sellees seller, fees_opensee opensea, fees fees_seller seller _udd fees, fees_opertt _ate__ppeeeaaaee pand price, gain, sad_price, usd _ed__ain, to_ to_eth, to
to__ssuuuueeeeests usd, created created_dated date] and binary representations of categorical feature fields [token, chain, token_type type, gepet_contracttype type, asset_type].
The response variable is set by binarizing benefit such that it is set to if benefit > 0, and 0 otherwise.
19

---

Published in Transactions on Machine Learning Research (06/2024)
Table 6: Raw data fields of OPEN SEA dataset
Variable
Type
Description
Processing
token id
String
The id of the NET this value is unique within the same collection
Anonymized
mum sales
Integer
progressive integer indicating the number of suc- cessful transactions involving the NFT up to the cur- rent timestamp (cf. *tx "tx__it__ee___________e_ee_e"ee_ "ix__imestamp") The name of the NET The description of the NFT as provided by the cre- ator
Original
name nft description
Vector ID
Anonymized Anonymized
Vector ID
nftt image collection name
Vector ID
The ID for accessing the NFT image vector The ID for accessing the Collection name vector
A Anonymized A Anonymized Anonymized
Vector ID
collection description
Vector ID
The ID for accessing the Collection description vec-
collection image fees seller
Vector ID
The ID for accessing the Collection image vector The absolute amount of fees the seller has gained from this transaction expressed in *token The absolute amount of fees OpenSea has gained from this transaction expressed in *token*
Anonymized
Float
Original
fees _peensea
Float
Original
fees sellerr usd
Float
The absolute amount of fees the seller has gained from this transaction expressed in US dollars (USD)
Original
fees opensea usd
Float
The absolute amount of fees OpenSea has gained from this transaction expressed in US dollars (USD) The wallet address where seller fees are deposited Timestamp of the transaction expressed in yyyy- mm mm-ddTHHe MM.SH.MM.SSS
Original
payout_com collection address tx timestamp
String
Anonymized
String
Original
price
Float Float
The price of the transaction expressed in token The gain after fees (i.e., gain price [ces_opeesccl * price fees seller * price)
Original
gain
Original
usd price
Float
The price of the transaction expressed US dollars (USD)
Original
usd _sd_gain
Float
The difference between the price and the fees ex- pressed in US dollars (USD)
Original
token
Categorical
The token type used to pay the transaction
Original Original
to te_
Float
The conversion rate to convert tokens into Ethereum the current timestamp, such that eth price
to eth
to usd
Float
The conversion rate to convert tokens US dollars (USD) at the current timestamp, such that usd price to usd
Original
from account
String
The address that sends the payment (i.e., win- ner ner/buyer)
Anonymized
account
String
The address that receives the payment (it often cor- responds the contract linked to the asset)
Anonymized
seller account
String
The address of the NFT seller
Anonymized
winner account
String
The address of the NFT buyer
Anonymized
contract
address address
String
The contract address the blockchain
Anonymized
created date
The date of creation of the contract The blockehain where the transaction occurs The schema of the token, i.c., ERC721 or ERC1155 Categorical The asset typology, i.e., non-fungible or semi- fungible
Timestamp
Original
chain
Catcgorical
Original
token type
Categorical
Original
asset contract type
Original
asset type
Categorical
Whether the asset was involved in a simple or bundle transaction
Original
20

---

Published in Transactions on Machine Learning Research (06/2024)
C
Edge Homophily Trends in Datasets
(a) EPIC GAMES
(b) YELPCHI
0.850
0.900
Average
Average
Edge Homophily
0.800
Edge Homophily
0.800
0.950
Positive Class
0.150
Positive Class
0.900
0.125
0.600
0.900
Negative Class
0.400
Negative Class
0.875
2
8
6
time
time
(c) WIKIPEDIA
(d) Mooc
0.997-
0.980
Average
Average
Edge Homophily
0.997
Edge Homophily
0.978
0.050
0.023
Pasitive Class
Positive Class
0.025
0.022
0.999
0.990
Negative Class
Negative Class
0.999
0.989
0
2
4
6
8
0
2
4
6
8
time
time
(e) REDDIT
(f) OPEN SEA
0.590
0.999
Average
Edge Homophily
0.999
Edge Homophily
0.580
Average
0.005
0.550
0.004
Positive Class
Positive Class
0.003
0.500
1.000
Negative Class
0.640
Negative Class
0.999
0.620
2
8
6
time
time
Figure 6: Edge homophily (measured using the metric Equation trends as a function of time (with time measured batches). In the case of WIKIPEDIA, Mooc, and REDDET, the overall edge homophily reaches as high as 99%, whereas the homophily among positive class edges falls below 5%, a phenomenon attributed to significant class imbalance. Across these three platforms, there is a noticeable trend of decreasing positive edge homophily over time, a pattern that emerges as banned users students who withdraw from courses exit. the network. Conversely, on YELPCHI, there's an observable increase in positive class homophily and a decrease in negative class homophily. This trend could stem from spar attacks on businesses, typically carried out by similar groups of reviewers whose numbers grow over time. A similar pattern is detected on Open Sea, suggesting an increasing ratio of sellers who profit over time. Nonetheless, those who profited in the system's carly stages remain active participants.
21

---

Published in Transactions on Machine Learning Research (06/2024)
D Parameter Sensitivity Analysis
FLP
FLP
FLP
0.85
0.85-
0.80
0.80
0.84
0.75
0.83
MCC
0.70
MCC
0.65
0.81
0.60
0.80-
0.55
0.79
0.50
0.50-
0.45
5.0
10.0
25.0
50.0
100.0
100
1000
Z0n0
30000
4000
Steio
7ha00
10900
0.77
10
15
20
28
Negative Sampling Ratio
Batch Size
Number of Neighbors
FLP
FLP
1.00
0.98
0.90
0.95
0.98-
APS
APS
0.88
0.07
0.70
0.97
0.60
1.
100
25.0
50.0
106.0
0.80
100
1000
2000
Sntel
dnted
Suan
7000
109000
0.96-
Negative Sampling Ratio
Batch Size
Number of Neighbors
FLP
FLP
0.98
0.98
0.97
0.98-
0.97
AUC
0.90
AUC
0.97
0.88
0.97-
0.05
0.96
0.96
2.0
10.0
25.0
50.6
100.0
1000
Zin0
Stro
Sto0
7000
10000
Negative Sampling Ratio
Batch Size
Number of Neighbors
Figure 7: Parameter Scnsitivity of FLP WIKIPEDIA
22

---

Published in Transactions on Machine Learning Research (06/2024)
RLC
RLC
0.20
0.20-
0.15
0.15-
MCC
0.10
MCC
0.10-
0.05-
0.05
0.00
0.00
-0.05
100
1000
2000
3000
4000
5000
7000
10000
25
Batch Size
Number of Neighbors
RLC
0.10
RLC
0.08
0.05-
0.06
0.04
APS
0.04
APS
0.03
0.02
0.02-
0.00
0.01
100
1000
2000
3000
4000
5000
7000
10000
2
25
Batch Size
Number of Neighbors
RLC
RLC
0.92
0.92-
0.90
0.88
0.90
AUC
0.86
AUC
0.88
0.84
0.82
0.86-
0.80
0.78
0.84
0.76
100
1000
2000
3000
4000
5000
7000
10000
0.82-
2
25
Batch Size
Number of Neighbors
Figure 8: Parameter Sensitivity of RLC - WIKIPEDIA
23

---

Published in Transactions on Machine Learning Research (06/2024)
E
TGN Modifications RLC
YELPCHI
0.32
0.32
0.28
MCC
0.274
80
0.51
0.31
0.26
0.30
mean
last
exp decay
mean
last
exp. decay
mean
lnst
exp. decay
(b) WIKIPEDIA
0.90
0.25-
0.88-
MCC
0.20
00 0.10
0,12
0.86
0.08
0.82-
0.05
0.04
0.00
0.00
0.02
mean
last
exp, decay
moan
last
exp. decay
moan
last
exp. decay
(c) Mooc
0.10
8.08
0.75-
MCC
8.05
AUC
0,04
0.04
0.22
enc
021-
0.04
moan
last
exp. decay
moan
last
exp. decay
moan
last
exp. decay
(d) REDDIT
815
0.01
0.68
012
818-
0.00
0.06
MGC
5o
AUC
0.64-
8.2
0.01
0.62-
0.00
0.60
0.00
mean
last
exp. decay
mean
last
exp. decay
mean
last
exp. decay
Figure 9: Aggragator Versions
24

---

Published in Transactions on Machine Learning Research (06/2024)
(a) YELPCHI
0.28
0.32
0.70
0.77
0.27
0.79
MCC
0.20
g
0.74
0.35
0.75
0.6-
0.72
8.8
0.2
0.11
0.70
0.2
Sro-dst
sre-dot-t
sre-dst- -re-dst-msg
art-magh mag-tt
dst- dst-mag-t
aro-dst-magt mag-
srodst
sre-doi-
pre-dat-nng
arc-meg- dst ma----
arc-dst-nn arc-dst-mag+ mags
sre-dst
sre-det-t
sxt-dat-mss
grc-mag dst mogg--
are-dst- mog
(b) WIKIPEDIA
0,90
0.12
0.65
MCC
0.20
0.08
0.1-
AUC
0.90
0.10
0.06
0.04
0.75
0.22
0.70
ere-dst
sre-dst-t
sre-dst-msg arc arc-mag-
mag-t
dst-mag-t
aro-dst-mag+4
ere-dst
sre-det-t
sro-dat-msg
arc-mag-t
dst Est-mag-t
are-dst mms--
are-dst
srt-dot-t
sme-dst-mss
arc.ma-4
dst Est-mag-t
arc pro-dst-mop-t
(c) Mooc
0.04
672
MCC
0.04
C
n.w
0.92
0.05
0.08
0.m7
Src-dol
aro-dst-t
smm do-- may
src-msg-t
dit- dis-msg-t
arc-dst- isgg
0.22
groast
are-dsb-t
sre-dst-mon
sre-mis--
der det-msg+ maggs
sre-dst-mas mag
6.80
grodst
arc-dstt
sme-dst-n sme-dst-mag
sre-mag-
dst mogst
sre-dst- mot-
(d) REDDIT
0.12
08:
0.70
0u6e
0.00
MCC
0.81
0.62
0.40
0.22
0.4
0.54
smv-ds
sre-dstet
sre-dat-mag
sre-mat re-mssg-t
dst-mag-
sro-dst- mng--
sre-dst
are-dstt
sro-daf-meg
sre-msg
dst Ast-mag-t
sro-dst- sro-dst-msg-t msg.t
sre-dst
sre-det-
sru-dat-msp
are-mag-t
dst-msg-t
smo-dst-meg
Figure 10: Readout Versions
25

---

Published in Transactions on Machine Learning Research (06/2024)
YELPCHI
0.33
0.28
0.32
MCC
4UC
0.2
learned
fixed
learned
fixed
learned
fixed
(b) WIKIPEDIA
0.80-
0.15
0,5
0.26
0.20
0,15
MCC
015
00
0.10
AUC
018
0.03
0.05
0.0
0.80
00.0
0.00
0.78
learned
Fixed
learned
fixed
learnad
fixed
(c) Mooc
810
0.04
8.08
0.04
0.73-
MCC
85
0.04
0.02
0.00
0.1-
-000
0.70-
learned
fixed
learned
fixed
learned
fixed
REDDIT
0.01
0.05
0.01
MCC
AUC
0.82
9.60
000
0.00
0.00
0.56
learned
fbxe
learned
fixed
Jearned
flsed
Figure 11: Time Encoding Versions
26

---

Published in Transactions on Machine Learning Research (06/2024)
F
Readout Variations RLC
WIKIPEDIA
Dimensson
Model Dim ommesson 100 Model Dimension 200
0.01
0.00
0.70
0.00
10 Number Neighbors
Number Neighbors
Number Neighbors
Number of Neighbors
Model Dimension
Model Dimension 200
Model Dimension
Model Dimension 200
0.20
0.08
MCC
0.30
Number Neighbars
Number Neighbors
Number Number Neighbors
(b) REDDIT
Modal Dimension M00 Dimension 200
Model Dim
100 Model Dimension 200
0.50
0.00
Number of Neighhors
Number Neighbars
Number of Neighbers
Number of Neighbors
Model Dimension 100 Model Dimensiio 200
Model
0.00
0.01
0.06
0.01
0.00
Numbar Neightors
Number Neighbors
Number of Neighibees
Numbor Neighbses
sec-dst
sre
dat
sre-dst-mng
sre-msg dst-mag
Figure 12: The performance on RLC using different variations of readout layer
27


Reviews:
[{"review_id": "PbXIemO4gx", "reviewer": "Reviewer_jUR9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "`Strengths:`\n--- \n1. Formulation of the new task of recent link classification is very interesting and important specifically in industry. Also, the provided benchmark datasets for this task can provide a great opportunity for future work.\n2. The authors design Graph Profiler, which is very intuitive and simple, yet effective for this task.   \n3. The paper is overall well-presented and its several illustrative figures help a lot in understanding of introduced concepts. \n  \n  \n  \n  \n`Weaknesses:`\n---\n1. What is the relationship between temporal anomaly detection and recent link binary classification? To my understanding, temporal anomaly detection methods [1, 2] can also be applied in this setting as they classify the recent links into 1 (anomaly) and 0 (normal). Also, there is a close connection with temporal multiplex networks (graphs with multiple types of edges) as the general recent link prediction can be seen as future link prediction in multiplex networks (at least with some modification they can be applied in this setting). To classify recent links, anomaly detection in temporal multiplex networks can also (see for example [3]) be applied in this setting. If this is the case, comparison and discussion of these methods are necessary.     \n2. It would be great if the authors could also discuss the relevance of recent link prediction with the task of dynamic link property prediction proposed in the Temporal Graph Benchmark paper [4]. Can existing methods on this task be simply modified for the recent link classification task?\n3. The ablation study does not support the importance of the architecture design of Graph Profiler. It would be great if the authors could also show the impact of removing each of the modules. Is the choice of time encoding arbitrary? How learnable time encoding can affect performance?\n4. Based on the reported results by [4], the state-of-the-art models can vary based on the datasets and their properties. The current choice of baselines needs improvement. It would be better if the authors could use more state-of-the-art models as baselines like DyGFormer [5] and CAWN [6].   \n\n\n`References:`\n--- \n[1] AddGraph: Anomaly Detection in Dynamic Graph Using Attention-based Temporal GCN. Zheng et al. 2019.     \n[2] Netwalk: A flexible deep embedding approach for anomaly detection in dynamic networks. Yu et al. 2018.   \n[3] Anomaly Detection in Multiplex Dynamic Networks: from Blockchain Security to Brain Disease Prediction. Behrouz et al. 2022.   \n[4] Temporal Graph Benchmark for Machine Learning on Temporal Graphs. Huang et al. 2023.  \n[5] Towards Better Dynamic Graph Learning: New Architecture and Unified Library. Yu et al. 2023.  \n[6] Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks. Wang et al. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces and formalizes a new task of recent link classification on temporal graphs and provides new benchmark datasets for this downstream task. To evaluate the models' performance, the authors in addition to AUC and AP further suggest using the Matthews Correlation Coefficient due to its robustness to imbalanced datasets. The authors further propose Graph Profiler, a new architecture that using its four modules (Profile Encoder, Destination Embedding, Message Encoder, Time Encoder) can encode the previous events’ class information on source and destination nodes.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SXneXdKWUU", "reviewer": "Reviewer_YZjS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Proposes a new and relevant problem setting for temporal graph learning rather than continuing to make incremental improvements to existing tasks such as link prediction.\n- Proposes a new measure of edge homophily to capture the importance of graph information for the RLC task.\n- Empirical findings about differences in training TGNNs for RLC compared to link prediction tasks may be useful to practitioners.\n\nWeaknesses:\n- The problem setting is not as novel as claimed. The RLC task is closely related to the task of link sign prediction, which aims to predict the sign of an edge given the nodes that formed the edge, as well as edge classification. There is some work also on link sign prediction for temporal graphs. See references below for a few examples of work on link sign prediction. These should be discussed, if not compared to. This is a major weakness.\n- The proposed name \"recent link classification\" is somewhat limiting. I'm not sure why \"recent\" is there, as the approach the authors consider applies to general link classification problems. The more general term \"edge classification\" may be the best term for this problem.\n- The authors do not explain why there is an asymmetry in how the source and destination nodes are treated. There is a complex profile encoder for the source node, while the destination node is basically a standard embedding representation for a look-up table.\n\nReferences on link sign prediction and edge classification:\n- Jushuo Chen, Feifei Dai, Xiaoyan Gu, Haihui Fan, Jiang Zhou, Bo Li, and Weiping Wang. 2023. Learning Pair-Centric Representation for Link Sign Prediction with Subgraph. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM '23). Association for Computing Machinery, New York, NY, USA, 256–265. https://doi.org/10.1145/3583780.3614951\n- C. Aggarwal, G. He and P. Zhao, \"Edge classification in networks,\" 2016 IEEE 32nd International Conference on Data Engineering (ICDE), Helsinki, Finland, 2016, pp. 1038-1049, doi: 10.1109/ICDE.2016.7498311. keywords: {Social network services;Gold;Context;Data mining;Electronic mail;Algorithm design and analysis;Probabilistic logic},\n- Song, D., & Meyer, D. A. (2015). Link sign prediction and ranking in signed directed social networks. Social network analysis and mining, 5, 1-14.\n- Dang, Q. V., & Ignat, C. L. (2018, October). Link-sign prediction in dynamic signed directed networks. In 2018 IEEE 4th International Conference on Collaboration and Internet Computing (CIC) (pp. 36-45). IEEE.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose the task of recent link classification (RLC) as an alternative to the typical future link prediction (FLP) and dynamic node classification tasks used to evaluate temporal graph learning (TGL) methods. In the RLC task, we are given four pieces of information about a recently-observed edge: source node, destination node, edge timestamp, and edge features. Given these four pieces of information, the objective is to predict the class label of the edge, which is not observed until later in time.\n\nThe main contributions I observe are as follows:\n- Formulation of a new task, recent link classification, for evaluating temporal graph learning methods.\n- Development of the Graph Profiler architecture for the RLC task on temporal graphs with edge attributes and classes.\n- Experimental evaluation on differences between RLC and FLP tasks and their implications towards designing and training temporal graph neural networks (TGNNs).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major issues:\n- Perform a literature search on related tasks such as link sign prediction and edge classification. These should be, at the very least, cited and discussed. It would be even stronger to have a comparison against a representative method from this category.\n\nMinor issues:\n- Better motivate the reasoning behind the asymmetry for the treatment of the source and destination nodes. Why such a complex profile encoder for the source node while having just a look-up table embedding for the destination node?\n- Table 1 notation inconsistency: caption uses $\\tilde{\\mathcal{H}}_e^b$ while the table uses $\\tilde{\\mathcal{H}}_e$.", "broader_impact_concerns": "No broader impact statement is present. I suggest that the authors add one and think about potential misuses of edge classification methods, if successful.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sCBi1o8afu", "reviewer": "Reviewer_qa8i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper formalizes a new task of Recent Link Classification and proposes a new architecture of Graph Profiler, with the design of each component clearly illustrated. \n2. The paper explores the usage of Matthews Correlation Coefficient (MCC) as a novel evaluation metric on the RLC task. MCC solves the problem of skewed measures of quality when applied to imbalanced datasets.\n3. The paper modifies existing TGL datasets to suit the RLC task, along with the introduction of new RLC datasets (e.g., Epic Games and Open Sea). This enriches the set of available datasets for this research problem.\n4. The experimental results demonstrate that the proposed Graph Profiler consistently attains an enhanced MCC across the majority of relevant cases.\n\nWeaknesses\n1. Some figures are not cited in the main text, e.g., Figure 1 and Figure 3. Please cite them in proper places of the paper.\n2. The markers in Figure 4 are hard to distinguish at the first glance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper formalizes the Recent Link Classification (RLC) task within temporal graphs as an established benchmark downstream activity in Temporal Graph Learning (TGL), accompanied by the provision of pertinent datasets. It elucidates the distinctions between RLC and other related tasks and presents a novel architecture named Graph Profiler. The model is specifically devised to acquire entity profiles or time-aggregated representations conducive to the RLC task. It not only expands the scope of TGL but also introduces a new evaluation metric, the Matthews Correlation Coefficient. The experimental study demonstrates the superiority of the proposed model over state-of-the-art methods in TGL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to Weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper is ready for publication with some minor changes.
Decision: Accept (poster)