Contextualized Messages Boost Graph Representations
Authors: Brian Godwin Lim
Galvin Brice Sy Lim
Renzo Roel Tan
Kazushi Ikeda
Venue: Accepted by TMLR ()

Abstract: Graph neural networks (GNNs) have gained significant attention in recent years for their ability to process data that may be represented as graphs. This has prompted several studies to explore their representational capability based on the graph isomorphism task. Notably, these works inherently assume a countable node feature representation, potentially limiting their applicability. Interestingly, only a few study GNNs with uncountable node feature representation. In the paper, a new perspective on the representational capability of GNNs is investigated across all levels—node-level, neighborhood-level, and graph-level—when the space of node feature representation is uncountable. Specifically, the injective and metric requirements of previous works are softly relaxed by employing a pseudometric distance on the space of input to create a soft-injective function such that distinct inputs may produce similar outputs if and only if the pseudometric deems the inputs to be sufficiently similar on some representation. As a consequence, a simple and computationally efficient soft-isomorphic relational graph convolution network (SIR-GCN) that emphasizes the contextualized transformation of neighborhood feature representations via anisotropic and dynamic message functions is proposed. Furthermore, a mathematical discussion on the relationship between SIR-GCN and key GNNs in literature is laid out to put the contribution into context, establishing SIR-GCN as a generalization of classical GNN methodologies. To close, experiments on synthetic and benchmark datasets demonstrate the relative superiority of SIR-GCN, outperforming comparable models in node and graph property prediction tasks.

Published in Transactions on Machine Learning Research (04/2025)
Contextualized Messages Boost Graph Representations
Brian Godwin Lim
lim.brian Boliiaannnnnn_____i_ www.sssll..........
Nora Institute of Science and Technology
Galvin Brice Lim De La Salle University- - Monila
galvin lim dim@dlsu.ed....
Renzo Roel Tan
rr. Laaaaainnnnaati naist.jpp
Nora Institute of Science and Technology Ateneo de Manila University
Kazushi Ikeda
Cannshiiis.niss....
Nara Institute of Science and Technology
Reviewed on OpenReview: // net/ forun? id= dd=xx1ffjjs1)
Abstract
Graph neural networks (GNNs) have gained significant attention in recent years for their ability to process data that may be represented as graphs. This has prompted several studies to explore their representational capability based on the graph isomorphism task. Notably, these works inherently assume a countable node feature representation, potentially limiting their applicability. Interestingly, only few study GNNs with uncountable node feature representation. In the paper, of GNNs is investigated across all levels nodeelvvllll neighborhoodd-evell and graph-level- when the space of node feature representation is uncountable. Specifically, the injective and metric requirements of previous works are softly relaxed by employing a pseudomelric distance on the space input to create a soft-injective function such that distinct inputs may produce similar outputs and only the pseudometric deems the inputs to be sufficiently similar on some representation. As a consequence, simple and computationally efficient sefl-isomorphic relational graph convolution network (SIR-GCN) that emphasizes the contextualized transformation of neighborhood feature representations via anisotropic and dynamic message functions is proposed. Furthermore, a mathematical discussion on the relationship between SIR-GCN and key GNNs in literature is laid out to put the contribution into context, establishing SIR-GCN as a generalization of classical GNN methodologies. To close, experiments on syuthetic and benchmark datasets demonstrate the relative superiority of SIR-GCN, outperforming comparable models node and graph property prediction tasks.
perspective the representational capability
1 Introduction
Graph neural networks (GNNs) constitute a class of deep learning models designed to process data that may represented as graphs. These models are well-suited for node, edge, and graph property prediction tasks across various domains, including social networks, molocular graphs, and biological networks, among others (Hu et al.) 2020a] Dwivedi et al., 2023). In the literature, GNNs predominantly follow the message-pastie scheme wherein each node aggregates the feature representation of its neighbors and combines them to create updated node fcature representation (Gilmer et al. 2017 Xu et al. 2018 2019). This allows the model to encapsulate both the network structure and the broader node contexts. Moreover, a graph readout function is employed to pool the node feature representations of a graph and create an aggregated representation for the entire graph (Ying et al. 2018 Murphy et al., 2019 Xu et al. 2019
Among the classical GNNs in literature include the graph convolution network (GCN) Kippf & Welling 2017), graph sample and aggregate (GraphSAGF) (Hamilton et al. 2017), graph attention network (GAT)

---

Published in Transactions on Machine Learning Research (04/2025)
Veličković et al. 2018 Brody al. 2022 and graph isomorphism network (GIN) Xu et al. 2019 which largely fall under the messagg-ppssingg neural network (MPNN) GGilmrr et al. 2017) framework. These models have gained popularity due to their simplicity and remarkable performance across various applications (Hu
ct al. 2020a Dwivedi ct al. 2023 Improvements of these foundational models are also constantly proposed
to achieve state-of-Che-arl performance (Wang et al. 2019b Bodnar et al. 2021 Bouritsas el al. 2023
Notably, advances in GNN have mainly been driven by heuristics and empirical results. Nonetheless, several studies have recently begun exploring the representational capability of GNNs (Garg et al. 2020 Sato et al,
2021 Azizian & Lelarge
2021 Bodnar et al. 2021 Böker et al. 2023). Most of these works analyzed GNNs in relation to the graph isomorphism task. In particular, Xu et al. (2019) was among the first to lay the foundations for creating a maximally expressive GNN based on the Weisfeller-Leman (WL) graph isomorphism test (Weisfeiler & Leman 1968 Subsequent works build upon their results by considering extensions to the original 1-WL test.. Crucially, the theoretical results of these works only hold with countable node feature representation which potentially limits their applicability. Mcanwhile, Corso et al. (2020) proposed using multiple aggregators to create powerful GNNs when the space of node feature representation is uncountable. Interestingly, there has been no significant theoretical progress since this work.
This paper presents a new perspective on the representational capability of GNNs when the space of node feature representation is uncountable. Specifically, the key idea is to define a psendometric distance on the space of input to create a voft-injective function such that distinct inputs may produce similar outputs if and only if the distance between the inputs is sufficiently small on some representation. This framework is comprehensively analyzed across all worel--node-leveds neighborhood-lever and graph-level. From the theoretical results, simple and computationally efficient soft-isomorphic relational graph convolution network (SIR-GCN) which emphasizes the contextualized transformation of neighborhood feature representations using anisotropic and dynamic message functions is proposed. This is further accompanied by a discussion on the mathematical relationship between SIR-GCN and key GNNs in literature to underscore the contribution and distinctive advantages of the proposed model. Finally, experiments on synthetic and benchmark datasets in node and graph property prediction tasks are performed to highlight the expressivily SIR-GCN, positioning the model as a strong candidate for practical GNN applications.
2 Graph neural networks
Let G = (Vg.Eg) be a graph and (u) S Vg be the set nodes adjacent to node u E Vg. The subscript G will be omitted whenever the context is clear. Suppose H is the space of node feature representation, henceforth feature, and hn H is the feature of node u. A GNN following the message-passing scheme can be expressed mathematically as
H. = (h. UU Eg(u))
ag = AGG (H.)
(1)
h" = COMB (hu, au),
where AGG and COMB are some aggregation and combination strategies, respectively, Hu is the multiset [Xu et al. 2019) of neighborhood features for node u, au is the aggregated neighborhood feature for node u, and h is the updated feature for node Since AGG takes arbitrary-sized mullisets of neighborhood features as imput and transforms them into single feature, it may be considered a hash function. Hence, aggregation and hash functions shall be used interchangeably throughout the paper.
Related works When H is countable, Xu al. (0199 showed that there exists a function f H. + S such that the aggrogation or hash function
F
(HH) = (h)
(2)
hEH
is injective or unique in the embedded feature space for each multiset of neighborhood features H of bounded size. This result forms the theoretical basis of GIN.
2

---

Published in Transactions on Machine Learning Research (04/2025)
Meanwhile, the result above longer holds when H. is uncountbbl. In this setting, Corso et al. (2020) proved that if 4 comprises multiple aggregators (e.g., mean, standard deviation, max, and min), the hash function
= m (h)
3
hEH
produces a unique output for every H of bounded size. This finding provides the foundation for the principal neighborhood aggregation (PNA) (Corso et al. 2020). Notably, for this result to hold theoretically, the number of aggregators in + must scale with the size of the multiset neighborhood features H which may be impractical for large and dense graphs.
3 Soft-injective functions
While injective functions and metrics necessary for tasks requiring graph isomorphism to ensure a unique mapping in the embedded feature space, many practical applications of GNN often do not require such strict constraints. For instance, in node classification tasks, GNNs must produce identical outputs for some distinct nodes. Thus, motivated by Xu et al. (2019) and Corso et al. (2020), this paper softly relaxes the injective and metric requirements within the MPNN framework by employing pseudometrics and ooft-injective functions.
Definition (Pseudometric). Let H be non-empty set. function d H x H - R2o a pseudometric on H. if the following holds for all (1) h (2) (3) E H::
I. d (10,110 h() 0;
(h .()))) d (h (109,10) and
(1 ))) s (), ,1)) + d (1(2, 1)).
Note that for a metric d, the first condition is replaced with d h h1 = h()) (2), ensuring points
in H are distinguishable and unique with respect to d. contrast, for a pseudometric d, (n (1) (2))
does not necessarily imply that (1) 12), relaxing the distinguishaailitty constraint of metric. The following assumption is then imposed on the pseudometric d, leveraging results from kernel theory.
Definition 2 (Conditionally positive definite kernel (Schölkopf 2000)). Let H. be a non-empty set. A symmetric function k H x H + R is a conditionally positive definite kernel on H. if for all N E N and H. h h ......(.)) (1))
cejy & ( (1),)() 0,
(4)
with C1. C2.... CN E R and Ei= Q ===0 Assumption 1. Let d H H. + Rzo be pseudometric on H. such that dd a conditionally positive definite kernel H.
The Euclidean distance is an example of a pseudometric satisfying Assumption satisfying this assumption provided below, see Schölkopf (2000) and Berg et al.
A class of pseudometrics
2012) for more.
Remark 1. Consider the pseudometrics dj and d2 on H satisfying Assumption 1 For a > 0 and 0 < p < 1,
the pseudomelrics d2, Vik+ + dl, and df also salisfy Assumplion
Assumption T thus ollers considerable flexibility in the choice of pseudometric d. The following theorem then softly relaxes the injective and metric requirements in previous works.
Theorem 1. Let H. be a non-empty set with a pseudometric d d: H. x H. + Rxo satisfying Assumption There erists feature map H. + S such that for every (1)) h22) E H and E E2
82 < 11 (nx)) (n (2) <s + 82 < h (1) (11 .2 < <1
(5)
3

---

Published in Transactions on Machine Learning Research (04/2025)
Theorem 1 shows that, for each node E V, given pseudometrie distance du that represents dissimilarity function on H. possibly encoded with prior knowledge, there exists corresponding feature map IN. that maps distinct inputs hu (2) hr (2) E H close in the embedded feature space if and only if dy determines hu (1) and hu (2) to be sufficiently similar on some representation. The lower bound E2 asserts the ability of Iu to separate elements of H in the embedded feature space S while the upper bound é1 ensures that Iu maintains the relationship between elements of with respect to da. The feature map Iu may then be described soft-injective Fig. D provides an illustration of how the soft-injective feature map g maps distinct elements of
H. to the same point in S since the corresponding pseudometric (h
deems
these points to be similar. Corollary extends this result for multisets.
s
s
g(d)
(1(20)
(h())
(2)
(33) h0)
9 ()
dH
(H. d)
Figure 1: A soft-injective feature map g H + S corresponding to a pseudometric on H.
3.1 Soft-isomorpii relational graph convolution network
Corollary 1. Let H be non-emply set with pseudometric D on bounded, equinumerous mullisels of H defined as
D2 H HD))
Σ
dhrh')
Σ dh,
Σ dhh,h')
(6)
hEHO) (
KEHO K
he hE
for some pseudometric d H x H + Rzo salisfying Assumplion and bounded, equinumerous mullisels
(1), (2), There erists feature map g H S such that for every H (1) H and e1 s2 0,
< G (H - G (H!3) -(H112)
<Ei = E2 < D H (
H()) < <E1
(7)
where
G(H) Σ g(h).
(8)
Similarly, Corollary I shows that, for each node E V, given pseudometric distance Dn, multisets of H. defined in Eqn. 6 with a corresponding pseudometric distance d, on H. there exists a corresponding feature map Ia and hash function G, defined in Eqn. 8 that produces similar outputs for distinct mullisets of neighborhood features H (1) H if and only Du deems H, (1) and H, (2) to be sufficiently similar on some representation. Likewise, the lower and upper bounds guarantee the ability of G, separate equinumerous multisets of H in the embedded feature space S while maintaining the relationship with respect to D,,- In this setting, the feature map Iu may be interpreted the soft-injective message function (Gilmer et al. 2017) that transforms the individual neighborhood features with corresponding soft-injective hash function G., Meanwhile, the pseudomelric D,, corresponds to the kernel distance (Joshi et al. 2011) which intuitively represents the difference between the cross cross-distance and self distaace between two multisets. The two necessary properties of the soft-injective message function Linctionddyaamil dynamic and anisotropic are then motivated below.
The pseudometric induces the equivalence class [h) = (h' E H: d h²) = 0} with the quotient space H. H dd [b]y E H} such that becomes metric and the correspanding feature map g becomes injective H. Schoenberg 1938
4

---

Published in Transactions on Machine Learning Research (04/2025)
Dynamic transformation To illustrate the role of the pseudometric, consider node with two neighbors and and the task of anomaly detection on the scalar node features he, and h representing zero-mean scores. If du simply corresponds to the Euclidean distance, then the corresponding hash function G, becomes linear as seen Fig. 2a. Crucially, the contour plot highlights collisions instancee where distinct inputs produce idantical outputs (K.., the equivalence class [H between dissimilar mullisets neighborhood features, resulting aggregated neighborhood features that are less useful for the task.
hn
(a) gh) h.
(b) gc = MLP(h).
(c) 9m = MLP + 1).
(d) gh)
-h
Figure 2: Hash functions G under different message functions Ju-
Nevertheless, other choices of pseudometrics, possibly incorporating prior knowledge, would correspond to more complex message functions Ju- This leads to non-trivial hash functions G, and contour plots where only the regions determined by Du to similar with respect to a given task may produce similar aggregated neighborhood features, making collisions more informative and controlled. In practice, this also highlights the significance of dynamic (Brody al. (022)) (f.e. a universal function approximator (Hornik et al. 1989) message functions Iu in the MPNN framework, which may be modeled multi-layer perceptrons (MLPs)
illustrated in Figs. 26 and 2c
As further illustration, if d, instead corresponds to the Euclidean distance of the squared score, then the corresponding hash function G. has the contour plot in Fig. [d]] The resulting hash collisions and equivalence classes then become more useful and meaningful for detecting anomalous scores.
Anisotropic messages It is also worth noting that Corollary I holds for each node E V independently. Hence, different nodes may correspond to different Du. du, Jus and Gu. For simplicity, especially in inductive learning contexts, consider a single pseudometric instead, defined as
2 H (1) H (2) 1m)
Σ
(h, hm))
Σ hEH, ( WEH, (2)
d h/:
Σ hEH, (2) K'
P(h,h':hu) (9)
hEH, (1) MEH, (2)
with a single hash function, defined as
G G(Huiha)
=
Σ heH
g (h; hu),
(10)
for every node E V. This approach makes D, d, and G anisotropic (Dwivedi et al. 2023 (i.e., a function of both the features the query (center) node hu and key (neighboring) nodes h E H.). In addition, contextualized on the features of the query node, may still be interpreted as a pseudometric controlling hash collisions with a corresponding soft-injectivv hash function G.
Furthermore, the integration of h, also allows for the interpretation of g soft-injective relational message function, guiding how features of the key nodes are to be embedded and transformed based the features of the query node. Figs. 2b and 2c provide intuition for this idca where the introduction of a bias term, assuming function of the features of the query node, shifts the contour plot to produce distinct aggregated neighborhood features au A aw for nodes u * u' V with identical neighborhood features Ha = H but distinct features h * har. Morcover, one may also inject stochasticity into the node features distinguish between nodes u + u' V with identical features h hy and neighborhood features H. H. with high probability (aat et al. 2021 and to imitate having distinct Du, du, Jus and G, for each node E V.

---

Published in Transactions on Machine Learning Research (04/2025)
Proposed model For a graph representation learning problem, one may directly model the anisotropic and dynamic sofl-injective relational message function two-layer MLP,, with implicitly learned pseudometrics, to obtain the soft-isomorphic relational graph convolution network (SIR-GCN)
h
E (EN(M)
WR (Wohu + WKhhe,
(11)
where a a non-linear activation function, Wq. WK E Rduidaon xdn, and WR Rdow Xduiaton lincarity, the model has computational complexity of
Leveraging
O (IV) daidden x dm + |8| duidden + [V] x dout X dhidden)
(12)
with only the application of an activation function along edges, making it comparable to classical GNNs in literature. Nevertheless, may also be modeled as a deep MLP in practice if modeling as a shallow
two-layer MLP becomes infeasible.
In essence, SIR-GCN is a simple instance the MPNN framework explicitly designed to handle uncountable node features while maintaining rigorous theoretical foundations. It emphasizes the anisotropic and dynamic transformation of neighborhood features, obtaining contextualized messages that enable it to learn complex relationships between neighboring nodes. Moreover, the proposed model is also computationally efficient, requiring only a single aggregator and applying only activation function along edges to facilitate effective pessage-passing of uncountable node features.
3.2 Soft-isomorphic graph readout function
Corollary also shows that, for each graph G, given a pseudometric distance Dg on multisets of H. defined in Equ. 0 with a corresponding pseudometric distance dg on H. there cxists a corresponding feature map ry and graph readout function Rg defined in Equ. [ While this result holds for each graph G independently, one may simply consider a single D, d, r, and R for every graph {Ga)aED under task D. Nevertheless, the graph context and structure may also be integrated into D, d, r, and R, through a virtual super node (Gilmer et al., 2017) for instance, to imitate having distinct Dg, dg, rg, and Rg for each graph & and to further enhance its representational capability.
Similarly, for a graph representation learning problem, the dynamic soft-injective feature map may also be directly modeled as an MLP, with implicitly learned pseudometrics, to obtain the soft-isomorphic graph readout function
hg Σ MLPR R (hv),
(13)
vEVe
where MLPR R corresponds to r and hg is the graph-level feature of graph G.
4 Mathematical discussion
The mathematical relationship between SIR-GCN and key GNNs literature GCN, GraphSAGE, GAT, GIN, and PNA are presented in this section to underscore the contribution and distinctive advantages of the proposed model. Tt is worth noting that while activation functions and MLPs applied after each GNN layer play a significant role in the overall performance, the discussions only focus on the core pressge-passine operation that defines GNNs. In addition, the relationship between SIR-GCN and the 1- 1- test is also presented to contextualize the representational capability of the former.
4.1 GCN and GraphSAGE
It may be shown that Corollary 1 holds up to a constant scale. Hence, the mean aggregation and symmetric mean aggregation, by extension, may be used in place of the sum aggregation Equ. []] If one sets a as identity or PRELU(( = 1), Wa = 0, WR = W, and N(w) N(w) fu), one obtains
h
Σ (EN(M)
VIV(u) Who
(14)
6

---

Published in Transactions on Machine Learning Research (04/2025)
and
h [(u)) NEN(")
Σ Who
(15)
which are equivalent to GCN and GraphSAGE with mean aggregation, respectively. Moreover, the sum aggregation may also be replaced with the max aggrogation, albeit without theorctical justification, to recover GraphSAGE with max pooling. Thus, GCN and GraphSAGHP may be viewed instances of SIR-GCN. The key difference lies in the isotropic (Dwivedi et al. 2023) nature (i.e., a function of only the features of the key nodes) of GCN and GraphSAGE and their non- con-linearities only after aggregating the neighborhood features.
4.2 GAT
Moreover, in Brody et al. (0022), the attention mechanism of GATv2 is modeled an MLP given by
Cu,v GAT LEAKYRELU (WQ,GAT ha WK,GG hw).
T
(16)
with the message from node to node proportional to exp (fu.u) WK,GAT h,- While the attention mechanism of GATv2 is amisotropic and dynamic, its messages nevertheless only linearly transformed with the query node only determining the degree of contribution of each message through the scalar eu,e. Meanwhile, STR-GCN directly applies the anisotropic and dynamic function in Equ. 16 to the message function, allowing the features the query node to dynamically transform messages. Specifically, if Wq = WQ,GAT, WK- WK,GAT, o = LEAKYRELU and WR adATs one obtains
GA LEAKYRELU (WQ,GAT hu + WK,GAT ho) vEN(u)
Σ
(17)
which shows Eqn. 16 becoming contextualized message in SIR-GCN. Nevertheless, GAT and GATv2 may also be recovered, up to normalizing constant, with the appropriate parameters.
4.3 GIN
Likewise, within the proposed SIR-GCN, one may explicitly add residual connection in the combination strategy to obtain
h MMPPRes Res (hw)
Σ
WR o (Wohu Wkhe),
(18)
vEN(u)
where MLPR Res is a learnable residual network. If MLPRes(th = (1+c) h == PRELU(a-= 1), We = 0,
and WRWK I, then
h = (1+ hu
Σ he veN(a)
(19)
is equivalent to GIN. Hence, SIR-GCN with residual connection generalizes GIN,
4.4 PNA
Furthermore, while SIR-GCN and PNA approach the problem of uncountable node features differently, both models highlight the significance of anisotropic message functions considering both the features the query and key nodes. The key difference lies with PNA using multiple aggregators and static (Brody et al. 2022 (i.e., a function approximator with limited expressivity; e.g., linear) message function
(h hw) Wkh, Woh =: WKh bu-
(20)
Consequently, the influence of the query node on the aggregated neighborhood uralure becomes limited. In particular, when using mean, max, or min aggregators, the influence of the query node is restricted to the bias term b, Wohn Moreover, with normalized moment aggrogators, the bias term is effectively canceled
2GraphSAGE with LSTM aggregation included in this discussion.

---

Published in Transactions on Machine Learning Research (04/2025)
out during the normalization process, further reducing the inffuence of the query node. Hence, PNA does not fully leverage its anisolropic nature, attributed to its heuristic application of multiple aggregators and scalers in a linear MPNN, thereby limiting its expressivity. In contrast, the dynamic nature of SIR-GCN allows for the non-lincar and contextualized embedding of the features of the query node within the messages, thereby fully leveraging its anisolropic nature while allowing employ only a single aggregator.
4.5 1-WL test
Additionally, in terms of graph isomorphism representational capability, SIR-GCN is comparable modified 1-WL test. Suppose w the WL node label of node a the Ith WT.test iteration. The modified update equation is given by
w hash (A[ w w :ee
(21)
where the modification lies concatenating the label the query node u with every element of the mullisse before hashing. This modification, while negligible when H countable, becomes significant when H is uncountable as highlighted in the previous section. Thus, SIR-GCN inherits the theoretical capabilities of the 1-WL test.
4.6 SIR-GCN
Overall, SIR-GCN is a simple MPNN instance that of- fers flexibility in two key dimensions GNN message transformation and aggregator. Consequently, it generalizes four classical GNNs in literature GCN, GraphSAGE, GAT, and GIN ensuring that it is least expressive these models. Notably, SIR-GCN distinguishes itself from other GNNs by employing both anisotropic and dynamic (i.e., contextualized) messages within the MPNN framework, enabling the non-uniform aggregation of neighboring nodes het- erophilous graphs (Zheng et al. 2024) while maintain- ing adaptability to homophilous graphs.
Anisokropie
STR-GON
GAT
Monn
Sum
GCN GrapiSAGE
GIN
Isotropic
Figure 3: SIR-GCN generalizes classical GNNs.
In addition, SIR-GCN also distinguishes itself from PNA in addressing the problem of uncountable node features by employing only a single aggregator that theoretically holds for graphs of arbitrary sizes, thus reducing computational complexity. Nevertheless, its expressivity is maintained through contextualized mes- sages via the application of only an activation function along edges, allowing to inherit the representational capability of the 1-WL test.
5 Experiments
Experiments on synthetic and benchmark datasets in node and graph property prediction tasks are performed to highlight the expressivity of SIR-GCN, Following the evaluation methodology of Xu et al. (2019) Corso ct al. (2020), and [Brody et al. (2022). the key GNNs in literature without advanced architectural design nor domain-specific features are used as primary comparisons to ensure fair evaluation.
5.1 Synthetic datasets
DictionaryLookup DictionaryLookup (Brody et al., 2022) consists of bipartite graphs with 2n nodes key nodes each with an attribute and value and n query nodes each with an attribute. The task to predict the values of query nodes by matching their attributes with the key nodes seen in Fig.
8

---

Published in Transactions on Machine Learning Research (04/2025)
B
B.2
Figure 5: HeleroFddeeCunn.
Figure 4: DietionaryLookin
Table 1: Test accuracy on DictionaryLookup.
Model
n = 10
n
n = 30
1 = 40
n 50
GCN
0.10 = 0.00
0.05 = 0.00
0.03 0.00
0.03 0.00
0.02 + 0.00
GraphSAGE
0.10 + 0.00
0.05 + 0.00
0.03 : 0.00
0.02 0.00
0.02 + 0.00
GATv2
0.99 + 0.03
0.88 + 0.18
0.74 + 0.28
0.56 + 0.37
0.60 + 0.40
GIN
0.78 + 0.07
0.29 + 0.03
0.12 + 0.03
0.03 + 0.00
0.02 + 0.01
PNA
1.00 = 0.00
+ 0.02
0.86 0.09
0.66 0.09
0.50 0.05
SIR-GCN
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
Table presents the mean and standard deviation of the test accuracy for SIR-GCN, GCN, GraphSAGE, GATv2, GIN, and PNA across different values of n. Notably, SIR-GCN and GATv2 can achieve perfect accuracy in this synthetic task attributed to their anisotropic and dynamic nature, enabling them to learn the relationship between every query and key node. Nonetheless, it may be observed that GATv2 suffers from performance degradation in some trials. Mcanwhile, the other models fail to predict the value of query nodes even for the training graphs due to their isotropic and/or static nature, hindering their ability to learn relationships between neighboring nodes. The results hence underscore the utility of a dynamic attentional or relational mechanism in capturing the relationship between the query and key nodes.
EegeroEdgeCounn HeleroEdgeeount is an original synthetic dataset consisting of randomly generated directed graphs with each node randomly labeled one of с classes. The task then to count the number of heterophilous directed edges in each graph connecting nodes with different class labels as illustrated in Fig. 5 This dataset is explicitly designed to highlight the limitations of key GNNs in literature, particularly the theoretically grounded GNNs such GIN and PNA, even in trivial tasks involving countable node features. Specifically, it underscores the utility of anisotropic and dynamic message functions in learning the relationships between neighboring nodes. Crucially, this dataset is not intended to assess the ability of GNNs to handle heterophilous graphs, as this falls beyond the scope of this work.
Table 2: Test mean squared error on HeteroFdge
Model
4
6
8
10
GCN
22749 1: 1242
50807 +:2 2828
62633 + 3491
68965 1 3784
72986 1 4025
GraphSAGE
22962 +:2 1215
36854 1: 2330
30552 +: 1574
21886 : 1896
16529 + 1589
GATv2
22329 +: 1307
44972 2834
49940 2942
50063 3407
= 49661
GIN
39.620 + 2.060
37.193 + 1.382
31.649 + 1.502
32.424 + 1.841
30.091 + 1.429
PNA
172.15 + 97.82
224.83 + 85.80
249.99 + 108.56
251.49 + 98.84
195.72 + 36.65
SIR-GCN
0.001 + 0.000
0.004 + 0.005
1.495 + 4.428
0.038 + 0.068
0.089 + 0.134
Table 2 presents the mean and standard deviation of the test mean squared error (MSE) for SIR-GCN, GCN, GraphSAGE. GATv2, GIN, and PNA across different values of c. Notably, SIR-GCN consistently achieves near-zero MSE loss due to its anisotropic and dynamic nature as well as its sum aggregation, allowing it to learn the relationship between the labels of neighboring nodes while retaining the graph structure. In fact, for We = I, WK I, RELU, and WR 1T it may be shown that SIR-GCN will always produce the correct output for every graph. In contrast, GCN, GraphSAGE, and GATv2 obtained large MSE losses due to their mean aggregation which fails to preserve the graph structure as noted by Xu et al. 2019

---

Published in Transactions on Machine Learning Research (04/2025)
Meanwhile, GIN and PNA successfully retain the graph structure with their sum aggregation but fail to differentiate neighboring nodes due to their static nature. The results thus illustrate the utility of anisotropic and dynamic message functions using sum aggregation even in simple tasks with countable node features, highlighting the limitations of cxisting GNNs.
5.2 Benchmark datasets
Benchmarking GNNs Benchmarking GNNs (Dwivedi et al. 2023 is collection of benchmark datasets consisting of diverse mathematical and real-world graphs across various GNN tasks. In particular, the WikiCS, PATTERN, and CLUSTER datasets fall under node property prediction tasks while the MNIST, CIFARIO, and ZINC datasets fall under graph property prodiction tasks. Furthermore, the WikiCS, MNIST, and CIFAR10 datasets have uncountable node features while the remaining datasets have countable node features. The performance metric of ZINC is the mean absolute error (MAE) while the performance metric of the remaining datasets is accuracy. These six benchmark datasets encompass diverse range of GNN tasks, enabling comprehensive and robust evaluation of model performance. Dwivedi et al. 2023 provides more information regarding the individual datasets.
Table 3: Test performance on Benchmarking GNNs.
Model
WikiCS (t)
PATTERN (t)
CLUSTER (1)
MNIST (t)
CIFAR1O (t)
ZINC (4)
GON
77.47 0.85
85.50 + 0.05
47.83 + 1.51
90.12 0.15
54.14 0.39
0.416 + 0.006
GraphSAGE
74.77 0.95
50.52 + 0.00
50.45 + 0.15
97.31 0.10
65.77 + 0.31
0.408 + 0.003
GAT
76.91 0.82
75.82 + 1.82
57.73 + 0.32
95.54 + 0.21
64.22 + 0.46
0.475 = 0.007
GATv2
67.48 + 0.53
0.447 = 0.015
GIN
75.86 = 0.58
85.59 + 0.01
58.38 = 0.24
96.49 0.25
55.26 1.53
0.387 = 0.015
PNA
97.19 - 0.08
70.21 + 0.15
0.320 = 0.032
EGC-S
66.92 + 0.37
0.364 + 0.020
EGC-M
71.03
0.42
0.281 + 0.007
SIR-GCN
78.06 + 0.66 85.75 + 0.03
63.35 + 0.19
97.90 + 0.08
71.98 + 0.40
0.278 + 0.024
Note: Missing values indicate that no results were published in previous works.
Table 3 presents the mean and standard deviation of the test performance for SIR-GCN, GCN, GraphSAGE, GAT, GATv2, GIN, and PNA across the six benchmark datasets with the experimental set-up, such as parameter count and model architecture, following that of Dwivedi et al. (2023), ) Corso et al. (2020), and
Tailor et al. ) to ensure a fair evaluation where performance differences are solely attributed to the GNN architectural design. The test performance for the efficient graph convolution single (EGC-S) and efficient graph convolution multiple (EGC-M) (Tailor et al.) 2022) are also presented as additional MPNN-based basclines. Notably, SIR-GCN consistently outperforms classical GNNs in litcrature by a substantial margin which may be altributed to its ability to generalize these models, complementing the mathematical discussions in the previous section. Moreover, despite employing multiple aggregators and incurring higher computational complexity to ensure injectivity, PNA still fails to outperform the simpler and more computationally efficient SIR-GCN on datasets with uncountable node features, even though the former is explicitly designed for such tasks. Furthermore, SIR-GCN also outperforms the more recent FGC-S and EGC-M despite their use of additional tricks, including multiple convolutional basis weights, regularization heads, and aggregators. Overall, the results underscore that, under the same constraints, SIR-GCN consistently outperforms MPNN-based baselines despite its simplicity, establishing it as promising alternative to existing MPNNs.
Open Graph Benchmark Open Graph Benchmark (Hu et al. 2020a) is another collection of datasets consisting of realistic, large-scale, and diverse benchmarks for GNNs. In particular, the ogbn-arxiv with its uncountable node features falls under node property prediction tasks. Mcanwhile, the ogbg-molhiv with its countable node features falls under graph property prediction tasks. The performance metric of ogbn-arxiv is accuracy while the performance metric of Lgbg-molhiv is the area under the receiver operating characteristic curve (ROC-AUC). Hu et al. (2020a) provides more information regarding the individual datasets.
10

---

Published in Transactions on Machine Learning Research (04/2025)
Table 4: Test performance on Open Graph Benchmark.
Model
ogbn-arxiv (*)
ogbg-molhiv (f)
GCN
71.92 + 0.21
76.14 = 1.29
GraphSAGE
71.73 + 0.26
75.97
1.69
GAT
71.81 + 0.23
77.17
1.37
GATv2
71.87 + 0.43
77.15 + 1.55
GIN
67.33 + 1.47
76.02 + 1.35
PNA
71.21 1 0.30
79.05 1.32
EGC-S
72.21 + 0.17
77.44 1.08
EGC-M
71.96 + 0.23
78.18 + 1.53
SIR-GCN
72.52 +: 0.16
77.63 + 0.84
Table 4 presents the mcan and standard deviation of the test performance for SIR-GCN, GCN, GraphSAGE, GAT, GATv2, GIN, PNA, EGC-S, and EGC-M across the two large-scale benchmark datasets with the experimental set-up following that of Corso et al. (2020) and TTillrr et al. (2022) to ensure a fair evaluation. Notably, SIR-GCN still outperforms both classical GNNs and EGC-S by a significant margin even in these large-scale graphs, which may be attributed to its anisotropic and dynamic message function. Unsurprisingly, however, PNA and EGC-M exhibit better performance relative to STR-GCN on ogbg-molhiv as this molecular property prediction task greatly benefits from maintaining graph isomorphism via multiple aggregators, scalers, convolutional basis weights, and regularization heads. Nevertheless, SIR-GCN outperforms both PNA and EGC-M by substantial margin on ogbn-arxiv as this 40-class node classification lask does not require maintaining graph isomorphism. Overall, the results highlight the significance of contextualized messages in enhancing GNN expressivity and the utility of softly relaxing the injective and metric requirements within the MPNN framework for most practical GNN applications.
6 Conclusion
In summary, the paper provides a new perspective for creating powerful GNNs when the space of node features uncountable. The key idea is to use pseudometric distances on the space of input to create soft-injective functions such that distinct inputs may produce sémülar outputs if and only the distance between the inputs is sufficiently small on some representation. From the theoretical results, SIR-GCN is proposed a simple and computationally efficient MPNN instance emphasizing contextualized message transformation. Notably, compared to existing MPNN instances, this distinctive feature enables it to learn complex relationships between neighboring nodes and allows it to better handle uncountable node features. Furthermore, the proposed model is also demonstrated to generalize classical GNN methodologies. Despite its simple architectural design and minimal computational requirements, empirical results on synthetic and benchmark datasets underscore the expressivily of SIR-GCN, making it a promising candidate for practical GNN applications. Overall, the paper contributes to GNN literature by theoretically and empirically demonstrating the necessity of both anisotropic and dynamic messages to enhance GNN expressivity. Future works may extend the present framework by considering more complex pseudomelric formulations for bounded, equinumerous multisets of H. in Corollary [ They may also consider a formal analysis of the relationship between contextualized messages and performance on heterophilous graph tasks.
Acknowledgments
This work is supported in part by the Japan Society for the Promotion of Science through the Grants-in-Ae Aid for Scientific Research Program (KAKENHI 18K19821) and in part by Kyoto University and Toyota Motor Corporation through the joint project titled "Advanced Mathematical Science for Mobility Society."
11

---

Published in Transactions on Machine Learning Research (04/2025)
References
Waiss Azizian and Mare Lelarge. Expressive power of invariant and equivariant graph neural networks. In International Conference on Learning Representations, 2021.
Christian Berg, Jens Peter Reus Christensen, and Paul Rossel. Harmonic Analysis on Semigroups: Theory of Posilive Definile and Related Panctions, volume 100. Springer, New York, NY, 2012.
Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolutional networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):3950- 3957, 2021.
Cristian Bodnar, Fabrizio Frasca, Nina Outer, Yuguang Wang, Pietro Liò, Guido F. Montufar, and Michael Bronstein. Weisfeiler and Lehman go cellular: CW networks. In Advances in Neural Information, Processing Systems, volume 34, pp. 2625 2640, 2021.
Jan Böker, Ron Levie, Ningyuan Huang, Soledad Villar, and Christopher Morris. Fine-grained expressivity of graph neural networks. In Advances in Neural Information Processing Systems, volume 36, pp. 46658-46700, 2023.
Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, and Michael M. Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):657-668, 2023.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In International Conference on Learning Representations, 2022.
Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovie. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, 2021.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pictro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. In Advances in Neural Information Processing Systems, volume 33, pp- 13260 13271, 2020.
Lun Du, Xiaozhou Shi, Qiang Fu, Xiaojun Ma, Hengyu Liu, Shi Han, and Dongmei Zhang. GBK-GNN:
Gated bi-kernel graph neural networks for modeling both homophily and heterophily. In Proceedings of the ACM Web Conference 2022, WWW '22, pp. 1550 1558. Association for Computing Machinery, 2022.
Vijay Prakash Dwivedi, Chaitanya K. Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research, 24(43) 48, 2023.
Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. Generalization and representational limits of graph neural networks. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3419 3430, 2020.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Leurning Research, pp. 1263- 1272, 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, volume 30, 2017.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359-366 1989.
Yi-Ling Hsu, Yu-Che Tsai, and Cheng-Te To Li. FinGAT: Financial graph attention networks for recommending top-k profitable stocks. IEEE Transactions on Knowledge and Data Engineering, 35(1):469 481, 2023.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advonces in Neural Information Processing Systems, volume 33, pp. 22118-22133, 2020a.
12

---

Published in Transactions on Machine Learning Research (04/2025)
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representalions, 2020b.
Sarang Joshi, Raj Varma Kommaraji, Jeff M. Phillips, and Suresh Venkatasubramanian. Comparing distributions and shapes using the kernel distance. In Proceedings of the Twenty-Sevente Annual Symposium Computational Geometry, SoCG '11, pp. 47-56. Association for Computing Machinery, 2011.
Byung- Hoon Kim and Jong Chul Ye. Understanding graph isomorphism network for rs-fMRI functional connectivity analysis. Frontiers in Neuroscience, 14:630, 2020.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Xiang Li, Renyu Zhu, Yao Cheng, Cailuua Shan, Siqiang Luo, Dongsheng Li, and Weining Qian. Finding global homophily in graph neural networks when meeting heterophily. In Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2022.
Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
Sunil Kumar Maurya, Xin Liu, and Tsuyoshi Murata. Simplifying approach to node classification in graph neural networks. Journal of Computational Science, 62:101695, 2022.
Ryan Murphy, Balasubraaaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Relational pooling for graph representations. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of 2019.
Proceedings of Machine Learning Research, pp.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Newral Information Processing Systems, volume 32, 2019.
Oleg Platonov, Denis Kuznedelev, Artem Babenko, and Liudmila Prokhorenkova. Characterizing graph datasots for node classification: Homophily-heterophhi dichotomy and beyond. In The Second Learning on Graphs Conference, 2023a.
Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of GNNs under heterophily: Are we really making progress? In International Conference on Learning Representations, 2023b.
Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pp. 333-341, 2021.
Isaac Jacob Schoenberg. Metric spaces and positive definite functions. Transactions of the Americon Mathemalical Sociely, 44:522 536, 1938.
Bernhard Schölkopf. The kernel trick for distances. In Advances in Neural Information Processing Systems, volume 13, 2000.
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjing Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 1548-1554, 2021.
Bharath K. Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert R.G. Lanckrict. Hilbert space embeddings and metrics on probability measures. Journal of Machine Leorning Research, 11 (56) 2010.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Shyam A. Tailor, Felix Opolka, Pictro Lio, and Nicholas Donald Lane. Do need anisotropic graph neural networks? In International Conference on Learning Representations, 2022.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.
Graph attention networks. In International Conference on Learning Representations, 2018.
Petar Veličković, Rex Ying, Matilde Padovano, Raia Hadsell, and Charles Blundell. Neural execution of graph algorithms. In International Conference on Learning Representations, 2020.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang, Deep graph library: A graph-centric, highly-performanl package for graph neural networks, 2019a. arXiv:1909.01315
Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S. Yu. Heterogeneous graph atlention network. In The World Wide Web Conference, WWW '19, pp. 2022 2032. Association for Computing Machinery, 2019b.
Xiyuan Wang and Muhan Zhang. How powerful are spectral graph neural networks. In Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23341 23362, 2022.
Boris Weisfeiler and Andrei Leman. The reduction of a graph to a canonical form and an algebra arising during this reduction. Nouchno-Techmichesie Technicheskaya Informatsia, 2(9): 16, 1968.
Keyulu Xu, Chenglao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5453-5462, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovee. Hierarchical graph representation learning with differentiable pooling. In Advances in Neural Information Processing Systems, volume 31, 2018.
Xin Zheng, Yi Wang, Yixin Liu, Ming Tí, Miao Zhang, Di Jin, Philip S. Yu, and Shirui Pan. Graph neural networks for graphs with heterophily: A survey, 2024. 07X02.220..07082.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra, Beyond homophily in graph neural networks: Current limitations and effective designs. In Advances in Neural Information Processing Systems, volume 33, pp. 7793 -8044, 2020.
Jiong Zhu, Ryan A. Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K. Ahmed, and Danai Koutra. Graph neural networks with heterophily. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12): 11168-11176, 2021.
14

---

Published in Transactions on Machine Learning Research (04/2025)
A Proofs
Theorem 2 (Hilbert space representation of conditionally positive definite kernels (Schoenberg. 1938
Schölkopf 2000 Berg et al. 2012)). ). Let H. be non-empty set and k : H. x H + conditionally positive definite kernel on H satisfying & (h, h) = 0 for all h E H. There crists Hilbert space S of real-valued functions on H and Jealure map g H S such that for every h h H,
|9 (na)) 9 (n=)) = -K(h,,,
(22)
Proof. See Schölkopi (2000).
Theorem Let be non-emply set with a pseudomebric d x - RRo satisfying Assumplion There erists feature map H + S such that for every (2) .h h2 E H and E E2
E2 9 (n
hh [1=))|<< = E2 d (1) h22) <E.
(5)
Proof. Let d x H + Rz0 be a pseudometric satisfying Assumption
By Theorem there exists a
feature map g H + S such that for every (t)) 12)2 (2
H.
(n) - (1t2) d (110)
(23)
Hence, for every E1 > E2 > O,
< 9 (h (h9) <E = e2 < (10)) < <EI-
(24)
Theorem 3. Suppose h (0) ()) (1, (2)
E H and 1: H x H - IR. is symmetric function. Then
k (n0 = 2
1 k (1) h h2) - 11 m0) (
(1)
(0) 12) + (0) 1 ()
(25)
is positive definite if and only if is conditionally positive definite.
Proof. See Schölkopf (2000).
Corollary 1. Let H be non-emply set with pseudometric D on bounded, equinumerous multisets of H defined as
D2 H
HD)
Σ @h,
Σ dh,,h)
Σ dh
(6)
2
h) ()
OEH K'EHO)
N'EH
for some pseudometric d H. x H. + Rz0 satisfying Assumption and bounded, equinumerous multisets
(1)) (2) There erists feature map g 7L S such that for every (!), H and E, E2 0,
82 < G (H G (2))) <el = ea < D ( (1)) H < <
(7)
where
G(H) Σ g(h).
(8)
hcH
15

---

Published in Transactions on Machine Learning Research (04/2025)
Proof. Let D be pseudometric on bounded, equimumerous multisets of H. defined
D2 ( H
E
& (h,
E hEHO) h'EHO)
d
Σ (h,,h')
(26)
hEH h'
hE
k'EH
for some pseudomelric d: H x H + Rz0 satislying Assumption and bounded, equinumerous mullisels
H(), (2) By Theorem 3 the pseudometric d has a corresponding positive definite kernel k: H. H + R.
A simple algebraic manipulation and using the fact that H and H are equinumerous results in
D²
H (1)
H (0
Σ OEH N'
h(h, +
Σ hEM
-2
Σ b(hhh')
(27)
AEMO
K'
N'
Note that D is indeed pseudometric since k is positive definite as noted by Joshi et al. 2011 8 reproducing property of and the linearity of the inner product, it may be shown that
By the
| ( - || -
(H)
(28)
where
G(H) Σ g(h)
(29)
PEH
and g is the corresponding feature map of the kernel k. Hence, for every E1 > E2 > 0,
< G ( - -G (H) <ei > < D ( (
H(2) <
(30)
B Additional benchmark experiments
Hcterophilous Datascts Hcterophilous Datascts Platoonov et al. 2023b) is a collection of node property prediction benchmark dalasets for evaluating GNNs under heterophily. In particular, the roman-empire, amazon-ratings, tolokers, and questions datasets have uncountable node features. Meanwhile, the minesweeper dataset has countable node fcatures. The performance metric of roman-cmpire and amazon-ratings is accuracy while the performance metric of the remaining datasets is ROC-AUC, Platonov et al. (2023b) provides more information regarding the individual datasets.
Table 5 presents the mean and standard deviation of the test performance for SIR-GCN, GCN, GraphSAGE and GAT across the five benchmark datasets with the experimental set-up closely following that of Platonov let al. (2023b). The test performance for the Transforaee (Shi et al. 2021 and heterophilous GNNs
HCGCN (Zhu et al. 2020), CPGNN (Zhu et al. 2021), GPR-GNN (Chien et al.] 2021). FSGNN (Maurya et al.
2022), GloGNN (Li el al. 2022), FAGCN (Bo el al. 2021). GBK-GNN (Du
2022 and JacobiConv WWang& & Zhang 2022) are also presented as additional baselines. Unsurprisingly, SIR-GCN performs poorly amazon-ratings, tolokers, and questions as these datasets exhibit near-zero label informativeness (Platonov el al. 2023a between key node labels and query node labels, which severely limits the ability of SIR-GCN to learn meaningful relationships between neighboring nodes. Despite this inherent challenge, the performance gap in ROC-AUC between SIR-GCN and the top-performing models on tolokers and questions remains minimal, reflecting its robustness. Conversely, SIR-GCN outperforms classical GNNs, Graph and all heterophilous GNNs on roman-empire and minesweeper, underscoring utility even in heterophilous settings. Overall, the results highlight that SIR-GCN remains competitive in heterophilous graph tasks. Future works may further explore its theoretical and empirical properties under heterophily.
3If & is also integrally strictiy positive definite (rrieermmuddr al. 2010 then hash function G becomes injective and D. becomes metric.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Table 5: Test performance on Heterophilous Datasets.
Model
roman-empire (t)
amazon-ratings (1)
minesweeper (t)
tolokers (t)
questions (t)
GCN
73.69 + 0.74
48.70 + 0.63
89.75 + 0.52
83.64 = 0.67
76.09 + 1.27
GraphSAGE
85.74 0.67
53.63 0.39
93.51 + 0.57
82.43 0.44
76.44 0.62
GAT
80.87 0.30
49.09 0.63
92.01 = 0.68
83.70 + 0.47
77.43 1.20
Transfrersf
86.51 t 0.73
51.17 + 0.06
91.85 + 0.76
83.23 = 0.64
77.95 + 0.08
H&GCN
60.11 0.52
36.47 0.23
89.71 + 0.31
73.35 1.01
63.59 1.46
CPGNN
63.96 + 0.62
39.79 + 0.77
52.03 + 5.46
73.36 = 1.01
65.96 1.95
GPR-GNN
64.85 + 0.27
44.88 0.34
86.24 + 0.61
72.94 0.97
55.48 + 0.91
ESGNN
79.92 0.56
52.74 0.83
90.08 + 0.70
82.76 0.61
78.86 0.92
GloGNN
59.63 0.69
36.89 0.14
51.08 + 1.23
73.39 = 1.17
65.74 1.19
FAGCN
65.22 +: 0.56
44.12 + 0.30
88.17 + 0.73
77.75 = 1.05
77.24 +: 1,26
GBK-GNN
74.57 + 0.47
45.98 0.71
90.85 + 0.58
81.01 0.67
74.47 + 0.86
JacobiConv
71.14 0.42
43.55 0.48
89.66 + 0.40
68.66 0.65
73.88
1.16
SIR-GCN
87.67 + 0.28
46.73 0.61
94.12 + 0.42
82.85 0.72
75.33 + 1.34
C Experimental set-up
All experiments are performed on a single NVIDIA Quadro RTX 6000 (24GB) card using the Deep Graph Library (DGL) (Wang et al. 2019a) with PyTorch (Paszke et al.] 2019) backend. For synthetic datasets, the reported results obtained from the models at the final epoch across 10 trials with varying seed values. For benchmark datasets, the reported results are obtained from the models with the best validation loss across the 10 trials. The codes to reproduce the results may be found at htttp://giiee wom:brranggodvviniiii SIR-GCN
C.1 Synthetic datasets
DictionaryLookup Adopting Brody al. (2022), the training dataset consists of 4,000 bipartite graphs, each containing 2n nodes with randomly assigned attributes and/or values, while the test dataset comprises 1,000 bipartile graphs with the same configuration. All models utilize single GNN layer with 4n hidden units. A two-layer MLP is also used for GIN and a of SIR-GCN while PNA uses the sum, max, and standard deviation aggregators, Model training is performed with the AdamW (Loshchilov & Hutter 2019) optimizer for 500 epochs with a batch size of 256 and a learning rate of 0.001 that decays by a factor of 0.5 with patience of 10 epochs based on the training loss.
HeteroEdgeCount The training dataset consists of 4,000 directed graphs, each containing a maximum of 50 nodes with uniformly selected edges using the rand aad__rraph function of DGL and uniformly assigned node labels from one of classes using the randint function of PyTorch. These measures ensure that the graphs sufficiently diverse with respect to graph structure and heterophily. Meanwhile, the test dataset comprises 1,000 directed graphs with the same configuration. All models utilize single GNN layer with 10c hidden units and sum pooling as the graph readout function. A feed-forward neural network is also used for GIN while PNA uses the sum, max, and standard deviation aggregators. Model training is performed with the AdamW optimizer for 500 epochs with batch size of 256 and a learning rate of 0.001 that decays by a factor of 0.5 with patience of 10 epochs based on the training loss.
C.2 Benchmark datasets
Benchmarking GNNs The datascts are obtained from dgl with data splits (training, validation, test)
following Dwivedi et al. (2023). In line with Dwivedì et. al. 2202)), Corso et al. (2020), and Tailor et al. (2022), all models utilize 4 GNN layers with batch normalization and residual connections while constrained to a parameter budget of 100,000. Regularization with weights in {1 10-7, x 10-6 10-5) and dropouts with rates in {0.1. 0.2, 0.3) are also used to prevent overfitting. The mean, symmetric mean, and max aggregators used since the aggregator is observed to not generalize well to unseen graphs noted
17

---

Published in Transactions on Machine Learning Research (04/2025)
Additionally, sum pooling is used as the graph readout function for ZINC while mean pooling is used for MNIST and CIFAR10. Model training is performed with the AdamW optimizer for a maximum of 500 epochs with a batch size of 128, whenever applicable, and a learning rate of 0,001 that decays by a factor of 0.5 with patience of 10 epochs based on the training loss. The reported results for the
by Veličković et al. (2020)
other models in Table 3 are obtained from Dwivedi et al. (2023) Corso et al. (2020), and Tailor et al. (0022).
Open Graph Benchmark The datasets are obtained from ogb, the Open Graph Benchmark Python
package, with data splits following Hu et. al. (2020a). In line with Corso et al. (2020) and [Taiorr et al. (002)); the model for ogbn-arxiv and ogbg-molhiv utilizes 3 and GNN layers, respectively, with batch normalization and residual connections while constrained to a parameter budget of 100,000. Weight decays in {1 x 10-4, 1 x 10-3} and dropouts with a rate of 0.2 are also used to prevent overfitting. The symmetric mean and max aggregators are used with mean pooling as the graph readout function for ogbg-molhiv. Model training is performed with the AdamW optimizer for a maximum of 1000 and 100 epochs, respectively, with a batch size of 64 for ogbg-molliv and learning rates in {0.001, 0.013 that decays by a factor of 0.5 with patience of 40 and 10 epochs, respectively, based on the training loss. The reported results for the other models in Table 4 are obtained from Corso al. 2020 and Tailor et al. (2022).
Heterophilous Datasets The datasets are obtained from dgl with data splits following Platonov et al. (2023)). In line with Platonov et al. (2023b), the number of GNN layers is chosen from {3, 5). employing residual connections and dropouts with a rale of 0.2. Moreover, the hidden dimension is chosen from (256, 512}, utilizing batch and layer normalization as well as the symmetric mean and max aggregators. Model training is performed with the AdamW optimizer for a maximum of 1000 epochs and a learning rate of 3 x 10-5, The reported results for the other models in Table are obtained from Platonov et al. (2023b).
Runtime analysis
As an additional evaluation, the inference runtime for each model in the synthetic datasets presented Tables and 7 The results, when considered alongside Tables [ and [ illustrate that SIR-GCN achieves a balance between computational complexity and model expressivity, specifically with regards to PNA which is also designed for uncountable node feat ures.
Table 6: DictionaryL.ookup inference runtime.
Model
10
20
30
n 40
50
GCN
0.3526s + 0.0778s
0.4734s = 0.0468s
0.4777s 1 0.0854s
0.5619s 1 0.0518s
0.5520s + 0.0679s
GraphSAGE
0.4565s + 0.0873s
0.5264s = 0.0317s
0.5716s 1 0.1132s
0.7742s 1 0.0597s
0.9193s + 0.0473s
GATv2
0.3950s
0.1017s
0.5276s = 0.0556s
0.6191s 0.0879s
0.7472s 1 0.0346s
1.0065s 1 0.0280s
GIN
0.3696s
0.0899
0.4610s
= 0.0459s
0.4670s + 0.0781s
0.5947s = 0.0548s
0.5194s = 0.0993s
PNA
0.8854s + 0.0412s
1.1913s = 0.1024s
1.4526s + 0.0684s
1.8793s + 0.0528s
2,8387s + 0.0603s
SIR-GON
0.4687s
0.0777s
0.6066s = 0.0398s
0.8053s
0.0485s
1.1496s + 0.0427s
1.7031s + 0.0458s
Table 7: HoteroEddeeounnt inference runtime.
Model
2
6
10
GCN
0.4243s 4 0.0520s
0.3852s + 0.0517s
0.3868s t 0.0743s
0.4166s : 0.0551s
0.4177s + 0.0494s
GraphSAGE
0.4691s + 0.0400s
0.4790s - 0.0440s
0.4399s
0.0629s
0.4501s + 0.0603s
0.4964s + 0.0601s
GATv2
0.4710s 0.0978#
0.4941s = 0.0567s
0.4718$ = 0.0361s
0.5514s = 0.0608s
0.5437s = 0.0724s
GIN
0.4085s + 0.0741s
0.3875$ = 0.0627s
0.3855$ + 0.0645s
0.4298s + 0.0566s
0.4329s = 0.0534s
PNA
2.2963s + 0.0413s
2.4238s = 0.0611s
2.4577s + 0.0533s
2.4741s + 0.0665s
2.5623s + 0.0425s
SIR-GON
0.5338s + 0.0353s
0.5264s + 0.0737s
0.5635s + 0.0695s
0.5764s + 0.0401s
0.6230s + 0.0388s
18

---

Published in Transactions on Machine Learning Research (04/2025)
Table 8 further complements these results by presenting the asymptotic computational runtime complexity of the different GNNs. In particular, SIR-GCN first computes the linear transformations Wohu and Wx Wxh, for every node which incurs O ([V) x dhidden x dm.). Afterward, a (Wohu + Wkh,) is computed for every odge, using the previously calculated values, and aggregated across the neighbors of cach node which incurs ( x dnidden). Finally, the aggregated values are linearly transformed with WR for every node which incurs O ((V) dout x dhidden f only an activation function along edges, its computational complexity is comparable to GCN, GraphSAGB, GAT, GATv2, and GIN. Specifically, these models achieve computational efficiency by maintaining linear complexity along edges attributed to activation functions and neighborhood aggregation. Despite this, SIR-GCN consistently outperforms these classical GNNs across all benchmarks. Notably, SIR-GCN also demonstrates a lower complexity than PNA due to the number of aggregalors used, yet delivers superior performance across most datasets. These additional analyses further underscore the practical utility of the proposed model.
In total, since SIR-GCN employs only linear transformations along nodes and
Table 8: Asymptotic runtime complexity.
Model
Complexity
GON
O (IV) x dow x di + [[] dout)
GraphSAGE
O ([V) x dom x di + [E] dout)
GAT/GATI2
O (DV) x doa x du + [8 dout)
GIN
O ([8) x di + IV x MLP)
PNA
O ([E) x di, [8| dn x + [V] dous din k))
SIR-GCN O (VV) x dhidden x dn + |8| x dhidttnn + [V] tont X dhidden)
Note: h represents the mumber of aggregatons and sealers in PNA.
E SIR-GCN extensions
Denote hu as the feature of the edge connecting node to node Following the intuition presented in Sention 3.1 STR-GCN with residual connection may be modified to leverage edge features to obtain
h MMLPRessMM (hu.) +
WR a (Wqhu + WElu + Wkh,), Σ vEN(u)
(31)
where WE E I RRlnideen xdin, Consequently, this also increases the computational complexity of the model to
( x dhidden X dm + [V x dout x dniddion + [V] x MLPRes),
(32)
where MLPRes denotes the computational complexity of MLP Ress making it comparable to PNA. Similarly, this extension may be viewed as a generalization of GIN with edge features Hu et al. 2020b).
Furthermore, one may also inject inductive bias into the pseudometrics which may correspond to specifying the architecture type for the corresponding message function g. For instance, if node features are known have a sequential relationship (e-g., stock (Hsu et al. 2023) and fMRT (Kim & Ye 2020) data), g may then be aptly modeled using recurrent or convolutional networks.
'In the case of STR-GCN with max aggregation, the linear transformation and max aggregator cannot be interchanged. Hence, the linear transformation WR must be performed along edges which incurs ([E) x dout dhidden).
19


Reviews:
[{"review_id": "j3hQKeWqOt", "reviewer": "Reviewer_u4dh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- This paper is well-written and easy to follow.\n- The case of uncountable node feature representation mentioned in this paper is currently less explored in the community.\n\n### Weaknesses\n- The paper repeatedly mentions the term \"metric\" but does not provide a definition. The authors should add more information to make the paper more accessible to readers with different research backgrounds, such as why it is necessary to define \"metric\" in the context of the research problem and its role within the existing theoretical framework.\n- The soft-injective message function proposed in the paper has two properties, i.e., dynamic transformation and anisotropic messages. The authors should provide further explanation to clarify what \"dynamic\" transformation means, why it is important, and what type of function corresponds to a static transformation.\n- The authors spend a considerable amount of space in Section 4 discussing the relationship between the proposed method SIR-GCN and various classical methods. This section is not very informative, and it is recommended to present the comparison directly in a table or other concise formats.\n- In R1, the authors prove that by tuning hyperparameters, classical GNN methods can also achieve competitive performance on heterophilic graphs. The authors should conduct experiments on more real-world datasets (e.g., larger graphs, real-world homophilic and heterophilic graphs, R2-R4) and compare the results with those in R1.\n\nR1. Classic GNNs are Strong Baselines. NeurIPS'24\n\nR2. Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods. NeurIPS' 21\n\nR3. A critical look at the evaluation of GNNs under heterophily. ICLR' 23\n\nR4. Open Graph Benchmark: Datasets for Machine Learning on Graphs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper improves the existing theoretical framework for GNN expressiveness in the case of uncountable node feature representation. It proposes a new message-passing aggregation method within this framework. This method can be seen as a generalization of classical GNN approaches. The results on synthetic and real-world datasets show that the proposed method outperforms simple yet classic GNN methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0YttICBXYH", "reviewer": "Reviewer_wxBX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(S1) This work highlighted the soft-injective function in understanding the representation capability of GNNs with uncountable node feature representation.\n\n(S2) The proposed SIR-GCN was simple and computationally efficient. The connections between SIR-GCN and existing GNNs were discussed\n\n(S3) Experimental results demonstrated the superior performance of SIR-GCN over baselines.\n\nWeaknesses:\n(W1) Figure 2 is not well-explained. It is unclear regarding the MLP-induced message functions.\n\n(W2) More recent GNN baselines can be included in the experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies GNNs with uncountable node feature representation. It relaxed the requirements for injection and metrics when designing the aggregation function. The key idea was to define a pseudometric distance in a soft-injective function, such that the inputs result in similar outputs if the distance between them is sufficiently small on some representation. A SIR-GCN framework was then proposed based on the anisotropic and dynamic message functions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The GraphHeterophily data set can be further explained to discuss the impact of graph heterophily on SIR-GCN based on the anisotropic and dynamic message functions. More heterophily-based GNNs can be considered to validate the effectiveness of SIR-GCN on GraphHeterophily.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MlHaVnsXlQ", "reviewer": "Reviewer_ypW5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n* The paper provides a mathematical analysis comparing SIR-GCN with well-known GNNs, establishing a theoretical basis of why the proposed method outperforms the existing GNNs.\n* The experimental results show that the proposed method outperforms existing GNNs.\n* By employing pseudometric distances and soft-injective functions, the proposed method relaxes strict injective and metric requirements, allowing the model to handle distinct inputs more accurately.\n\nWeaknesses\n* The performance gain could come from the additional parameters such as MLPs, compared to the baselines.\n* Even though the proposed SIR-GCN outperforms the existing GNNs, I wonder if the proposed GNN can outperform the recent powerful models such as graph transformers. Otherwise, it would be better to discuss whether learning the pseudometric distances could be possible in attention-based architectures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes the Soft-Isomorphic Relational Graph Convolution Network (SIR-GCN), for understanding the representational capabilities of GNNs when the node feature space is uncountable. \nSpecifically, it introduces the concept of soft-injective functions by employing pseudometric distances, allowing distinct inputs to produce similar outputs only if they are deemed sufficiently similar by the pseudometric. This relaxes the strict injective and metric requirements in previous GNN models.\nThe proposed SIR-GCN performs the contextualized transformation of neighborhood feature representations using anisotropic and dynamic message functions by modeling the anisotropic and dynamic soft-injective relational message passing with a two-layer MLP that can implicitly learns the pseudometrics.\nThe authors provide a mathematical analysis comparing SIR-GCN with well-known GNNs such as GCN, GraphSAGE, GAT, and PNA, showing that other GNNs can be a special case of SIR-GCN.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Minor Request\n* It would be better to relate the ability to model the distinct inputs to Figure 1, to emphasize the importance of the proposed concepts.\n* It is promising to have the comparable computational complexity from existing GNNs. However, the in-depth comparison should be provided to enhance the readability.", "broader_impact_concerns": "The authors show that the proposed GNN architecture outperforms in predicting molecular properties, which can be maliciously used to select toxic compounds.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is an overall consensus that the paper should be accepted. The authors provided their source code, and it looks thorough.
Decision: Accept (poster)