Node-Specific Space Selection via Localized Geometric Hyperbolicity in Graph Neural Networks
Authors: See Hian Lee
Feng Ji
Wee Peng Tay
Venue: Accepted by TMLR ()

Abstract: Many graph neural networks have been developed to learn graph representations in either Euclidean or hyperbolic space, with all nodes' representations embedded in a single space. However, a graph can have hyperbolic and Euclidean geometries at different regions of the graph. Thus, it is sub-optimal to indifferently embed an entire graph into a single space. In this paper, we explore and analyze two notions of local hyperbolicity, describing the underlying local geometry: geometric (Gromov) and model-based, to determine the preferred space of embedding for each node. The two hyperbolicities' distributions are aligned using the Wasserstein metric such that the calculated geometric hyperbolicity guides the choice of the learned model hyperbolicity. As such our model Joint Space Graph Neural Network (JSGNN) can leverage both Euclidean and hyperbolic spaces during learning by allowing node-specific geometry space selection. We evaluate our model on both node classification and link prediction tasks and observe promising performance compared to baseline models.

Published in Transactions on Machine Learning Research (12/2024)
Node-Specific Space Selection via Localized Geometric Hy- perbolicity in Graph Neural Networks
See Hian Lee
sechian001 &e. nee.tuueeee ntu..du.sp
Deportment of Electrical and Electronic Engineering Nonyang Technological University
Ji Feng
fifeng@nttr eednnttuuudd....
Department of Electricol and Electronic Engineering Nonyang Technological University
Tay Wee Peng
uplay@ntu. Settoyyntuuedu.... edu.sg
Department of Electricol and Electronic Engineering Nonyang Technological University
Reviewed on OpenReview: https: openreview. pet/pdj2 tHteJFeNly
Abstract
Many graph neural networks have been developed to learn graph representations in cither Euclidean or hyperbolic space, with all nodes' representations embedded in single space. However, graph can have hyperbolic and Euclidean geometries at different regions of the graph. Thus, it is sub-optimal to indifferently embed an entire graph into single space. In this paper, we explore and analyze two notions of local hyperbolicity, describing the underlying local geometry: geometric (Gromov) and model-based, to determine the preferred space embedding for each node. The two hyperbolicities' distributions are aligned using the Wasscrstein metric such that the calculated geometric hyperbolicity guides the choice of the learned model hyperbolicity. As such our model Joint Space Graph Neural Network (JSGNN) can leverage both Fuclidean and hyperbolic spaces during learning by allowing node-specific geometry space selection. We evaluate model on both node classification and link prediction tasks and observe promising performance compared to baseline models.
1 Introduction
Graph neural networks (GNNs) are neural networks that learn from graph-structured data. Many works such as Graph Convolutional Network (GCN) (Kipf & Welling, 2016), Graph Attention Network (GAT) (Veličković et al., 2018), GraphSAGR (Hamilton et al., 2017) and their variants operate the Euclidean space and have been applied in many areas such as recommender systems (Ying et al., 2018; Chen et al., 2022a), chemistry (Gilmer et al., 2017) and financial systems (Sawhney et al., 2021). Despite their remarkable accomplishments, their performances are still limited by the representation ability of Fuclidean space. They are unable to achieve the best performance in situations when the data exhibit non-Euclidcan characteristics such scale-free, tree-like, hierarchical structures (Yang et al., 2022).
As such, hyperbolic spaces have gained traction in research as they have been proven to better embed tree-like, hicrarchical structures compared to the Euclidean geometry (Bachmann et al., 2019; Cho et al., 2019). Intuitively, encoding non-Euclidean structures such as trees in the Euclidean space would result in more considerable distortion since the number of nodes in a tree increases exponentially with the depth of the tree while the Euclidean space only grows polynomially (Zhu et al., 2020). In such cases, the hyperbolic geometry serves as an alternative to learning those structures with comparably smaller distortion as the hyperbolic space has the exponential growth property (Yang et al., 2022). As such, hyperbolic versions of

---

Published in Transactions on Machine Learning Research (12/2024)
GNNs such HGCN (Chami et al., 2019), HGNN (Lin et al., 2019), HGAT (Zhang et al., 2021a) and LGCN (Zhang et al., 2021b) have been proposed.
Nevertheless, real-world graphs are often complex. They are neither solely made up of Euclidean nor non- Euclidean structures alone but a mixture geometrical structures. Consider a localized version of geometric hyperbolicity, a concept from geometry group theory measuring how tree-like the underlying space is for each node in the graph (refer to Section 3.1 for more details). We observe mixture of local geometrie hyperbolicity values in most of the benchmark datasets employ for experiments that the graphs contain mixture of geometries and thus, it is not ideal to embed the graphs into single geometry space, regardless of Euclidean hyperbolic as it inevitably leads to undesired structural inductive biases and distortions (Yang et al., 2022).
in Fig. 2. This implies
Taking a graph containing both lattice- and tree-like structures as an example, Fig. 1c and Fig. If shows that 15 of the bluc-colored nodes in the tree structure are calculated to have 2-hop local goometric hyperbolicity value of zero, while 12 of the purple nodes have value of one and the other 3 purple nodes (at the center of the lattice) have value of two (the smaller the hyperbolicity value, the more hyperbolic). This localized metric can therefore serve indication during learning on which of the two spaces is more suitable to embed the respective nodes.
(a)
(b)
(c)
Lattice-like graph
Tree-like graph
Graph with Tree & Lattice Structure
12
14
14
Number of Nodes
10
Number of Nodes
Number of Nodes
8-
10:
10
8
4
2
0.0 0.5 1.0 1.5
2.0
2.5
0.0
1.0 1.5 2.0 Geometric Hyperbolicity
0.5
2.5
0.0
0.5
1.0
1.5
2.0 2.5
Geometric Hyperbolicity
Geometric Hyperbolicity
(d)
(o)
(f)
Figure 1: Example graphs. (a) Lattice-like graph. (b) A tree. (c) A combined graph containing both lattice and tree structure. (d-f) The histograms reflect the geometric hyperbolicity in the respective graphs.
Here we address this mixture of geometry graph and propose Joint Space Graph Neural Network (JSGNN) that performs learning on a joint space consisting of both Euclidean and hyperbolic geometries. To achieve this, we first update all the node features in both Euclidean and hyperbolic spaces independently, giving rise to two sets of updated node features. Then, employ exponential and logarithmic maps to bridge the two spaces and an attention mechanism is used as form of model hyperbolicity, taking into account the underlying structure around each node and the corresponding node ures. The learned model hyperbolicity is guided by geometric hyperbolicity and is used to "softly decide" the most suitable embedding space for each node and reduce the two sets updated features into only one set. Idcally, a node should be either hyperbolic or Euclidean and not both simultancously, thus, we also introduce an additional loss term achieve this non-uniform characteristic. Related works discussed in Appendix E.
2

---

Published in Transactions on Machine Learning Research (12/2024)
Cora
Citeseer
Pubmed
Disease
0.4
0.4
Probabilicy
0.3
Probability
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.5
1.0
1.5
2.0
0.0
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0 1.5
2.5
Goometric Hyporbolicity
Geometric Hyperbalicity
Goometric Hyperbolicity
Geometric Hyperbalicity
(a)
(b)
(c)
(d)
Airport
CS
Photo
0.7
0.6
0.6
0.5
0.8
20.5
00.4
Probabilisy
50.4
0.3
80.3
0.4
0.
0.2
0.1
0.1
0.2
0.0
0.5
0.0
0.0
Geometric Hyporbolicity
Geometric Hyperbalicity
Geometric Hyperbolicity
(e)
(f)
Figure Distributions of geometric hyperbolicity for all datasets, obtained by computing ōg., on each nodes' 2-hop subgraph. See Appendix C for more results.
2
Background
In this section, we give a brief overview of hyperbolic geometry that will be used the paper. Readers are referred to Lee (2018) for further details. Moreover, we review GAT and hyperbolic version.
2.1 Hyperbolic geometry
A hyperbolic space non-Fuclidean space with constant negative curvature. There are different but equivalent models describe the same hyperbolic geometry. In this paper, we work with the Poincaré ball model, in which all points
inside ball. The hyperbolic space with constant negative curvature is denoted by (m> g)) It consists of the n-dimensional hyperbolic manifold D>" = (x E R" c||x|| < 1} with the Riemannian metric gf = (Ag) where X = 2/(1 - c||x and gf I, is the Euclidean metric.
each x E D there is a tangent space TxDd, which can be viewed as the first-order approximation of the hyperbolic manifold at (Bachmann et al., 2019). The tangent space then useful to perform Euclidean operations that we are familiar with but are undefined hyperbolic spaces. hyperbolic space and the langent space at point connected through the exponential map expx T TxD2 D and logarithmic map log D + TxDD specifically defined as follows:
expx(v) x @e taanh Vc
Aallvll
)
v
(1)
2
Velvv/
log2 (y)
tanh -1 (Voll x yll)
x $
(2)
Vexe
- C y
where y, E D E T.D" and O. is the Möbius addition. For convenience, we write ID for D if no confusion arises.
A salient feature of hyperbolic geometry is that it is "thinner" than Euclidean geometry. Visually, more points can squeezed in hyperbolic subspace having the same shape as its Euclidean counterpart, due to the different metrics in the two spaces. We discuss the graph version in Section 3.1 below.
2.2 Graph attention and message passing
Consider graph G EV, where is the set of vertices, E is the set of edges, and each node in V is associated with node feature h.. Recall that GAT is GNN that updates node representations using
3

---

Published in Transactions on Machine Learning Research (12/2024)
message passing by updating edge weights concurrently. Specifically, for one layer of GAT (Veličkovié et al., 2018), the node features are updated as follows:
hy Σ (EN(2)
Oay Wh;):
(3)
exp(ev))
Oyj LkEN(e) exp(enk)
(4)
Gvj = LeakyRel.Ula Why)),
(5)
where || denotes the concatenation operation, denotes activation function, a represents the learnable attention vector, W is the weight matrix for linear transformation and a denotes the normalized attention scores.
This model has been proven to be successful in many graph-related machine learning tasks.
2.3 Hyperbolic attention model
To derive hyperbolic version of GAT, we adopt the following strategy. We perform feature aggregation in the tangent spaces of points in the hyperbolic space. Features mapped between hyperbolic space and tangent spaces using the pair of exponential and logarithmic functions: expx and logx
With this, we denote Euclidean features as ha and hyperbolic features as hn. Then one layer of message propagation in the hyperbolic GAT is as follows (Zhu et al., 2020):
Kep o
Σ JEN(M)
auj log ( & hgD t. bi).
(6)
Crj = LeakyReT.U
a Niw |l hi] x do (lr,D, hy.m));
(7)
2
dp (h,,D, hy,D) tanh (voll hED + hawll),
Vc
Oej = softmax,(ew))
where da the normalized hyperbolic distance, hy logo Be Beh)), while 8 and th represent the Möbius matrix multiplication and addition, respectively.
3 Joint Space Learning
In this section, we propose our joint space learning model. The model relics on comparing two different notions of hyperbolicity: geometric hyperbolicity and model hyperbolicity. We start by introducing the former, which also serves as the motivation for the design of our GNN model.
3.1 Local geometry and geometric hyperbolicity
Gromov's 6-hyperbolicity is mathematical notion from geometry group theory to measure how tree-like metric space is in terms of metric or distance structure (Adeock et al., 2013; Chami et al., 2019). The precise definition is given as follows.
Definition 1 (Gromov 4-point 5-hyperbolicity (Bridson & Haefliger, 1999) p.410). For metric space X with metric d(.,)) ), it is 5-hyperbolic, where 2 0 if the four-point condition holds:
d(x.v) u) + d(z, max(d(x, 2) + d(u, (), d(z,v) + d(x,, ()) + 28,
(10)
for any x. 2, X. X is hyperbolic: it is 5-hyperbolic for some 8 0.
4

---

Published in Transactions on Machine Learning Research (12/2024)
This condition of 5-hyperbolicity is equivalent to the Gromov thin triangle condition. For example, any tree is (0-)lyperbolic, and IR", where 2 2 is not hyperbolic. However, if X is a compact metric space, then X is always d-hyperbolic for some ò large enough such as é = diameter(XX.. Therefore, it is insufficient to just label X as hyperbolic or not. We want to quantify hyperbolicity such that a space with smaller hyperbolicity resembles more of a tree.
Inspired by the four-point condition, we define the cc-version and the 1-version of hyperbolicity as follows. Definition 2. For compact metric space X and y, 2, E X, denote infs2o (10) holds for 2, y, 2,2,t) by Tx(x, W. 2. Define
8x,x0 sup
Tx [x v, 2, t): x.U.i,EXX
6x,1 = Bu.. [Tx 2,t],
where Unif represents the uniform distribution.
In order for these invariants to be useful for graphs, we require them to be almost identical for graphs with similar structures. We shall see that this is indeed the case. Before stating the result, we need a few more concepts.
Let G be the space of weighted, undirected simple graphs Though for most experiments, the given graphs are unweighted. However, aggregation mechanisms such as attention essentially generate weights for the edges. Therefore, for both theoretical and practical reasons, it makes sense to expand the graph domain to include weighted graphs.
For each G EV E G, it has a canonical path metric de, and de makes G into metric space including non-vertex points on the edges. For > 0, there is the subspace Ge of g consisting of graphs whose edge weights are greater than e.
On the other hand, there is a metric on the space G and Ge, called the Gromov-Hausdorf metric (Bridson & Haefliger (1999) p.72). To define it, we first introduce the Hausdorff distance. Let X and Y be two subsets of a metric space (M.d), Then the Hausdorff distance du (X. Y) between X and Y is
d=(X, max d(æ, sup =EX
sup ((X,)); yEY
where d(x, Y)= inr d(x,y) y), d(X,y) infzex d(z, y). The Hausdorff distance measures in the worst case, how far away point in X is away from Y and vice versa.
In general, we want to also compare spaces that do not a priori belong to common ambient space. For this, if X, Y are two compact metric spaces, then their Gromov-Hausdorff distance dcH(X,,, is defined as the infimum of all numbers (f g(Y) for all metric spaces M and all isometric embeddings f: - M.g: Y + M. Intuitively, the Gromov-Hausdore distance measures how far X and Y are from being isometric. The following proved in the Appendix.
Proposition 1. Suppose G and its subspaces have the Gromou-Housdory metric. Then 00 is Lipschitz continuous w.r.t. G E G and 8G, is continuous w.r.l. G G.G. for any >> 0.
Consider a graph G. We fix either 60.00 or 5G.1 as a measure of hyperbolicity, and apply to each local neighborhood of G. To be more precise, is studied (Chen et al., 2020b; Rong et al., 2020) that many popular GNN models have a shallow structure. It customary to have a 2-layer network possibly due oversmoothing (Chen al., 2020a; Chamberlain et al., 2021b; Zeng et al., 2021) and oversquashing (TTpppin et al., 2022) phenomena. In such models, each node only aggregates information in small neighborhood.
Therefore, if we fix small l and let G, be the subgraph of the k-hop neighborhood of E V, then it is more appropriate to study the hyperbolicity 5,, either ôG.. or dG..1, of G., For our experiments, the former is utilized. We call 8, the geometric hyperbolicily at node v. The collection Av = (6, : v E V} allows us to obtain an empirical distribution μl of geometric hyperbolicity on the sample space R20-
For instance, we can build histograms to acquire the distributions as observed in Fig. 2. In Cora, we observe that substantial number of nodes have small (local) hyperbolicity, despite its high global hyperbolicity value
5

---

Published in Transactions on Machine Learning Research (12/2024)
exponential mup
logarithmic map
hy..R
Euclidean space model
2e,5
21
soft space selection
Ze
Zr, logo (Zu.))
BI,II
Hyperbolic space model
(a)
heR
Euclidean space model
scaled by distance similarity based score
Z,,
Ç du (Zu,R, log, (Z..))
<dp expo,,,)
Hyperbolic
scaled by distance similarity based score
Ze,D
space model
(b)
Figure 3: Comparison between JSGNN and GIL (Zhu et al., 2020) in leveraging Euclidean and hyperbolic spaces. Both models utilizes GAT Section 2.2 and HGAT in Section 2.3 for message passing in Euclidean and hyperbolic space respectively. (a) Soft space selection mechanism of JSGNN where trainable selection weights Re B..DD are non-uniform, effectively selecting the better of the two spaces considered. (b) Feature interaction mechanism of GIL where 5,5' R are trainable weights and dp, dR the hyperbolic distance (cf. (8)) and Euclidean distance respectively. The node embeddings of both spaces in GIL are adjusted based on distance, potentially introducing more noise to the branches as there is minimal information in the sub-optimal space to "enhance" the representation in the better space.
(Chami al., 2019; Liu et al., 2022a). Meanwhile, Airport argued be globally hyperbolic, but a large proportion of nodes has large local hyperbolicity. However, this is not contradiction we are considering the local structures of the graph. We call pe the distribution of geometric hyperbolicity. It depends only on G and k.
3.2
Space selection and model hyperbolicity
In this section, describe the backbone of our model and introduce the notion of model hyperbolicity. Our model consists of two branches, one using Euclidean geometry and the other using hyperbolic goometry. We primarily use GAT (cf. Section 2.2) for the Euclidean model and HGAT (cf. Section 2.3) for the hyperbolic model. Other pairs Euclidean or hyperbolic models (e.g., GCN and HGCN) can also be applied to the corresponding branches. In Section 4.3, we show the experimental results on two variants of JSGNN.
After the respective message propagation, we would have sets of updated node embeddings, the Euclidean embedding Zp and the hyperbolic embedding Zz. The two sets of embeddings combined into a single embedding Z = {2v, E VJ through an attention mechanism that serves space selection procedure. The attention mechanism is performed in Euclidean space. Thus, the hyperbolic embeddings are first mapped into the tangent space using the logarithmic map. Mathematically, the normalized allention score indicating whether a node should be embedded in the hyperbolic space år,D or Euclidean space 30.R R is as follows:
W..R = q" tanh (M + b),
(11)
Wp = q" tanh (ahhMMonn lM (Z. + b),
(12)
exp(W,.R)
8,.B = exp(We.R) exp(we,D)
(13)

---

Published in Transactions on Machine Learning Research (12/2024)
exp(w (ep(we.D B.. exp( W,,R) + exp (w,,
(14)
where q refers to the learnable space selection attention vector, M is a learnable weight matrix, b denotes a learnable bias and D. + f,, = 1, for all e E V. The weights SvDD and B. are conditioned to be non-uniform as illustrated in Section 3.4. The two sets of space-specific node embeddings can then be combined via a convex combination using the learned weights as follows:
Zy = 2 + Pe,
(15)
This gives one layer of the model architecture of JSGNN, as illustrated in Fig. 3.
The parameter Be,R, v E V controls whether the combined output, consisting of both hyperbolic and Euclidean components, should rely more on the hyperbolic components or not. We call 8. the model hyperbolicity at the node v. The notion of model hyperbolicity depends on node features as well as the explicit GNN model. Similar to geometric hyperbolicity, the collection F = {B..R E V} gives rise to an empirical distribution vG on [0, 1]. We call vG the distribution of model hyperbolicity.
To motivate the next subsection, from (15), notice that the output depends smoothly on B., If we wish to have similar output for nodes with similar neighborhood structures and features, we want their selection weights to have similar values. On the other hand, we have seen (ef. Proposition 1) that geometric hyperbolicities, which can be computed given G, are similar for nodes with similar neighborhoods. suggests that we may use geometric hyperbolicities to "guide" the choice of model hyperbolicities.
3.3 Model hyperbolicity geometric hyperbolicity
We have introduced geometric and model hyperbolicities in the previous subsections. In this subsection, we explore the interconnections between these two notions.
Let θ be the parameters of proposed GNN model. We assume that the model has the pipclinc shown in Fig. 4. Given node features {hv, v E V} and model parameters 6, the model generates (embedding) features {zv, D }} and selection weights or model hyperbolicity {Bo.R, v E V} the intermediate stage. For each
V, there combination function & such that the final output (0., v E V} satisfies d. = (o(z,,
e, u V}
{Ep, V}
(er V}
[RuR, V)
(8, E)
Figure 4: The model pipeline is shown in the (blue) dashed box, while the geometric hyperbolicity can be computed independently of the model.
In principle, we want to compare (B.R. E VV and (6,, v E V) so that the geometric hyperbolicity guides the choice of model hyperbolicity. However, comparing pairwise 8, and 8, for each v V may lead to overfitting. An alternative is to compare their respective distributions va and PG,, even coarser statistics (e.g, mean) of ng and MG (cf. Fig. 5). The latter may lead to underfitting. We perform an ablation study the different comparison methods in Section 4.5.
We advocate choosing the middle ground by comparing the distributions μa and VG- The former can be computed readily as long as the ambient graph G is given, while the latter is part the model that plays a crucial role in feature aggregation each node. Therefore, Nc can be pro-determined but not vG. We propose to use the known μG to constrain PG and thus the model parameters O A widely used comparison tool the Wasserstein metric.
7

---

Published in Transactions on Machine Learning Research (12/2024)
VE vs + PG H Stat. vG
H PG
H Stat. of μG
Figure Different ways of comparing geometric and model hyperbolicities.
Desinition (Wasserstein distance). Given p 1, the p- Wasserstein dislance metric Villani (2008) mensures the difference between two different probability distributions Gao et al. (2021). Let IIlve, PG) be the set of all joint distributions for random variables and where and PG- Then the p- Wasserstein distance between PG and vG is as follows:
W Wp(ve,Pa) = - inf B(aa)))llx=- - y/lp 1/p (Ell((re.pG)
(16)
To compute the Wasserstein distance exactly is costly given that the solution of an optimal transport problem is required (Rowland et al., 2019; Chen et al., 2022b). However, for one-dinensional distributions, the p- p-Wasserstein distance can be computed by ordering the samples from the two distributions and then computing the average p-distance between the ordered samples (Kolouri et al., 2019; Rowland et al., 2019).
In ideal circumstances, considering the distributions do not lose much information. We first notice that for both B., and de, a smaller value means more hyperbolic in an appropriate sense. Suppose BI is increasing w.r.t. dey ie., & 6g implies that B.R BuRR- Then, W2("e, V M EvevlBuR 5,12.
3.4 Non-uniformity of selection weights
A node is considered to be more suitable to be embedded in the hyperbolic space when B.D > B..z. Meanwhile when B. < Sv,R, the node is considered to be Euclidean. Nevertheless, to align with our motivation that each node can be better embedded in one the two spaces and the less suitable space would result in distortion in representation, we require JSGNN to learn non- uniform attention weights, meaning that each pair of attention weights (Be.D, B,.R) should signíficantly deviate from the uniform distribution. This is because soft selection without a non-uniformity constraint may result in the assigument of nodes to be partially Euclidean and partially hyperbolic with P..R Bu,D 0.5. Hence, we include additional component to the standard loss function encouraging non-uniform learned weights as follows:
Lm
Σ VEV
( + B.D))
(17)
V
Since -1 - - P.DD) I -0.5 and BR, + P. = 1, minimizing the term would favor non-uniform attention weights for each node.
In summary, we may combine hyperbolicity matching discussed in Section 3.3 and the non-uniformity loss to form the loss function to optimize JSGNN.
Loverall = Ltask + Wnu Luu + Wwas W2 pG),
(18)
where Ltask is the task-specific loss, while Wau and Wwas balancing factors. For the node classification task, Irask refers to the cross-entropy loss over all labeled nodes while for link prediction, it refers to the cross-entropy loss with negative sampling. This completes the description of the JSGNN model.
We speculate that the non-unifora component Lmu should push the model hyperbolicities towards the two extremes and 1. On the other hand, as we have seen in Section 3.3, compute W2(":: Ma), we need order (d,)evv, (Be.))eEV respectively, and compute their pairwise differences. Therefore, W2(PG.Ha) aligns the shapes of va and pG-
8

---

Published in Transactions on Machine Learning Research (12/2024)
Table 1: Node classification result on Cora, Citeseer and Pubmed datasets. Performance score averaged over ten runs. The best performance boldfaced while the socond-best performance is underlined.
Standard split
60/20/20% split
Mothod
Cora
Citrseer
Puhmed
Cora
Citeseer
Pubmed
GCN
81.53 = 0.84
70:47 + 0.64
78.30
0.63
91.99
0.79
84.13
0.98
58.79
GAT
81.68 = 1.06
70.96 + 0.96
78.05
0.50
91.63
0.57
83.93
0.85
89.99
1.31
GraphSAGE
1.06
65.26
2.91
77.90
0.71
91.25
84.08
89.62
CurGNN
81.58 0.51
71.14 + 0.67
78.17
0.48
91.60
0.25
84.18
0.37
86.65
0.14
CGNN
82.15 = 0.60
71.31
1.16
78.34
0.83
91.96
0.27
84.29
0.34
86.86
0.16
HIGN
79.28 0.77
70.00 + 0.74
77.45
+ 1.40
89.73
0.84
80.27
0.21
88.27
0.51
HGCN
78.68 0.77
67.25 + 1.45
76.72 0.92
91.37 0.28
83.65
0.52
86.83 0.31
HGAT
78.81 1.49
68.16 + 1.34
77.43 1.20
90:27 0.81
81.29 0.79
86.27 0.47
LGCN
78.93 = 0.79
68.59 + 0.64
78.08 = 0.65
92.35 + 0.57
85.03 0.28
89.59 + 0.11
X-GCN (D)
78.64 0.83
67.17 + 0.73
78.01 + 0.67
89.16 0.59
84.57
0.20
88.37 + 0.37
DeepHGGN
80.66 0.89
72.11 = 0.60
78.13 = 1.67
88.51 + 1.52
80.45
0.36
$6.90 + 0.42
x-GCN (D x R
78.71 1.02
66,96 + 1.13
77.67 = 0.74
88.85 + 0.85
84.01 = 0.81
85.59 + 0.53
GIL
79.97 = 1.93
67.54 + 1.23
76.62 + 0.81
91.90 + 0.84
82.39
0.90
87.39 + 0.21
JSGNN (GCN+HGCN)
81.79 = 0.80
70.55 + 1.09
78.38 + 0.74
93.26
0.92
84.95
0.31
89.68 + 0.60
JSCNN (GAT HGAT)
82.94 + 0.55
71.26 + 1.13
78.57 + 0.90
93. 10 + 0.86
85.10 + 0.64
90.53 = 0.32
4
Experiments
In this section, we evaluate JSGNN on node classification (NC) and link prediction (LP) tasks. Datasct statistics, model settings, and model size and complexity are discussed in Appendix A and Appendix B.
4.1 Datasets
A total of seven benchmark datasets are employed for both NC and LP. Specifically, three citation datasets: Cora, Citeseer, Pubmed; a flight network: Airport; discase propagation tree: Discase; an Amazon co-purchase graph dataset: Photo; and a coauthor dataset: CS.
4.2 Baselines
For JSGNN, we consider two variants: GAT as the Euclidean model with HGAT as the hyperbolic model, and GCN as the Euclidean model with HGCN as the hyperbolic model. We compare against the following models: (a) Euclidean methods: GCN (Kipf & Welling, 2016), GraphSAGE (Hamilton al., 2017) and GAT (Veličković al., 2018); (b) hyperbolic models: HGCN (Chami et al., 2019), HGNN (Liu et al., 2019), HGAT (Zhang et al., 2021a), LGCN (Zhang et al., 2021b), DeepHGCN (Liu el al., 2024) and R-GCN (D)6) (Bachmann et al., 2019), which is a constant (negative) curvature graph neural network based on the x-stereographic model; (c) CurvGN (Ye et al., 2020) and CGNN (Li et al., 2021), which utilizes graph curvature information to filter messages differently based upon different local structures; (d) mixed models: GIT. (Zhu et al., 2020) and k-GCN (1D16, x TR 1R:1)), which similar JSGNN, leverages multiple spaces or a mixed curvature space. k- GCGNN (ID26 1216) employs two-component product space of negative and zero curvature.
4.3 Node classification
For the node classification task, each of the nodes in dataset belongs to one of the C' classes in the dataset. With the final set of node representations, we aim to predict the labels of nodes that are in the testing set.
To test the performance of each model under both semi-supervised and fully-supervised settings, two data splits are used in the node classification task for the Cora, Citeseer and Pubmed datasets. In the first split, we followed the standard split for semi-supervised settings used Kipf & Welling (2016); Veličković et al. (2018); Monti al. (2017); Chamberlain al. (2021b): Zhu al. (2020): Chamberlain et al. (2021a); Hamilton et al. (2017); Liu al. (2022b); Feng et al. (2020). The train set consists of 20 train examples per class while the
9

---

Published in Transactions on Machine Learning Research (12/2024)
Table 2: Node classification result on CS, Photo, Airport and Disease datasets. 00M corresponds to out-of-
memory.
Method
CS
Photo
Airport
Disease
GCN
96.53 1 0.10
94.05
1 0.27
79.62 i 1.28
83.32 1 1.37
GAT
96.36 1 0.38
94.45
+ 0.92
83.07 + 1.52
86.05 + 1.08
GraphSAGE
96.45 + 0.91
96.13
+ 1.61
81.73 + 0.98
83.47 + 1.77
CurvGN
96.91 + 0.09
94.21 + 0.22
87.25 +: 0.88
84.35 + 3.20
CGNN
96.27 1 0.08
93.93 1 0.18
87.21 1: 1.08
85.75 1 2.08
HGNN
96.72 1 0.19
94.74 1 0.66
84.37 1 1.19
87.40 1.66
HGCN
96.58 1: 0.10
95.27 0.25
89.30
1.52
87.93 1.61
HGAT
96.65 1 0.15
96.62 0.28
89.31
1.09
90.04 1.50
LGON
00M
96.71 0.24
88.53
1.26
91.15 1.02
G.GON (D:))
97.04 + 0.10
95.56
0.54
89.08
1.15
92.39 0.73
DeepHGGN
95.80 1: 0.88
94.37 +: 1.43
90.24
1.88
90.38 1.76
R-GON (D x _
96.97 1. 0.10
94.31 1 0.41
88.38 1 0.62
89.47 1 1.56
GIL
95.83 1 0.30
94.41 1 0.57
90.78 1 1.74
90.67 1 1,98
JSCNN (GCN (HGN))
97.08 1: 0.04
95.69
+ 0.22
90.59 1: 1.75
90.73 + 1.44
JSGNN (GAT+HGAT)
97.40 + 0.14
97.16 + 0.44
90.33 + 1.61
90.88 + 1.54
validation set and test set consist of 500 samples and 1,000 samples, respectively.' Meanwhile, in the second split, all labels are utilized and the percentages of training. validation, and test sets are set as 60/20/ 2002 For the Photo and CS datasets, the labeled nodes are also split into three sets where 60% of the nodes made up the training set, and the rest of the nodes were divided equally form the validation and test sets, Airport and Discase datasets were split in similar settings as Zhu al. (2020).
In Table 1 and Table 2, the mean accuracy with standard deviation is reported for node classification, except for the case of Airport and Disease datasets where the mean F'l score is reported. Our empirical results demonstrate that JSGNN frequently outperforms the baselines, especially HGAT and GAT which are the building blocks of JSGNN. Even though the performance of the variant JSGNN (GCN+HGCN) is often slightly lower than JSGNN HGAT+HGAT), we have similarly observed building blocks GCN and HGCN. This is not necessarily observed for other mixed space models. Thus, this shows the superiority of not only using both Euclidean and hyperbolic spaces but also our method of incorporating the two spaces for graph learning as compared to GIL and k-GCN (ID x I.
consistently outperform its
We also observe that Euclidean models such as GCN, GAT, GraphSAGE, CurvGN and CGNN perform better than hyperbolic models in general on the Cora, Citeseer, and Pubmed datasets for both splits. Meanwhile, hyperbolic models achieve better results on the CS, Photo, Airport, and Disease datasets.. This means that Euclidean features are more significant for representing Cora, Citescer and Pubmed datasots while hyperbolic features are more significant. for the others. Nevertheless, JSGNN is able perform relatively well across all datasets. We note that JSGNN exceeds the performance of single-space baselines on all datasets except for Disease. This can be explained by the fact that Disease consists of a perfect tree and thus, does not exhibit different hyperbolicities in the graph. However, JSGNN still outperforms 3 hyperbolic benchmarks and all the other mixed models.
We also particularly note that the difference in results between single-space models using only the Euclidean embedding space and hyperbolic models is not significant. This means that many of the node labels can be potentially predicted even without the best representation from the right space. This might be the reason why the gain in performance for the node classification task is not exceptional from embedding nodes in the better space. Nevertheless, still see improvements in predictions for cases where there is mixture of local hyperbolicities. Moreover, embedding nodes in more suitable space can benefit other tasks that require more accurate representations such as link prediction.
Note that the top results on httpss//paperewittcoode (either semi-superviser settings with larger number training samples fully-supervised settings such the split) which give much higher accuracies
used different data splits
10

---

Published in Transactions on Machine Learning Research (12/2024)
Table 3: Link prediction result averaged over ten runs.
Method
Cora
Citesocr
Pubmed
Airport
Discuse
GCN
88.22 1.01
90.60 + 1.10
87.63 + 3.25
91.79 + 1.48
61.60 + 3.76
GAT
85.47 + 2.28
85.31 4: 1.89
85.30 +: 1.46
93.70 1 0.65
61.23 4: 2.75
GraphSAGE
88.94 0.81
91.61 1: 1.00
88.42 + 1.14
91.63 1 0.81
68.31 +: 2.94
CurvGN
94.40 2.13
95.38 + 2.33
94.55
0.89
93.90 + 0.37
95.47 + 0.81
CGNN
94.26 1.36
96.54 + 0.78
94.71 + 3.14
95.13 + 0.97
95.35 + 1.22
HGNN
91.48 0.38
93.63 1 0.14
92.95 + 0.35
96.31 + 0.30
82.98 = 0.98
HGCN
93.72 + 0.26
96.72 1: 1.69
96.68 +: 0.04
97.55 1 0.08
87.14 :: 1.34
IIGAT
94.06 + 0.11
95.60 1 0.20
95.78 + 0.05
97.86 1: 0.08
86.61 : 1.67
LGCN
93.10 + 0.30
93.40 + 0.70
95.45 + 0.08
97.88 1: 0.19
95,99 + 0.58
K-GON (D!
92.43 + 0.63
94.38 1 0.51
94.89 + 0.07
96.78 1 0.19
93.58 - 0.31
x-GCN [D x 2%)
91.32 + 0.38
92.87 1 0.27
93.53 + 0.06
97.17 1 0.10
90.15 = 0.77
GIL
98.04 + 1.64
99.95 + 0.09
92.50 + 0.50
97.20 + 1.04
100.00 + 0.00
JSGNN (GCN+HGCN)
98.83 + 0.92
99.97 + 0.09
97.67 + 0.08
98.94 + 0.91
100.00 + 0.00
JSGNN (GAT+HGAT)
99.43 0.21
99.98 + 0.05
96.95 + 0.03
99.26 + 1.23
99.97 + 0.08
Table 4: Ablation study of JSGNN (GAT+HGAT) for node classification task. Cora, Citeseer and Pubmed on the standard split.
Method
Cora
Citesser
Pubmed
Airport
Discuse
JSGNN (GAT+
97.40
1 0.14
97.16
1 0.44
82.94 a 0.55
71.26
1.13
78.57
0.90
90.33 1 1.61
90.88
1:554
w/a NU W2
97.15
0.10
95.95
+ 0.41
81.65
+ 1.08
70.87
1.22
78.14
1.02
89.67 + 1.26
89.85
1.61
w/a Wa
82.36
71.15
1.17
0.53
90.02
2.22
w/a NU
97.38 0.15
96.42
1: 0.37
82.67
1 0.51
70.86
1.45
78.48
0.47
89.98
1.72
90.37
2,12
4.4 Link prediction
We employ the Farmi-Dirac decoder with a distance function model the probability of an edge basod on our final output embedding, similar Zhu et. al. (2020); Sun et al. (2022); Chami et al. (2019). The probability that an edge exists is given by P(e.j E E () (e(d(z)-2))-))/ + 1)- where r,t > 0 are hyperparameters and d is the distance function. The edges of the datasets are randomly split into 85/5/10% for training, validation, and testing. The average ROC AUC for link prediction is recorded in Table 3. We observe that JSGNN HGAT+HGAT) performs better than the baselines in most cases. For the link prediction task, we notice that hyperbolic models consistently outperform Euclidean models by a significant margin. Moreover, Euclidean mothods such as CurvGN and CGNN benefit from using topological information during lcarning. Empirical results also suggest that predicting the existence of edges seems to benefit from dual space models, L..., GIL and JSGNN, except for the case of R-GCN (ID26 R10).
This finding is similar to that reported in Bachmann et al. (2019); Xiong et al. (2022) where despite slightly different settings, the constant (negative) curvature k-stereographic model frequently outperforms the k- x-GCN leveraging on the product of multiple constant curvature spaces. We hypothesize that the simple concatenation to combine the embeddings of the two different component spaces k-GCN (D46 x IR might be insufficient and might have resulted noise from the other space being passed the negatively curved space which was performing well standalone as seen in x-GCN (D16). As such, our aim learn select the better space for each node is potentially capable of offering better representations with reduced distortions.
4.5 Ablation study
We conduct an ablation study the node classification task by introducing three variants of JSGNN (GAT+HGAT) to validate the effectiveness of the different components introduced:
Without the non-uniformity constraint (w/o NU): This does not enforce the model to learn non- uniform selection weights.
Without the Wasserstein metric (w/o W2): The learning of model hyperbolicity is not guided by geometric hyperbolicity.
11

---

Published in Transactions on Machine Learning Research (12/2024)
Without the non-uniformity loss and Wasserstein distance (w/o NU & W2): Only guided by the cross entropy loss, i.e., Whn = O, Wwass = 0 (cf. (18)).
Table 4 summarizes the results of our study, from which we observe that all variants of JSGNN (GAT+HGAT) with some components discarded perform worse than the full model. Morcover, JSGNN (GAT+HGAT) without W2 always achieves belter results than JSGNN (GAT+HGAT) without NU and W2, signifying the importance of selecting the better of the two spaces instead of combining the features with relatively uniform weights. Similarly, JSGNN (GAT+HGAT) without NU performs better than JSGNN (GAT+HGAT) without NU and 22 in most cases, suggesting that incorporating geometric hyperbolicity through distribution alignment does help to improve the model. A comparison of the time taken for each variant is provided in Appendix B.
To further analyze our model, we present a study regarding our method of incorporating the guidance of geometric hyperbolicity through distribution alignment. The result is as seen in Table 5. We test and analyze empirically different variants of our model based the different comparisons shown in Fig. 5. Pairwise match indicates minimizing the mean squared error between elements of Ia and Av (without sorting) while mean match minimizes the squared loss between the means of I'G and Av. We observe that comparing the distributions VG and μa consistently outperforms comparing their mean, demonstrating the insufficiency of utilising coarse statistics for supervision. Secondly, pairwise matching gave better results than mean matching, though still lower than ristribution matching, suggesting the importance of fine-scale information yet, a need to avoid potential overfitting.
Table 5: Node classification results of different comparison methods to incorporate geometric hyperbolicity to guide model hyperbolicity.
Dataset
Pairwise match
Distribution
Mean match
Cora
82.35 + 1.06
82.94 = 0.55
81.36 + 1.50
Citesver
70.06 2.05
71.26
1.13
69.64 1 L.18
Pubmed
78.46 0.86
78.57
0.90
78.08 # 0.62
Aiport
90.13
1.53
90.33
1.61
80.31 1 2.22
Disease
90.66
1.91
90.88
1.54
87.53 + 6.24
Photo
96.17 + 0.23
97.16 + 0.44
95.96 + 0.59
CS
97.20 + 0.16
97.40 + 0.14
97.17 + 0.11
4.6 Analysis of hyperbolicities
We have speculated the effects of different components of our proposed model at the end of Section 3.4. To verify that our model can learn model hyperbolicity that is non-uniform and similar in distribution as geometric hyperbolicity, we analyze the learned model hyperbolicities (Br,R)vEV of JSGNN (GAT+HGAT) and the model w/o NU & W2 for the node classification task. Specifically, we extract the learned values from the first two layers of JSGNN and its variant for ten separate runs. The learned values from the first two layers were then averaged before determining W2(vG, Unif) and W2(vo.HG).
In Fig. 6, it can be inferred that JSGNN's Iearned model hyperbolicity is always less uniform than that of the model w/o NU W2 given JSGNN's 's larger W2(PG, Unif) score, demonstrating a divergence from uniform distribution. Meanwhile, for most cases, JSGNN's W2(vo, HG) smaller than that of the model w/o NU & W2, suggesting that the shape between PG and na of JSGNN is relatively more similar. At times, JSGNN's We(Me. (G) is larger than the model w/o NU & W2, suggesting tradeoff between NU and W2 as choose the optimal combination for the model's best performance.
5 Conclusion
In this paper, we have explored the learning of GNNs in a joint space setting given that different regions of a graph can have different geometrical characteristics. In these situations, it would be beneficial to embed different regions of the graph in different spaces that are better suited for their underlying structures, to
12

---

Published in Transactions on Machine Learning Research (12/2024)
0.5
ISGM
0.5
JoONA NU
0.4
Wolog. HG
Walag,
0.3
0.2
0.1
Cora
Creseer
Airport
Ussaase
Photo
Cate
Ulteseer
Airport
Disease
Phoce
(a)
(b)
Figure 6: Analysis of hyperbolicities on different datasets. (a) W2 Unif). (b) W2(vG, MG).
reduce the distortions incurred while learning node representations, Our method JSGNN utilizes soft attention mechanism with non-uniformity constraint and distribution alignment between model and geometric hyperbolicities to select the best space-specific feature for each node. This indirectly finds the space that is best suited for each node. Experimental results of node classification and link prediction demonstrate the effectiveness of JSGNN against various baselines. In future work, we aim further improve our model with adaptive mechanism to determine the appropriate, node-level specific neighborhood to account for cach node's s hyperbolicity. Limitations ions are discussed in Appendix F.
6
Acknowledgements
The first author is supported by Shopee Singapore Private Limited under the Economic Development Board Industrial Postgraduale Programme (EDB IPP). The programme is a collaboration between Shopee and Nanyang Technological University, Singapore. This research is supported in part by the Singapore Ministry of Education Academic Rescarch Fund Tier grant MOE-T22FP2222000002 This work carried out during the first author's participation in the EDB IPP.
References
Aaron В. Adcock, Blair D. Sullivan, and Michael W. Mahoney. Tree-like structure in large social and information networks. In 2013 IEEE 13th International Conference on Data Mining, pp. 1-10, 2013. doi: 241109/TCDMM2001.77
Gregor Bachmann, Gary Bécigneul, and Octavian-Eugen Ganca. Constant curvature graph convolutional networks. In Proceedings of the 7th International Conference on Learning Represenlations, 2019.
M. Bridson and A. Haefliger. Metric Spaces of Non-Positive Curvature. Springer, 1999.
Benjamin Paul Chamberlain, James Rowbottom, Davide Eynard, Francesco Di Giovanni, Dong Xiaowen, and Michael M Bronstein. Beltrami flow and neural diffusion on graphs. Proceedings of the Thirty-fifthh Conference on Neural Information Processing Systems (NeurIPS), 2021a.
Benjamin Paul Chamberlain, James Rowbottom, Maria I. Gorinova, Stefan D Webb, Emanuele Rossi, and Michael M. Bronstein. GRAND: Graph neural diffusion. In The Symbiosis of Deep Learning and Differential Equations, 2021b.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. In Advances in Newral Information Processing Systems, pp. 4869-44 4880, 2019.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Mcasuring and relieving the over-smoothing problem for graph neural networks from the topological view. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):3438 3445, 2020a. doi: 10. 10.1609/aaai, (0.11009/aaa..344444.77777
13

---

Published in Transactions on Machine Learning Research (12/2024)
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learning, pp. 1725-1735. PMLR, 2020b.
Yankai Chen, Menglin Yang, Yingxue Zhang, Mengchen Zhao, Ziqiao Meng, Jianye Hao, and Irwin King. Modeling scale-free graphs with hyperbolic geometry for knowledge-awar recommendation In Proceedings of the 15th ACM International Conference on Web Search and Data Mining, WSDM '22, pp. 94- 94-102, 2022a. ISBN 45034300...
Yidong Chen, Chen Li, and Zhonghua Lu. Computing wasserstein-p distance between images with linear cost. In 2022 LEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 509-518, CVVPP55688.2022.00000
2022b. doi:
Hyunghoon Cho, Benjamin DeMeo, Jian Peng, and Bonnie Berger. Large-margin classification in hyperbolic space. In Proceedings of the 22nd International Conference Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Reseorch, pp. 1832-1840. PMLR, 16-18 Apr 2019.
Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, Evgeny Kharlamov, and Jie Tang. Graph random neural networks for semi-supervised learning graphs. In Proceedings of the 34th International Conference on Newral Information Processing Systems. Curran Associates Inc., 2020. ISBN 9781713829546.
Ji Gao, Xiao Huang, and Jundong Li. Unsupervised graph alignment with wassorstein distance discriminator. In Proceedings of the 27th ACM SIGKDD Conference on Knowledlge Discovery & Dala Mining, KDD '21, pp. 426 435, New York, NY, USA, 2021, Association for Computing Machinery. ISBN 9781450383325. doi: 10.1145/3445488346733..
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, ICML'17, pp. 1263 1272. JMLR.org, 2017.
W. L. Hamilton, R. Ying, and J. Leskovec. Inductive representation learning large graphs In Advances in Neural Information Processing Systems, 2017.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks, 2016.
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. Generalized sliced wasserstein distances. In Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
John M. Lee. Introduction to Riemannian Manifolds. Springer Cham, 2018.
Haifeng Li, Jun Cao, Jiawei Zhu, Yu Liu, Qing Zhu, and Guohua Wu. Curvature graph neural network. Information Sciences, 2021. doi:
1277.7
Jiahong Liu, Menglin Yang, Min Zhou, Shanshan Feng, and Philippe Rournier-Viger Enhancing hyperbolic graph embeddings via contrastive learning, 2022a. URL https: httpp/////xxi.oor///////00// 08554.
Jiaxu Liu, Xinping Yi, and Xiaowci Huang. Deephgen: Toward deeper hyperbolic graph convolutional networks. IEEE Transactions on Artificial Intelligence, pp. 14, 2024. doi: 10.1100/AA.2222.444222
Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 8230 2230-8241, 2019.
Songtao Liu, Rex Ying, Hanze Dong, Lanqing Li, Tingyang Xu, Yu Rong, Pellin Zhao, Junzhou Huang, and Dinghao Wu. Local augmentation for graph neural networks. In International Conference on Machine Learning, 2022b.
14

---

Published in Transactions on Machine Learning Research (12/2024)
F. Monti, D. Boscaini, J. Masci, E. Rodola, J. Svoboda, and M. M. Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In 2017 TEEE Conference on Compuler Vision and Pottern Recognition (CVPR), pp. 5425 5434, Los Alamitos, CA, USA, jul 2017. LEEE Computer Society. doi:
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In International Conference on Learning Representations, 2020. URL https://openreviign
Mark Rowland, Jiri Hron, Yunhao Tang, Krzysztof Choromanski, Tamas Sarlos, and Adrian Weller. Orthogonal estimation of wasserstein distances. In Proceedings of the 22nd Intervational Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 186-195. PMI.R, 16 18 Apr 2019.
Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, and Rajiv Shah. Exploring the scale-free nature of stock markets: Hyperbolic graph learning for algorithmic trading. In Proceedings of the Web Conference, WWW 221, pp. 11 22, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383127. doi: 10.1145/3444 /44/444443311..550555
Li Sun, Zhongbao Zhang, Junda Ye, Hao Peng, Jiawei Zhang, Sen Su, and Philip S. Yu. A scle-supe mixed-curvature graph neural network. In Association for the Advancement of Artificial Intelligence (AAAI), 2022.
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. Tn International Conference Learning Representations, 2022. URL
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
Cédric Villani. Oplämal Transport: Old and New. Springer Science & Business Media, 2008
Bo Xiong, Shichao Zhu, Nico Potyka, Shirui Pan, Chuan Zhou, and Steffen Staab. Pseudo-riemamiin graph convolutional networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
wwwpeerriii..eee//rrrnnrr met/forum?id=
KeIuNChobiH.
Menglin Yang, Min Zhou, Zhihao Li, Jiahong Liu, Lujia Pan, Hui Xiong, and Irwin King. Hyperbolic graph neural networks: A review of methods and applications, 2022.
Ye, Kin Sum Liu, Tengfei Ma, Jie Gao, and Chao Chen. Curvature graph network. In Proceedings of the 8th International Conference on Learning Representations (ICLR 2020), April 2020.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovee. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '18, pp. 974 983, 2018. ISBN 450355505.
Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kannan, Viktor Prasanna, Long Jin, and Ren Chen. Decoupling the depth and scope of graph neural networks. In Advances in Neural Information Processing Systems, 2021.
Yiding Zhang, Xiao Wang, Chuan Shi, Xunqiang Jiang, and Yanfang Fanny Ye. Hyperbolic graph attention network. IEEE Transactions on Big Data, 2021a.
Yiding Zhang, Xiao Wang, Chuan Shi, Nian Liu, and Guojie Song. Lorentzian graph convolutional networks. Proceedings of the Web Conference 2021, WWW '21, pp. 1249 1261, New York, NY, USA, 2021b. Association for Computing Machinery. ISBN 9781450383127. doi: 10.1145/3442381.344900
Shichao Zhu, Shirui Pan, Chuan Zhou, Jia Wu, Yanan Cao, and Bin Wang. Graph geometry interaction learning. In Advances in Neural Information Processing Systems, 2020.
15

---

Published in Transactions on Machine Learning Research (12/2024)
A
Dataset statistics and model settings
Dataset statistics are provided in Table
Table 6: Dataset statistics.
Dataset
Nodes
Edges
Classes
Features
Cora
2708
5429
1433
Citeseer
3327
4732
3703
Pubmed
19717
44338
3
500
Aiport
3188
18631
4
4
Disease 1044
1043
2
1000
Photo 7650
119081
8
745
CS
18333
81894
15
6805
For all models, the hidden units are set 16. We set the carly stopping patience to 100 epochs with a maximum limit of 2000 epochs. The hyperparameler settings for the baselines are the same as Zhu al. (2020) if given. The only difference is that the hyperparameter h-drop for GIL in Zhu et al. (2020) (which determines the dropout to the weight associated with the hyperbolic space embedding) is set to for all datasets as setting large value essentially explicitly chooses one single space. Else, the hyperparameters are chosen to yield the best performance. For JSGNN, we perform a grid search on the following search spaces: Learning rate: [0.01, 0.005]; Dropout probability: [0.0, 0.1, 0.5, 0.6]; Number of layers: [1, 2, 3]; whu and wwss: [1.0, 0.5, 0.2, 0.1, 0.01, 0.005]; q (cf. (11)): [16, 32, 64). The Wasscrstein-2 distance employed in all variants of JSGNN.
The source code can be found at http::://gittab..c/////lle////// mentation is based on the setup in https://githuu: httpt/tbbb..o///hhersee////////a
BaceaiNN_iiee__SeS Saaeeedd_ppee_G_aeee GNN. The imple-
B Complexity, run-time and model size
We first incur overhead to compute the geometric hyperbolicity 8 for cach node. We follow the approach as described below? rather than using the naive implementation with time complexity of O(V2). Instead of considering all possible 4-tuples in the graph, we heuristically sample K 4-tuples from each nodes' two-hop subgraph. Consequently, the complexity is O(N) x (O(N2) + O(K x Esubgraph)) ) z O(NKd²) where d denotes the average degree in the graph, K denotes the number of samples and Esubgraph represents the number of edges in the two-hop subgraph each node. Assuming the graph is sparse, d is small. This runtime can be further optimized by parallelization, as each node's calculation is independent. Also, notice that this step is performed only once during pre-processing, and the step is model-agnostic. For dataset, the computed Jeev can be stored and re-used for different trainings and also for different base models. Moreover, if the dataset is updated, we only need to adjust 8, for nodes whose neighborhood structures are changed during the update, which can be done efficiently.
For training the model, we usually use two base models Me (e.g., GCN) and M₁ (e.g., HGCN) for handling Euclidean and hyperbolic structures respectively. The complexity of the message passing our model is OCC, Cz), where C. and Cn are the message passing complexities of M. and Mr respectively.
Regarding the implementation of the extra loss terms, calculating the Wasserstein distance for one-dimensional distributions requires time complexity of O(N N) where N is the number of nodes in the graph, primarily dominated by the time needed to sort the distributions. Meanwhile, obtaining the non-uniformity loss has a time complexity of O(N).
In Table 7, we present the time taken for 300 epochs across different variants of JSGNN that we examine in our ablation study in Section 4.5. The results show that, despite the added time complexity introduced by
"https://gtthubb.om Reseeaa Research htgg///i///mattee//t//////yrree/
16

---

Published in Transactions on Machine Learning Research (12/2024)
Table 7: Time taken (in seconds) for 300 epochs for node classification task across different model variants.
Method
Cora
Citeseer
Pubmed
CS
Photo
JSGNN HGAT+HGAT)
18.53 + 1.05
21.60 + 0.62
24.73
3.49
49.35 + 0.62
37.03 = 0.51
w/o W2
17.29 + 0.66
20.06 + 0.27
23.56
1.85
47.31 + 0.34
34.45 = 0.41
w/o NU
18.06 + 0.13
19.98 i 0.32
23.42
+ 3.49
48.36
+ 0.41
36.71 = 0.32
w/o NU & W/2
17.68 + 0.25
19.67 + 0.34
23.09
1.97
46.97 + 0.60
34.16 = 0.23
Table 8: Time taken (in seconds) training cach model. The ratio is between the run-time of JSGNN and that of HGCN plus GCN.
Method
Cora
Cileseer
Pubmed
GCN
1.08
1.07
1.10
HGCN
3.87
3.96
4.44
JSGNN
18.53
21.60
24.73
Ralão
3.74
4.29
4.45
additional loss terms, JSGNN incurs only a small increase in time compared its simpler variants across the evaluated datasets.
Additionally, in Table 8, we show the empirical run-time comparison. We see that the run-time ratio between JSGNN and HGCN plus GCN remains stable for graphs of different sizes. Therefore, the scalability of the well-studied models GCN, and HGCN implies that of JSGNN.
We show in Table 9 the model size comparison. We see that our model size is comparable to all the other hybrid models and approximately double those the single-type models.
C Distributions of hyperbolicity
We include Table that tabulates the percentage of nodes whose geometric hyperbolicity &, (approximately) the prescribed value. Here, the hyperbolicity taken neighborhood of each node account for the local nature of message passing. We see, for example, for the Cora dataset, there are more than 52.4% of nodes whose 6, is at least 1. Hence, Euclidean model is preferred. On the other hand, there are 23.0% of nodes whose neighborhood is tree-like, and this is why hybrid model can further improve the performance.
D Proof of Proposition 1
Proof. We first consider 60 For two graphs G1 = (V1, E1) and G2 (V2, E2), let G: f2 G2 M be isometeric embeddings into a metric space (M, d) such that dG G2) (G(( Denote d (2) by 77. For z, 2, t in G1, there are x', 1 2 t' E G2 such that d(
Table 9: Number of trainable parameters.
Method
Cora
Citeseer
Pubmed
CS
Photo
JSGNN (GAT+HGAT)
47128
119696
17720
219967
25928
R-GCN
46112
118720
16128
218272
24128
GIL
46700
119286
19155
219036
25380
GAT
23452
59752
8843
109244
12144
GCN
23335
59638
8339
109151
12072
HGNN
23335
59638
8339
109151
12072
HGCN
23336
59638
8339
109151
12072
HGAT
23431
59734
8435
109199
12120
17

---

Published in Transactions on Machine Learning Research (12/2024)
Table 10: Hyperbolicity distribution in each dataset computed based on each node's two-hop subgraph. The lower the hyperbolicity, the more hyperbolic the node's neighborhood is.
Dataset
0
0.5
1.5
2
2.5
Disease
1.0000
0.0000
0.0000
0.0000
0.0000
0.0000
Cora
0.2296
0.2230
0.5240
0.0225
0.0007
0.0000
Citescer
0.5648
0.1350
0.2958
0.0045
0.0000
0.0000
Pubmed
0.3064
0.2225
0.3880
0.0814
0.0017
0.0000
Airport
0.0471
0.2566
0.6963
0.0000
0.0000
0.0000
Photo
0.0251
0.0482
0.8825
0.0442
0.0000
0.0000
CS
0.0657
0.1284
0.5718
0.2332
0.0009
0.0000
d(fi(e) ((()))) are all bounded by 17. We now estimate:
da. (2,1) (2, d(f] ((f(4)))
s +
+ 44
= das(# (x' ( + da, (=' +
S max foax|das(0 (x', + dc, (1/ do. (2' ./'' + (c,(,
+0 + 4n)
(19)
S
d(fi()))
f()( ((f()))))))))))
+ 26 + 87)
= max max(da, (I. 2) + da, (v, 1), da, (2.1) + da, (x,1))
+ +66 + 81.
Therefore, o0 I 602.20 + 4n. By the same argument swapping the role of G1 and G2, we have 862,00 S 80,,00 + 4n.. Therefore j0,, - do2. S 4n and 80 is Lipschitz continuous w.r.t. G.
The proof of the continuity of 8G.1 is more involved. Consider G1 and G2 in G.. Let fi, f2, (M, d), n be as carlier and assume 17 < c, for example, 1) for a is smaller than all the numerical constants in the rest of the proof.
We adopt the following convention: for any non-vertex point of a graph, its degree is 2. By subdividing the edges of G1 and G2 if necessary, we may assume that the length of each edge in Eg or Eg satisfies c/2 S e < c. As a consequence, for (u, v) Ei (resp. E2), de, vu,v) (resp. da, (u, v)) is the same as the length of (u, v). We define a map é : G1 - G2 as follows. For G1, there is a v' in G2 such that (f) f2 $ m, Then we set d(v) = v'. The map & is injective on the vertex set Vi. Indeed, for u A e E V1, (u, v) 2 e/2 and hence do, (o(u), ((v)) 2 c/2 2m > 0. The strategy is to modify o by small perturbation such that the resulting function v G. + G2 is homeomorphism that is almost an isometry.
For U E V1, let No, be the 5n neighborhood of v. It is a star graph and its number of branches is the same as the degree of v, say k. Let uI,..., Uy be the endpoints of No. The convex hull (of shortest paths) C. of (ő(v1),....
((vx)) ((())) G2 is also a star graph. This is because Co is contained the 7n neighborhood of ((v) and it contains at most vertex in V/2
We claim that C, has the same number of branches as No. First of all, C. cannot have fewer branches. For otherwise, there is Φ(v) in the path connecting (0v) and @(v,) for some j * i. Hence,
da, (b(v;), o(v;)) S da, (o(()())) ((v)) < 7n
< 10n 2n da, (v,, (y), - 2n.
This is a contradiction with the property of Φ. It cannot have more branches than as the convex hull of at most k points.
18

---

Published in Transactions on Machine Learning Research (12/2024)
We next consider different cases for k. For k * 2, as Co is star graph, it has a unique node v' with degree k (in Co), and do, (v', ((())) > 0, 1 < j S j. We claim that e' has degree exact k in G2. Suppose on the contrary, its degree in G2 is larger than k. Then there is a branch not contained in Co. Let w' be a node on the new branch such that 6n I da, (w', <(v)) S 7q. Morcover, there is a node w in Ne such that 4n S da, (w, u) & 9n and w' = ((w). Moreover, w on the branch containing "/ for some j. and hence de, (ue, vy) S 4n.. Therefore,
da, (w, S 6n 2n
< do ,((;)) + dG (w', 27
= do (6(w), o(1))) 27/
which is contradiction. In this case, we define v1v) = v' E G2. If 2 when N, is path, by similar argument, we have that C2 is path. We set w(v) <(v). An illustration is given in Fig. 7.
N v
C U
((V)
v
Figure 7: Illustration of db.
For each E Vi, we now enlarge the neighborhood and consider its e/G-neighborhood N₁. It does not contain another vertex and hence is also star graph. Moreover, if * u E V1, then N n. = 0 for otherwise do, (u, v) I </3, which is impossible. We may similarly consider the e/8-ncighborhoods C² Cy of rb(u) and ((V). Both C and C do not contain any vertex in 1/2 with degree / 2.
As No. and C are star graphs with the same number of branches, there is an isometry (also denoted by) vb : N + C² such that da, ((w)) s 2n. By disjointedness of c/6 neighborhoods, we may combine all the maps above together to obtain vp UvEv, - U.ev,
For the rest of G1, consider any edge (u,v) Ej. Without loss of generality, let un and be the leaves of N2 and N2 contained in (u, We claim that the shortest open path connecting vuu) and s(v1) is disjoint from Ueev, C. For otherwise, da, (ui v1) 2 2c/3, while da, (ō(()) d(uz)) S (b,()))))) (uu))) - 4n 2 c/2 + 2 4n.. Therefore, 2e/3 - 21j 2 5e /6 - 4n, which is impossible as <
Let Pa.. and Qu be the shortest paths connecting un, v1 and (u1)), v(v1) respectively (illustrated in Fig. 8). Then the length of Pa.v and Qu differ at most by 4n, We may further extend vi Pa,v + Qu, by a linear scaling such that do, (v(w),, ((w)) I 3n for E Pu,.- For different edges (u, v), (u', it is apparent. Qu, Du ,u are disjoint, as the minimal distance between points Pa,e and Par is at least e/3. Therefore, we obtain a continuous injection va G + G2, which maps homeomorphically onto its image.
We claim that & is onto. If not, there is a vertex of E V22 that is not in (VV)) but it has a neighboring vertex u' = (u). However, this implics that the degree of u' is strictly larger than that of u, which is impossible as we have shown.
In summary, 3 G G2 is a homeomorphism such that |da, (u, v) do, (u, v)] S 6n for any u, G1. Moreover, & is piccewise linear whose gradient e is in the interior of N2, E Vi and satisfies
6n S ((w)) < + 6n
(20)
for w contained in the interior of some Pa,,, (u, v) E Ey.
19

---

Published in Transactions on Machine Learning Research (12/2024)
C%
N
(t(e)
N
Pu,y
C
Qu.
d
((()
Figure 8: Illustration of Pa.e and Qu
We are ready to estimate 1ōc.,1 8. Let |Gil be the total edge weights of G,, For convenience, denote typical tuple (u, u, w,t) Gf vector v, and (vb(u), (b()), (bw),, (()) by (v The map & Gf G2, + vb(v inherits the properties of its counterpart v,, which is piecewise linear homeomorphism. In particular, its Jacobian J(v) is defined almost everywhere. Using Definition 2, we have:
106,,1 - -G2.11
|G1 (GI
(21)
|G2| [[())())
I sup |Ta (v) (v) |G1|
J(v) (b(v))
GG2l
vEG
Similar to (19), we estimate
sup TG, [G, TG, (vb(v)))) 24n.
(22)
vEGi
Morcover, we have seen in the proof that & can only have distortion when restricted to Pa,e for (u, v) E Er. As
2x
67
2x + 67
<
/lpool//Qu.cl
2x
2
the same bounds holds for |G, Both upper and lower bounds be arbitrarily close to 1 if is small enough. Similarly, by (20), J(v) as a fourth power of t can also be made arbitrarily close to 1. In conjunction with (21) and (22), |ōG,,1 - &c,,a can be arbitrarily small if r is chosen to be small enough. This proves that 6G, is continuous in G.
E Related works
In this appendix, models that utilize multiple spaces and advanced topological information such as curvature are reviewed.
CurvGN (Ye et al., 2020) and Curvature Graph Neural Network (CGNN) (Li et al., 2021) learn to reweigh messages propagated between nodes, in Euclidean space, using curvature information. Curvature measures how casily information flows between two nodes. These works assume that the edges with low curvatures indicate the class boundaries, thus low weights are assigned when the edges are of low curvature. In our work, we use Gromov hyperbolicity instead of Ollivier's Ricci curvature to choose the appropriate space and we do not reweigh the edges.
To the best of our knowledge, the closest works to ours are Geometry Interaction Learning (GIL) (Zhu et al., 2020) and K-GNN (Bachmann et al., 2019). x-GNN utilizes the (Cartesian) product space to model data
20

---

Published in Transactions on Machine Learning Research (12/2024)
in different spaces and employs the k-stereographic model in each of the spaces. The Cartesian product enables a combinatorial construction of the mixed curvature space, thus the representations are first learned independently in the respective spaces and then concatenated (Sun et al., 2022). In terms of implementation, this similar to our framework but instead of concatenation, we introduce a space selection mechanism guided by hyperbolicity to fuse the representations.
On the other hand, GIL proposes feature interaction scheme leverage different spaces and a probability assembling module to combine the classification probabilities for obtaining the final prediction. The feature interaction scheme is where the node features in the respective two spaces enhanced based the distance similarity of the two sets of spatial emboddings. The larger the distance between the different spatial embeddings, the more significant the portion of features from the other space is summed to itself as seen in Fig. However, this may introduce noise to the respective spaces, which we explain further below.
Our approach differs from GIL (Zhu al., 2020) in some key aspocts. Firstly, we leverage the distribution of geometrie hyperbolicity to guide our model to learn to decide if each node better embedded in a Euclidean hyperbolic space instead of performing feature interaction learning. This is done by aligning the distribution of the learned model hyperbolicity and geometric hyperbolicity using the Wasserstein distance. Our motivation is that if a node can be best embedded in one of the two spaces, encoding it in another space other than the optimal one would result. in comparably larger distortions. Minimal information would be present in the sub-optimal space to help "enhance" the representation in the better space. Hence, promoting feature interaction could possibly introduce more noise to the corresponding spaces. The ideal situation then to learn normalized selection weights that are non-uniform for each node that we select for each node a single, comparably better space's output embedding. To achieve this, we introduce an additional loss term that promotes non-uniformity. Lastly, we do not require probability assembling since only have one set of output features at the end of the selection process.
F
Limitations
The paper does not have a comprehensive numerical study on extremely large datasets. Therefore, we do not have a definite answer to how JSGNN will perform on these graphs. However, the datasets being studied in our paper are diverse enough in terms of geometric properties. We have theoretically analyzed the complexity, which depends on the complexities of the hyperbolic and Euclidean base models. Therefore, its scalability also depends on the based models. For example, GCN, GAT, HGCN, and HGAT all scale well with graph size. Notice that message passing is a local operation, and hence model performance on graphs of medium sizes should be illustrative enough.
In future works, it is worthwhile to test our model on extremely large graphs to verify whether the above speculation holds.
G Broader impact
Since this research focuses on graph neural networks applied to publicly available datasets, there is no immediate broader impact concerns. The datasets used are standard benchmarks in the field, and there is no indication that the work introduces any significant ethical risks.
21


Reviews:
[{"review_id": "TtGOoTb6XY", "reviewer": "Reviewer_HXvf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Combining Euclidean and hyperbolic spaces for node-specific embeddings would be a promising and reasonable way to handle heterogeneous graph geometries.\n2. The motivation of the paper in Figure 1 is clear and makes sense to me.\n\n\n\n\n**Weaknesses**\n1. The experiments in Tables 1 and 2 are not fair to me. For example, JSGNN is a combination of GCN and HGCN, so the number of parameters will roughly double compared to GCN (correct me if I am wrong here). If this is the case, the results put the proposed method in an unfair position.\n2. If the above is true, the marginal improvement in Table 1 can be ignored and is not promising at all.\n3. While the motivating example in Figure 2 makes sense to me, this paper does not show that it is reflected in real graphs (despite the distributions of geometric hyperbolicity in Figure 2). This paper should also plot real graphs that include tree and lattice structures.\n4. Figure 3 is of low quality and not professional to me.\n5. As this paper claims that node representation is geometric (Gromov) and model-based, to determine the preferred embedding space for each node, the visualization of the node embeddings is a must-have experiment to show the learned node representations and compare them to hyperbolic and Euclidean representations.\n6. Based on Weakness 1, this paper should include a discussion on the scalability of JSGNN and analyze the time complexity of JSGNN, particularly regarding computational costs compared to GNN and HGNN. Providing empirical results on how well the method scales with graph size would also be helpful.\n7. I would also recommend expanding the experiments to include synthetic graph datasets like in Figure 1, especially where both Euclidean and hyperbolic structures are observed. This would help demonstrate the effectiveness of the proposed method.\n8. The datasets used in the experiments are quite small at this time. The largest graph is Pubmed, which may limit the scope of the experiment. Adding experiments on larger datasets would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Considering that real-world graphs are a combination of hyperbolic and Euclidean geometries, this paper presents a new graph representation learning approach by integrating Euclidean and hyperbolic space representation learning, called the Joint Space Graph Neural Network (JSGNN). The model uses two notions of hyperbolicity—geometric and model-based—and aligns their distributions through the Wasserstein metric to determine optimal embeddings.\n\n\nThe claimed contributions of this paper are:\n1. Exploring and analyzing two notions of local hyperbolicity—geometric and model-based—to describe the underlying local geometry and determine the preferred embedding space for each node.\n2. Proposing the Joint Space Graph Neural Network (JSGNN), which leverages both Euclidean and hyperbolic spaces during learning by allowing node-specific geometry space selection.\n3. Empirical evaluations show that JSGNN outperforms baseline models in node classification and link prediction tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see **Weaknesses**", "broader_impact_concerns": "I have no Broader Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ILwUpFi5F6", "reviewer": "Reviewer_RicP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The introduction of node-specific space selection in GNNs through localized geometric hyperbolicity is an innovative approach that addresses the limitations of using a single geometric space for the entire graph. This allows for better handling of complex graphs with mixed geometrical properties.\n\n- The authors provide a thorough evaluation of their model, including ablation studies that highlight the importance of key components like non-uniformity loss and Wasserstein alignment. This provides useful insights into the functioning and benefits of the proposed architecture.\n\n- The paper provides strong theoretical backing for the proposed model, including detailed explanations of geometric and model-based hyperbolicity. The use of the Wasserstein metric to align these hyperbolicities adds mathematical rigor to the space selection process.\n\n\nWeaknesses:\n\n- The paper mentions the computational cost of calculating geometric hyperbolicity but lacks a detailed analysis of how this affects scalability, particularly for large graphs. A more thorough discussion of the model's computational overhead would be useful for practical implementations.\n\n- There is limited discussion regarding potential limitations of the model. For instance, the model may not perform as well on graphs that are either entirely Euclidean or entirely hyperbolic, and this issue is not fully explored. Additionally, the trade-offs between performance and computational complexity are not sufficiently discussed.\n\n- The dual-space architecture and the additional loss terms (non-uniformity loss and Wasserstein alignment) add complexity to the model. Although the empirical results justify these components, the authors could discuss whether simpler variants of the model might still achieve competitive results with reduced computational burden.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel approach to embedding nodes in Graph Neural Networks (GNNs) by leveraging both Euclidean and hyperbolic spaces. The authors propose that real-world graphs often contain regions with mixed geometrical properties, making it inefficient to embed all nodes into a single geometric space. To address this, they introduce two forms of localized hyperbolicity—geometric (Gromov) and model-based—that help determine the most suitable space for each node. Their proposed Joint Space Graph Neural Network (JSGNN) architecture learns node embeddings in both Euclidean and hyperbolic spaces simultaneously, using an attention mechanism guided by the Wasserstein metric to select the optimal embedding space for each node. Additionally, the authors introduce a non-uniformity loss to encourage the model to select either Euclidean or hyperbolic space for each node, avoiding distortions caused by partial embeddings in both spaces. Through evaluations on node classification and link prediction tasks across multiple benchmark datasets, JSGNN demonstrates superior performance compared to baseline models by effectively leveraging both Euclidean and hyperbolic geometries in graph learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Some citations are not properly formatted. The authors are advised to carefully check and correct all instances of \\cite and \\citep to ensure consistent and accurate referencing.\n\n2. In the definition of $ \\mathbb{D}^n_c$, please explicitly specify the type of norm used for clarity and to avoid ambiguity in the interpretation of the mathematical space.\n\n3. In Definition 2, equation (10) presents an inequality. It is unclear how the authors achieve $\\inf_{\\delta>0} \\{(10)\\}$. Please provide additional explanation or steps to clarify this derivation.\n\n4. Figure 3 could be made clearer. It is recommended that the authors improve the clarity of the figure by using higher resolution graphics or adjusting the visual elements to make the relationships in the diagram easier to understand.\n\n5. Provide a more detailed analysis of the computational complexity of the proposed model, particularly the pre-processing step of calculating geometric hyperbolicity. This analysis should include time and memory complexity, as well as scalability considerations for large graphs. This is crucial for assessing the model’s practicality in real-world applications.", "broader_impact_concerns": "Since this research focuses on graph neural networks applied to publicly available datasets, I do not foresee any immediate broader impact concerns. The datasets used are standard benchmarks in the field, and there is no indication that the work introduces any significant ethical risks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nL2flTxKqs", "reviewer": "Reviewer_mCbs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n• The necessary explanations and background provided as to why solely GAT or hyperbolic GAT are not ideal for complex graph, justifying the opportunity and motivation of this paper. It’s a novel idea to combine Euclidean and hyperbolic geometries for each node representation\n\n• The experiments on multiple datasets are informative and convincing. The results show JSGNN exceeds the performance of single-space baselines on most datasets and demonstrate the benefit of including both euclidean spaces and hyperbolic spaces\n\n• The ablation study isolates and studies the impact of individual components (non-uniformity constraint, Wasserstein metric) and provides insightful analysis into their contributions.\n\n• The thoroughness of including Non-uniformity of selection weights in the loss function to deviate attention weights from 0.5 and maximize the use of JSGNN\n\n\nWeaknesses:\n\nOverall the paper is well-written and convincing, with enough theories and experiments backing up. Below are a few points where the paper can be improved\n\n• In node classification part, I didn’t see the dataset characteristics description provided to explain why hyperbolic models work well for certain datasets, Euclidean models work well for the others, and why JSGNN can exceed the performance on most datasets.\n\n• It would be beneficial to discuss the tradeoff between computational costs of JSGNN and performance as model scales with larger datasets", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper starts by calling out that real-world graphs are complex which could be made of both euclidean and hyperbolic structures, therefore explores the idea of integrating both Euclidean and hyperbolic space for node representation based on the hyperbolicity of local nodes in a graph neural network. The authors introduced JSGNN, which is a joint space graph neural network where dual-space embeddings are combined through attention mechanism. To optimize this model, distribution match is conducted between the geometric hyperbolicity of local graph structure and the model hyperbolicity of the GNN, ensuring that both Euclidean and hyperbolic embeddings are available where needed in the graph. The authors also empirically prove through different tasks on multiple benchmark datasets that JSGNN demonstrated competitive performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "• Discuss the computation complexities of nodes representation in JSGNN and how it grows with larger scaled dataset\n\n• A description of the benchmark datasets on hyperbolic and euclidean features", "broader_impact_concerns": "The broader impact of this paper has not been clearly justified yet.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I suggest the authors follow the reviewers’ suggestions and revise the draft. The following is the list of suggestions (which may not be comprehensive):

- Add statistics and evaluation results raised in the discussions with the reviewers. More specifically:
  - The distribution of the hyperbolicity values of datasets shown [here](https://openreview.net/forum?id=tHteJFeN1y¬eId=DgJcS1jEUL).
  - The comparison of computation times shown [here](https://openreview.net/forum?id=tHteJFeN1y¬eId=s7N9j8A9Rf) and [here](https://openreview.net/forum?id=tHteJFeN1y¬eId=gxm2goCxVA).
  - The comparison of parameter size of models shown [here](https://openreview.net/forum?id=tHteJFeN1y¬eId=63OfxFh4W7)
- Reflect discussions with the reviewers on the manuscript. More specifically:
  - Discussions of the computational complexity of JSGNN in terms of time and memory shown, e.g., [here](https://openreview.net/forum?id=tHteJFeN1y¬eId=RBi8nScNQb)
  - Discussions of the prediction performance on graphs with either hyperbolic or Euclidean properties shown, e.g. [here](https://openreview.net/forum?id=tHteJFeN1y)
- Fix the format issues pointed out by the reviewer. More specifically:
  - Replace Figure 3 with a clearer image.
  - Fix the format of citations.
- Discuss the limitations of this paper, for example, evaluations on larger graphs.
- Add a discussion of broader impact, as described in [Submission Guidelines and Editorial Policies](https://jmlr.org/tmlr/editorial-policies.html)
Decision: Accept (poster)