Neural networks on Symmetric Spaces of Noncompact Type
Authors: Xuan Son Nguyen
Shuo Yang
Aymeric Histace
Venue: ICLR 2025 Poster (2025)
Keywords: geometric deep learning
symmetric spaces
hyperbolic spaces
SPD manifolds
Primary area: learning on graphs and other geometries & topologies

Abstract: Recent works have demonstrated promising performances of neural networks on hyperbolic spaces and symmetric positive definite (SPD) manifolds. These spaces belong to a family of Riemannian manifolds referred to as symmetric spaces of noncompact type. In this paper, we propose a novel approach for developing neural networks on such spaces. Our approach relies on a unified formulation of the distance from a point to a hyperplane on the considered spaces. We show that some existing formulations of the point-to-hyperplane distance can be recovered by our approach under specific settings. Furthermore, we derive a closed-form expression for the point-to-hyperplane distance in higher-rank symmetric spaces of noncompact type equipped with G-invariant Riemannian metrics. The derived distance then serves as a tool to design fully-connected (FC) layers and an attention mechanism for neural networks on the considered spaces. Our approach is validated on challenging benchmarks for image classification, electroencephalogram (EEG) signal classification, image generation, and natural language inference.

Published as a conference paper at ICLR 2025
NEURAL NETWORKS ON SYMMETRIC SPACES OF NONCOMPACT TYPE
Xuan Son Nguyen, Shuo Yang, Aymeric Histace ETIS, UMR 8051, CY Cergy Paris University, ENSEA, CNRS, France (xuan-son nggyen, shuo, yang, aymeric hisstace) fr
ABSTRACT
Recent works have demonstrated promising performances of neural networks hyperbolic spaces and symmetrie positive definite (SPD) manifolds. These spaces belong to a family of Riemannian manifolds referred to symmetric spaces of noncompact type. In this paper, we propose a novel approach for developing neu- networks on such spaces. Our approach relies on a unified formulation of the distance from a point to a hyperplane on the considered spaces. We show that some existing formulations of the point-to-hyperplamm distance can be recovered by our approach under specific settings. Furthermore, we derive a closed-for ex- pression for the siintto-hhppeplant distance higher-rank symmetric spaces of noncompact type equipped with G-invariant Riemannian metries. The derived dis- tance then serves as a tool to design fully-connected (FC) layers and an attention mechanism for neural networks the considered spaces. Our approach is vali- dated on challenging benchmarks for image classification, electroencephalogran (EEG) signal classification, image generation, and natural language inference.
INTRODUCTION
Neural networks in non- EunEuucidean spaces have become powerful tools for addressing problems in a wide range of domains such as natural language processing (Chami et al. [019 Ganea et al. (0188), computer vision (Huang& & Gool, 2017 Nguyen et al. 2019), and medicine (Liu et al. 2019). There There rich existing literature focusing hyperbolic neural networks (HNNs) due to the ability of hyperbolic spaces to represent hierarchical data with high fidelity in low dimensions (Chami et al., 2021. Other examples of non-Euclidean spaces which have been commonly encountered are SPD manifolds. In this paper, we restrict our attention to neural networks with manifold-valued output.
The concept of hyperplanes has proven useful in the construction of HNNs (Ganea et al. 2018b
Shimizu et al. 2021) and classification algorithms in hyperbolic spaces (Fan et al. 2023). There exist
two classes of hyperplanes hyperbolic spaces, namely, Poincaré hyperplanes (Ganea et al.
20 8b 2021) which are identified as sets of geodesics, and horocycles (Fan et al. 2023
Shimizu et al.
Helgason 1 1984) which are described as manifolds orthogonal to families of parallel geodesies. Re-
cently, some approaches (Chen et al. 2024a; Nguyen & Yang 2023 Nguyen et al. 2024) have suc- cessfully developed matrix manifold analogs of Poincaré hyperplanes. However, these approaches either work for SPD manifolds associated with special families of Riemannian metrics (Chen al., 2024a), or require rich algebraic structures the considered spaces, which limits their generality.
In this paper, present novel approach for building neural networks symmetric spaces of non- compact type (Helgason 1979). These include hyperbolic spaces and SPD manifolds and are gener- ally regarded being among the most fundamental and beautiful objects in mathematics (Bridson & Häfliger 2011 Helgason] 1994). Our contributions are summarized as follows:
We propose novel method to construct the point-to-hyperplans distance in symmetric spaces of noncompact type. Compared to Ganea et al. (018b)) Nguyen & Yang (202)); Chen et al. (2024a) which only concern with hyperbolic spaces (Ganea et al.) 2018b) or SPD manifolds (Nguyen & Yang 2023 Chen et al. 2024a). our method deals with all those spaces and gives unified formulation for this distance.

---

Published as a conference paper at ICLR 2025
We derive expression for the -hhpe-plaaee distance in symmetric space of non- compact type equipped with a G-invariant Riemannian metric.
We propose FC layers and an attention mechanism for neural networks on symmetric spaces of noncompact type. Within the context of this work, such building blocks to the best of our knowledge.
the first to develop
We provide experimental results on image classification, EEG signal classification, image generation, and natural language inference showing the efficacy of our approach.
RELATED WORKS
2.1 HYPERBOLIC SPACES
HNNs have gained growing attention since the seminal work in Ganea et al. (2018b), which was inspired by [eeanon & Lafferty (2004) and proposed hyperbolic analogs of several building blocks of deep neural networks (DNNs). Some missing building blocks in Ganea al. (20T8b) (e.g., FC and convolutional layers) were then introduced in Shimizu et al. (2021). Both the works in Ganea iet al. (2018b): Shimizu et al. (2021) rely primarily on the construction of Poincaré hyperplancs. Another concept of hyperplanes
hyperbolic spaces (horocycles) was studied in Fan et al.
2023
This approach derives the distance between point and horocycle using horospherical projections. which were originally used for dimensionality reduction in hyperbolic spaces (Chami et al. 2021
Motivated by the impressive performance of graph neural networks (GNNs) (Veličković et al. GNNs in hyperbolic spaces were also investigated (Chami al. 2019
2018
Gulcehre et al. 2018
2.2 MATRIX MANIFOLDS
Most existing works concern with neural networks on SPD and Grassmann manifolds, and special orthogonal groups SPDNet, LieNet, and GrNet were among the first networks designed on those
spaces (Huang & Gool 2017 Huang et al. 2017 2018). In Brooks al. 2019);:5 Juu & Guan (2023);
Kobler et al. (2022; Nguyen (021 Nguyen al. (2019); Pan et al. (0022) Wang et al. (202 1). the
authors either introduced Riemannian batch normalization layers or improved Bimap layers (Huang
& Gool 2017). The works in López al. I222I;; Nguyen (2022a b); Nguyen & Yang (2023)
Nguyen et al. al. (2024) leverage rich algebraic structures of SPD and Grassmann manifolds to general- ize some basic operations and concepts in Euclidean spaces to these manifolds. Inspired by Nguyen
& Yang (2023), the work in Chen et al. (2024a) generalized multinomial logistic regression (MLR)
SPD manifolds under two families of Riemannian metrics.
2.3 GENERAL RIEMANNIAN MANIFOLDS
There have also been attempts develop more general frameworks for Riemannian manifolds. The works in Chakraborty et al. (2020); Zhen et al. (2019) advocated the use of weighted Fréchet mean to build number of layers (c.g., convolutional and residual layers) for neural networks on Riemannian manifolds. In Katsman et al. (2023), the authors parameterized vector fields to design Riemannian residual neural networks. Our work can connected to this work as one can use our derived distance to parameterize such vector fields. Extensions of SPD batch normalization layers (Brooks) ct al. 2019) on Lie groups were also proposed (Chen et al. 2024b).
3
MATHEMATICAL BACKGROUND
3.1 HYPERBOLIC SPACES AND SPD MANIFOLDS
We briefly discuss the geometries of two families of symmetric spaces commonly encountered in machine learning applications.
Hyperbolic Spaces by the manifold Bm,, = {a E R : ||æ||| < 1} equipped with the Riemannian metric (u, v)2 = (1- |ل] (u, 2) where u, U R". The Riemannian distance between two points x, y E Bn is given
The Poincaré model Bn, of m-dimensioonl hyperbolic geometry is defined
2

---

detailed discussion of hyperbolic spaces from a Published as a conference paper at ICLR 2025
by dz y) = cosh -1 (1+ 2 2 [1-1=1 PX 1
symmetric space perspective is given in Appendix
SPD Manifolds Here we consider PEM (Chen et al. 2024c) (see Appendix L.2) which is more
general than the well-established Log-Euclidean framework (Arsigny al. 2005). Let Sym,, be the space of m symmetric matrices. Under PEM, the SPD manifold Sym is defined by Symm = {ar E Symm uuf 0 for all u E Rm u 0} equipped with the met- (Dzø(u), DzΦ(v)), where & Sym + Symm is a diffeomorphism, D2 T2 Sym + To(u) Symm is the directional derivative of map at point x. T is the tangent space of X x X. The Riemannian distance between two points 2, Sym is given by dIzM(1, y) = [Ø(x) Q.))l. A detailed discussion of SPD manifolds from a symmetric space perspective is given in Appendix
ric U)
Existing point-to wointtoohyppee hyperplaee distances on Riemannian manifolds are generally built for one of the above families of symmetric spaces, except for the composite distance (Helgason 1984 1994). However, the use of the composite distance for our purposes is not straighttorward since it is vector-valued distance in higher-rank symmetric spaces (e.g.. SPD manifolds). In the following, we develop a unified framework to address this limitation of existing works.
3.2
SYMMETRIC SPACES OF NONCOMPACT TYPE
This section briefly recaps important concepts used in the paper. We refer the reader to Ballmann 2012 Bridson & Häfliger (2011): Helgason (1979) for further reading.
Roughly speaking, a symmetric space X is connected Riemannian manifold which reflectionally symmetric around any point. That is, for any x X, there exists local isometry 8, of X such that (x) = & and the differential Dz =- -id,x- x- Every (simply-connected) symmetric space Riemannian product of irreducible symmetric spaces. symmetric space irreducible, if cannot be further decomposed into Riemannian product of symmetric spaces. There are two types of (nonflat) irreducible symmetric spaces: compact type and noncompact type. Those two types are interchanged by Cartan duality. Please refer to Appendix L.4 for further discussion. In the following, we restrict our attention to those of noncompact type.
Formally, let G be a connected noncompact semisimple Lie group with finite center, K be maximal compact subgroup of G. Then the symmetric space of noncompact type consists of the left cosets
X = G/K = fI g G}.
The action of G on X = G/K is defined as ||0] = g(hk] ghK for x hK E X, g, h E G. Let o be the origin in X, then the map yp :gK + glo] is a diffeomorphism of G/K onto X.
Let G = KAN be the Iwasawa decomposition HeepooiiittiooHHlssooooooo Helgason 1979); Sawyer (016) of G, and let and a be the Lie algebras of G and A, respectively. For any linear form a on a, set 8a := {u E g/vu e [u, v] = a.))u). Let a' be the dual space of Then the set of restricted roots is defined by E = {a E a" 10Hga * {O}}. The kernel of each restricted root is a hyperplane of a. A Weyl chamber in is connected component of UsEx ker(a). We fix a Weyl chamber at and denote by at its closure.
Geometric boundary In a symmetric space X of noncompact type, boundary (ideal) points can be regarded as generalizations of the concept of directions in Euclidean spaces. Intuitively, bound- ary points represent directions along which points in X can move toward infinity (Chami et al.) 2021]. The set of boundary points 0X of X is referred to as the (geometric) boundary of X. For instance, the Poincaré disk model (a model of 2-dimensional hyperbolic geometry) is given by D {(11,2) #2) xf + 2² < 1) (one can think of this set as the set of all complex numbers with length less than 1, i.e. IB {r C |||||| < 1)). The boundary ADD of D is the unit circle JD = ((x1,02) xf + x3- = 1}.
Let d(...) be the distance induced by the Riemannian metric. A geodesic ray in X is a map 8 [0, oo) + X such that d(8(t), ((t)) (t')))= -t',t,, Vt, 2 0. A geodesic line in X is a map 8 R R- X such that d(5(1),(((')) |t 1'), Vt, R. Two geodesic rays 8' are said to be asymptotic if
3

---

Published as a conference paper at ICLR 2025
d(((t), (t)) is bounded uniformly in t. This is equivalence relation the set of geodesic rays in X. The set ax boundary points of is the set of equivalence classes geodesic rays. The equivalence class of geodesic ray 8 is denoted by ò(co).
Angular metric For x E X and E E axx there exist unique geodesic rays ó and 8' which issue from and lie in the classes and &', respectively (Ballmann 2012). One can then define (E(E, to be the angle at a between and (see Appendix L.5p. The angle Z(E: ((::E') )') is defined as
<(5,5") sup (5, zeX
The function (E,E)) E(E defines the angular metric (Bridson & Hälliger 2011 ax.
Busemann functions Busemann functions (coordinates) can be regarded as generalizations of the concept of coordinates in Euclidean spaces. In Euclidean space, given a point æ and a unit vector w (which represents a direction). one has
- (x, w) lim (d(2,, (w) d(0, tw)) = lim (d(a,, tw) - 1), t-+00
t-
where tw, t > can be seen ray that moves toward infinity in the direction of as t oo. Note that the inner product (z, w) gives the coordinate of in the direction of w. This observation can be used compute coordinates and & 8(0o) 0X. Then, by replacing tw with geodesic ray 8(t), one defines the Busemann coordinate of a point x E X in the direction of & as
X. Let 5 oo) - X be (unit-speed) geodesic ray
Bę(œ)= = lim (d(x, 8(1)) - t). 1000
The function Bg X IR is called the Busemann function associated to the geodesic ray 5.
Horocycles Like a Euclidean hyperplane which is orthogonal to a family of parallel lines, a horo- cycle is orthogonal to a family of parallel geodesics (Helgason) 1984 1994). Thus, horocycles can be regarded as symmetric space analogs of Euclidean hyperplanes. Let M be the centralizer of A in K, i.e., M == (A( = {K K ka ak for all E A). The space E of horocycles can be identified (Helgason) 1994) with
E: G-MNN = {n = gMN E G).
Composite distances The notion of composite distance is a symmetric space analog of the Eu- clidean inner product (Helgason) 1984 1994). Let vj g be horocycle where g E G, and let = kan where E K, E A, and n. E N. Then 5 = kM E ax is said to be normal to y, and log(a) a) is the composite distance from the origin o to 77 More generally, if gK e X, and hMM E E where g, h E G, then H [1) is the composite distance from æ to where the determined by 91 h exp ((g)) n1 with 9I E G, k1 K, and n N.
map
PROPOSED APPROACH
We define hyperplanes and propose general formulation for the hoint-to-hyperplane distance the considered spaces in Sections 4.1 and 4.2 respectively. We then examine the proposed formulation
for hyperbolic spaces and SPD manifolds in Section 4.3 In Section 4.4 our distance is derived
for spaces equipped with G-invariant Riemannian metrics. In Section 5 we show how to build FC layers and attention mechanism for neural networks the considered spaces.
4.1 HYPERPLANES SYMMETRIC SPACES
In Euclidean space R hyperplane H. is defined by
H. {Ir E T (I, - 0},
where a E 1R" (0), b E R, and (., is the Euclidean inner product. The hyperplane H.E can be reformulated as
H for E R. :p x, 0).

---

Published as a conference paper at ICLR 2025
d(2, H
H S.P
Figure 1: The distance between point and hyperplane H
E.p"
where p R and (p, a) = b.
In order generalize Euclidean hyperplanes matrix manifolds, the work in Nguyen & Yang 2023) treats parameter as point on the considered manifold X. The equation of hyperplane H. is then generalized to the matrix manifold setting by defining matrix manifold analogs of operations and as well that of the Euclidean inner product. Here take a different approach by rewriting (p- I, a} as Busemann function. Let be the equivalence class of the geodesic ray o(t) t a Tall where is the Euclidean norm. Using the expression of the Busemann function in Rm (see Appendix L.6), we have that
ip I, ||l|| Bc(-p+2)).
Assuming that one can define appropriate operations @ and @ on X that are symmetric space analogs of operations and +, respectively. This leads us to the following definition.
Definition 4.1 (Hyperplanes a Symmetric Space). For p X and 0x, hyperplanes on X are defined as
He.p = {a E X Bg(Ep @ x) 0}, where @ and @ are the inverse and binary operations on X, respectively.
In symmetric space, horocycle is manifold which is orthogonal families of parallel geodesics (Helgason 1994). Thus horocycles generalize the idea of Euclidean hyperplanes which are orthogonal families of parallel lines. In our approach, hyperplane contains fixed point p X and every point E X such that the segment Gp @ x is orthogonal to fixed direction Segments of the form ep @ x can be regarded as symmetric space analogs of Fuclidean lines. Therefore, those hyperplanes also generalize the idea of Euclidean hyperplanes in natural way.
4.2 POINT-TO-HYPERPLAAN DISTANCE ON SYMMETRIC SPACES
Let E.p be hyperplane R". Then the distance d((x, H.F between point x E R and E.
can be computed (see Fig. M as
d(z, HE d(x, p) (o)
(1)
where 3 is the angle between the segments [z, p] and [x, ql with being the projection of on H By convention, d(æ, 1.. for any z E H.E Note that Eq. can be rewritten as
d(z, E.p = d(x, p) cos (),
where & and g' are the equivalence classes of the geodesic rays 8 and 5' which issue from x and whose images are the segments [x, q] and [z. pl, respectively. Let = 5(t), then
(o)
dx, H.E.)) d(x, cos (E( -d(x, p) lim t1+000
The last expression Kapovich et 2017) is remarkable because it relates the distance d(.,.) to Busemann function. Note also that
()) - (8) -(ta',

---

Published as a conference paper at ICLR 2025
where 8(t) 8t) ta', is unit vector, and a' = p-2 Therefore
11p-x|
d(x, H d)x, a) d(x, (){
p a) d(x,p) p)
Be(-p
1p x1
||
x
This motivates the following definition.
Definition 4.2. Let Hg.p be a hyperplane as given in Definition 4.1 and let || lls be a norm on X. Then the (signed) distance d(x, Hg,p) between a point x X and He.p is defined
dxr Hep) = d(r, B (5p
e p 0xls
4.3 POINT-TO-HYPERPLANN DISTANCES ON HYPERBOLIC SPACES AND SPD MANIFOLDS
We now derive the hyperplaee distance for the symmetric spaces discussed in Section 3.1
Hyperbolic Spaces The following result is straightforward.
Corollary 4.3. Let 3 and @ be the Möbius subtraction 9M and Möbius addition BM in 18m. respectively, and let ||- [|s be the Euclidean norm and let He.p be hyperplane point x Bm and Hggpp is computed by
(see Appendix(L7.1] Let p Bys E E OB Then the distance d(æ, He.x) between
given in Definition 4.1
d3 dg(x,,)
- BM x
d(z, Hg,p)
log
x||
p @M x E|
SPD Manifolds Proposition shows that the 2024a is special case of proposed distance (see Appendix M. for the proof of Proposi- tion 4.4)
distance studied in Chen et al.
Symm be a diffeomorphism. Let and @ be the binary and Proposition 4.4. Let á Sym inverse operations defined by
I + ( +(y)).
=x= -
where x. y E Sym Let || I|s be the norm induced by the inner product (:)s given as
(x, le.y)8 = (0(x),()))))))))))))))) o(y)).
Let 5(t) (ta be a geodesic line in Sym where a E Sym and ||a|| = Ler & 8(00), p Sym and let He be hyperplane as given in Definition 4. Then the distance d(xr, He.r) between point Sym and Hg computed as
dx, He.p) = (a, d(p) {(x))
A direct consequence of Proposition 4.4 that the distance between SPD matrix and an SPD hypergyroplane (Nguyen & Yang 2023) is also special case of our proposed distance under Log-
Euclidean and Log-Cholesky frameworks (e.g.. the map é is the matrix logarithm the case of Log-Euclidee Euclidean framework).
4.4 POINT-TO-HYPERPLANE DISTANCE ASSOCIATED WITH A G-INVARIANT METRIC
In the preceding section, closed-form expressions the for hyperbolic spaces and SPD manifolds under PEM. In this section, shall derive this distance in a higher-rank symmetric space X of noncompact type equipped with a G-invariant Riemannian metric. This requires us (1) define the binary operation @ and inverse operation @ on X; (2) to define the norm || ||s on X; and (3) to compute the Busemann function.
distance are computed
Let gK, y h.K X, where g, G.
Definition 4.5 (Binary Operation). The binary operation @ is defied as
+ ghK.

---

Published as a conference paper at ICLR 2025
Definition 4.6 (Inverse Operation). The inverse operation @ is defined as
The motivation for the above definitions is that the space G/K with the operation admits group structure (the identity element is K and the inverse of any element is given by the inverse operation).
In order to compute the norm || lls. we shall define an inner product (t, iss whose construction based on the following natural view points:
The inner product (s should agree with the Riemannian distance.
The inner product -g should be invariant under the action of K. This property holds for the ones proposed in Helgason 1994); Nguyen & Yang (2023).
We thus consider the following inner product. Definition 4.7 (The Inner Product on Symmetric Spaces), Let x = gk, y = hK e g, Then the inner product ( 1) on X is defined as
(x,0)s = (H())
where the map (Cartan projection) μ G at is determined by kexp(u(g)) with G and k, k E K (this follows from the Cartan decomposition Helgason 1979) of where pu.)) is continuous, proper, surjective map to the closed Weyl chamber a
Proposition 4.8
states that the aforementioned properties hold for the considered inner product (see
Appendix M.2 for the proof of Proposition 4.8).
Proposition 4.8. Let gK, hK X, g. G. and let (s be the inner product as given Then
in Definitions 4.7
(i) We have that:
e d(x, y): where the norm 1. Is is induced by the inner product )s
(ii) For any le E K, we have that:
(x, y)s (le k(y))s
Finally, closed-form expression of the Busemann function is provided in Proposition 4.9 Appendix for the proof Proposition (49)..
(see
Proposition 4.9. Let d(t) k exp(ta) K be a geodesic ray, where k K, a, ||a|| and let Ç 6(co).. Then
(x(x) = (a, (.gg(gg)))
where x E X. and G is given by k [x] =gK.
As consequence of Proposition 4.9 Corollary [4.10] gives the expression of the distance between
point and a hyperplane in symmetric space (see Appendix Mor A
the proof of Corollary 4.10)
Corollary 4.10. Let 8(t) = k exx((ta) K be geodesic ray, where lo E K, E ||al| = 1. and let {= 5(xc). ). Let p = hK X, G, and lei He.p be hyperplane given in Definition Then the distance d(æ, Hg.p) between point =gK- E X, g and He.p is computed as
dxz H.pp) = (a, H(g hk))-
(2)
The connection of the distance in Eq. with existing works is discussed in Appendix G
NEURAL NETWORKS ON SYMMETRIC SPACES
In this section, shall develop symmetric space analogs of two important building blocks in DNNs, ie., FC layers and attention mechanism. Our starting point is the construction of the point-to- hyperplane distance presented in the preceding section.

---

Published as a conference paper at ICLR 2025
Clas prbsbiiliiss
Mudpoini
Facidean layer
Concatanate
Acternion
Remannian 4)
map
Allantion mobule
FC
SPD matrix
Imput
Input data
(a]
(b)
Figure 2: Our proposed attention block (a) and the network architecture for EEG classification (b).
5.1 FC LAYERS
An FC layer can be described by the following linear transformation:
ar b,
(3)
where Rmxm', Rm and y, E Rm each for one dimension, i.e., the j-th dimension yj. j
Eq.
be rewritten system of equations, 1..., m of the output is given as
Vj (z. az) bj.
where aj E "R" by E R. Let É E 0x and let H2,,K be the hyperplane that contains the origin (i.e., K) and is orthonormal to the j-th axis of the output space. Then yj can be interpreted as the signed distance d(y, Hi k)) from the output y to hyperplane Hg We thus have
d(y, Hg, = (ar, ag) - by-
From Definition 4.1 & E OX. Therefore
we can write the expression (I., az)) bj as Bg, (ep; e Ox), where pj E X and
d(y, Hgg, B5, (Ep, @x).
(4)
Since the axes of the output space are orthonormal, it is tempting to construct a set of orthonormal boundary points (5 for which the output is related to the input x via Eq. (4). Two boundary points E and L.j 1.....m.. m. + j are said be orthonormal if Z &
Such set of
boundary points can be identified from Proposition 5.1 (see Appendix M.5 for its proof).
Proposition 5.1. Let 8(t) exp(ta) K and 8' (f) exp(ta') be geodesic rays, where a and a' are stundard basis vectors Rm, + a'. Lei { = 8(2x), 5' 8'(0)). Then and g are orthonormal.
We now formulate our proposed FC layers (see Appendix M.6]ffor the proof of Proposition (.2)).
Proposition 5.2. Let (t) ky exp(ta, K,j I..... m be geodesic rays, where k, K, aj E a, [|ayll = 1. Let 1z(2) = Bg, (Ep, @ r),j = 1, m, where & = dz (00), Pj X, and x E X is the input of an FC layer. Then the output y of the FC layer can be expressed as
y = nexp((-u1(()
vm (x))),
where n E N.
In our approach, the transformation performed by an FC layer is designed to be a symmetric space analog of the linear transformation in Eq. which makes approach distinct from existing ones (Huang & Gool, 2017 Huang et al., 20188] Chakraborty et al. 2020)) Wang 2021 [onoda et al. 2022 Please refer to Appendix for comparison of our approach against those approaches.
8

---

Published as a conference paper at ICLR 2025
Table Different formulations of the point-to-hyperplane distance on 1m--
g- g-distance
h-distance
b-distance
sinh
-
a allgg
du(z.p)
log II pibar E1
[1
P||a|
a E1
pG0 M=
p.P E B. a E T,,Bm.
(0)
B mms 0, I. a 0.5 E dBm
2, p E Bm C E dL
Ganca ct al., 2018b
(Fan ct al. 2023
This work
Table 2:
Accuracies (%) Hybrid ResNet-18 18 models for image classification.
Method
CIFAR-10
CIFAR-I00 100
Hybrid Poincaré Guo et al.
2022
95.04:10.13
77.19+0.50
Poincaré ResNet vvnn Spengler et al.
2023
94.51+0.15
76.60++0.32
Muclidean--Poicarte H Fan al.
2023
81.7247.84
44.35+22.93
Buclidean-Poincare-- G Ganea 2018b
95.1410.11
77.7810.09
Euclidean- Ponncann-Pincrr--- (Ours)
95.23310.08
77.7810.11
5.2 ATTENTION MECHANISM
We use an approach similar to Shimizu et al. (2021).
The scaled dot product attention
Vaswann
al. 2017 formulated
q2" m
abb(g. 2, v) = softmax
v,
(5)
where y, E TRixm= and v E R..m. are the queries, keys, and values, respectively. / is the sequence length, my and m, are the hidden dimensions of the queries (keys) and values, respectively, and function softmax(.) produces matrix of the same size as its input matrix by applying the softmax function each row of this matrix.
The matrix product q2" corresponds to an attention function that determines the similarities be- tween all query-key pairs. The product function softmax(.) and e :..:: of produces the weighted means of values vj, ...... / and thus can be seen as a midpoint operation. In self- attention, the queries, keys, and values are different linear projections of the same input sequence. Therefore, Eq. (5) can be reformulated as
att Jiin Pinm (1z))=1)
( (13))=)))
for all j' = I,...,I, where (1z))=1 is the input sequence, f (-), (.), (.) are linear functions that project the input points to the queries, keys, and values, respectively, (zyz)/=1 softmax (ffat (zy), fim,)) (z;))) fate(..)) is the attention function, and fmiid.) is the mid- point operation. We use our proposed FC layers (see Fig. 2 (a)) to perform linear projections in f (-), f (-), and (.). The attention function (Gulcehre al. 2018 Shimizu 2021 given
Sate (at (1
(i,(),)) (in
-cid(/i (Mzy) Jin (zz)) C2
where C₁ C2 E R. > 0 are learnable parameters. We adopt the weighted Fréchet mean (wFM) for the midpoint operation.
6 EXPERIMENTS
In this section, report experimental evaluation the image classification and EEG signal classification tasks. We refer the reader to Appendix [ for experimental details and Appendices C and D for experimental evaluation image generation and natural language inference.

---

Published as a conference paper at ICLR 2025
Table 3: Accuracies (%) of our networks and state-of-the-art methods for EEG signal classification.
Method
BCIC-IV-2a
BCI-NER MAMEM-SSVEP-II
EEG-TCNet [ngolfisonn et al.
2020
67.0914.6
55.45=7.66
77.05=2.4
MBEEGSE Altuwaijri 2022
64.58+6.0
56.45+7.2
75.46=2.3
MAtt (Pan et al.
2022
74.71+5.0
65.50+8.2
76.01+2.2
Graph-CSPNet Ll & Guan
2023
11.95+13.3
AttSymSpd-La (Ours)
78.24 + 5.4
70,96 = 8.6
78.02 = 2.3
AtrSymSpd-Gi (Ours)
78.08 + 4.8
67.24 + 7.4
75.88 = 2.2
6.1 HYPERBOLIC SPACES
We follow Bdeir al. 2024) and design hybrid architecture which consists of the ResNet- (He
whiih
et al., 2016) and the Poicare MLR (Ganea et al. 2018b). The output of the ResNet- 18 is mapped to
the Poincaré ball before is fed to the Poincaré MLR. We employ our proposed point-to-hypperplaee
distance as well as those from Ganea et al. (01 8b); Fan et al. (2023) (see Tab. 11 in the Poincaré
ML.. Experiments are conducted on CIFAR- 10 and CTFAR- 100 datasets (Krizhevkyy 2009). Tab. P
shows the results of the three resulting networks and those of Hybrid Poincaré (Guo et al. 2022) and
Poincaré ResNet (van Spengler et al. 2023) taken from Bdeir et al. (2024). Hybrid Poincaré only in the Poincaré MLR which uses the reparameterization method in Shimizu et al.) (202T) Our network gives the best mean accuracies on both datasets. In particular, outperforms all the HNN models from [Beii et al. (2024) including the fully hyperbolic model on CIFAR-10 10 dataset. Note that the g-distance is the closest distance from point to Poincaré hyper- plane, and the b-distance designed to be a symmetric space analog of the closest distance from point to a Euclidean hyperplane. However, the h-distance is obtained by horospherical projections which aim preserve an important property Principal Component Analysis, i.c., distances be- tween points are invariant to translations along orthogonal directions. Therefore, the h-distance does not have the same nature as the g-distance and b-distance. This probably explains why Euclidean- Poincaré-H is inferior to the other models. The inferior performance of the h-distance can also be observed in our experiments for image generation and natural language inference (see Appendices C DD. Furthermore, those experiments demonstrate that: (1) for image generation, the b-distance outperforms the g- g-distance in terms of mean performance in all cases; and (2) for natural language inference, the former performs favorably compared to the latter in terms of mean performance in most cases. This indicates that our proposed distance has the potential to improve existing HNNs.
differs from Euclidean-Poincant
6.2 SPD MANIFOLDS
We validate the proposed building blocks (see Section 5) for SPD neural networks on three EEG sig- nal classification datasets: BCIC-IV-2a (Brunner al. 2008), MAMEM-SSVEP-I (Nikolopoulos) 2021], and BCI-NER (Perrin et al. 2012)) We test two variants of the network architecture illustrated Fig. (b). The FC layers used in the attention block of the first network AuSSmmSp-LL are built
upon Log-Euclidean metrics (see Section 4.3) while those of the second network AttSymSpd- built upon G-invariant metrics (see Sections and
Tab. shows the results of our networks and some state-of-the-ar methods. Most of these methods are selected (Pan et al., 2022) based on two criteria: (1) code availability and completeness; and (2) solid evaluation (e.g., cross-session) without additional auxiliary procedures. As can be observed, AuSSmmSpd-L performs the best on all the datasets. AuSymSpd-OG is on par with AtSSmSppd-L on BCIC-IV-2a dataset. Although AutSmmSpd-C is outperformed by AuSSmSpp-LL on MAMEM- SSVEP-II and BCI-NER datasets, the former enjoys an advantage having much smaller numbers of parameters than the latter. For example. AttSymSpd-GI and AttSymSpd-LL use 0.007 MB and 0.034 MB learnable parameters on BCIC-IV-2a dataset, respectively (see also Appendix B.2.3).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We are grateful for the constructive comments and feedback from the anonymous reviewers.
REFERENCES
Ghadir Ali Altuwaijri, Ghulam Muhammad, Hamdi Altaheri, and Mansour Alsulaiman. A Multi- Branch Convolutional Neural Network with Squeeze-and-Excitatee Attention Blocks for EEG- Based Motor Imagery Signals Classification. Diagnostics, 12(4))995, 2022.
Vincent Arsigny, Pierre Fillard, Xavier Pennec, and Nicholas Ayache. Fast and Simple Computa- tions on Tensors with Euclidann Metrics. Technical Report RR-5584, INRIA, 2005.
Werner Ballmann. Lectures on Spaces of Nonpositive Curvature. Birkhäuser, 2012.
Francesca Bartolucci, Filippo De Mari, and Matteo Monti. Unitarization of the Horocyclic Radon Transform on Symmetric Spaces. CoRR, abs/2108.04333. 2021. URL http:://axiv..rrrt////
abs/2108. 04338
Ahmad Bdeir, Kristian Schwethelm, and Niels Landwehr. Fully Hyperbolic Convolutional Neural Networks for Computer Vision. In ICLR, 2024.
Clément Bonet, Benoit Malézieux, Alain Rakotomamoniy, Lucas Drumetz, Thomas Moreau, Matthieu Kowalski, and Nicolas Courty. Sliced- Wasserstein on Symmetric Positive Definite Ma- trices for M/EEG Signals. In ICML, pp. 2777-2805, 2023.
Silvère Bonnabel. Stochastic Gradient Descent Riemannian Manifolds. IEEE Transactions Automatic Control, 580))2217-2229, 2013.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A Large An- notated Corpus for Learning Natural Language Inference. In EMNLP, pp. 632-642, 2015.
M.R. Bridson and Häfliger. Metric Spaces of Non-Positive Curvature. Springer Berlin Heidel- berg, 2011.
Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geo- metric Deep Learning: Going beyond Euclidean Data. IEEE Signal Processing Magazine, 34(4): 18 18-42, 2017.
Daniel A. Brooks, Olivier Schwander, Frédéric Barbaresco, lean-Yves Schneider, and Matthieu Cord. Riemannian Batch Normalization for SPD Neural Networks. In NeurlPS, pp. 15463- 15474, 2019.
Clemens Brunner, Robert Leeb, Gernot Müller-Putz, Alois Schlögl, and Gert Pfurtscheller. BCI Competition 2008-Gral data set A. Institute for Knowledge Discovery (Laboratory of Brain- Computer Interfaces), Graz University of Technology. 16:1-6, 2008
Rudrasis Chakraborty, Jose Bouza, Jonathan H. Manton, and Baba C. Vemuri. ManifoldNet: A Deep Neural Network for Manifold-valuee Data with Applications. TPAMI, 44(2):799 810, 2020.
Ines Chami, Rex Ying, Christopher R, and Jure Leskovec. Hyperbolic Graph Convolutional Neural Networks. CoRR, 1ati1190.19933 2019. URL URL
Ines Chami, Albert Gu, Dat Nguyen, and Christopher Ré. HoroPCA: Hyperbolic Dimensionality Reduction via Horospherical Projections. In ICMI., pp. 1419-1429, 2021.
Ziheng Chen, Yue Song, Gaowen Liu, Ramana Rao Kompella, Xiaojun Wu, and Nicu Sebe. Rie- mannian Multinomial Logistics Regression for SPD Neural Networks. CoRR, abs/2305. 12288, 2024a.
wwwpxiivorrgaaaao abs /200. 12888
Ziheng Chen, Yue Song, Yunmei Liu, and Nicu Sebe. A Lie Group Approach Riemannian Batch Normalization. CoRR, abs/2403. 11261, 2024b. URL http: arxiv. htt.:oggg/abbaaaogeo
11261
11

---

Published as a conference paper at ICLR 2025
Ziheng Chen, Yue Song, Tianyang Xu, Zhiwu Huang, Xiao-Jun Wu, and Nicu Sebe. Adaptive Log-Euclidean Metrics for SPD Matrix Learning. IEEE Transactions on Image Processing. 33: 5194-5205, 2024c.
Taco S. Cohen, Mario Geiger, Jonas Koehler, and Max Welling, Spherical CNNs. CoRR, abs/180 100,, 2018. wtR.ittts://arxiv..oo abs/ 000.001.100 10130
Terrance DeVries and Graham W. Taylor. Improved Regularization of Convolutional Neural Net- works with Cutout. CoRR, abs// 708.04552, 2017. URL URt.htttaa//arxiv.ooo hbrgg//bb/// 708.
04552
Xiran Fan, Chun-Hao Yang, and Baba C. Vemuri. Honospherical Decision Boundaries for Large Margin Classification in Hyperbolic Space. CoRR, abs/2302.06807, 2023. URL http:
7/ //aaxii..rg/ab///2233.... 06807
Octavian Ganea, Gary Becigneul. and Thomas Hofmann. Hyperbolic Entailment Cones for Learning Hierarchical Embeddiings. In ICML., pp. 1646-1655, 2018a
Octavian-Fugen Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. In NeurlPS, pp. 5350-5360, 2018b.
Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, and Nando de Freitas. Hyperbolic Attention Networks. CoRR, Abs/1805.09758 2018. htRL/ttss////xi.vooom
abs 0/5//8805.0777
Yunhui Guo, Xudong Wang, Yubei Chen, and Stella X Yu. Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifiers. In CVPR, pp. 11-20, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pp. 770-778, 2016.
S. Helgason. Differential Geometry, Lie Groups, and Symmetric Spaces. ISSN. Elsevier Science, 1979.
S. Helgason. Groups and Geometric Analysis: Integral Geometry, Invariant Differential Operators, and Spherical Functions. Number vol. I in Groups and Geometric Analysis. Academic Press, 1984.
S. Helgason. Geometric Analysis on Symmetric Spaces. Mathematical surveys and monographs. American Mathematical Society, 1994.
Zhiwu Huang and Luc Van Gool, A Riemannian Network for SPD Matrix Learning. In AAAI, pp. 2036-2042, 2017.
Zhiwu Huang, Chengde Wan, Thomas Probst, and Lue Van Gool. Deep Learning on Lie Groups for Bheleton-Basee Action Recognition. In CVPR, pp. 6099-610 2017.
Zhiwu Huang, Jiqing Wu, and Lue Van Gool. Building Deep Networks on Grassmann Manifolds. In AAAI, pp. 3279-3286, 2018.
Thorir Mar Ingolfsson, Michael Hersche, Xiaying Wang, Nobuaki Kobayashi, Lukas Cavigelli, and Luca Benini. BEG-TCNet: An Accurate Temporal Convolutional Network for Embedded Motor- Imagery Mrcinne Interfaces. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 2958-2965, 2020.
Catalin Ionescu, Orestis Vantzos, and Cristian Sminchisescu. Matrix Backpropagation for Deep Networks with Structured Layers. In ICCV, pp. 2965-29277 2015.
Ce Ju and Cuntai Guan. Graph Neural Networks SPD Manifolds for Motor Imagery Classifica- tion: A Perspective From the Time- Fene-FFreyency Analysis. IEEE Transactions on Neural Nenworks and Learning Systems, pp. 1-15, 2023.
12

---

Published as a conference paper at ICLR 2025
Michael Kapovich, Bernhard Leeb, and Joan Porti. Anosov Subgroups: Dynamical and Geometric Characterizations. CoRR, abs/1 2017. URL BtLLhttpt///rraat org/ haapg//bs//770 /ttp://rww......///////3
01647]
Hermann Karcher. Riemannian Center of Mass and Mollifier Smoothing. Communications Pure and Applied Mathematics, 30(5):509-54) 1977.
Fanny Kassel. Proper Actions on Corank-one Reductive Homogeneous Spaces.
CoRR,
abs/0807. .3900, 2009.
Isay Katsman, Eric Ming Chen, Sidhanth Holalkere, Anna Asch, Aaron Lou, Ser-Nam Lim, and Christopher De Sa. Riemannian Residual Neural Networks. CoRR, abs/2310.10013, 2023. URL
http::/araa www../xwxivv...///vv org/ abs 2310 10013
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
Thomas N. Kipf and Max Welling, Semi-Supervised Classification with Graph Convolutional Net- 02907
works. CoRR,
6009002977 2017.
Reinmar J. Kobler, Jun ichiro Hirayama, Qibin Zhao, and Motoaki Kawanabe. SPD Domain- specific Batch Normalization to Crack Interpretable Unsupervised Domain Adaptation in EEG. In NeurlPS, pp. 6219-6235, 2022.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian Optimization in PyTorch. CoRR, abs/2005.0281 022119, 2020. httpptts///axxi...o//oopoee
02819
Alex Krizhevsky, Learning Multiple Layers of Features from Tiny Images. pp. 32-33, 2009. URL
https://wwat /ttp:///wwww... cs. http://w....ooooooo../////////ee
features
pdE.
Vernon J Lawhern, Amelia Solon, Nicholas Waytowich, Stephen M Gordon, Chou Hung, and Brent Lance. EEGNet: A Compact Convolutional Neural Network for EEG-based Braincom- puter Interfaces. Journal of Neural Engineering, 15(5):056013, 2018.
Guy Lebanon and John Lafferty. Hyperplane Margin Classifiers on the Multinomial Manifold. In ICML, pp. 66, 2004.
Mario Lezcano-Casade Trivializations for Gradient-Based Optimization on Manifolds. In NeurIPS, pp. 9154-9164, 2019.
Zhenhua Lin. Riemannian Geometry of Symmetric Positive Definite Matrices via Cholesky Decom- position. SIAM Journal Matrix Analysis and Applications, 1354)11353- 2019.
Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic Graph Neural Networks. In NeurlPS, pp- 8828-82211 2019.
Federico López, Beatrice Pozzetti, Steve Trettel, Michael Strube, and Anna Wienhard. Vector- valued Distance and Gyrocalculus on the Space of Symmetric Positive Definite Matrices. In NeurlPS, pp. 18350-18366, 2021.
Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. Differentiating through the Fréchet Mean. In ICML, pp. 6393-6403, 2020.
Ravikiran Mane, Effie Chew, Karen Chua, Kai Keng Ang, Neethu Robinson, A. P. Vinod, Seong- Whan Lee, and Cuntai Guan. FBCNet: A Multi-view Convolutional Neural Network for Brain- Computer Interface. CoRR, abs/2104.01233, 2021. URLhttts://aaxiii wttp:rgggabb/agooeeee//e/g
01233
Yazeed K. Musallam, Nasser AlFassam. Ghulam Muhammad, Syed Umar Amin, Mansour Al- sulaiman, Wadood Abdul, Hamdi Altaheri, Mohamed A. Bencherif, and Mohammed Algabri. Ebectroencophhloograhbbbbbe based Motor Imagery Classification Using Temporal Convolutional Net- work Fusion. Biomedical Signal Processing and Control, 69:102826, 2021.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A Wrapped
Normal Distribution on Hyperbolic Space for Gradient-Based Learning. In ICML, pp. 4693-
2019.
13

---

Published as a conference paper at ICLR 2025
Galileo Namata, Ben London, Lise Getoor, and Bert Huang. Query-driven Active Surveying for Collective Classification. Workshop on Mining and Learning with Graphs, 2012.
Xuan Son Nguyen. GeomNet: A Neural Network Based on Riemannian Geometries of SPD Matrix Space and Cholesky Space for Saeleton-Basee Interaction Recognition. In ICCV, pp. 13379- 13389, 2021.
Xuan Son Nguyen. A Gyrovector Space Approach for Symmetric Positive Semi-definite Matrix Learning. In ECCV. pp. 52-68, 2022a.
Xuan Son Nguyen. The Gyro-Structure of Some Matrix Manifolds, In NeurIPS, pp. 26618-26630, 2022b.
Xuan Son Nguyen and Shuo Yang. Building Neural Networks Matrix Manifolds: A Gyrovector Space Approach. In ICML, pp. 26031-26062, 2023.
Xuan Son Nguyen, Luc Brun, Olivier Lézoray, and Sébastien Bougleux. A Neural Network Based on SPD Manifold Learning for Skeleton-basee Hand Gesture Recognition. In CVPR, pp. 12036- 12045, 2019.
Xuan Son Nguyen, Shuo Yang, and Aymeric Histace. Matrix Manifold Neural Networks++. In ICLR, 2024.
Spiros Nikolopoulos. MAMEM EEG SSVEP Dataset II (256 channels, 11 subjects, 5 frequencies presented simultancously). 2021.
Yue-Ting Pan, Jing-Lun Chou, and Chun-Shu Wei. MAtt: A Manifold Attention Network for EEG wttp///....... 01986
Decoding. CoRR, abs/2210.01986, 2022.
Margaux Perrin, Emmanuel Maby, Sébastien Daligault, Olivier Bertrand, and Jérémic Mattout. Ob- jective and Subjective Evaluation of Online Error Correction during P300-Based Spelling. Adv. Hum. Comput. Interact., 0012:5782551155829555 2012.
Sawyer. Computing the Iwasawa Decomposition of the Classical Lie Groups of Noncompact Type Using the QR Decomposition. Linear Algebra and its Applications, 493:573-579, 2016.
Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Hggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep Learning with Convolutional Neural Networks for EEG Decoding and Visual- ization. Human Brain Mapping, 38(1 11:5531-554200 2017.
Prithviraj Sen, Galileo Mark Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi-Rad. Collective Classification in Network Data. Al Magazine, 29(3)993-106, 2008.
Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic Neural Networks++. CORR, abs/2006.08210, 2021. URL http::/pp...//allllooo///aaaaae/
08210
Sho Sonoda, Isao Ishikawa, and Masahiro Ikeda. Fully-Comnected Network Noncompact Sym- metric Space and Ridgelet Transform based on Helgason-Founr Fourier Analysis. ICML, pp. 20405- 20422, 2022.
Max van Spengler, Erwin Berkhout, and Pascal Mettes. Poincaré ResNet. In ICCV, pp. 5396-5405, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In NIPS, pp. 1998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li, and Yoshua Bengio. Graph Attention Networks. CoRR, 1091000001090.... 2018. URL https://arxxx
axg/abs/171in 10900
Ming-Xi Xi Wang. Laplacian Eigenspaces, Horocycles and Neuron Models on Hyperbolic Spaces, 2021.
14

---

Published as a conference paper at ICLR 2025
Rui Wang, Xiao-Jun Wu, and Josef Kittler. SymNet: A Simple Symmetric Positive Definite Mani- fold Deep Learning Method for Image Set Classification. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-15, 2021.
Chun-Shu Wei, Toshiaki Koike-Akino, and Ye Wang. Spatial Component-wise Convolutional Net- work (SCCNet) for Motor- Imagery EEG Classification. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). pp. 328-331, 2019.
Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate Independent Convo- lutional Networks Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds. CoRR, abs/2106.06020, 2021. URL htttss::///aatt wttp://rgg.bb/......///// 06020
Tao Yu and Christopher De Sa. Random Laplacian Features for Learning with Hyperbolic Space. In ICLR, 2023.
Xingjian Zhen, Rudrasis Chakraborty, Nicholas Vogt, Barbara B. Bendlin, and Vikas Singh. Di- lated Convolutional Neural Networks for Sequential Manifold- Valued Data. In ICCV, pp. 10620- 10630, 2019.
A IMPLEMENTATION DETAILS
In this section, we provide details our implementations of FC layers and the attention module as well as the associated backpropagation procedures.
A.1 FC LAYERS
Input: I E X.
Trainable parameters: ky E K, aj E a, 1, E X,j I,..., m. m. N N.
Output: y E X.
In the case of SPD manifolds, we note that:
= Om (the group of m x m orthogonal matrices).
A is the subgroup of x diagonal matrices with positive diagonal entries.
.N is the subgroup of m m upper- tripe-triangular matrices with diagonal entries
The dimensions of input, output and trainable parameters then can be inferred accordingly (a, Rm. = 1,..., m..
The computations performed by an FC layer are follows.
Step 1: Compute Spj @ h g where p; = h,,,I = gK,hj.8 E G.j = I,...,m. m
Step 2: Compute gj E GG.j 1, ...., m such that kj (ep, @ x| = gy
Step 3: Compute H) = 1,..., m from the Iwasawa decomposition of 95 i.e., we need to
determine the map H G + such that 9, K (x
Step 4: Compute Be, (Op; @ x).j 1,
Bg, (Op, e (aj, H( 1)
Step 5: Compute the output of the FC layer as
B5 (Opr @ @x)
Be. (8pm e @x) 1) K
y = nex
Be, (ep2 2x)
15

---

Published as a conference paper at ICLR 2025
Backpropagation Below desribe the backpropagation procedure for FC layers in the case of SPD manifolds.
Step 1: Let p, o ujs,uy and Ug be eigen decompositions of pj and æ, respectively, where Uj Ur are orthogonal matrices and sj. 8x are diagonal matrices. Then
Py + x - sj 2 Ug K.
Eigen decompositions are differentiable operations in the Tensorflow and Pytorch frameworks. Please refer to lonescu al. (2015) for detailed discussion of gradient computations for singu- lar value decompositions and eigen decompositions.
Step 2: Compute 9j = k 5 " U.s..
Step 3: Let 9 G and g kan with l K A, and E N. Then
g g= a² k2 kan = n² a2n = n² a(n a)².
Let g² g= ca. be the Cholesky decomposition of g¹ (c is a lower triangular matrix), and let be the diagonal matrix that contains the main diagonal of Then n (cs-1)7 and The map H : G + is then given by (g) = log(a). We see that the map H can be determined from Cholesky decomposition which is differentiable operation in the Tensorflow and Pytorch frameworks.
Step 4: Backpropagation proceeds as normal with the computation of the inner product (aj, (g; )}.
Step 5: The output of the FC layer computed
Bg (Op1 @ x)
1) nT.
y nxx
(ep2 x)
-Be.. (8pm + xx)
which involves only a matrix product.
A.2 ATTENTION MODULE
The pipeline of the attention module is given in Fig. (()).
Input: A sequence of points zy E X,j = I,....I.
Trainable parameters: Those include trainable parameters of the three FC layers and c1, 02 IR: (used by the attention function).
Output: A sequence of points y
In the case of SPD manifolds, the dimensions of input, output and trainable parameters follow from those in FC layers (see above).
The computations performed by this module are as follows.
Step 1: Apply the three FC layers the sequence input points xy and obtain three sequences of points (a:) (queries), (zz)/=1 (keys), and (u;)}=1 (values):
(g)))=1 fi.m ((=;))=1); (2,) fin. ((;))))) (0,))=1= fin
where lū (.), fin (-), and (.,(() (.) are the linear transformations performed by the three FC layers.
Step 2: Compute the similarities between queries and keys R'J,J'; = =,.... from the Rieman- nian distance function as
テ = = , C2. Step 3: Compute the attention weights J'' = 1, ,.....,. as
(TTz)) softmax ((TT)))
Step 4: Perform the midpoint operation to get the sequence of output points yy. 1,...,
.,.
yy fmid( Tys1/
where my is the attention weight associated with V/-
16

---

Published as a conference paper at ICLR 2025
Backpropagation We desribe the backpropagation procedure for the attention module in the case of SPD manifolds. Here we only concern with the two blocks "Riemannian distance" and "Midpoint operation". For AttSymSpd- GI, the Riemannian distance used in the "Riemannian distance" block given by
d(x, y) = log(y 1zy- *)]l,
where 2;, y E Sym The term y 2 is computed
= us uu
where y usu is eigen decomposition of y. is orthonormal matrix and is diagonal matrix. The log(.) function also obtained from an eigen decomposition of its argument.
The "Midpoint operation" block performs the wFM operation under Log-Euue Log-Euclidean framework. Let {æz, wy)f=1 be set of points zj E Sym, with associated weights w; E IR, where wi > 0 and Σj=1 wy 1. Then the wFM of these points is given by
(PM((z,,yy)++) = exp Σ my log(1z))
The exp(.) function is computed from an eigen decomposition as exp(y) u exp(s)u", where is an orthonormal matrix and is diagonal matrix. Thus all functions involved in the computation of wFM are based on eigen decompositions and so backpropagation proceeds as explained above.
B EXPERIMENTAL DETAILS
B. IMAGE CLASSIHICATION
B.1.1 DATASETS
CIFAR-10 and CIFAR-100 datasets contain 60K 32 x 32 colored images from 10 and 100 different classes, respectively. We use the dataset split implemented in PyTorch, which has 50K training images and 10K testing images.
CIFAR-10 and CIFAR-100 (Krizhevsky 2009)
1.1.2 EXPERIMENTAL SETTINGS
BonidearnPPonccr--BB.. and Euclidean- -oinidarnn-iccar----- H have the same architecture which consists the ResNet-18 and the Poincaré MLR. Here we only present the Poincaré MLR. Let L be the number of classes, then MLR computes the probability of each the output classes as
Network architecture
Ceaidean-PPiicarée
explaf x b1) prop(y = (1x) E/=1 exp(a? x -b,)
x exp(af x br),
(6)
where E R the input, bj RR 0, IRm j I,...,L. L are model parameters. One can express (Lebanon & Lafferty 2004) Eq.
6 as
prop(y = (|x) x exp(sign(a (af bi)llaalld(a Ha,,t))
where d(x, H by is the distance between r and hyperplane H.E (see Section 4.1 In the Poincaré
MLR (Ganea et al. 20185), Eq (7) is written as
2
(1-11-
2 Pi M x, oz)
pt DM x ax ))
prop(y (|x) exp (1
|pp1 (0).
12)
| sinh
(1-
(8)
where x, Pi E Bm, 01 Tp B,
L.
Enclideam-PPincate uses Eq. 8 to compute the probability of each of the output classes. constructed by replacing the point-to- hyperplane distance in Eq. (8) with our proposed distance and the one from Fan et al. (2023).
and Muclidean-Poiccate
respectively.
17

---

Published as a conference paper at ICLR 2025
Table 4: Computation times (seconds) per epoch for image classification experiments (measured on a Quadro RIX 8000 GPU).
Method
Euclidean- Haliidaann--oncre---H
Fuclidean-Poincareae
CIFAR-10
52
57
59
CIFAR-100
90
114
118
Hyperparameterss We follow closely the settings in DeVries & Taylor (2017); ; Bdeir et al. 20224) Random mirroring and cropping are used for training. Thee batch size and number of epochs are set to 128 and 200, respectively. The learning rate and weight decay set to le and 5e 4, respectively. The training epochs are set to 60, 120, and 160 for adaptive learning rate scheduling where the gamma factor set to 0.2.
Optimization and evaluation All models are implemented in Pytorch. We use the library Geoopt (Kochurov et al., 2020) for Riemannian optimization. RiemamnianSGD is used to train the networks. Results averaged over 5 runs for each model. We use Quadro RTX 8000 GPU for all experiments.
B.1.3 MORE RESULTS
The computation times of for image classification experiments are presented in Tab. 4 can be observed that Euclidean- Poincaré-B B has high computational costs compared to its competitors, while Euclidean-Poincart the fastest method among the three methods.
H. Soniddaan-oincar---G and
B.2 EEG SIGNAL CLASSIFICATION
B.2.1 DATASETS
BCIC-IV-2a consists EEG data captured from 9 subjects. The cue-based BCI paradigm consists of 4 different motor imagery tasks, namely the imagination of movement of the left hand (class 1), right hand (class 2), both feet (class 3), and tongue (class 4). Two sessions on different days are recorded for each subject. Each session comprised of runs separated by short breaks. One run consists of 48 trials (12 trials for each of the 4 possible classes), yielding a total of 288 trials per session. The signals are recorded with 22 Ag/AgCI sensors (with inter-electrode distances of 3.5 cm) and sampled at 250 Hz. They are bandpass- filtereed between 0.5 Hz and 100 Hz.
MAMEM-SSVEP-II It consists of EEG data with 256 channels captured from 11 subjects exe- cuting SSVEP-based experimental protocol. Five different frequencies (6.66. 7.50, 8.57, 10.00 and 12.00 Hz) are used for the visual stimulation, and the EGI 300 Geodesic EEG System (GES 300), using 256-channel HydroCel Geodesic Sensor Net (HCGSN) and a sampling rate of 250 Hz is used to capture the signals.
It consists of EEG data captured from 26 subjects. The EEG electrode placement fol- lows the extended 1020 system. Five sessions (60 trials for the first sessions and 100 trials for the last session) are recorded for each subject, and the duration of single EEG trial is 1.25 seconds. The signals are recorded with passive Ag/AgCI sensors (VSM-CTF compatible system) and sam- pled at 600 Hz. Sixteen subjects released in the early stage of the Kaggle competition are used in our experiments.
BCI-NER
B.2.2 EXPERIMENTAL SETTINGS
Network architecture Inspired by Huang Gool (017). our network applies a number of convo- lutional layers the input data to extract features. The sequence of extracted features is divided into nonoverlapping subsequences, each of them forms an SPD matrix (Huang & Gool 2017). These
https:// htt////www www. www.wwwwww....w......wwww.wwwwww Beggle... com/ c/wnriia..... -eaichhlll 1enge
18

---

Published as a conference paper at ICLR 2025
procedures create sequence of SPD matrices, which are fed to the attention block (see Section 5.2) Each output SPD matrix of the attention block is projected to the tangent space at the identity matrix via the logarithmic map (Huang & Gool 2017). The resulting matrices transformed into vectors, which are then concatenated to create final features for classification. The network architecture is illustrated in Fig. (b).
For AudLLS we use the distance derived Proposition 4.4 with o(.) = log(.) to build FC layers in the attention module. As noted in Section our definition of hyperplanes and our derived distance match the definition of SPD hypergyroplanes and the distance between an SPD matrix and
an SPD hypergyroplane, respectively. Thus we can use the method in Nguyen et al. (2024) for our purposes. Let r E Sym, be the input of an FC layer, and let ((.n) = (EtePu..) BBe
F, ",.aa,))" P(t.). Q(Ij)) #y 1 j, ......" (see Appendix L.7.2 for the definitions of operations tBte Ste and the SPD inner product (...)a). Then the output y of the FC layer computed as
= exp ((=0.)))i>-)))
where 2(1)) given by
(u))(()))
if j
(an()))
ifl <
2(15)
(o))
if i
For AttSymSpd-GI, the output y of the FC layer in the attention module is computed
y == exp((
u. (2))),,
where (y(x) (so) @ z). Py E Sym ( exp(tay) K., 1..... m. For param- eters P. 9, we model them on the space of symmetric matrices, and apply the exponential map to obtain SPD matrices (López et 2021).
The map H G + a is computed as follows. Let E G and = kan with k E K, a E A, and kTTTann kan n n2, which shows that a cxp(H(g)) and n can be determined from LDL decomposition of g"
E N. Then g" 9 n. 1
To compute the wFM for the midpoint operation, we rely on Log-Euclidean framework. Let {zj, my IL m1j]==1 be a set of points Ij E Sym with associated weights wy E R, where wj > 0
and Σ Wj = 1. Then the wFM of these points is given by
L wy log(z)))
WF = exp Σ
Hyperparameters To create sequences of SPD matrices for the attention block, the numbers of nonoverlapping subsequences are set to 4, 6, and 4 on BCIC-IV-2a, MAMEM-SSVEP-II and BCI- NER datasets, respectively. The number of convolutional layers is set to 2. The numbers of output channels of the first and second convolutional layers are set to 20 and 15, respectively. The sizes of output SPD matrices of FC layers in the attention block are set to 6 6, 4 x 4, and 4 4 BCIC- IV-2a, MAMEM-SSVEPA and BCI-NER datasets, respectively. The numbers of epochs are set to 400, 100, and 100 BCIC-IV-24 MAMEM-SSVEP-IN and BCI-NER datasets, respectively. The batch sizes are set to 128, 64, and 64 on BCIC-IV-2a, MAMEM-SSVEP. and BCI-NER datasets, respectively (Pan et al. 2022). The learning rate and weight decay respectively.
le 3 and le 1,
Optimization and evaluation All models are implemented in Tensorflow. Cross-entropy loss and Adam (Kingma& & Ba 2015) are used to train the network. Our evaluation protocol is based on Mane jet al. (2021): Pan et al. 2022);; Wei et al. 2019). For BCIC-IV-2a dataset, the session data of subject used as the training set whose 1/8 is used the validation set. The session 2 data of the same subject is used as the test set. For MAMEM -SVEPII (BCI-NER) dataset, the first sessions of subject are used as the training set whose 1/4 is used as the validation set. The fifth session of the same subject is used as the test set. In all experiments, the models that obtain the lowest losses on the validation sets used for testing. The results BCIC-IV-2a and HAMEM-SSVERD datasets
19

---

Published as a conference paper at ICLR 2025
Table 5:
Comparison of the numbers of parameters (MB) of AttSymSpd-GI and AttSymSpd-L.E
Dataset
BCIC-IV-2a
MAMEM-SSVEP-IT
BCT-NER
AuSymSpd-LE
0.034
0.024
0.034
AuSymSpd-Gi
0.007
0,013
0.022
Table 6: Effectiveness of the proposed FC layers and attention module on BCIC-IV-2a dataset.
Method
CovNet
JuSymSpd-Gl-Bimm
AuSSmmSpd-Gi 78.0814.8
73,0416.34
75.8245.1
computed from accuracies obtained over 10 runs for each subject, while those on BCI-NER dataset based on the AUC score. Results are averaged over 10 runs for each model. We use a Quadro RTX 8000 GPU for all experiments.
B.2.3 MORE RESULTS
Tab. reports the numbers learnable parameters of AtSSymSpd- GI and AtSSymSpd-LE, Results clearly show that AttSymSpd-GI uses far fewer parameters than AttSymSSd-LE, It is interesting to note that these networks give similar accuracies on BCIC-IV-2a dataset, but AttSymSpd-C has about 5x fewer parameters than AttSymSpd-L.E.
We also study the effectiveness of the proposed FC layers and attention module for EEG signal classification. To this end, we evaluate the performance of AtSSymSpd-GI in two cases:
The attention module is removed from the network. The resulting network is called Cov- Net. This allows us to validate the contribution of the attention module.
The FC layers in the attention module are replaced with Bimap layers (Huang & Gool 2017 Bimap layers are referred as FC convolution-like layers and arguably the most commonly used analogs of FC layers in SPD neural networks. The resulting network is called AttSymSpd-Gl-Bimee
Tab. 6 reports results of our experiments. can be observed that both the building blocks effective. In particular, the use of attention module leads to more than 5% improvement in mean accuracy, and the network based on our FC layers outperforms the one based on Bimap layers by more than 2% in terms of mean accuracy.
Finally, Tab. shows all the results from Tab. and additional results of some tfaee-o-thee-e meth- ods. It can be seen that AttSymSpd-LE outperforms state-of-the-arr methods on all the datasets, while AuSSmmSpdGG outperforms them on BCIC-IV-2a and MAMEM-SSVEP-II datasets.
C IMAGE GENERATION
In this section, perform image generation experiments using CIFAR-I0 and CIFAR-100 datasets. We design a new hyperbolic variational autoencoder (VAE) from HCNN Lorentz (Bdeir et al. 2024) in which we replace the hyperbolic wrapped normal distribution the Lorentz model with that in the Poincaré ball, and replace the Lorentz MLR with the Poincaré MLR. We use three different point-to-hypee hhypepl-y distances in the Poincaré MLR in our image classification experiments.
C.I THE LORENTZ MODEL
The Lorentz model Im of m-dimensional hyperbolic geometry is defined by the manifold Ly {z Vo...... .....,y Fm m= To > z Σn 2 -1} equipped with the Rie- mannian metric (.,,)z diag(-1.... 1). The Riemannian distance between two points x =
[co, Tm.) Imm² 1 [yo...., vml E Imm is given by dr, y) cosh -1 (wowo - -m Tays).
20

---

Published as a conference paper at ICLR 2025
Table 7: Accuracies of networks and state-of-the-art methods for EEG signal classification.
Method
BCIC-IV-2a
MAMEM-SSVEP-I
BCI-NER
ShallowNet (Schirrmeister et al.
2017
61.84=6.39
56.93:66977
71.86=2.64
EEGNet Lawhern et al. 2018
57.43-16.25
53.72:17.23
74.28:2.47
SCCNet Wei el al. 2019
71.95=
62.11+7.70
70.93+2.31
EEG-TCNet (Ingolisson al.
2020
67.0914.6
55.45.1.66
77.05
TCNet-Fusion MMusal lam
2021
56.52++3.0
45.00+6.4
70.46+2.9
FBCNet (Mane et al. 2022
71.45=4.4
53.0915.6
60.47=3.0
MBEEGSE Altuwaiji et al.
2022
64.58.16.0
56.4517.22
75.46.12.3
MAtt (Pan et al.
2022
74.71
65.5018.2
76.01
Graph-CSPNet Ju & Guan
2023
11.95++1.3
AttSymSpd-LE (Ours)
78.24 1 5.4
70.96 1 8.6
78.02 1 2.3
AttSymSpd-Gr (Ours)
78.08 1 4.8
67.24 + 7.4
75.88 1 2.2
Table The network architecture for image generation. The PROJ layer maps data I to Lmm. The H-PROJ -Bn and H-PROJ layers map data between Lm and B3m (see the text). The CONV and CONVTR layers are Lorentz analogs of the convolutional and transposed convolutional layers, respectively. The BN and RELU layers are Lorentz analogs of the batch nor- malization and ReLU layers, respectively. The FC-MEAN, FC-VAR, and FC layers are Lorentz analogs of the Euclidean FC layer, respectively. The SAMPLE layer generates random samples in Bn from the latent distribution of the network. The MLR layer is the Poincaré MLR. Convolutional layers and transposed convolutional layers have kernel sizes of 3 and of 4 x 4, respectively. and denote stride and zero padding, respectively.
Layer
CIFAR-10/ CIFAR- 100
ENCODER:
PROJ TT
8 8 X 3
CONV 65, BN RELU
4 x
65
FLATTEN
1025
> FC-MEAN129
129
+ FC-VAR 129 SOFTPLUS
129
DECODER:
H-PROJ L... Bm
128
+ SAMPLE (Bm)
128
- H-PROJ Bm -L.
129
FC257 BN RELU
257
RESHAPE
2 x
65
>> CONVTRB3.2.p + BN - RELU
4 X 33
+ 11.
+ BN - RELU
8 8 x 17
+ CONV 05,81.p1
8 8 x 65
H-PROJ, L,,.J.
8 8 x 64
MLR (B,))
8 8 8 3
C.2 NETWORK ARCHITECTURE
The network architecture is given in Tab. 8 The H-PROJ, 1oy B,,, layer maps the Lorentz model into the Poincaré ball via the diffeomorphism given as
(x), ........ Fm
T(Do, D1, Lmn)
Z0 +
21

---

Published as a conference paper at ICLR 2025
The H-PROJ Bm -Lm layer maps the Poincaré ball into the Lorentz model via the diffeomorphism given as
( x + ||=|1 201..... 2mm)
||
We briefly present the other layers below. Please refer to Bdeir et al. (0024) for details.
Lorentz FC layer
Vilefur b][
y LFC(x)
p(wx + b)
where are the input and output of the layer, respectively, w R pmm(x(+1)) and b Rm' and denote the bias and activation, respectively.
Lorentz convolutional layer Lorentz model. Thus the image can be seen as an ordered set of m-dimensional hyperbolic feature vectors. The Lorentz convolution is then performed as
Given an image, the feature of each image pixel mapped the
Vh.w LFC((Cat(( (1= (# + h,
H.W
M.W where {T w" ee h, are the features within the receptive field of the kernel, HCat(.) de- notes the concatenation of hyperbolic vectors, (h', w') denotes the starting position, and is the stride parameter.
Lorentz transposed convolutional layer The transposed convolutional layer works by swapping the forward and backward passes of the convolutional layer. This is achieved in the Lorentz model through origin padding between the features.
Lorentz batch normalization Given batch B of m features Iú, the traditional batch normaliza- tion algorithm can be described as
BN(x:) ④ T; mean(B)
+ v,
Vvar(B) +
where mean(B) Ex Li. var(B) ii (xt mean(B))² and v are parameters to re-scale and re-center the features.
For the Lorentz batch normalization layer, the Lorentzian centroid and the parallel transport opera- tion are used for re-centering, and the Fréchet variance and straight geodesics at the origin' 's tangent space used for re-scaling.
Lorentz ReLU
y= ( 3m
(1 ReLU ( Fmmm
where æ [zo.... ... 2m and y are the input and output of the layer, respectively.
Wrapped normal distribution The SAMPLE layer the method in Nagano et al. (2019) to generate random samples on B, Given a normal distribution parameterized by a hyperbolic mean vector h E Bh,, and a Fuclidean variance matrix Σ Rmxmm the layer performs the following operations:
Sample a Euclidean vector ū from the normal distribution. Saciiiionn/// N(00 2).
2. Compute 2.
3. Parallel transport v from the tangent space of the origin 0 to the tangent space of the hy- perbolic mean h to obtain tangent vector E T7, Bm
To-)((v)
||^]]
22

---

Published as a conference paper at ICLR 2025
Table 9: Reconstruction and generation FID of hyperbolic VAEs (lower is better).
CIFAR-10
CIFAR 100
Method
Rec. FID
Gen. FID
Rec. FID
Gen. FID
Lorentz-Poincar- Fan et al. 2023)
125.53:5594
69.11
110.36-11.50
6..32:16.34
Lorentz- Lorentz-Poincaré G Ganea et al. 2018b
39.68:1
49.9112.00
42.82:12.48
60.24:14.01
Corentz-Poincaré-B (Ours)
38.32++2.11
48.45+1.31
42.05
59.76++1.11
4. Map to Bm by applying the exponential map
expx (n) h dBM
( (Lanh
(1
|||||
|
||
where @M is the Möbius addition (see Appendix L.7.1 and z the final sample in Bm.
C.3 EXPERIMENTAL SETTINGS
Hyperparameters We adopt the hyperparameters from [deii et al. (2024). The curvature for the Lorentz model and the Poincaré ball is to 1. The learning rate and weight decay to 5e and 0, respectively. The batch size and number of epochs are set to 100. The KI. loss weight is set 0.024.
Optimization and evaluation All models are implemented in Pytorch. We use the library Geoopt (Kochurov et al. 2020) for Riemannian optimization. RiemannianAdam is used to train the networks. We use the reconstruction FID and generation FID to evaluate the networks. The reconstruction FID is computed by comparing test images with reconstructed validation images. A fixed random portion of 10K images in the training set is used as the validation set (Bdeir et al.) 2024). The generation FID is computed by generating random images from the latent distribution and comparing them with the test set. Results are averaged over 5 runs for each model. We use Quadro RTX 8000 GPU for all experiments.
C.4 RESULTS
Results are shown in Tab. 9 Our method achieves the best performances in terms of mean recon- struction FID and generation FID in all cases. We also observe that the h-distance is significantly outperformed by the g-distance and b-distance in these experiments.
D NATURAL LANGUAGE INFERENCE
In this section, we compare our method for constructing the point-to-hyperrpanee distance in Poincaré ball against those in Ganea et al. (2018b); Fan et al. (2023) by performing the same ex- periments Ganea et al.) (2018b) for textual entailment and detection of noisy prefixes. For the first task, one has to predict whether a sentence can be inferred from another sentence. The second task consists determining conducted on SNLI (Bowman al. 2015) and PREFIX datasets (Ganca et al., 2018b) for the first and second tasks, respectively. Our implementation is based on the open-source implementation of HypGRU (Ganea et al.) 2018b) that uses the Poincaré MLR classification layer. The compet- ing networks differ only in the computation of the point-to-hyperplart distance (see Tab. I in the Poincaré MLR.
sentence noisy prefix of another sentence. Experiments are
htttas///ithubb.cc//ooottt/////
https:///iittbbb..mm//////////// in
23

---

Published as a conference paper at ICLR 2025
Class probabilities
Hyperbolic MLR
Hyperbolic FFNN
Hyperbolic GRU
Hyperbolic GRU
Word embedding
Word embedding 2
Figure 3: The network architecture for natural language inference.
D.1 DATASETS
SNLI (Bowman et al. 2015) It consists of 570K training, 10K validation and 10K test sentence pairs. Similarly to Ganea et al. 2018b), The "contradiction" and "neutral" classes are merged into single class of negative sentence pairs, while the "entailment" class gives the positive pairs.
PREFIX (Ganea et al. 2018b) PREFIX-10%, PREFIX-30%, and PREFIX-500 are synthetic datasets, each of them consists of 500K training, 10K validation, and 10K test pairs. Each dataset built as follows. For each random first sentence of random length at most 20 and one random prefix of it, a second positive sentence is generated by randomly replacing Z% (Z is 10, 30, or 50) of the words of the prefix, and second negative sentence of same length is randomly generated. Word vocabulary size is 100.
D.2 EXPERIMENTAL SETTINGS
Network architecture We use the architecture the fully hyperbolic GRU (Ganea al. 2018b) illustrated in Fig. 3 The network consists of a hyperbolic GRU, hyperbolic feed forward neu- ral network (FFNN), and the Poincaré MLR (see Appendix B.1.2 The update equations of the hyperbolic GRU are given
a [logo (W" @M h 0M U 8 D b*)),
8 2z 0 b²)) The = s ((W diag(r:)) @M h BM @M di @ar b),
2t = a (Wo 8M h-1 0M U²
h4 h4 &M diag(=;) 8M (-h @M he).
where Ie E Bm, is the input at frame t, h4-1, bu Im the hidden states at frames t and t, respectively, W", W W e " b", b b E IR Bm,, "T,U=,U U E Xm are model parameters, @M is the Möbius addition (see Appendix L.7.I. multiplication (see Appendix L.7.1. a, y0 are activation functions, logo()) is the logarithmic map at (see Appendix L.7.1 and the function 88 (.) defined as
&M the Möbius matrix-vector
(z) expo (x(oov(()
where x E Bm and cxpo(.) ) is the exponential map at O (see Appendix L.7.1
Let T2, E Bm, be the outputs of the hyperbolic GRU corresponding the first and second sentences, respectively. Then the output of the hyperbolic FFNN is computed as
2 == T(W 8M @M W2 @M 22 @M bi @M da (#1, @ b2),
where W, W2 E Rm b, by E BB Bm, are model parameters, @ is the Möbius scalar multipli- cation (see Appendix I.7.1 and is activation function.
24

---

Published as a conference paper at ICLR 2025
Table 10: Accuracies of the competing networks for natural language inference.
Method
SNLI
PREFIX- 10%
PREFIX-30 30%
PREFIX-500
HypGRU Ganea et al.
2018b 2023
80.8910.17 17
96.75_0.40
87.59
76.45.10.61
HypGRU-H (Fan et al.
80.66+00.46
92.20-8.33
83.29+6.34
74.67+3.12
HypGRU-B (Ours)
01.01=00.35
97.03=0.11
87.69=0.04
76.25=0007
Hyperparameters The word and hidden state embedding dimensions as well as the number of output channels of the hyperbolic FFNN are 5. The number of epochs and batch size to 30 and 64, respectively. The learning rates for word embeddings and hyperbolic weights are set Le 1 and 1e- respectively. All activation functions in the hyperbolic GRU and hyperbolic FFNN are set to the identity function.
Optimization and evaluation We use cross-entropy as the loss function. Euclidean and hyper- bolic parameters are optimized using Adam (the learning rate set to 1e - 3) and Riemannian stochastic gradient descent (RSGD) (Bonnabel) [013]] Ganea et al. (0188). respectively. Results are averaged over 5 runs for each model. We use a Quadro RTX 8000 GPU for experiments.
D.3 RESULTS
Results of the competing networks are shown in Tab. I0 Our network outperforms HypGRU-H in terms of mean accuracy and standard deviation on all the datasets. Results also demonstrate that our proposed distance has the potential improve existing HNNs on the considered task.
E NODE CLASSIFICATION
In this section, we compare our method for constructing the point-t-hhyperple distance in Poincaré ball against those in Ganea et al. (2018)); Fan al. (2023) by performing node classi- fication experiments.
E. DATASETS
Disease (Chami et al., 2019) is the transductive variant of a dataset created by simulating the SIR diseasc spreading model where the label of node is whether the node was infected or not, and node features indicate the susceptibility to the disease.
a flight network dataset from OpenFlights.org where nodes represent airports, edges represent the airline Routes, and node labels are the populations of the country where the airport belongs.
Airport (Chami et al. 2019)
Pubmed Namata et al. 2012) It is standard benchmark describing citation networks where nodes represent scientific papers in the area of medicine, edges are citations between them, and node labels are academic (sub).
Cora/Sen et al. (2008) It is citation network where nodes represent scientific papers in the area of machine learning, edges are citations between them, and node labels are academic (sub)areas. Each publication in the dataset is described by 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.
The statistics of the four datasets summarized in Tab. 11 Following Chami al. (2019), 30/10/60 percent splits for Disease dataset, 70/15/15 percent splits for Airport dataset, and standard splits (Kipf & Welling 2017) with 20 train examples per class for Pubmed and Cora datasets.
25

---

Published as a conference paper at ICLR 2025
Datasct
#Nodes
#Edges
#Classes
#Fcatures
Disease
1044
1043
1000
Airport
3188
18631
4
Pubmed
19717
44338
3
500
Cora
2708
5429
1433
Table 11: Description of the datasets for node classification experiments.
Dataset
Discase
Airport
Pubmed
Cora
Hyperbolicity 6
8 = 0
8 =
6 3.5
= 11
HGCN-H
85.67 + 2.58
69.82 + 2.08
75.26 +:2 1.82
77.09 + 2.02
HGCN-G
88.98 + 1.96
84.78 + 1.48
76.02 + 1.09
77.47 + 1.15
HGCN-B (Ours)
89.05 0.78
85.04 0.97
75.89 + 0.78
77.90 + 1.00
Table 12:
Results of HGCN models for node classification. HGCN-H, HGCN-G, and HGCN-B are built on h-distance, g-distance, and b-distance, respectively. A lower hyperbolicity value ó means more hyperbolic.
E.2 EXPERIMENTAL SETTINGS
Network architecture We use the HGCN architecture in Chami et al. (2019) which consists of graph convolutional network (GCN) and MLR as the final layer for classification. Both the GCN and the MLR layer built the Poincaré ball.
Hyperparameters We follow closely the experimental settings Chami al. (2019). The num- ber of epochs is set to 5000. We use early stopping based on validation set performance with a patience of 100 epochs. The learning rate set to for all experiments. The weight decays for Diease and Airport datasets are set to 0. The weight decays for Pubmed and Cora datasets are set to le 4 and - 3, respectively. The number of dimensions set to 16. The number of layers the GCN is set to 3.
Optimization and evaluation The network is implemented in Pytorch and is trained using cross- entropy loss and Adam optimizer. Results are averaged over 10 random parameter initializations on the final test set.
E.3 RESULTS
Table 12 shows results of our experiments. We can observe that the b-distance gives the best per- formance in most cases, which is similar to our results on natural language inference experiments. can also be observed that the performance of h-distance can nearly match those of g-distance and h-distance on less hyperbolic datasets (high hyperbolicity values), i.e., Pubmed and Cora datasets. However, h-distance significantly outperformed by the g-distance and b-distance on more hy- perbolic datasets, i.e., Disease and Airport datasets. For those datasets, the b-distance achieves the best performances, demonstrating its effectiveness for datasets with stronger hierarchical structures. Furthermore, in all cases, the b-distance achieve the lowest standard deviation, suggesting that it can offer stable results in the considered application.
F COMPLEXITY ANALYSIS
We analyze the complexities of the two network building blocks based on G-invariant metrics on SPD manifolds. Let d and dout be the dimensions of input and output matrices of an FC layer, respectively. Let / be the number of input matrices of the attention module.
/
26

---

Published as a conference paper at ICLR 2025
FC layer: It has memory complexity O dout) and time complexity O dout).
Attention module: has memory complexity O(3 dows). The time complexity of the FC layers is (31d, dout). Both the "Riemannian distance" and "Midpoint operation" blocks have time complexity ((P-dout)
G CONNECTION OF OUR DERIVED POINT-TO-HHYEERPLANE DISTANCE WITH EXISTING WORKS
In the case where 5(t) = exp(taa K, the distance given in Eq. (2) has a direct connection with the composite distance (see Section 3.2). That is, the former is obtained by the action of functional a E on the latter, which is a vector-valued distance. This is similar to how the Helgason-Fourr Fourier transform (Helgason 1994) of a function X is formed. In particular, for any function f X, its Helgason-Foorree transform is defined as
f(),E)= //x (xt))xx((-i+++ ())(z,)))dz
where X, e E a", 0X, and A(x, E denotes the composite distance from the origin to the horocycle passing through the point E X with normal ç. Here the exponent (-iX + 2) (xx (xr is regarded as the action of functional -iÀ + o E a" on the vector-valued distance A(x, E) (Helgason) 1994 Sonoda et al. 2022).
The distance given in Eq. (2) is also closely related to random features on hyperbolic spaces Yu &
Sa 2023). Those features generated from map HyLa(.) B. + given
HyLaxne(o) exp where E Bm, & OR.. and b, E R. By generating m' random samples of tuple g) from appropriate distributions, isometry-invarinn kernel over hyperbolic space Bm- In the higher-rank symmetric space setting, the term Ax (2, {) will be replaced with the Euclidean inner product of vector and vector-valued distance, resulting in a similar formulation of the distance in Eq. (2)
E)) cos(AA b),
obtains feature map B., + R that approximates an
H COMPARISON OF OUR FC LAYERS AGAINST EXISTING ONES
In Huang & Gool (2017): Huang et (2018), the authors introduce Bimap and FRMap layers and refer them as FC convolution- layers. For both types of layers, each element of the output matrix linear combination of the elements of the input matrix, which is not the case in our FC layers. FRMap layers and our FC layers also differ in their outputs as the former do not produce points on the considered manifolds.
In Chakraborty et al. (2020), the weighted Fréchet Mean (wFM) is adopted to develop Riemannian convolutional layers. These layers cannot be easily extended build natural generalizations of Euclidean FC layers by simply treating FC layers as special cases of convolutional layers with full kernel size. This because the resulting FC layers will take as input set points on the considered manifold and therefore have no obvious connection with the linear transformation in Eq. (3)
FC layers for neural networks on hyperbolic spaces (Wang 2021 and symmetric spaces (Sonoda al. (022) include activation functions which are not used in our FC layers. Also, FC layers
in Sonoda et al. 2022); Wang (202I) do not output points on the considered spaces.
In Nguyen et al. (0024); Shimizu et al. (2021). FC layers are not built upon Busemann functions. Although the method in Nguyen et al. (2024) is designed for matrix manifolds, some of which are not covered by method (c.g., Grassmann manifolds), the former relies on differentiable forms of certain geometric quantities (c.g., the logarithmic map and parallel transport) which are not required by the latter.
Another line of work (Cohen et al. 2018 Weiler et al. 2021 develops Riemannian neural networks which are functions f X Th where is the number of output channels. The context of these works is different from ours since we aim to build neural networks which are functions ʃ X + X.
Otter adaptations also needed hut they heyond the scope of paper.
27

---

Published as a conference paper at ICLR 2025
D
P M
P tu
P GM 2.
{
(a)
(b)
Figure 4: Comparison of Poincaré hyperplanes and our hyperplanes in the Poincaré disk model.
I ADVANTAGES AND LIMITATIONS OF G-INVARIANT AND PEM METRICS
Our motivation for using G-invariant metrics is that such metrics always exist on the considered spaces (e.g., those induced from the Killing form (Helgason) 1979)). As result, the framework proposed in Section 4.4 is valid for all those spaces. Since Busemann functions associated with G-invariant metrics are determined (see the proof Proposition 4.9) by Busemann functions on the maximal abelian A of G' (recall that G KAN) which has much lower dimension than X, a major advantage of G-invariant metrics compared to PEM metrics is that the former allow FC layers scale better high-dimensional input and output matrices. However, due to the same reason, Busemann functions associated with G-invariant metrics do not fully capture the structure of G which might lead poor performance.
The use of PEM metrics is motivated by the fact that like Log-Euclidean metrics, they turn SPD manifolds into flat spaces (i.e., their sectional curvature is null everywhere) which considerably simplifies computation and analysis (e.g., Riemannian computations become Euclidean computa- tions in the logarithmic domain (Arsigny al.) 2005)). PEM metrics are also more general than Log-Euclidean ones. On the downside, those metrics do not yield full affine-invarianee (Arsigny) al., 20055] Lin 2019). This might break the geometric stability principle that plays a crucial role in geometric deep learning architectures (Bronstein al. 2017).
J COMPARISON OF THE B-DISTANCE AND G-DISTANCE
The difference between the b-distance and g-distance can be best explained in the Poincaré disk
model (see Appendix L.I). A Poincaré hyperplane (Ganea et al. 2018b defined as
Ha.p {ar E (log,(()) a)p = =),
where p E D, a E T,D' been shown (Ganea al. 20185) that hyperplane Ha,p can also be described
(0). and
(, the Riemannian metric of the Poincaré disk model. It has
Ha.p (ar ID: (-p BM xr a) 0}.
which is illustrated by the segment (in purple) in Fig. aa).
Since we use the Möbius addition @M to define the binary operation @), a hyperplane in our approach is characterized by
Hepp = fx E D: By(- x) = -].
where p D and & E OD. One can interpret Be(-p @M x) as the signed distance between the origin and the horocycle 1/(pp @M E) which is the unique horocycle (Hlggaonn) 1979) through -p @a x with normal g. It implies that o must lie on the horocycle 77 + x, {) Hence, points -p 4M must lie the (unique) horocycle (in purple) through the origin with normal E (see Fig. 4(b)). can be seen that the characterization our hyperplanes and that of Poincaré hyperplanes different, resulting in different formulations for the hhypeplo- distance.
28

---

Published as a conference paper at ICLR 2025
K LIMITATIONS OF OUR APPROACH
In our work, the distance between point and hyperplane is derived for all higher-rank symmetric spaces of noncompact type, and the proposed FC layers also designed for neural networks on those spaces. However, the attention module relies on the computation wFM which does not have glosed-form solution in the general case. To address this issue,
consider using the Karcher algorithm (Karcher 1977) which has proven effective the implementation of batch normalization layers in SPD neural networks (Brooks et al. 2019). This algorithm can be computationally expen- sive in practice due to its iterative nature. To alleviate this challenge, the authors of (Chakraborty (2020) introduced an efficient wFM estimator which worth investigating. One can also consider using the methods proposed in Lou et al. (2020). In particular, the one that relies on an exponential map reparameterization [Lezcano-aasaoo 2019) can offer an effective solution to our problem.
To obtain a closed form for the point-to-hyperplane distance, one has to derive closed form for the Busemann function on the considered spaces and Riemannian metrics which is not always trivial.
Our FC layers have memory complexity Odd dout) and time complexity O dowt). ). Even though our method significantly reduces the complexity of FC layers proposed in Nguyen et al. (2024), it still does not scale well to high-dimensional input and output matrices of FC layers. Our attention block is built from these FC layers and thus suffers from the same issue.
Finally, since our approach developed for symmetric spaces of noncompact type, cannot be applied to those of compact type, e.g., Grassmann manifolds that are also encountered in many machine learning applications.
L DEFINITIONS AND BASIC FACTS
L.1 HYPERBOLIC SPACES SYMMETRIC SPACES OF NONCOMPACT TYPE
Here we describe the Poincaré disk model of the 2-dimensional hyperbolic space from a symmetric space perspective. Denote by DD {x C ||a||| Riemannian metric
1} the open unit disk in C equipped with the
(u, v), (u, v)
|
where E T.D are tangent vectors at E ID Let G be the group defined as
{
G = SU.
a. b b,
|b] =
Let 80m be the group of m orthogonal matrices of determinant 1. Then ID can be identified as D SU 1)/S00..
The subgroups K, A, and N in the Iwasawa decomposition G KAN and the centralizer M of A in K are given by
10
0 E [0, 2m)
10
cosh
sinht
A
:tE R
sinht
cosh f
1 +
is
is
N
8 E R
is
is
[-1
0
M
0
-1
The group G acts on Di by isometries via the Möbius action defined as
ar + b bz +
g |
This map is conformal and maps circles and lines into circles and lines.
29

---

Published as a conference paper at ICLR 2025
L.2 PULLBACK EUCLIDEAN METRICS
Under PEM, the Riemannian operations are given by:
exp, (u) = x (x(x) + Da Du)),
(Ø(y) log, (1) De(z) (x(y) 6(x));
Tu- D Do D=
where expx()) (-), log,() and (.) are the exponential map, logarithmic map, and parallel trans- port, respectively.
L.3 SPD MANIFOLDS AS SYMMETRIC SPACES OF NONCOMPACT TYPE
The SPD manifold Symm is differentiable manifold of dimension m(m + 1)/2. The tangent space Tz Sym at point x E Sym of the manifold is isomorphic to Symm- The Riemannian metric is given by
(u, v)2 Tr( - -u= Tv),
where u, v E Ty Sym This metric is G-invariant.
Let G.Lm be the group of m invertible matrices, and let Om be the group of m X orthogonal matrices. Then Sym,, can be identified as
Sym GT.m/
Let G = GLm Then the subgroups K, A, and N in the Iwasawa decomposition G = K ANN are
given by
K Om-
A is the subgroup of x diagonal matrices with positive diagonal entries.
N is the subgroup of m upper- trianutri matrices with diagonal entries 1.
Any E G can be written g kan for exactly one triple (A: a, n) E K A x N, and the map K A N G sending (k, a, n) to kan is a diffeomorphism. The centralizer M of in K M = Ck (A) {k E K Kakaa ak for all A). which is the set of diagonal matrices with entries +1. The (transitive) action of G on Sym is defined as g[æ]] == grg" for any E G and E Sym
L.4 SYMMETRIC SPACES NONCOMPACT TYPE
Symmetric spaces of compact type and of noncompact type are interchanged by Cartan duality, Each of those can be further categorized into two classes.
Symmetric spaces of compact type have non-negative sectional curvature. The two classes of sym- metric spaces of compact type are:
Homogeneous spaces of compact Lie group defined by involution (class 1).
Compact Lie groups with bi-invariant metrics (class 2).
Symmetric spaces of noncompact type have non-positive sectional curvature and are diffeomorphic to Euclidean spaces. The two classes of symmetric spaces of noncompact type are:
Homogeneous spaces of a noncompact, noncomplex Lie group, by a maximal compact subgroup (class 3, dual to class 1).
Homogeneous spaces of a complex Lie group by real form (class 4, dual to class 2).
There is correspondence between symmetric spaces of noncompact type and semisimple Lie groups with trivial centre and compact factors: For any Lie group G with trivial centre and no
30

---

Published as a conference paper at ICLR 2025
TT0
Figure 5: The boundary OD of the Poincaré disk model ID is illustrated by the green circle. The boundary point& is normal both the horocycle r and the basic horocycle f/o The distance between the origin and the horocycle is -d.
compact factors, if we take maximal compact subgroup K, then the quotient X = G/K endowed with G-invariant Riemannian metric forms a symmetric space of noncompact type.
An important invariant of a symmetric space of noncompact type is its rank. The rank of symmetric space X of noncompact type is the maximum dimension of flats in X (flats in X are subspaces isometric to Euclidean spaces). A symmetric space of noncompact type can be of rank one or of higher-rank.
Rank one symmetric spaces noncompact type: The real, complex and quaternionic hy- perbolic spaces and the hyperbolic plane over the Cayley numbers.
Higher-rank symmetric spaces of noncompact type: All symmetric spaces of noncompact type of rank greater than one. Typical examples are SPD manifolds.
Many geometric properties of rank one symmetric spaces of noncompact type and those of higher- rank are distinet. Thus, might be challenging to extend a method designed for hyperbolic spaces to the higher-rank symmetric space setting. For instance, in the Poincaré model, the point-to- hyperplane distance resulted from geodesic projections (Chami et al.] (021) can be computed in closed-form (Ganea al. 20188). However, this is not the case for SPD manifolds (except those under specific Riemannian metrics, e.g. Log-Fuclidean metrics) (Nguyen & Yang 2023).
In the following, provide several examples to illustrate some concepts reviewed in Section 3.2
Geometric boundary Some examples of geometric boundaries symmetric spaces of noncom- pact type are given below:
(I) For the Poincaré disk model the 2-dimensional hyperbolic space described Section L. its geometric boundary OD is the unit circle 0D {a C: ||>||| = 1) (see Fig. 5).
(2) Consider the Poincaré model By of m-dimensionall hyperbolic geometry discussed in Sec- tion 3.1 Two geodesic rays 5 and 8' in Bmm asymptotic if and only if ((t) and (t()) converge to the same point of RT as t - co. Thus the geometric boundary ²Bm of IB, is naturally identified with the sphere of Euclidean radius centred at _
(3) For S, its geometric boundary i Sym consists of the left cosets
0Sy K/M = {E KM.
(4) If X is a complete mn-dimensional Riemannian manifold non-positive sectional curvature, then 0X is homeomorphic to gm- the (m 1) s)-sphe This is because for given base point X, there exists homeomorphism which associates to each unit vector u tangent to at z the class of the geodesic ray 5 which issues from with velocity vector
31

---

Published as a conference paper at ICLR 2025
Horocycles The horocycles of ID and Sym can be described follows.
(1) The geodesics in are circular ares perpendicular to the boundary ØD. All circular ares perpen- dicular the same point at OD can be seen parallel lines. Thus natural notion of horocycle is that circle tangent to OD (see Fig. 5).
(2) Let In be the for some k E K and a E A. In particular, it is the set of matrices having a² as diagonal matrix in the U DU decomposition with respect to the Ram-basis kie,),m where {e; hem,im, I....... is the standard basis of R (Bartolucci et al. (022).
identity matrix. Then any horocycle be written 17 Ko.N [m]]
Composite distances For Sym the composite distance A(z, {) from the origin o the horocy- cle passing through a point x Sym with normal is given (Sonoda al. 2022) by
A(x = 5 kM) = 2 log (k= (x)),,
where the map y Sym + A is determined by y vyy)v" with E Sym E N.
L.5 ANGLES
Definition L.I. A comparison triangle in E² for a triple of points (p, q. in X is triangle in the Euclidean plane with vertices p. Q, F such that d(p,9) d(p, F), d(g,() d(q. f), and d(p.r) d(p,F) Such triangle is unique up to isometry, and shall be denoted gp,,grr) The interior angle of (pp, q, at is called the comparison angle between q and p and is denoted Zp(g,r). The comparison angle is well-defined provided and r are both distinct from p.
Definition L.2. Let ô : [0, a] + X and 8' : [0, a'] + X be two geodesic paths with 8(0) = 80). Given t E (0,a) and E (0,a'), we consider the comparison triangle (88 8(t), ((')), and the comparison angle Zato) (8(1), ((()). The (Alexandrov) angle or the upper angle between the geodesic paths ō and 8' the number Lo,8' E [0, π] defined by:
L8,8 == lim sup Zo()) = lim sup 20000
((())
8-00 --0,e
L.6 BUSEMANN FUNCTIONS
Euclidean spaces Brrdsoon & Häfliger 2011) Let 5(t) = tu be ray in R", where u is a unit
vector. Then the Busemann function B B8=0( is given by
Bς(x) (I, u).
SPD manifolds under Log-Euclidean framework (Bonet et al. 2023) Let 5(t) = cxp(ta) be geodesic line in Sym where a E Sym,, and |||||| = Then the Busemann function B (.) given by
Be(x) = Tr(a log())).
L.7 OPERATIONS
L.7.1 POINCARÉ MODEL AND MÖBIUS GYROVECTOR SPACES
In the Poincaré model Bm, of m-dimensional hyperbolic geometry, the logarithmic map logo and exponential map expo(.) (.) are given
loga (x) = tanh -1 ([|=]] |>|||
x
expo(e) tanh([|e])) v
||ll|
where x E Bm \ (0) and v E ToBm \ (0).
The Möbius addition M is defined as
(1 2(x, 3) + |/ .r + (1 ||0 (2l))) 12)y
+ y
1 2(x,v) + 1 |
32

---

Published as a conference paper at ICLR 2025
where 2, y E Bm The Möbius subtraction @M is then defined as
e 2
Z.
The Möbius scalar multiplication @M is defined as
M I Sanh(rtanll -aah--((0 ([1==]))
|
where r R and x E Bm \ (0).
The Möbius matrix-vector multiplication @M is defined as
| ([1==1))
xM
M @M x = tanh
tanh
|
|IM
where E Im,. M E R" and M @M if xM 0. Note that the same notation
@ for the Möbius scalar multiplication and Möbius matrix-vector multiplication as in Ganea et al. (018b).
L.7.2
LE GYROVECTOR SPACES
For v,y E Sym the binary operation Ote and inverse operation Ote are given (Nguyen 2022a
Ste (I) + log(y)), =t===2"
where exp(.) denotes the matrix exponential The SPD inner product is defined
(x, v))²
M MATHEMATICAL PROOFS
M.1 PROOF OF PROPOSITION 4.4
Proof. We first recast a result from Chen et al. (2024a) (Lemma 3.5) in form of the following propo- sition.
Proposition M.1. Let Φ Sym + Symm be a diffeomorphism, E Sym E Sym V(0), and let Hib be the hyperplane defined
Hpb = Sym (2), a') =0},
where (p..)p the PEM at point p as given in the definition of SPD manifolds. Then the distance
drd Hob ,) between point Sym and hyperplane Hbb is given by
(z Heppp) [(((x) d(p), D (a)(a
||a'
where 11-1 is the norm induced by the Riemannian inner product (., .p.
By the triangle inequality,
d(5(0), d(xx,(00)) < (()) < d(5(0), 8(1) + d(x, 5(0)),
which gives
t - d(z, 5(0)) d(x, o(t)) t d(x,6(0)).
Thus
Mx, 8(0)
d( 8(1)
((0)
1
2t
2t
2t
As for function log(-), the meaning of function exp(.) should be clear from the context.
33

---

Published as a conference paper at ICLR 2025
which leads to lime++00
d(2,8(0))+t 1. Therefore
(z=
lim d(x, 0(1)
1-00
lim (d(x;, -1) d(r, 8(1)) +
t-00
2t.
lim 1
(d(x,; 8(1))2 - 12)
2/
1
lim
(11(xz) +(8(1))) (?)
2/
lim
(11(2))1 + ([(((())11 2(0(((()). +))
2/
lim (I1(2))12 + |taa| 2(ta, x(x)) -t)
++ 2t
1
lim
(lo(x)))² 2t(a, ((x)))
<o 2t
lim 1
(10(x))13) (a, (()))
2t
(a, (()))
By the definitions of the binary operation and inverse operation 3 we have
Pp + z = o o (Ø(x) (Ø(x) - d(p)).
Hence
|| @ llo(ep @ x) = 1Ø(z) - <(p)).
We then get
d(z, Hgpp) = d(r, (c(
-d(, (a.0(5p)z))
[0(x)) x())
= - -
(a, x(p)).
From Proposition M.
K(b()) x(p), Dpp(aa))
(((())=(() o(D), Datlaa) ||a!
dab (x, Hob
||a'11%
By the property of pullback metrics,
(a1, a2) (D)b(i))),
where 01,00 a2 T S. We deduce that
Dro(a')
|a'1l
can be seen that the unsigned distance [d( Hg,p)l has precisely the same form as d Hob )).
M.2 PROOF OF PROPOSITION 4.8
Proof. To prove (i), need a result from Kassel 2009
34

---

Published as a conference paper at ICLR 2025
Lemma M.2. Let p X at be the map sending =glo] X to p(g), where g G. For all F, X.
ll{(z) p(2')(( d(x,z').
Moreover, aa x' E exp((+))ob then d(xra²') = ||((x) p(a') Il|
Let -gK,y y hK where g, h G. Since the distance d(...) is G-invariant, have d(x,y) d(g ' "000"~1|.1111100 d(o, g h(o)).
Let g kak where a e exp(at), k k' K. Then
d(o,g "|oo)) - d(k |o), k kak
alo))
By Lemma M.2
Ko,a(ol) = |o(()) - p(a[o)))
||{(a)]
|u(g [h]]l-
Note that
1er = 1gg hKlls
Vgg 1(---1hl -KK, 9hkK9
VMmlg- 1h),(g (h)-
=Il(s=1h))
Therefore
ee d(x,y) y).
To prove (ii), note that for any &o E K, we have k[x] kgK kkiainı keainiK where
= kia,n l22 = kkj, kn E K, aj E A, n2 E N. Thus p(kg) = p(g). Similarly, we deduce that
(kh) = p(h). Therefore
(k[x), k(yl)s ((lkg))((kh)) (p(9),((h))
(z. v)s.
M.3 PROOF OF PROPOSITION 4.9
Proof. We first recast result from Bridson & Häfliger (001) (Lemma 10.26) in form of the fol- lowing lemma.
Lemma M.3. Let X be symmetric space of noncompact type and let G be group acting by isometries X. Suppose that h G leaves invariant geodesic line d R X and that h[8(t)] = ö(t + c) where e > 0. Let 8(0) and let N C G be the set of elements g E G such that - gh [zo] + zo as - 30. Then N fixes 8(00) E OX and leaves invariant the Busemann function associated to 8.
Let 6(1) = kexp(la) h exp(a) G. where |al| 1, k E K. Setting 8'(1) ---(()) Then
h() exp(a+ tt)
1-- +1) = 5
35

---

Published as a conference paper at ICLR 2025
Since the distance d(.) is G-invariant, for any X, we have
d(z, ((t)) = d(k [x) (t).
Let g E be such that [2]] = gK, and let y = 11 exp A(g)k: where nı E N, ki k For any
E N, n [o] + + cc. By Lemma M.3 (c 1), we deduce that ( (k [x]]
(o== (ni [x]). Hence
(x) d(ni "on
We thus have the following chain of equations
d(k [x), (t))) = d(ni exp A
d(exp A(g) K, 8'())²
(10)
(A(g) - la, A(g) - la)
= (A(g).A(g)) - 2t A(g)) ++.
By the triangle inequality,
((6' (t)) d(k [(k= (0)) & d(k-"(=).8 s d(8'(0), + d(k
which gives
t -- d(k- ((()) - d(k
(c)
Thus
d(k [z]] (0))
d(k [z] [x) (()) +t 1 2).
2t
2t
2t
which results in 1om
d(k
(e))4 1. Therefore
21
B (x)=
lim 5(1) -
1-:00
lim ((k--((a) [xt). (1))) -6
lim [x), -t) d(k-'[x), (t))) +
2t
lim 1 (2) 1).
2t
Using Eq. 10
we get
Bę(x) lim (A(g) 2t(a)
2t
- (o, A(g))
(a, H (g 1));
which concludes Proposition 4.9
M.4 PROOF OF COROLLARY 4.10
Proof. We have
Be Be 3 x)
d(x, H.,p) = d(x,p) p).
|| e p @ ælIs
Bg (1-
= =((x p)
e 0 x||s
We use the same notation A for the composite distance from the origin a horacycle as in Helgason
1994..
36

---

Published as a conference paper at ICLR 2025
Note that k - 1h- [ h- lelhh'ggh gg. By Propositions 4.8 and 4.9
d(a, p). Be (((h'gK) g
(a, H( hkk))
θ p + 5lls
which concludes Corollary 4.10
M.5 PROOF OF PROPOSITION 5.1
Proof. We first recast result from Bridson & Häfliger (201T) (Proposition 9.8) in form of the following proposition.
Proposition M.4. Lel X be a symmetric space of noncompact type with basepoint zo. Let E
OX and let 6, 8' be geodesic rays with 6(0) = (0) = 3o. 8(x)) = = and 8' ('(x)) = = Then
1
2sin(Z(5, lim ((6(t),8'())
1000 t
For any E o0), we have that
d()(t),8(1)) d(exp(ta) exp(ta')
||t(a a')ll
2t.
By Proposition M.4
2 2sin(((E,E))2) lim -d(8(1)8(())
22
We thus deduce that <() =
M.6 PROOF OF PROPOSITION 5.2
Proof. Let (ez))=1.....m be the standard basis of _,, and let E, = _00) where 6y(t)
exp(te;)K, m be geodesic rays. Then for y gK e X and any j E {l...., m}.
y (2)2) d(y, He (ey, H( )) (g
where H(g denotes the j-th dimension of (gg Thus
H(g = [x] ... um vm (T))².
Note that g = n exp(-11 )) with n E N and k E K. Therefore
g nexp((_u2(2 ... vm k,
which leads to y nexppp(p1pp) (1)))= ... Una (x)))
37


Reviews:
[{"review_id": "7bk57Qc0sc", "reviewer": "Reviewer_dvJ8", "summary": "This paper proposes neural network structures on symmetric spaces of noncompact types, such as hyperbolic spaces or symmetric positive-definite (SPD) matrix manifolds. For this purpose, an expression for the distance between a point and a hyperplane is derived and utilized to generalize the Euclidean fully connected (FC) layer and attention layer. The proposed neural network is applied to various problems to demonstrate its performance advantages.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed ideas of generalizing FC and attention layer to symmetric spaces are novel and seem reasonable. \n- They are also general enough to incorporate both hyperbolic spaces and SPD manifolds, which are non-Euclidean spaces of interest and frequent use in ML.\n- Most mathematical derivations seem rigorous (I could not follow all the details).\n- The paper made a great effort to show the empirical benefits of the proposed neural network by considering diverse benchmarks.", "weaknesses": "- Understanding this paper requires a solid background in the geometry of symmetric spaces, and Section 3.2, in particular, is highly abstract and challenging to follow. This complexity may reduce the paper’s accessibility for a broader machine learning audience. Maybe incorporating the materials about decomposition equations from Appendix G.1 to G.3 in the manuscript, along with an explanation of their significance, would improve readability.\n- The methods to forward propagate the FC layer and attention layer are explained, but it is not clearly defined what the inputs, outputs, dimensionalities, and trainable parameters of each layer are. Including equations or diagrams that summarize these details would enhance clarity. Additionally, there is limited discussion on the backward propagation process in each layer, particularly regarding gradient calculations. It would be helpful to specify whether gradient computation is feasible and, if so, describe the method.\n- The rationale for considering both PEM and G-invariant metrics, as well as the differences between using each metric is not sufficiently discussed. A brief comparison of PEM and G-invariant metrics, with an overview of their respective strengths and limitations within the proposed neural network architecture, would help readers understand the motivation for including both metrics and their potential effects on network performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MWUZX6PY4G", "reviewer": "Reviewer_dL7j", "summary": "This paper introduces a framework for calculating point-to-hyperplane distances within symmetric spaces of noncompact type. Building upon this theoretical foundation, the authors propose novel manifold learning blocks tailored for neural networks, particularly designing fully connected (FC) layers and an attention mechanism applicable to these spaces. The paper demonstrates the effectiveness of this approach through numerical experiments, particularly on EEG classification tasks.", "questions": "1. In Line 143, Add references following “Iwasawa decomposition of G” to provide readers with foundational context.\n2. In Line 233, Correct “\\(\\Vert\\ldot\\Vert\\)” to “\\Vert\\cdot\\Vert” for accuracy in notation.\n3. In Corollary 4.3, There seems to be an extraneous dot in the formula. \n4. In Definition 4.5, Shouldn’t the definition of “addition” assume an abelian group structure for coherence with traditional addition in symmetric spaces?\n5. In Definition 4.7, The formula for “ g = k \\exp(\\mu) ” in line 342 could benefit from an intuitive explanation.\n6. In Proposition 4.12, The construction of the FC layer from this proposition is not immediately intuitive. Adding a diagram or further expanding on its practical implications would help readers grasp the proposed structure better.\n7.  Is there complexity analysis for the two proposed blocks. such as in the supp.?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work presents a well-constructed theoretical basis by generalizing point-to-hyperplane distance formulations on symmetric spaces of noncompact type, encompassing both hyperbolic and SPD manifolds. This unified approach is a notable advancement that addresses the limitations in existing methodologies which often focus on narrower manifold types (e.g., Nguyen & Yang, 2023). The paper’s theoretical contribution strengthens its foundation, offering a comprehensive framework applicable across various symmetric spaces, potentially enhancing applications in machine learning on non-Euclidean geometries .\n2. The experimental results, particularly on EEG datasets, demonstrate the approach’s capability. Despite minor performance gains, the proposed model achieves competitive accuracy, and in some cases, it outperforms existing methods such as EEG-TCNet, Graph-CSPNet, and MBEEGSE. This highlights the framework’s potential for real-world applications in EEG signal processing.", "weaknesses": "Sections 4.5.1 and 4.5.2, which are the core practical contributions of this work, are difficult to follow. While the theoretical sections are clearly presented, the implementation of the proposed FC layers and attention mechanism in symmetric spaces feels briefly discussed and lacks an intuitive explanation. A more thorough discussion, with a step-by-step breakdown or additional illustrative examples, would greatly improve accessibility and clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H9ccQGfTML", "reviewer": "Reviewer_75bJ", "summary": "The paper presents a new method for defining neural networks on symmetric spaces of noncompact type. The method is derived by deriving consider how hyperplanes are defined in such spaces and how point-to-hyperplane distance can be computed. This follows from expressing inner products via Busemann functions. This is then used for defining neural networks be generalizing the observation that affine functions in Euclidean space can be expressed as a function of a point-to-hyperplane distance. By replacing quantities in Euclidean space with their symmetric space counter parts, linear layers are defined to define neural networks on these spaces.", "questions": "Questions / Remarks:\n\n1. What are the specific connection between the proposed formulation versus the previous approaches presented in Table 1. I may be incorrect, but my understanding is that Ganea et al., 2018b is specialized for Hyperbolic spaces; and the b-distance approach is the Busemann function specialized to Hyperbolic spaces via Section 4.3 / Corollary 4.3. Is there a deeper reason why the point-to-hyperplane distances would not reduce to be the same? A reason for this question is that my initial perception was that the proposed look into symmetric spaces generalized the hyperbolic space, and thus when specializing to hyperbolic spaces one should obtain the same distance function. In summary, it would be great if you could provide a detailed summary / comparison for why the distances obtained in Ganea et al. 2018b differs from those obtained through your Busemann function approach.\n\n2. Unsure if it is the original citation for the technique of affine maps as point-to-hyperplane distance functions, but Ganea et al., 2018b cites \"Hyperplane margin classifiers on the multinomial manifold\" by Lebanon & Lafferty. To this end, it would be useful to add this citation to Section 2.1 and perhaps further elaborate on the historical development of using point-to-hyperplane distances for affine maps.\n\nTypos / Minor Mistakes:\n - Definition of hyperbolic distance is incorrect on Line 106. There is a type in the inner term's denominator. It should be \"$(1 - \\Vert x \\Vert^2)$, the squared is in the wrong position.\n - wFM used on Line 470 before defined (in appendix Line 1192).\n - Missing \".\" punctuation on Line 425.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The approach provide a novel generalization of defining neural networks in the more general symmetric space of noncompact type. It is very appealing that the approach can be utilized on several types manifolds (Section 4.3)\n- The paper provides is mostly written well to explain the technical background of the material (caveat below).\n- Experimental results seem promising.", "weaknesses": "- The connection between the proposed approach and previous ones are not exactly clear. Mostly in how / why the are different (see Questions)\n- I think the narrative of eventually defining the FC layers in section 4.5 (and the attention mechanism) could be improved. Particularly, the I feel like the connection of expressing affine functions via point-to-hyperplane distances should be further elaborated (L396-404)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dUeu5bzWVs", "reviewer": "Reviewer_yrjK", "summary": "The paper presents a new formulation for the distance between a point and a hyperplane in symmetric spaces, i.e., spaces that generalize the commonly used hyperbolic spaces and the manifold of SPD matrices. Based on this distance, new FC and attention components of neural networks on such symmetric spaces are proposed. These neural networks are then demonstrated in the context of hyperbolic spaces and the SPD manifold, showing an advantage in performance compared to other existing methods.", "questions": "- Could the authors provide clarification on the choice of CIFAR-10 and CIFAR-100 datasets in light of the results and the available alternative datasets with stronger hierarchical structures?\n- Please explain why more hierarchical datasets were not included.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- **Solid Mathematical Foundation**: The paper introduces new definitions grounded in a well-developed mathematical framework.\n- **New Neural Network Architecture**: The authors propose a new neural network design, including fully connected (FC) layers and an attention mechanism, that operate on symmetric spaces, extending the current capabilities of neural networks in non-Euclidean geometries.", "weaknesses": "- **Presentation and Organization**: The paper's structure is fragmented, making it challenging to follow and fully appreciate its main contributions. Specifically:\n    - **Sections 2 and 3** contain too many short, disconnected paragraphs or subsections that do not establish a cohesive narrative or logical organization.\n    - **Section 4** would benefit from being divided into two distinct parts: one focused on the mathematical framework and the other on the proposed neural network. Since the neural network is the primary outcome, separating these would clarify the focus.\n    - Additional minor structural suggestions:\n        - **Section 4.3**: Consider renaming the section titled \"Examples\" to make its purpose clearer.\n        - **Figure 2(b)**: Rotating this figure by 90 degrees could improve readability.\n\n- **Experimental Results**: Some limitations in the experimental study reduce the impact of the findings:\n    - **Section 5.1**: There are many other datasets with a clearer hierarchical structure that typically benefit more from hyperbolic space representation. For example, gene expression and word-document datasets are often considered as benchmarks of embedding in hyperbolic space. Therefore, the choice of CIFAR-10 and CIFAR-100 as benchmarks is questionable and should be explained, especially considering the marginal improvements reported in Table 2.\n    - **Section 5.2**: There are multiple differences between the proposed methods and the baselines, with several moving parts. It remains unclear which component contributes most to the observed (marginal) improvement. Consider conducting an ablation study to isolate the effects of the different components. This would help clarify which aspects are most responsible for the performance improvements.\n    - **Runtime Analysis**: Since the accuracy improvements are relatively small, including runtime metrics alongside accuracy would provide a fuller perspective on the method's practical benefits.\n\n- **Limitations Statement**: The limitation statement at the end is very specific and could be extended to cover broader aspects of the proposed framework, e.g., scalability and practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)