Riemannian Residual Neural Networks
Authors: Isay Katsman
Eric Ming Chen
Sidhanth Holalkere
Anna Asch
Aaron Lou
Ser-Nam Lim
Christopher De Sa
Venue: NeurIPS 2023 poster (2023)
Keywords: neural network
riemannian
manifold
resnet
TL;DR: We construct generalized residual neural networks that can learn over Riemannian data.

Abstract: Recent methods in geometric deep learning have introduced various neural networks to operate over data that lie on Riemannian manifolds. Such networks are often necessary to learn well over graphs with a hierarchical structure or to learn over manifold-valued data encountered in the natural sciences. These networks are often inspired by and directly generalize standard Euclidean neural networks. However, extending Euclidean networks is difficult and has only been done for a select few manifolds. In this work, we examine the residual neural network (ResNet) and show how to extend this construction to general Riemannian manifolds in a geometrically principled manner. Originally introduced to help solve the vanishing gradient problem, ResNets have become ubiquitous in machine learning due to their beneficial learning properties, excellent empirical results, and easy-to-incorporate nature when building varied neural networks. We find that our Riemannian ResNets mirror these desirable properties: when compared to existing manifold neural networks designed to learn over hyperbolic space and the manifold of symmetric positive definite matrices, we outperform both kinds of networks in terms of relevant testing metrics and training dynamics.

Riemannian Residual Neural Networks
Isay Katsman" Yale University isay. kattsma@ayale.edee
Eric M. Chen', Sidhanth Holalkere" Cornell University {emc348, sh844)@cornell.cm
Anna Asch Cornell University aca89@cornell. edu
Aaron Lou Stanford University aaronlou@stanffrrd edu
Ser-Nam Lim
Christopher De Sa Cornell University cdesa@cs wornell. edu
University of Central Florida sernamθucf edu
Abstract
Recent methods in geometric deep learning have introduced various neural net- works to operate over data that lie Riemannian manifolds. Such networks are often necessary to learn well over graphs with hierarchical structure or to learn over manifold-valued data encountered in the natural sciences. These networks are often inspired by and directly generalize standard Euclidean neural networks. However, extending Euclidean networks difficult and has only been done for select few manifolds. In this work, we examine the residual neural network (ResNet) and show how extend this construction to general Riemannian man- ifolds in geometrically principled manner. Originally introduced to help solve the vanishing gradient problem, ResNets have become ubiquitous in machine learning due to their beneficial learning properties, excellent empirical results, and casy-to-incorporaee nature when building varied neural networks. find that our Riemannian ResNets mirror these desirable properties: when compared to existing manifold neural networks designed to learn over hyperbolic space and the manifold of symmetric positive definite matrices, outperform both kinds of networks in terms relevant testing metrics and training dynamics.
Introduction
In machine learning, common to represent data as vectors in Euclidean space (i.e. R"). The primary reason for such a choice is convenience, as this space has classical vectorial structure, a closed-form distance formula, and a simple inner-product computation. Moreover, the myriad existing Euclidean neural network constructions enable performant learning.
Despite the ubiquity and success of Euclidean embeddings, recent research [41 has brought attention to the fact that several kinds of complex data require manifold considerations. Such data are various and range from covariance matrices, represented as points on the manifold of symmetric positive definite (SPD) matrices [26), to angular orientations, represented as points on tori, found in the context of robotics [43] However, generalizing Euclidean neural network tools to manifold structures such these can be quite difficult in practice. Most prior works design network architectures for specific manifold
thereby inefficiently necessitating a specific design for each new manifold.
We address this issue by extending Residual Neural Networks 23
Riemannian manifolds in way that naturally captures the underlying geometry. We construct our network by parameterizing vector fields and leveraging geodesic structure (provided by the Riemannian exp map) to "add" the learned
indicates equal contribution. Work done while Meta AI.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

vectors to the imput points, thereby naturally generalizing a typical Euclidean residual addition. This process is illustrated in Figure Note that this strategy is exceptionally natural, only making use of inherent geodesic geometry, and works generally for all smooth manifolds. We refer to such networks as Riemannian residual neural networks.
Though the above approach is principled, it is under- specified, constructing an efficient learnable vector field for given manifold is often nontrivial. To re- solve this issue, we present general way to induce a learnable vector field for manifold M given only map M - ARA, Ideally, this map should cap- ture intrinsic manifold geometry. For example, in the context of Euclidean space, this map could consist of series of k projections onto hyperplanes. There is a natural equivalent of this hyperbolic space that instead projects to horospheres (horospheres corre- spond to hyperplanes in Euclidean space). More gen- erally, we propose feature map that once more relies only on geodesic information, consisting of projec- tion to random (or learned) geodesic balls. This final approach provides a fully geometric way to construct vector fields, and therefore natural residual networks, for any Riemannian manifold.
T TpM
10
p
7(t)
exp, (v)
Figure 1: generalized residual addition. The traditional Fuclidean formula <- p + v is generalized to p &- exp, where exp is the Rieman- nian exponential map. M is the manifold and TyM is the tangent space at p.
An illustration of manifold-
After introducing our general theory, we give con- crete manifestations of vector fields, and therefore residual neural networks, for hyperbolic space and the manifold of SPD matrices. We compare the performance of our Riemannian residual neural networks to that of existing manifold-speciff net- works hyperbolic space and on the manifold of SPD matrices, showing that our networks perform much better in terms of relevant metrics due to their improved adherence to manifold geometry.
Our contributions are as follows:
1. We introduce novel and principled generalization of residual neural networks to general Riemannian manifolds. Our construction relies only on knowledge of geodesics, which capture manifold geometry.
2. Theoretically, we show that our methodology better captures manifold geometry than pre- existing manifold-specine neural network constructions. Empirically, we apply general construction to hyperbolic space and to the manifold of SPD matrices. On various hyperbolic graph datasets (where hyperbolicity is measured by Gromov d-hyperbolicity) our method considerably outperforms existing work on both link prediction and node classification tasks. On various SPD covariance matrix classification datasets, similar conclusion holds.
3. Our method provides way to directly vary the geometry of a given neural network without having to construct particular operations on per-manifold basis. This provides the novel capability to directly compare the effect of geometric representation (in particular, evaluating the difference between given Riemannian manifold (M. g) and Euclidean space (R", ll2)) while fixing the network architecture.
Related Work
Our work is related to but distinctly different from existing neural ordinary differential equation (ODE) [9] literature as well a series of papers that have attempted generalizations of neural networks to specific manifolds such as hyperbolic space [17] and the manifold of SPD matrices (66].
2.1 Residual Networks and Neural ODEs
Residual networks (ResNets) were originally developed to enable training of larger networks, previ- ously prone to vanishing and exploding gradients [23]. Later on, many discovered that by adding
learned residual, ResNets are similar Euler's method ( 21 37 45 53). More specifically,
the ResNet represented by h411 = h [h, 0.) for h, E RD mimics the dynamics of the ODE
(h((t, t, 0). Neural ODEs are defined precisely as ODEs of this form, where dh(t) defined by di

---

the local dynamics are given by parameterized neural network. Similar to our work, Falorsi and Forré [1], Katsman al. [29]. Lou et al. [36], Mathieu and Nickel [8 generalize neural ODEs to Riemannian manifolds (further generalizing manifold-specific work such as Bose et al. 3 that does this for hyperbolic space). However, instead of using manifold's s vector fields to solve a neural ODE, we learn an objective by parameterizing the vector fields directly (Figure 2 Neural ODEs and their generalizations to manifolds parameterize a continuous collection of vector fields over time for single manifold in dynamic flow-like construction. Our method instead parameterizes a discrete collection of vector fields, entirely untethered from any notion of solving an ODE. This makes our construction strict generalization of both neural ODEs and their manifold equivalents
15 29 36 38
2.2
Riemannian Neural Networks
Past literature has attempted generalizations of Euclidean neural networks to number of manifolds.
Hyperbolic Space Ganea et al. [17 extended basic neural network operations (e.g. activation function, linear layer, recurrent architectures) to conform with the geometry of hyperbolic space through gyrovector constructions [5]] particular, they use gyrovector constructions [5T] to build analogues of activation functions, linear layers, and recurrent architectures, Building on this approach, Chami et al. [8] adapt these constructions to hyperbolic versions of the feature transformation and neighborhood aggregation steps found in message passing neural networks. Additionally, batch normalization for hyperbolic space was introduced Lou et al. [35]; hyperbolic attention network equivalents were introduced in Gülçehre et al. [20]. Although gyrovector constructions are algebraic and allow for generalization of neural network operations to hyperbolic space and beyond, we note that they do not capture intrinsic geodesic geometry. In particular,
note that the gyrovector-basee hyperbolic linear layer introduced Ganea et al. [17] reduces to a Euclidean matrix multiplication followed by a learned hyperbolic bias addition (see Appendix D.2p. Hence all -un-Euclidean learning for this case happens through the bias term. In an attempt to resolve this, further work has focused on imbuing these neural networks with more hyperbolic functions [10, 49] Chen et al. [10] notably constructs a hyperbolic residual layer by projecting an output onto the Lorentzian manifold. However, we emphasize that our construction is more general while being more geometrically principled as we work with fundamental manifold operations like the exponential map rather than relying on the niceties of Lorentz space.
Yu and De Sa (55) make use of randomized hyperbolic Laplacian features to learn in hyperbolic space. We note that the features learned are shallow and are constructed from specific manifestation of the Laplace- Laplace-Beltrami operator for hyperbolic space. In contrast, our method is general and enables non-shallow (i.e., multi-layer) feature learning.
SPD Manifold Neural network constructs have been extended to the manifold of symmetrie positive definite (SPD) matrices as well. In particular, SPDNet (26) is an example of widely adopted SPD manifold neural network which introduced SPD-specific layers analogous to Euclidean linear and ReLU layers. Building upon SPDNet, Brooks et al. [5] developed a batch normalization method to
be used with SPD data. Additionally, López et al. 34 adapted gyrocalculus constructions used in
hyperbolic space to the SPD manifold.
Symmetric Spaces Further work attempts generalization to symmetric spaces. Sonoda et [50] design fully-connected networks over noncompact symmetric spaces using particular theory from Helgason- Fourier analysis 25). and Chakraborty et al. (7 attempt to generalize several operations such as convolution to such spaces by adapting and developing a weighted Fréchet mean construction. We note that the Helgason-Fourte Fourier construction in Sonoda et al. [50] exploits a fairly particular structure, while the weighted Fréchet mean construction in Chakraborty et al. 77] is specifically introduced for convolution, which is not the focus of our work (we focus on residual connections).
Unlike any of the manifold-specific work described above, our residual network construction can be applied generally to any smooth manifold and constructed solely from geodesic information.
3 Background
In this section, we cover the necessary background for our paper; in particular, we introduce the reader to the necessary constructs from Riemannian geometry. For a detailed introduction Riemannian geometry, we refer the interested reader to textbooks such as Lee [32].
3

---

y00 & (z
x11)
exp200 (v!2))
(0))
o12)
Eg (2x
(1) exp,00 (vl1))
Figure 2: A visualization of a Riemannian residual neural network on a manifold M. Our model parameterizes vector fields manifold. At each layer in network, take a step from point in the direction of that vector field (brown), which is analogous to the residual step in ResNet.
3.1 Riemannian Geometry
topological manifold (M,g) of dimension n is a locally Euclidean space, meaning there exist homeomorphic functions (called "charts") whose domains both cover the manifold and map from the manifold into R (i.e. the manifold "looks like" R locally). smooth manifold is topological manifold for which the charts are not simply homcomorphic, but diffcomorphic, meaning they smooth bijections mapping into IR" and have smooth inverses. We denote T,MM as the tangent space point p of the manifold M. Further still, a Riemannian manifold (M,g) an n-dimensional smooth manifold with smooth collection of inner products (Ep)pE. for every tangent space T..MM The Riemannian metric y induces distance d M x M - IR on the manifold.
3.2 Geodesics and the Riemannian Exponential Map
Geodesies A geodesic is curve of minimal length between two points p.q E .M, and can be seen as the generalization of straight line in Euclidean space. Although a choice of Riemannian metric on .M appears to only define geometry locally on M, it induces global distances by integrating the length (of the "speed" vector the tangent space) of shortest path between two points:
d(p.g) = =inf /''' (nto)(((()))))))) dt
(1)
where 7 E ( M) is such that 70) and 7(1) q.
For p E M and v E Tp.M. there exists a unique geodesic % where - = p. 7(() = v and the domain of is large as possible. We call the maximal geodesic [32.
Exponential Map The Riemannian exponential map is way to map Tp.M to a neighborhood around p using geodesics. The relationship between the tangent space and the exponential map output can be thought of local linearization, meaning that we can perform typical Fuclidean operations in the tangent space before projecting to the manifold via the exponential map to capture the local on-manifold behavior corresponding to the tangent space operations. For p E M and v E T,M, the exponential map at is defined as exp, (v) = (1).
One can think of exp manifold generalization of Euclidean addition, since in the Euclidean case we have exp,(v) (v) v.
homeomorphism a continuous bijection with continuous inverse. Note that imposing Riemannian structure does not considerably limit the generality of our method, as smooth manifold that is Hausdorff and second countable has Riemannian metric (32).

---

x 11 (z
w(t) (xx
x h (x
w/ expxxs (vl
0 = exprol (w
( = (x()
(
M(1)
M
Figure 3: An overview of our generalized Riemannian Residual Neural Network (RResNet) method- ology. We start by mapping z²(0) E ( to ( E _M2) using a base point mapping h1 Then, using our paramterized vector field &, we compute residual x(1) := & ()) Finally, we project v(1) back onto the manifold using the Riemannian exp map, leaving with (1), This procedure can be iterated to produce a multi-layer Riemannian residual neural network that is capable of changing manifold representation on per layer basis.
3.3 Vector Fields
Let T be the tangent space to manifold M at point p. Like in Euclidean space, vector field assigns to each point p E M tangent vector X, E T,.MM smooth vector field assigns a tangent vector Xp E M... to each point M such that Xp varies smoothly in p.
Tangent Bundle The tangent bundle of a smooth manifold M the disjoint union of the tangent spaces M, for all p E M, denoted by TM := UpEM TpMM = L....epppp............._. v) le E T.M}.
Pushforward A derivative (also called a pushforward) a map f M + N between two manifolds is denoted by Dzf M - N. This is generalization of the classical Euclidean Jacobian (since R" is a manifold), and provides a way to relate tangent spaces at different points on different manifolds.
Pullback Given Φ MM + V a smooth map between manifolds and f : R. smooth function, the pullback of ff by & is the smooth function o f on M defined by (Ø* f))) f()(z)). When the map é implicit, we simply write f to mean the pullback of f by d.
3.4 Model Spaces in Riemannian Geometry
The three Riemannian model spaces are Euclidean space R", hyperbolic space H and spherical space S", that encompass all manifolds with constant sectional curvature. Hyperbolic space manifests in several representations like the Poincaré ball, Lorentz space, and the Klein model. We use the Poincaré ball model for our Riemannian ResNet design (see Appendix for more details on the Poincaré ball model).
3.5 SPD Manifold
Let SPD(n) be the manifold of symmetric positive definite (SPD) matrices. We recall from Gallier and Quaintance [16] that SPD(n) has Riemannian exponential map (at the identity) equivalent the matrix exponential. Two common metrics used for SPD(n) are the log- log-Euclidean metric [16], which induces flat structure on the matrices, and the canonical affine-invarian metric 12 which induces non-constant negative sectional curvature. The latter gives SPD(n) considerably less trivial geometry than that exhibited by the Riemannian model spaces [2] (see Appendix A for more details on SPD(n)).
4
Methodology
In this section, we provide the technical details behind Riemannian residual neural networks.
4.1 General Construction
We define a Riemannian Residual Neural Network (RResNet) on manifold M be a function M M defined by
5

---

f(z) = (m)) x0 x
(2)
20 = expa 1: (f((x((-1))
(4)
for x E M, where m is the number of layers and & : . + TM is a neural vector field over M.. This residual network construction is visualized for the purpose of intuition in Figure In practice, parameterizing function from an abstract manifold M to its tangent bundle is difficult. However, by the Whitney embedding theorem [3], can embed M + IR smoothly for some dimension D 2 dim.. As such, for a standard neural network nig R RRD + RD we can construct & by
4(z) == (r.)
(5)
where we note that T&M C RR. is linear subspace (making the projection operator well defined). Throughout the paper we call this the embedded vector field design We note that this is the same construction used for defining the vector field flow in Lou et al. [36], Mathicu and Nickel [38] Rozen et al. [4).
We also extend our construction to work in settings where the underlying manifold changes from layer to layer. In particular, for a sequence of manifolds _M (() (..,,,(m)) with (possibly learned) maps h4 ( + M our Riemannian ResNet f :M + M(m)) is given by
fxx x (m)
(6)
x (0) == x
(7)
x(() = (41 e [m]
(8)
with functions & (M TMO given as above. This generalization is visualized in Figure In practice, our ( will be different dimensional versions of the same geometric space (e.g. H" or I" for varying n). the starting and ending manifolds are the same, the maps h4 will simply be standard inclusions. When the starting and ending manifolds are different, the h; may be standard neural networks for which we project the output, or the h; may be specially design learnable maps that respect manifold geometry. As concrete example, our h, for the SPD case map from an SPD matrix of one dimension to another by conjugating with a Stiefel matrix (26). Furthermore, as shown in Appendix D our model is equivalent to the standard ResNet when the underlying manifold is IR?..
Comparison with Other Constructions We discuss how our construction compares with other methods in Appendix B but here we briefly note that unlike other methods, our presented approach is fully general and better conforms with manifold geometry.
4.2 Feature Map-Induced Vector Field Design
Most of the difficulty in application of our general vector field construction comes from the design of the learnable vector fields 4 ( + (M, Although we give an embedded vector field design above, it is not very principled geometrically. We would like to considerably restrict these vector fields so that their range informed by the underlying geometry of M. For this, we note that it is possible to induce vector field & : + TM for manifold M with any smooth map f:M + R In practice, this map should capture intrinsic geometric properties of M and can be viewed as a feature map, de facto linearization of M. Given an M, need only pass through ff to get its feature representation in R2 then note that since:
Dof TyM 12
we have an induced map:
(T,M)" where (Dpf)* is the pullback of D,J. Note that TU ~ R and (R=*)* = IR by the dual space isomorphism. Moreover (TpM)* = M, by the tangent-cotangent space isomrrphism 33 Hence,
(Dpf)*
we have the induced map:
(Drf); R + T.M.
$ 'Ideal vector tield design is in general nontrivial and the embedded vector field is not a good choice for all manifolds (see Appendix B).

---

Feature Map Design One possible way to simplify the design of the above vector field to further break down the map f M - E into k maps fi...., ........ f& M - R. where ideally, each map fi is constructed a similar way (c.g. performing some kind of geometric projection, where the ft vary obtained from (Dpf)", simply by both precomposing and postcomposing the aforementioned iso- morphisms, where relevant. (D,f); provides natural way to map from the feature representation to the tangent bundle. Thus, we may view the map e :M TM given by:
ly(x) = (Dz.f)(((x))
as deterministic vector field induced entirely by f.
Learnable Feature Map-Induced Vector Fields We can easily make the above vector field con- struction learnable by introducing a Euclidean neural network ng IR - I after to obtain (x)(()) = (D..)) (n)()())))
only in terms of the specifying parameters). As we shall see in the following subsection, this ends up being a very natural design decision.
In what follows, we shall consider only smooth feature maps f :M- + R& induced by a single parametric construction ge M - R, i.e. the dimensions of the output of f are given by different choices of 0 for the same underlying feature marl] This approach also has the benefit of a very simple interpretation of the induced vector field. Given feature maps ................................ 90, M - that comprise our overall feature map f M + RR our vector field is simply linear combination of the maps Vgo, :M- + TM. If the Io, are differentiable with respect to 0j we can even learn the f; themselves.
4.2.1 Manifold Manifestations
In this section, an effort to showcase how simple it is to apply our above theory to come up with natural vector field designs, present several constructions of manifold feature maps ge .M T that capture the underlying geometry of M for various choices of M. Namely, in this section we provide several examples f M + E that induce & .M + T.M, thereby giving rise Riemannian neural network by Section
Euclidean Space To build intuition, begin with instructive case. We consider designing a feature map for the Fuclidean space TR... A natural design would follow simply by considering hyperplane projection. Let a hyper- plane w² b- 0 be specified by IR.", b R. Then a natural feature map Iw, R >> IR parameterized by the hyperplane parameters is given by hyperplane projection I TuTx* x+b|
14
||
Hyperbolic Space We wish to construct a natural feature map for hyperbolic space. Seeking to follow the construc- tion given in the Euclidean context, we wish to find hyperbolic analog of hyperplanes. This provided to us via the notion of horospheres (24] Illustrated in Figure 4 horospheres naturally generalize hyperplanes to hyper- bolic space. We specify horosphere in the Poincaré ball model of hyperbolic space H" by a point of tangency S--1 and real value E R. Then natural feature map lw.b H + RR parameterized by the horosphere parameters would be given by horosphere projection [4]
Figure 4: Example of horosphere in the Poincaré ball representation of hy- perbolic space. In this particular two- dimensional case, the hyperbolic space Hi, is visualized via the Poincaré disk model, and the horosphere, shown in blue, is called horocycle.
(x)) -- log
+ b.
Symmetric Positive Definite Matrices The manifold of SPD matrices an example of manifold where there is no innate representation of hyperplane. Instead, given X E SPD(n), a reasonable feature map 9k SPD(n) R. parameterized by k. is to map X its kth largest eigenvalue: 9% (X) = Ap.
*We use the term "feature map" for both the overall feature map f .M + 1 and for the inducing construction M + R. This is well-defined since in our work we consider only feature maps f ..M. + 12 that are induced by some go M - IR.

---

General Manifolds For general manifolds there is no perfect analog of a hyperplane, and hence there is no immediately natural feature map. Although this is the case, possible to come up with reasonable alternative. We present such an alternative in Appendix B.4 together with pertinent experiments.
Example: Euclidean Space One motivation for the vector field construction = (faf) that in the Euclidean case, f will reduce a standard linear layer (because the maps and (D,f)* are linear), which, in combination with the Euclidean exp map, will produce a standard Euclidean residual neural network.
Explicitly, for the Euclidean case, note that our feature map f R" + R will, for example, take the form fx) Wx, W E & (here we have b = 0 and W has normalized row vectors). Then note that we have D = W and (Df) = WT, We see for the standard feature map-based construction, vector field f (3:) (P=f) takes the form (x) = WTW..
For the learnable case (which is standard for us, given that we learn Riemannian residual neural net- works), when the manifold is Euclidean space, the general expression ff,0(x) = (D,f)* becomes f (x) W no (Wr). When the feature maps are trivial projections (onto axis-aligned hyperplanes), we have W and (x)(()) = (x)). Thus our construction can be viewed as generalization of standard neural network.
Dataset
Discase
Airport 8
PubMed 3.5
CoRA
Hyperbolicity
Task
LP
NC
LP
NC
NC
LP
NC
Fac
59.8
32.5 32.5
92.01o.
60.9
83.3go.
48.210.7
82.5go.
23.8
Shallow
Hyp
63.5
45.5ss.
94.5so.
70.2
87.5
68.5
$7.62ua
22.0a1a
Eue-Mixed
49.611.1
35.21344
91.510a
68.3
86.011.
63.01ma
84.41u2
46.1
Hyp-Mixed
55.1g1a
56.911.5
93.3100
69.61n
83.810.
73.9_aa
85.610.
45.92na
MLP
72.61n.
28.812.
89.81o.s
68.61o.
84.110.9
72.41ma
83.1ass
51.511
HNN 17
75.110.
41.011.
90.81
80.5
94.9_oa
69.819.4
89.01m.1
54.610.4
RResNet Horo
98.410.3
76.8
95.220.1
96.9
95.02oa
72.3217
86.7a6.
52.4
Table 1: Above give graph task results for RResNet Horo compared with several non-graph-based
neural network baselines (baseline methods and metries are from Chami et al. (8). Test ROC AUC is the metric reported for link prediction (LP) and test Fl score is the metric reported for node classification (NC). Mean and standard deviation are given over five trials. Note that RResNet Horo considerably outperforms HNN on the most hyperbolic datasets, performing worse and worse as hyperbolicity increases, to a more extreme extent than previous methods that do not adhere to geometry closely (this is expected).
5 Experiments
In this section, we perform a series of experiments to evaluate the effectiveness of RResNets on tasks arising on different manifolds, In particular, we explore hyperbolic space and the SPD manifold.
5.1 Hyperbolic Space
We perform numerous experiments in the hyperbolic setting. The purpose is twofold:
1. We wish to illustrate that our construction in Section is not only more general, but also intrinsically more geometrically natural than pre-existing hyperbolic constructions such as HNN
[1]. and thus able to learn better over hyperbolic data.
2. We would like to highlight that non non-Euclidean learning benefits the most hyperbolic datasets. We can do this directly since our method provides way to vary the geometry of a fixed neural network architecture, thereby allowing us to directly investigate the effect of changing geometry from Euclidean to hyperbolic.
5.1.1 Direct Comparison Against Hyperbolic Neural Networks 17
To demonstrate the improvement of RResNet over HNN 17 we first perform node classification (NC) and link prediction (LP) tasks on graph datasets with low Gromov ǒ-hyperbolicity (8) which means the underlying structure of the data is highly hyperbolic. The RResNet model is given the
8

---

AFEW 13
FPHA 18
NTU RGB+D
HDM05 (3
SPDNet
33.24+0055 58
65. 1..39+1.48
=0.47
66. #0.992
SPDNeiBNN
35.39+0.93
65.03+1.s5
41. =0.92
67.25=+0.44
RResNet Invarantn
35.17 11.78
66.53 103. 64.66
41.00 10.50
67.91 11.27
RResNet Log- Euclidean
36.38
6/.58
42.99=0.23
69.80=1.51
Table 2:
We run our SPD manifold RResNet on four SPD matrix datasets and compare against SPDNet (26] and SPDNet with batch norm [. We report the mean and standard deviation of validation accuracies over five trials and bold which method performs the best.
name "RReNNet Horo." It utilizes a horosphere projection feature map-induced vector field described in Section 4 All model details given in Appendix C.2 We find that because we adhere well to the geometry, we attain good performance on datasets with low Gromov d-hyperbolicities (e.g. 6 0,8 = 1). As soon as the Gromov hyperbolicity increases considerably beyond that (e.g. 5 ===3. 3 = 11), performance begins to degrade since we are embedding non-hyperbolic data in an unnatural manifold geometry. Since we adhere to the manifold geometry more strongly than prior hyperbolic work. we see performance decay faster as Gromov hyperbolicity increases, as expected. In particular, we test on the very hyperbolic Disease (8 0) 8 and Airport (6 = 1) g datasets. We also test on the considerably less hyperbolic PubMed (6 = 3.5) 47 and CoRA (Ō 11) [46]
datasets. We use all of the non-grap oon-graph-bases baselines from Chami et al. 8 since we wish to see how much we can learn strictly from proper treatment of the embeddings (and no graph information). Table summarizes the performance of "RResNet Horo" relative to these baselines.
Moreover, we find considerable benefit from the feature map-induced vector field over an embedded vector field that simply uses a Euclidean network to map from a manifold point embedded in R". The horosphere projection captures geometry more accurately, and if we swap to an embedded vector field we see considerable accuracy drops on the two hardest hyperbolic tasks: Disease NC and Airport NC. In particular, for Disease NC the mean drops from 76.8 to 75.0, and for Airport NC we see a very large decrease from 96.9 to 83.0, indicating that geometry captured with well-designed feature map is especially important. We conduct a more thorough vector field ablation study in Appendix C.5
5.1.2 Impact of Geometry
A major strength of our method is that it allows one to investigate the direct effect of geometry in obtaining results, since the architecture can remain the same for various manifolds and geometries (as specified by the metric of given Riemannian manifold). This is well-illustrated in the most hyperbolic Disease NC setting, where swapping out hyperbolic for Euclidcan geometry in an RResNet induced by embedded vector field decreases the Fl score from 75.0 mean to 67.3 mean and induces a large amount of numerical stability, since standard deviation increases from 5.0 to 21.0. We conduct a more thorough geometry ablation study in Appendix C.5
5.2 SPD Manifold
A common application of SPD manifold-based models is learning over full-rank covariance matrices, which lie on the manifold of SPD matrices. We compare our RResNet to SPDNet (26) and SPDNet with batch norm (5] on four video classification datasets: AFEW [3, FPHA (18] NTU RGB+D
[8]. and HDMO5 39 Results are given in Table 2 Please see Appendix C.6 for details on the
experimental setup. For our RResNet design, we try two different metrics: the Bug-Buclidean metric [16] and the affine-invariant metric [2,42]] each of which captures the curvature of the SPD manifold differently. We find that adding learned residual improves performance and training dynamics over existing neural networks on SPD manifolds with little effect on runtime. We experiment with several vector field designs, which we outline in Appendix B The best vector field design (given in Section 4.2), also the one we use for all SPD experiments, necessitates eigenvalue computation. We note the cost of computing eigenvalues is not a detrimental feature of our approach since previous works (SPDNet (26). SPDNet with batchnorm (51) already make use of eigenvalue computatior[] Empirically, we observe that the beneficial effects of our RResNet construction are similar to those of the SPD batch norm introduced in Brooks et al. (5] (Table 2 Figure Appendix (C.)). In addition, we find that our operations are stable with ill-conditioned input matrices, which commonly oceur in the wild. To contrast, the batch norm computation in SPDNetBN, which relies on Karcher flow
One needs this computation for operations such as the Riemannian exp and log over the SPD manifold.

---

Dataset
Disease d 0
Airport 8 ==1
PubMed = 3.5
CoRA d = 11
Hyperbolicity
GCN 31
69.7
81.410.6
81.31oa 18.1
GNN
GAT [52
70.410.4
81.510.3
79.010.3
83.0.a.7
SAGE 22
69.1
82.1
77.4+22
77.9+2.4
SGC [54]
69.5
80.6
78.9
81.0+o.1
HGCN 8
74.5+0.9
80.3
79.9
GONN
Fully HNN
10
96.021.0
90.9
78.021.0
80.221.8
G-RResNet Horo
95.4=1.0
97.4=0.1
75.5+0.5
64.4+7.6
Table 3: Above we give node classification results for G-RResNet Horo compared with several graph- bash- neural network baselines (baseline methods and metrics are from Chami et al. (8). Test Fl score is the metric reported. Mean and standard deviation are given over five trials. Note that G-RResNet Horo obtains state-of-the-art result on Airport. As for the less hyperbolic datasets, G-RResNet Horo does worse on PubMed and does very poorly on CoRA, once more, expected geometry. The GNN label stands for "Graph Neural Networks" and the GGNN label stands for "Geometric Graph Neural Networks."
due to unsuitability of
suffers from numerical instability when the input matrices are nearly singular. Overall, we observe our RResNet with the affine-invariant metric outperforms existing work on FPHA, and our RResNet using the log-Euclln Euclidean metric outperforms existing work on AFEW, NTU RGB+D, and HDM05. Being able to directly interchange between two metrics while maintaining the same neural network design is an unique strength of our model.
28 35),
Riemannian Residual Graph Neural Networks
Following the initial comparison to non- won-graph-bare methods in Table we introduce a simple graph- based method by modifying RResNet Horo above. We take the previous model and pre-multiply the feature map output by the underlying graph adjacency matrix A in a manner akin to what happens with graph neural networks (54). This is the simple modification that we introduce the Riemannian ResNet to incorporate graph information; we call this method G-RResNet Horo. We compare directly against the graph-based methods in Chami et al. [ as well as against Fully Hyperbolic Neural Networks [0] and give results in Table 3 We test primarily on node classification since we found that almost all LP tasks are too simple and solved by methods in Chami et al. g (i.e., test ROC is greater than 95%). We also tune the matrix power of A for given dataset; full architectural details are given in Appendix C.2 Although this method is simple, we see further improvement and in fact attain state-of-the- art result for the Airport [8] dataset. Once more, as expected, we see considerable performance drop for the much less hyperbolic datasets, PubMed and CoRA.
Conclusion
We propose a general construction of residual neural networks on Riemannian manifolds. Our approach is natural geodesically-oriente generalization that can be applied more broadly than previous manifold-speciff work. Our introduced neural network construction is the first that de- couples geometry (i.e. the representation space expected for input to layers) from the architecture design (i.e. actual "wiring" of the layers). Moreover, we introduce a geometrically principled feature map-induced vector field design for the RResNet. We demonstrate that our methodology better captures underlying geometry than existing manifold-specifit neural network constructions. On a variety of tasks such node classification, link prediction, and covariance matrix classification, our method outperforms previous work. Finally, our RResNet's principled construction allows us to directly assess the effect of geometry a task, with neural network architecture held constant. We illustrate this by directly comparing the performance of two Riemannian metrics on the manifold of SPD matrices. We hope others will use work to better learn over data with nontrivial geometries in relevant fields, such as lattice quantum field theory, robotics, and computational chemistry.
I Limitations We rely fundamentally on knowledge of geodesics of the underlying manifold. As such, we assume that a closed form (or more generally, easily computable, differentiable form) is given for the Riemannian exponential map as well as for the tangent spaces.
10

---

Acknowledgements
We would like thank Facebook AI for funding equipment that made this work possible. addition, we thank the National Science Foundation for awarding Prof. Christopher De Sa a grant that helps fund this research effort (NSF IIS-2008 102) and for supporting both Isay Katsman and Aaron Lou with graduate research fellowships. We would also like to acknowledge Prof. David Bindel for his useful insights on the numerics of SPD matrices.
References
[1] Cem Anil, James Lucas, and Roger B. Grosse. Sorting out lipschitz function approximation. In ICML, 2019.
[2] Rajendra Bhatia. Geometry of positive matrices. In Positive Definite Matrices, pages 201-236. Princeton University Press, 2007. ISBN 9780691 129181. URL http://www. jstor org/ stable/j.ctt7rxx22.9
[3] Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton. Latent variable modelling with hyperbolic normalizing flows. In Proceedings of the 37th International Conference on Machine Learning, pages 1045-1055, 2020.
[4] Martin R. Bridson and Andr'e e Haefliger. Metric spaces of non-positive curvature. 1999.
[5] Daniel A. Brooks, Olivier Schwander, Frédéric Barbaresco, Jean-Yves Schneider, and Matthieu Cord. Riemannian batch normalization for spd neural networks. In NeurlPS, 2019.
[6] Mario Lezcano Casado and David Portínez-Rubic Cheap orthogonal constraints in neural networks: A simple parametrization the orthogonal and unitary group. ArXiv, abs/1901.08428. 2019.
[7] Rudrasis Chakraborty, Jose J. Bouza, Jonathan H. Manton, and Baba C. Vemuri. Manifoldnet: A deep neural network for manifold- valued data with applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:799- 810, 2018.
[8] Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. In Advances in neural information processing systems, pages 4868-4879, 2019.
[9] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, volume 31, pages 6571-6583, 2018.
[10] Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou, Fully hyperbolic neural networks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2021.
[11] Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling, Spherical CNNs. In International Conference on Learning Representations, 2018.
[12] Calin Cruceru, Gary B' eecigneul, and Octavian- Ganea. Computationally tractable rie- mannian manifolds for graph embeddings. AAAI, 2021.
[13] Abhinav Dhall, Roland Göcke, Simon Lucey, and Tom Gedeon. Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pages 2106-2112, 2011.
[14] Laurent El Ghaoui. Hyper-textbook: Optimization models and applications, 2021. URL https://inst. eecs ber.eley.e..............oe httpp//ag...chooo///////////o//s
html
[15] Luca Falorsi and Patrick Forré. Neural ordinary differentiall equations manifolds. arXiv preprint arXiv:2006.06666 06663, 2020.
[16] Jean Gallier and Jocelyn Quaintance. Differential Geometry and Lie Groups: Computational Perspective, volume 12. Springer, 2020.
2020.

---

[17] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. In Advances neural information processing systems, pages 5345-5355, 2018.
[18) Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. 2018 IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 409-419, 2018.
[19] lan J. Goodfellow, Yoshua Bengio, and Aaron C. Courville. Deep learning. Nature, 521: 436 444, 2015.
[20] Caglar Gülgehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter W. Battaglia, Victor Bapst, David Raposo, Adam Santoro, and Nando de Freitas. Hyperbolic attention networks. ArXiv, abs/ 805.09786, 2019.
[21] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34, 2017.
[22] William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, 2017.
[23] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[24] Ernst Heintze and H Hof. Geometry of horospheres. Journal of Differential Geometry, 12: 481-491, 1977.
[25] Sigurdur Helgason. Radon-fourier transforms symmetric spaces and related group represen- tations. Bulletin of the American Mathematical Society, 71:757-763, 1965.
[26] Zhiwu Huang and Lue Van Gool. A riemannian network for spd matrix learning. In AAAI, 2017.
[27] Mohamed E. Hussein, Marwan Torki, Mohammad Abdelaziz Gowayyed, and Motaz Ahmad El-Saban. Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations. In IJCAI, 2013.
[28] H. Karcher. Riemannian center of mass and mollifier smoothing. Communications on Pure and Applied Mathematics, 30(5):509 541, 1977. doi: https///ooooor///....oooo/..1311 URL https:://onlinnllirrrrrrrrr/ httpuubrrrrrrrrllll........r wiley. com commddoi/abbl aas///b//00 /0. 1002/ cpa. 3160300502
[29] Isay Katsman, Aaron Lou, Derek Lim, Qingxuan Jiang, Ser-Nam Lim, and Christopher De Sa. Equivariant manifold flows. ArXiv, abs/2107.0885p 2021.
[30] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
[31] Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ArXiv, abs/1660.00207,, 2017.
[32] John M. Lee. Riemannian Manifolds: An Introduction to Curvature. 1997.
[33] John M Lee. Introduction Smooth Manifolds. Graduate Texts in Mathematics. Springer New York, 2013.
[34] Javier López, Béatrice Pozzetti, Steve J. Trettel, Michael Strube, and Anna Wienhard. Vector- valued distance and gyrocalculus on the space of symmetric positive definite matrices. ArXiv, abs/21 10. 13475, 2021.
[35] Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. Differentiating through the fréchet mean. In International Conference on Machine Learning, 2020.
12

---

[36] Aaron Lou, Derek Lim, Isay Katsman, Leo Huang, Qingxuan Jiang, Ser-Nam Lim, and Christopher De Sa. Neural manifold ordinary differential equations. In Advances in Neural Information Processing Systems, volume 33, pages 17548-17558, 2020.
[37] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. ArXiv, abs/1710. 10121, 2017.
[38] Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. In Advances in Neural Information Processing Systems, volume 33, pages 2503-2515, 2020.
[39] M. Müller, T. Röder, M. Clausen, B. Eberhardt, B. Krüger, and A. Weber. Documentation mocap database hdm05. Technical Report CG-2007-2, Universität Bonn, June 2007.
[40] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
[41] Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical represen- tations. In Advances in neural information processing systems, pages 1338-6347, 2017.
[42] Xavier Pennec, Pierre Fillard, and Nicholas Ayache. A riemannian framework for tensor computing. International Journal of Computer Vision, 66:41-66, 2005.
[43] Danilo Jimenez Rezende, George Papamakarios, Sebastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. In Proceedings of the 37th International Conference on Machine Learning, pages 8083-8092. 2020.
[44] Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman. Moser flow: based generative modeling on manifolds. In Neural Information Processing Systems, 2021.
Divergence-
[45] Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, 62:352-364, 2018. 2018.
[46] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi- Rad. Collective classification in network data articles. Al Magazine, 29:93-106, 09 2008 doi: 10. vo09//amaa..22933.22
[47] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi- Rad. Collective classification in network data. Al Mag., 29:93-106, 2008
[48] Amir Shahroudy, Jun Liu, Tian-Tsong for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010-1019, 2016.
Tsong Ng, and Gang Wang. Ntu rgb+d: A large scale dataset
[49] Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic neural networks++, 2020.
[50] Sho Sonoda, Isao Ishikawa, and Masahiro Ikeda. Fully-connected network on noncompact sym- metric space and ridgelet transform based helgason-fourie analysis. ArXiv, abs/2203.01631, 2022.
[51] Abraham Albert Ungar. A gyrovector space approach to hyperbolic geometry. In A Gyrovector Space Approach to Hyperbolic Geometry, 2009.
[52] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio', and Yoshua Bengio. Graph attention networks. ArXiv, abs/17110.10003 10903, 2018.
[53] W. Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, March 2017. ISSN 2194-6701. doi: 10.1007/
540304-017-0103-z.
[54] Felix Wu, Tianyi Zhang, Amauri H. de Souza, Christopher Fifty, Tao Yu, and Kilian Q. Wein- berger. Simplifying graph convolutional networks. ArXiv, abs/1992.07115 2019.
[55] Tao Yu and Christopher M De Sa. Hyla: Hyperbolic laplacian features for graph learning. ArXiv, abs/2202.0688. 2022.
13


Reviews:
[{"review_id": "Wd60Ncbgz9", "reviewer": "Reviewer_1LjF", "summary": "The paper proposes an extension of standard ResNets called Riemannian Residual Neural Networks. The extension is done based on Riemannian manifolds as discussed in Equation (2). Some numerical results on node classification problems are presented in section 5 to show the improvements of the proposed generalization of ResNets.\n\n-- Post-rebuttal Review Update --\n \nI thank the authors for the detailed responses to my comments. I find the responses satisfactory and raise my score to 6.", "questions": "How did the computational costs of training a Riemannian ResNet compare to that of a standard ResNet? Could the Riemannian ResNet demand more computational power for training than the normal ResNet?", "limitations": "Please see my previous responses.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1- The idea of Riemannian ResNets sounds interesting and as the numerical results suggest could help improve the performance of ResNet models in applications where the chosen Riemannian geometry suits the dataset.", "weaknesses": "1- The paper's presentation remains abstract in the main body, and I do not find the current presentation accessible enough to deep learning practitioners. For example, Section 3 spends about 2.5 pages explaining Riemannian geometry but does not discuss a concrete example where the exponential map and vector fields can be discussed. The examples in section 4.2.1 appear late in the draft and also do not derive the expression for the exponential map that appears in Riemannian ResNets.\n\n2- Since the paper has not discussed the algorithmic steps of training and evaluating a Riemannian ResNet, it is not that easy to see how the network can be trained for non-Euclidean Riemannian geometries. I suggest adding one or two algorithms to the draft to discuss the steps of training a Riemannian ResNet for the cases discussed in Section 4.2.1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and the constructive comments. We appreciate that you think our idea is interesting and has the potential to improve the performance of ResNets over datasets with chosen Riemannian geometry. We address your comments from the “Weaknesses” section below as well as your questions from the “Questions” section.\n\n---\n\n> The paper's presentation remains abstract in the main body, and I do not find the current presentation accessible enough to deep learning practitioners. For example, Section 3 spends about 2.5 pages explaining Riemannian geometry but does not discuss a concrete example where the exponential map and vector fields can be discussed. The examples in section 4.2.1 appear late in the draft and also do not derive the expression for the exponential map that appears in Riemannian ResNets.  \n‎   \nSince the paper has not discussed the algorithmic steps of training and evaluating a Riemannian ResNet, it is not that easy to see how the network can be trained for non-Euclidean Riemannian geometries. I suggest adding one or two algorithms to the draft to discuss the steps of training a Riemannian ResNet for the cases discussed in Section 4.2.1.\n\nA: We apologize for any confusion. Due to page limits, we decided to offload these details to the appendix. We would like to refer the reader to the Appendix where we extensively document how to implement a Riemannian ResNet: \n\n1. In Appendix A, we provide a table of the expressions we use to compute the exponential map on various manifolds. All operations can be performed with standard PyTorch operations. \n2. In Appendix B, we elaborate on the Riemannian ResNet design presented in Section 4, discussing the tradeoffs of each design. \n3. Appendix C outlines more experimental details. In Appendix C.7, we give a concrete example of how we constructed a Riemannian ResNet for covariance matrix classification. \n\nWe will include more of these details in the main body in a revised version.\n\n---\n\n> How did the computational costs of training a Riemannian ResNet compare to that of a standard ResNet? Could the Riemannian ResNet demand more computational power for training than the normal ResNet?\n\nA: One of the key benefits of a Riemannian ResNet is that it can capture geometric invariants of the training data. This can reduce the parameter count of the neural networks used, improving both performance and  efficiency. For example, because the Riemannian ResNet for SPD matrices acts on eigenvalues, it is invariant to change of eigenbasis. While the size of a $n \\times n$ matrix grows at an $O(n^2)$ rate, the number of eigenvalues grows at only an $O(n)$ rate, leading to computational efficiency. We demonstrate these benefits in Appendix C.10, where Euclidean ResNet performs worse than the Riemannian ResNet across all SPD datasets. However, computation can be a challenge when closed-form solutions to the exponential maps are unknown. \n\nWe believe that our work can highlight these computational benefits of geometric machine learning, and will clarify this in a revised version. \n\n"}, {"review_id": "Wc516i1WUI", "reviewer": "Reviewer_1vkd", "summary": "The paper generalizes the ResNet layer to non-euclidean geometries by replacing the Euclidean sum with the exponential map. The theory is general and applies to any smooth metric. They propose a way to parametrize a vector field on the manifold which is more geometrically principled than the trivial vector field embedding. Empirically they show improvement in performance on hyperbolic datasets and PSD spaces compared to some baselines.", "questions": "Please reply to the weaknesses above.\n\nFurthermore:\n\nMajor:\n* Line 215 and Appendix B4. The function $l_f(x)$ is inconsistent. In the main paper is a map $M \\rightarrow T M$ while in the appendix is a map $M \\rightarrow (T M)^*. Can you explain the inconsistency?\n\nMinor:\n* Line 189 $f_nn$ should be $f$?\n* The Log-Euclidean metric amounts to a Euclidean metric in the tangent space at the identity. In that case, does $f$ reduce to being $f: T_I M \\rightarrow R^k$? If so, is your construction just a standard Euclidean ResNet in a pre-specified tangent space? That would be good to state explicitly, if so.\n* It's quite common to run into numerical instabilities (esp. on hyperbolic manifolds). Is that something you face?", "limitations": "The parting \"Limitations\" paragraph discusses a relevant assumption. However, we feel that a discussion of the tractability of (projections onto) \"pseudo-hyperplanes\" (Eq. 17 in the appendix) is lacking.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper theory is general and applies to any smooth manifold metric. In ResNet a neural network produces a vector field and the output of the ResNet is the input plus such vector field. The proposed generalization, assuming access to a vector field on the manifold, defines the output of the ResNet as the exponential map of the input in the direction of such a vector field. This is consistent with the Euclidean case.\n\nA vector field in Euclidean space can simply be the output of a neural network, while on a manifold more care is required. Using an embedded vector field is straightforward, but, as correctly pointed out by the authors, it is not very principled geometrically. The authors propose a computationally tractable and geometrically principled way of defining a parametric vector field on a manifold. The idea is to make use of a collection of $\\mathbb{R}$ valued functions (obtained as a projection on hyperplanes or similar) to define, through push forward and pullback, a vector field on the manifold.\n\nExtensive specific examples of such a collection of functions are given for the hyperbolic and PSD manifolds. Experiments are also performed in these cases, and the proposed approach appears to outperform the current state of the art.\n\nIn the general manifold case (Appendix B.5), the idea of projection on pseudo-hyperplane is appealing and well-argued. And the further generalization to “hyper-disks” allows proper formal extension also to the non-geodetically-complete manifolds.", "weaknesses": "The definition of the vector field (Appendix B.4) is not sufficiently formal and contains mistakes. Specifically, the author assumes access to a smooth function $f:M\\rightarrow \\mathbb{R}^k$, a so-called “feature map”. The differential $D_x f$ is then a linear map from the tangent space in $x$, $T_x M$ to the tangent space in $f(x)$, $T_{f(x)} R^k = R^k$. Observing that the dual of $R^k$ is isomorphic to $R^k$ itself, the pullback of the differential $(D_x f)^*$ can be seen as a map from $R^k$ to $(T_x M)^*$, for every $x\\in M$. This function is evaluated in $f(x)\\in R^k$ such that the vector field (as defined in line 215) is a map\n$$ l_f: x \\rightarrow (D_x f)^*(f(x)) $$ which, as we saw, take values in $(T_x M)^*$ and NOT on $T_x M$, as line 215 is saying.\n\nWe would like to see this inconsistency explained. Is this based on the observation that both $(T_x M)^*$ and $T_x M$ are isomorphic to $\\mathbb{R}^{dim(M)}$?\n\nAnd also, can you reason about the choice of evaluating the pullback of the differential (line 215) in $f(x)$? Is this somehow principled?\n\nThe equivalence with Euclidean ResNet shown in Appendix D is a proper extension in the case of an embedded vector field, but it is rather difficult to follow in the case of the feature map. Specifically, Proposition 1 is trivially proved for an embedded vector field, but the same argument should also apply to the case of a feature-map-induced vector field. It would be helpful if the discussion on $g_{w,b}$ could e.g. focus on the case of axis-aligned planes (i.e. each $w$ should be an element of a standard basis and $b = 0$), such that the differential $D_x f$ reduces to an identity. We found this to be significantly more intuitive.\n\nIn the general manifold case (Appendix B.5), the idea of projection on pseudo-hyperplane is, although well explained, not at all investigated. First, it is not clear how to practically implement such projections of these pseudo-hyperplanes (not in the geodesically complete case, and even worse in the general case). Second, there are no experiments regarding general manifolds, and there is also no dissertation on the increased computational complexity with respect to hyperbolic and PSD cases. This reduces “contribution 3” in the statement in lines 72-76.\n\n*Minor:*\n* The related work section argues that the proposed construction is different from a neural ODE (which also generalizes ResNets), but honestly, we found this argument to be incomplete. It does seem like the proposed construction is a neural ODE.\n* Line 126, we found it unclear which \"specific structure\" is being exploited in methods using Frechet means (these averages apply on practically all manifolds).\n\n*Regarding the score:*\nWe are willing to increase the score if these concerns are appropriately discussed in the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and the constructive comments! We address your comments from the “Weaknesses” and “Questions” sections below.\n\n---\n\n> The definition of the vector field... are isomorphic to $\\mathbb{R}^{\\dim(M)}$?\n\nA: Since our manifold is equipped with a Riemannian metric, there is a canonical isomorphism, induced by the metric, between $(T_x M)^*$ and $T_x M$. Composing the differential operator $(D_x f)^* : R^k \\rightarrow (T_x M)^*$ with this yields the map that we notatate as $(D_x f)^*_r : R^k \\rightarrow T_x M$ in Appendix B.4, which we precompose with $f$ to obtain $\\ell_f : M \\rightarrow TM$. For more information about the isomorphism between $(T_x M)^*$ and $T_x M$ we encourage the reader to consult “The Tangent-Cotangent Isomorphism” section from chapter 13 of [33].\n\nFor simplicity, by abuse of notation we overload $(D_x f)^*$ to mean $(D_x f)^*_r$ in the main paper, and point the reader to appendix B.4 for details. However, we agree that this is confusing and will add details to the main paper about this step, thereby resolving any inconsistencies.\n\n---\n\n> And also, can you reason... somehow principled?\n\nA: We can describe the idea here, more broadly. The differential provides a natural way to map from $T_p M$ to a Euclidean space (since $T_{f(p)} R^k$ is Euclidean). We seek a natural map into the tangent space, so we take the pullback to obtain a natural map from $R^k$ into the dual space $(T_p M)^*$, and then dualize (i.e. use the tangent-cotangent isomorphism) to obtain a map from $R^k$ into $T_p M$, as desired.\n\nOne motivation is that in the Euclidean case, because the maps $f(x)$ and $(D_x f)^*$ are linear, $\\ell_{f}$ will reduce to a standard linear layer, which, in combination with the Euclidean $\\exp$ map, will produce a standard Euclidean residual neural network.\n\n---\n\n> The equivalence with... more intuitive.\n\nA: Yes, you are precisely correct. You can generalize the argument in Proposition 1 to the case of feature map-induced vector fields, by noting that if feature maps are projections to standard axis-aligned planes, the same reduction happens as that which was shown to hold in Proposition 1 for embedded vector fields.\n\n---\n\n> In the general manifold... the statement in lines 72-76.\n\nA: The idea of the general manifold case feature projection is proposed mostly as a natural theoretical extension of the feature projection theory we had been developing up until that point. Please note that we tested this approach experimentally in Appendix C.4 for the hyperbolic case, and compared the performance of these pseudo-hyperplanes with the horosphere-projection based feature map approach. However, in general, applying this method may require more specific investigation of the manifold over which optimization is occurring (i.e. there may be an easier approach to obtaining geodesic distances than explicitly solving the geodesic equations). \n\nOn a separate note, our claim stated in lines 72-76 refers to the ability to change the metric on the same manifold, as done in the SPD experiments in Section 5.2.\n\n---\n\n> (minor) The related work section... is a neural ODE.\n\nA: First, it is worth pointing out that the construction is quite different from a neural ODE, in that the vectors exist on the tangent space of a manifold, not just in $R^n$. Second, speaking of neural manifold ODEs, for which the residual vectors exist in tangent spaces, our construction is effectively a generalized neural manifold ODE. Looking more closely at a manifold ODE [36, page 4], we see the neural network depends on time and generates a “flow”. A riemannian ResNet requires only the provision of a vector field, entirely untethered from any notion of solving an ODE/untethered from a time variable. This makes it a strict generalization, suitable for use in a general manifold neural network context.\n\n---\n\n> (minor) Line 126, we found it... all manifolds.\n\nA: We meant mostly that the Helgason-Fourier construction exploits a fairly particular structure, but we also believe it is worth noting that weighted Frechet means are specifically introduced for convolution, which is not the focus of our work (we focus on residual connections). We will augment the writing to make this clear.\n\n---\n\n> (question, major) Line 215... explain the inconsistency?\n\nA: By the tangent-cotangent isomorphism, we pass from $(T_p M)^*$ to $T_p M$. Please see our above comments regarding this for more details. The map we actually use in practice goes from $M$ to $TM$. We will make this explicit for clarity.\n\n---\n\n> (question, minor) Line 189 f_n n should be f?\n\nA: Yes, thank you for pointing out this typo. This will be fixed.\n\n---\n\n> (question, minor) The Log-Euclidean metric... if so.\n\nA: For the embedded vector field, the answer is yes. This is one shortcoming we observed with SPDNet, which only operates with the Log-Euclidean metric, motivating our additional use of the affine-invariant metric. For the feature map-induced vector field, where the feature maps are eigenvalue projections, the answer is no. Learning happens by way of repeated spectral remapping.\n\n---\n\n> (question, minor) It's quite common... you face?\n\nA: This is something we have encountered, but there are ways of limiting the effect of such instabilities on the results. In particular, for the Poincare ball model we use, instabilities occur at large distances away from the origin, where $r \\in (1-\\epsilon, 1)$ for $\\epsilon < 0.001$. As a consequence, we limit the hyperbolic distances and do projections in order to keep our optimization numerically stable. But an approach that is fully naive will likely run into some numerical difficulties, as you suggested.\n\n---\n\n> (limitations concern) The parting \"Limitations\" paragraph... is lacking.\n\nA: We will add a separate discussion about tractability of pseudo-hyperplanes. We give this construction as an appealing theoretical generalization, but application to particular manifolds will require some care.\n"}, {"review_id": "CGv2w9dcXc", "reviewer": "Reviewer_5HC1", "summary": "This paper extends the well known residual network which are usually applied to Euclidean data to a variant defined on manifold. The main novelty is to replace the \"addition / plus\" operation in Euclidean space to exponential operation. Specifically, given an input point on a certain manifold (e.g., hyperbolic or SPD), it first learns a vector in the tangent space of the given input point through a neural network layer, and then maps the learned vector back to the manifold using exponential. By utilizing pushforward and pullback operations, the proposed method can transform input between different manifolds with different dimensions. Experiments are conducted on hyperbolic and SPD spaces to demonstrate the superior performance of the proposed method over HNN and SPDNet.", "questions": "In general:\n\n1. Except for Euclidean, hyperbolic and SPD, any other manifold are applicable to the proposed residual network?\n\nMethod part:\n\n2. For the function h_i defined in Line 202 parameterized by a neural network, given an input on a mainfold M_{i-1}, how to ensure the output is also on a manifold M_i?\n\n3. For SPD matrix, what's the advantage of the proposed Riemannian Residual Neural Networks over SPDNet (the AAAI paper) in theory?\n\n4. In Line 225 - 231, what's the relationship between f_i and g_{\\theta_i}?\n\n5. Still in Line 225 - 231, why does $\\nabla g_{\\theta_i}$ is a map from M to TM?\n\n6. In SPD case of Line 258 - 260, since g_k does not contain any learnable parameters, how to train such a network for SPD?\n\nExperiment part:\n\n7. How many links on average per graph used in the experiment in Section 5.1.1? It is important to verify the proposed method could perform well on medium-to-large datasets.\n\n8. In Table 2, the standard deviations of the proposed method is obviously larger than SPDNet, does it imply the proposed method is unstable than SPDNet? (Similar phenomena also happen in Table 1, though not that obvious.)\n\nImplementation part:\n\n9. As some parts (e.g., Line 225 - 231) are not that obvious to implement, will the code be released?\n\n ", "limitations": "The authors mentioned the limitations of the proposed method in Line 357 - 359.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposed to extend residual network from Euclidean space to non-linear manifold by replacing the conventional addition/plus operation with manifold exponential operation. The theoretical part is sound and the experiments are effective in supporting the proposed method.", "weaknesses": "The biggest weakness, as also mentioned in by the authors at the end of the paper, is whether the proposed method is only applicable to hyperbolic and SPD matrix? Is it possible to apply this residual network to other non-linear manifolds that have closed-form exponential manifolds? Furthermore, Is it possible to apply this residual network to other non-linear manifolds that do $\\textbf{not}$ have closed-form exponential manifolds? If so, please list such manifolds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and the constructive comments. We appreciate that you think our idea is theoretically sound and our experiments are supportive. We address your comments from the “Weaknesses” and \"Questions\" sections below.\n\n---\n\n> The biggest weakness... closed-form exponential maps?\n\nA: Riemannian ResNets can be applied to more than just hyperbolic space and the manifold of SPD matrices; the answer to both questions posed is an unequivocal “yes.” So long as a closed-form exp map is provided, our method becomes immediately applicable. In fact, we include experiments on spherical space (which does have a closed form exponential map) in sections C.11 through C.13 of the appendix. We provide a general feature map construction in B.5 which is applicable to any Riemannian manifold. Please see the question below for an explicit list of some of these manifolds.\n\n---\n\n> Except for Euclidean, hyperbolic and SPD, any other manifold are applicable to the proposed residual network?\n\nA: Yes, as mentioned above our method is easily applicable to myriad manifolds assuming there is a closed form exponential map provided. Common cases outside of the Euclidean, hyperbolic, and SPD manifolds include the manifold of spherical space and the Grassmannian manifold. Every matrix Lie group is also included, so for example the manifold of special unitary matrices (SU(n)), the manifold of special orthogonal matrices ($SO(n)$), and the manifold of all invertible matrices ($GL(n, \\mathbb{R})$).\n\n---\n\n> Is it possible to apply this residual network to other non-linear manifolds that do not have closed-form exponential manifolds?\n\nA: We note that without a closed-form exponential map, we would need a differentiable ODE solver for the geodesic equations, immediately complicating the computation by a significant amount. However, we note that this is not a considerable limitation of our method in that nearly all prior work in this subfield requires at least a closed-form exponential map, see for examples Neural Manifold ODEs [36], Riemannian Continuous Normalizing Flows [38], and Riemannian Convex Potential Maps [https://arxiv.org/abs/2106.10272].\n\n---\n\n> For the function $h_i$ defined in Line 202 parameterized by a neural network, given an input on a manifold $M_{i-1}$, how to ensure the output is also on a manifold $M_i$?\n\nA: We assume in our construction that $h_i$ maps $M^{(i-1)}$ to $M^{(i)}$. In general, one has to carefully construct these maps to ensure the output stays on manifold. As a concrete example, our h_i for the SPD case map from an SPD matrix of one dimension to another by conjugating with a Stiefel matrix [26].\n\n---\n\n> For SPD matrix, what's the advantage of the proposed Riemannian Residual Neural Networks over SPDNet (the AAAI paper) in theory?\n\nA: Besides the empirical benefits over SPDNet, theoretically, our work also generalizes SPDNet. SPDNet only operates on matrices under the log-Euclidean metric, which endows the manifold with flat geometry (the sectional curvature is zero at every point). This model fails to provide a way to capture more nontrivial geometry of the SPD manifold, a drawback that our approach removes.\n\nWe demonstrate that the proposed Riemannian ResNet can learn over the log-Euclidean metric, and the affine-invariant metric, which has non-constant sectional curvature.\n\nIn Appendix A.2, we provide a table comparing the different operations involved with both metrics, indicating that the difference is substantial and we provide nontrivial flexibility.\n\n---\n\n> In Line 225 - 231, what's the relationship between $f_i$ and $g_{\\theta_i}$?\n\nA: Each coordinate of the output of f is the output of a $g_{\\theta_i}$, i.e. $f(p) = (g_{\\theta_1}(p), g_{\\theta_2}(p), …, g_{\\theta_k}(p))$.\n\n---\n\n> Still in Line 225 - 231, why does $\\nabla g_{\\theta_i}$ is a map from M to TM?\n\nA: $\\nabla g_{\\theta_i}$ maps from M to TM because the gradient generates a vector field (tangent to the underlying space). If you wish to see an explicit reference, please refer to Chapter 13 of Introduction to Smooth Manifolds [33] where on page 342 (of the second edition) the gradient of $f$, a real-valued function over a smooth manifold is defined.\n\n---\n\n> In SPD case of Line 258 - 260, since g_k does not contain any learnable parameters, how to train such a network for SPD?\n\nA: While the feature map itself has no learnable parameters, we incorporate learnable parameters from other parts of the neural network. For instance, we can learn over the extracted features $g_k$ with other neural network layers to obtain a vector field. Appendix B.2.3 describes this in detail. \n\n> How many links on average per graph used in the experiment in Section 5.1.1? It is important to verify the proposed method could perform well on medium-to-large datasets.\n\nA: We provide the total number of edges and the number of edges used in training for each of the graphs below:\n\n- Total edges\n    - Airport: 18631\n    - Disease: 2664\n    - Cora: 5278\n    - Pubmed: 44327\n- Train edges\n    - Airport: 15837\n    - Disease: 2265\n    - Cora: 4488\n    - Pubmed: 37679\n\nAs can be seen, most of the edges are used as links for training. These datasets range from small/medium to relatively large (e.g. Pubmed).\n\n---\n\n> In Table 2, the standard deviations of the proposed method is obviously larger than SPDNet, does it imply the proposed method is unstable than SPDNet?\n\nA: Although in Table 2 the standard deviations for our method are larger than those for SPDNet for the datasets of AFEW and HDM05 (arguably, they are similar for the datasets of FPHA and NTU), in Figure 5 (referenced in Appendix C.8), we find that our proposed method converges faster and to a higher value than SPDNet, despite sometimes experiencing increased standard deviation. In particular, even for HDM05, which has higher standard deviation in the table, this is exceptionally clear.\n\n---\n\n> Will the code be released?\n\nA: We will release code for our experiments.\n"}, {"review_id": "Df0M4HzMkv", "reviewer": "Reviewer_nyBA", "summary": "EDIT: Having read everything here. I am increasing my score slightly. However, I still think the paper is not clearly explained. It is unclear why you first introduce the method without the feature map. It seems like there are two different versions of the method and it is not clear which is which. I am also confused by the experimental set up for datasets such as CORA. Do you first embed the graphs into a manifold? \n\nAdditionally, I think that requiring a closed form expression for geodesics is somewhat strong and limits the applicability.\n\n\nThe authors claim to attempt RESNET structures to hyperbolic manifolds", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper is applicable to a wide variety of manifolds", "weaknesses": "The proposed methods seem to not really leverage the manifolds intrinsice geometry and depend entirely on the embedding on the manifold into ambient space $R^D$. \n\nIndeed $n_i$ is defined on all of $R^D$. This means that there is no gauranteed that there would be any notion of consistence if $M$ was embedded into $R^D$ and $R^{D'}$ two different ways (where $D'$ may or may not equal $D$). This seems to be a major limitation of this that is not properly discussed. at a bare minumum, there should be some level of invariance to, e.g., Rotations and Translations of the manifold in $R^D$ after its been embedded. \n\nAdditionally, much of the paper is hard to understand such as the construction of the feature maps, which appears to take place in local coordinate systems which will not be consistent across the manifold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The proposed methods seem to not really leverage the manifolds intrinsice geometry and depend entirely on the embedding on the manifold into ambient space $R^D$\n\nA: Our approach uses intrinsic geometry (we use local coordinates, as mentioned by the reviewer).\n\n---\n\n> Indeed $n_i$ is defined on all of $R^D$. This means that there is no gauranteed that there would be any notion of consistence if $M$ was embedded into $R^D$ and $R^{D’}$ two different ways (where $D’$ may or may not equal $D$). This seems to be a major limitation of this that is not properly discussed. at a bare minumum, there should be some level of invariance to, e.g., Rotations and Translations of the manifold in $R^D$ after its been embedded.\n\nA: $n_i$ is not invariant, but this is a necessary cost to realize our manifolds for computation and is standard practice in the literature [36, 38]. However, our feature map-based layers are notably invariant to embedding, which was the motivation for introducing them.\n\nOur feature maps are invariant to local coordinates and are consistent (e.g. SPD eigenvalues are invariant to conjugation and the hyperbolic space construction is invariant to choice of model).\n"}]

Author rebuttal:

Meta-review: The paper deals with ResNets and how to deal with manifold structures there. While some reviewers found the paper to be a natural extension to existing works, others found the paper interesting for a subset of the NeurIPS community. Overall, after discussions, I am of the opinion that the paper merits acceptance with the hope that many of the suggestions should be incorporated.
Decision: Accept (poster)