SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Feature Map Design One possible way to simplify the design of the above vector field is to further break down the map  $f: \mathcal{M} \to \mathbb{R}^k$  into  $k$  maps  $f_1, \ldots, f_k: \mathcal{M} \to \mathbb{R}$ , where ideally, each map  $f_i$  is constructed in a similar way (e.g. performing some kind of geometric projection, where the  $f_i$  vary only in terms of the specifying parameters). As we shall see in the following subsection, this ends up being a very natural design decision.
In what follows, we shall consider only smooth feature maps  $f: \mathcal{M} \to \mathbb{R}^k$  induced by a single parametric construction  $g_{\theta}: \mathcal{M} \to \mathbb{R}$ , i.e. the  $k$  dimensions of the output of  $f$  are given by different choices of  $\theta$  for the same underlying feature map<sup>4</sup>. This approach also has the benefit of a very simple interpretation of the induced vector field. Given feature maps  $g_{\theta_1}, \ldots, g_{\theta_k}: \mathcal{M} \to \mathbb{R}$  that comprise our overall feature map  $f: \mathcal{M} \to \mathbb{R}^k$ , our vector field is simply a linear combination of the maps  $\nabla g_{\theta_i}: \mathcal{M} \to T\mathcal{M}$ . If the  $g_{\theta_i}$  are differentiable with respect to  $\theta_i$ , we can even learn the  $\theta_i$  themselves.
4.2.1 Manifold Manifestations
In this section, in an effort to showcase how simple it is to apply our above theory to come up with natural vector field designs, we present several constructions of manifold feature maps  $g_{\theta}:\mathcal{M}\to \mathbb{R}$  that capture the underlying geometry of  $\mathcal{M}$  for various choices of  $\mathcal{M}$ . Namely, in this section we provide several examples of  $f:\mathcal{M}\rightarrow \mathbb{R}$  that induce  $\ell_f:\mathcal{M}\to TM$ , thereby giving rise to a Riemannian neural network by Section 4.1.
Euclidean Space To build intuition, we begin with an instructive case. We consider designing a feature map for the Euclidean space  $\mathbb{R}^n$ . A natural design would follow simply by considering hyperplane projection. Let a hyperplane  $w^{T}x + b = 0$  be specified by  $w\in \mathbb{R}^n,b\in \mathbb{R}$ . Then a natural feature map  $g_{w,b}:\mathbb{R}^n\to \mathbb{R}$  parameterized by the hyperplane parameters is given by hyperplane projection [14]:  $g_{w,b}(x) = \frac{|w^Tx + b|}{||w||_2}$ .
Hyperbolic Space We wish to construct a natural feature map for hyperbolic space. Seeking to follow the construction given in the Euclidean context, we wish to find a hyperbolic analog of hyperplanes. This is provided to us via the notion of horospheres [24]. Illustrated in Figure 4, horospheres naturally generalize hyperplanes to hyperbolic space. We specify a horosphere in the Poincaré ball model of hyperbolic space  $\mathbb{H}^n$  by a point of tangency  $\omega \in \mathbb{S}^{n - 1}$  and a real value  $b\in \mathbb{R}$ . Then a natural feature map  $g_{\omega ,b}:\mathbb{H}^n\to \mathbb{R}$  parameterized by the horosphere parameters would be given by horosphere projection [4]:
$$
g _ {\omega , b} (x) = - \log \left( \begin{array}{c} 1 - | | x | | _ {2} ^ {2} \\ | | x - \omega | | _ {2} ^ {2} \end{array} \right) + b.
$$
Figure 4: Example of a horosphere in the Poincaré ball representation of hyperbolic space. In this particular two-dimensional case, the hyperbolic space  $\mathbb{H}_2$  is visualized via the Poincaré disk model, and the horosphere, shown in blue, is called a horocycle.
Symmetric Positive Definite Matrices The manifold of SPD matrices is an example of a manifold where there is no innate representation of a hyperplane. Instead, given  $X \in SPD(n)$ , a reasonable feature map  $g_{k}: SPD(n) \to \mathbb{R}$ , parameterized by  $k$ , is to map  $X$  to its  $k$ th largest eigenvalue:  $g_{k}(X) = \lambda_{k}$ .
<sup>4</sup>We use the term "feature map" for both the overall feature map  $f: \mathcal{M} \to \mathbb{R}^k$  and for the inducing construction  $g_{\theta}: \mathcal{M} \to \mathbb{R}$ . This is well-defined since in our work we consider only feature maps  $f: \mathcal{M} \to \mathbb{R}^k$  that are induced by some  $g_{\theta}: \mathcal{M} \to \mathbb{R}$ .
General Manifolds For general manifolds there is no perfect analog of a hyperplane, and hence there is no immediately natural feature map. Although this is the case, it is possible to come up with a reasonable alternative. We present such an alternative in Appendix B.4 together with pertinent experiments.
Example: Euclidean Space One motivation for the vector field construction  $\ell_f(x) = (D_x f)_r^*(f(x))$  is that in the Euclidean case,  $\ell_f$  will reduce to a standard linear layer (because the maps  $f$  and  $(D_x f)^*$  are linear), which, in combination with the Euclidean exp map, will produce a standard Euclidean residual neural network.
Explicitly, for the Euclidean case, note that our feature map  $f: \mathbb{R}^n \to \mathbb{R}^k$  will, for example, take the form  $f(x) = Wx$ ,  $W \in \mathbb{R}^{k \times n}$  (here we have  $b = 0$  and  $W$  has normalized row vectors). Then note that we have  $Df = W$  and  $(Df)^* = W^T$ . We see for the standard feature map-based construction, our vector field  $\ell_f(x) = (D_x f)^*(f(x))$  takes the form  $\ell_f(x) = W^T W x$ .
For the learnable case (which is standard for us, given that we learn Riemannian residual neural networks), when the manifold is Euclidean space, the general expression  $\ell_{f,\theta}(x) = (D_x f)^*(n_\theta(f(x)))$  becomes  $\ell_{f,\theta}(x) = W^T n_\theta(Wx)$ . When the feature maps are trivial projections (onto axis-aligned hyperplanes), we have  $W = I$  and  $\ell_{f,\theta}(x) = n_\theta(x)$ . Thus our construction can be viewed as a generalization of a standard neural network.
Dataset Hyperbolicity	Disease δ = 0	Airport δ = 1	PubMed δ = 3.5	CoRA δ = 11
Task	LP	NC	LP	NC	LP	NC	LP	NC
Shallow	Euc	59.8±2.0	32.5±1.1	92.0±0.0	60.9±3.4	83.3±0.1	48.2±0.7	82.5±0.3	23.8±0.7
Hyp [41]	63.5±0.6	45.5±3.3	94.5±0.0	70.2±0.1	87.5±0.1	68.5±0.3	87.6±0.2	22.0±1.5
Euc-Mixed	49.6±1.1	35.2±3.4	91.5±0.1	68.3±2.3	86.0±1.3	63.0±0.3	84.4±0.2	46.1±0.4
Hyp-Mixed	55.1±1.3	56.9±1.5	93.3±0.0	69.6±0.1	83.8±0.3	73.9±0.2	85.6±0.5	45.9±0.3
N	MLP	72.6±0.6	28.8±2.5	89.8±0.5	68.6±0.6	84.1±0.9	72.4±0.2	83.1±0.5	51.5±1.0
HNN [17]	75.1±0.3	41.0±1.8	90.8±0.2	80.5±0.5	94.9±0.1	69.8±0.4	89.0±0.1	54.6±0.4
RResNet Horo	98.4±0.3	76.8±2.0	95.2±0.1	96.9±0.3	95.0±0.3	72.3±1.7	86.7±6.3	52.4±5.5
Table 1: Above we give graph task results for RResNet Horo compared with several non-graph-based neural network baselines (baseline methods and metrics are from Chami et al. [8]). Test ROC AUC is the metric reported for link prediction (LP) and test F1 score is the metric reported for node classification (NC). Mean and standard deviation are given over five trials. Note that RResNet Horo considerably outperforms HNN on the most hyperbolic datasets, performing worse and worse as hyperbolicity increases, to a more extreme extent than previous methods that do not adhere to geometry as closely (this is expected).
In this section, we perform a series of experiments to evaluate the effectiveness of RResNets on tasks arising on different manifolds. In particular, we explore hyperbolic space and the SPD manifold.
5.1 Hyperbolic Space
We perform numerous experiments in the hyperbolic setting. The purpose is twofold:
1. We wish to illustrate that our construction in Section 4 is not only more general, but also intrinsically more geometrically natural than pre-existing hyperbolic constructions such as HNN [17], and is thus able to learn better over hyperbolic data.
2. We would like to highlight that non-Euclidean learning benefits the most hyperbolic datasets. We can do this directly since our method provides a way to vary the geometry of a fixed neural network architecture, thereby allowing us to directly investigate the effect of changing geometry from Euclidean to hyperbolic.
5.1.1 Direct Comparison Against Hyperbolic Neural Networks [17]
To demonstrate the improvement of RResNet over HNN [17], we first perform node classification (NC) and link prediction (LP) tasks on graph datasets with low Gromov  $\delta$ -hyperbolicity [8], which means the underlying structure of the data is highly hyperbolic. The RResNet model is given the
AFEW[13]	FPHA[18]	NTU RGB+D[48]	HDM05[39]
SPDNet	33.24±0.56	65.39±1.48	41.47±0.34	66.77±0.92
SPDNetBN	35.39±0.93	65.03±1.35	41.92±0.37	67.25±0.44
RResNet Affine-Invariant	35.17 ±1.78	66.53±01.64	41.00 ±0.50	67.91 ±1.27
RResNet Log-Euclidean	36.38±1.29	64.58±0.98	42.99±0.23	69.80±1.51
Table 2: We run our SPD manifold RResNet on four SPD matrix datasets and compare against SPDNet [26] and SPDNet with batch norm [5]. We report the mean and standard deviation of validation accuracies over five trials and bold which method performs the best.
name "RResNet Horo." It utilizes a horosphere projection feature map-induced vector field described in Section 4. All model details are given in Appendix C.2. We find that because we adhere well to the geometry, we attain good performance on datasets with low Gromov  $\delta$ -hyperbolicities (e.g.  $\delta = 0, \delta = 1$ ). As soon as the Gromov hyperbolicity increases considerably beyond that (e.g.  $\delta = 3.5, \delta = 11$ ), performance begins to degrade since we are embedding non-hyperbolic data in an unnatural manifold geometry. Since we adhere to the manifold geometry more strongly than prior hyperbolic work, we see performance decay faster as Gromov hyperbolicity increases, as expected. In particular, we test on the very hyperbolic Disease ( $\delta = 0$ ) [8] and Airport ( $\delta = 1$ ) [8] datasets. We also test on the considerably less hyperbolic PubMed ( $\delta = 3.5$ ) [47] and CoRA ( $\delta = 11$ ) [46] datasets. We use all of the non-graph-based baselines from Chami et al. [8], since we wish to see how much we can learn strictly from a proper treatment of the embeddings (and no graph information). Table 1 summarizes the performance of "RResNet Horo" relative to these baselines.
Moreover, we find considerable benefit from the feature map-induced vector field over an embedded vector field that simply uses a Euclidean network to map from a manifold point embedded in  $\mathbb{R}^n$ . The horosphere projection captures geometry more accurately, and if we swap to an embedded vector field we see considerable accuracy drops on the two hardest hyperbolic tasks: Disease NC and Airport NC. In particular, for Disease NC the mean drops from 76.8 to 75.0, and for Airport NC we see a very large decrease from 96.9 to 83.0, indicating that geometry captured with a well-designed feature map is especially important. We conduct a more thorough vector field ablation study in Appendix C.5.
5.1.2 Impact of Geometry
A major strength of our method is that it allows one to investigate the direct effect of geometry in obtaining results, since the architecture can remain the same for various manifolds and geometries (as specified by the metric of a given Riemannian manifold). This is well-illustrated in the most hyperbolic Disease NC setting, where swapping out hyperbolic for Euclidean geometry in an RResNet induced by an embedded vector field decreases the F1 score from a 75.0 mean to a 67.3 mean and induces a large amount of numerical stability, since standard deviation increases from 5.0 to 21.0. We conduct a more thorough geometry ablation study in Appendix C.5.
5.2 SPD Manifold
A common application of SPD manifold-based models is learning over full-rank covariance matrices, which lie on the manifold of SPD matrices. We compare our RResNet to SPDNet [26] and SPDNet with batch norm [5] on four video classification datasets: AFEW [13], FPHA [18], NTU RGB+D [48], and HDM05 [39]. Results are given in Table 2. Please see Appendix C.6 for details on the experimental setup. For our RResNet design, we try two different metrics: the log-Euclidean metric [16] and the affine-invariant metric [12, 42], each of which captures the curvature of the SPD manifold differently. We find that adding a learned residual improves performance and training dynamics over existing neural networks on SPD manifolds with little effect on runtime. We experiment with several vector field designs, which we outline in Appendix B. The best vector field design (given in Section 4.2), also the one we use for all SPD experiments, necessitates eigenvalue computation. We note the cost of computing eigenvalues is not a detrimental feature of our approach since previous works (SPDNet [26], SPDNet with batchnorm [5]) already make use of eigenvalue computation<sup>5</sup>. Empirically, we observe that the beneficial effects of our RResNet construction are similar to those of the SPD batch norm introduced in Brooks et al. [5] (Table 2, Figure 5 in Appendix C.6). In addition, we find that our operations are stable with ill-conditioned input matrices, which commonly occur in the wild. To contrast, the batch norm computation in SPDNetBN, which relies on Karcher flow
One needs this computation for operations such as the Riemannian exp and log over the SPD manifold.
Dataset Hyperbolicity	Disease δ = 0	Airport δ = 1	PubMed δ = 3.5	CoRA δ = 11
GNN	GCN [31]	69.7±0.4	81.4±0.6	78.1±0.2	81.3±0.3
GAT [52]	70.4±0.4	81.5±0.3	79.0±0.3	83.0±0.7
SAGE [22]	69.1±0.6	82.1±0.5	77.4±2.2	77.9±2.4
SGC [54]	69.5±0.2	80.6±0.1	78.9±0.0	81.0±0.1
GGNN	HGCN [8]	74.5±0.9	90.6±0.2	80.3±0.3	79.9±0.2
Fully HNN [10]	96.0±1.0	90.9±1.4	78.0±1.0	80.2±1.3
G-RResNet Horo	95.4±1.0	97.4±0.1	75.5±0.8	64.4±7.6
Table 3: Above we give node classification results for G-RResNet Horo compared with several graph-based neural network baselines (baseline methods and metrics are from Chami et al. [8]). Test F1 score is the metric reported. Mean and standard deviation are given over five trials. Note that G-RResNet Horo obtains a state-of-the-art result on Airport. As for the less hyperbolic datasets, G-RResNet Horo does worse on PubMed and does very poorly on CoRA, once more, as expected due to unsuitability of geometry. The GNN label stands for "Graph Neural Networks" and the GGNN label stands for "Geometric Graph Neural Networks."
[28, 35], suffers from numerical instability when the input matrices are nearly singular. Overall, we observe our RResNet with the affine-invariant metric outperforms existing work on FPHA, and our RResNet using the log-Euclidean metric outperforms existing work on AFEW, NTU RGB+D, and HDM05. Being able to directly interchange between two metrics while maintaining the same neural network design is an unique strength of our model.
6 Riemannian Residual Graph Neural Networks
Following the initial comparison to non-graph-based methods in Table 1, we introduce a simple graph-based method by modifying RResNet Horo above. We take the previous model and pre-multiply the feature map output by the underlying graph adjacency matrix  $A$  in a manner akin to what happens with graph neural networks [54]. This is the simple modification that we introduce to the Riemannian ResNet to incorporate graph information; we call this method G-RResNet Horo. We compare directly against the graph-based methods in Chami et al. [8] as well as against Fully Hyperbolic Neural Networks [10] and give results in Table 3. We test primarily on node classification since we found that almost all LP tasks are too simple and solved by methods in Chami et al. [8] (i.e., test ROC is greater than  $95\%$ ). We also tune the matrix power of  $A$  for a given dataset; full architectural details are given in Appendix C.2. Although this method is simple, we see further improvement and in fact attain a state-of-the-art result for the Airport [8] dataset. Once more, as expected, we see a considerable performance drop for the much less hyperbolic datasets, PubMed and CoRA.
We propose a general construction of residual neural networks on Riemannian manifolds. Our approach is a natural geodesically-oriented generalization that can be applied more broadly than previous manifold-specific work. Our introduced neural network construction is the first that decouples geometry (i.e. the representation space expected for input to layers) from the architecture design (i.e. actual "wiring" of the layers). Moreover, we introduce a geometrically principled feature map-induced vector field design for the RResNet. We demonstrate that our methodology better captures underlying geometry than existing manifold-specific neural network constructions. On a variety of tasks such as node classification, link prediction, and covariance matrix classification, our method outperforms previous work. Finally, our RResNet's principled construction allows us to directly assess the effect of geometry on a task, with neural network architecture held constant. We illustrate this by directly comparing the performance of two Riemannian metrics on the manifold of SPD matrices. We hope others will use our work to better learn over data with nontrivial geometries in relevant fields, such as lattice quantum field theory, robotics, and computational chemistry.
Limitations We rely fundamentally on knowledge of geodesics of the underlying manifold. As such, we assume that a closed form (or more generally, easily computable, differentiable form) is given for the Riemannian exponential map as well as for the tangent spaces.
We would like to thank Facebook AI for funding equipment that made this work possible. In addition, we thank the National Science Foundation for awarding Prof. Christopher De Sa a grant that helps fund this research effort (NSF IIS-2008102) and for supporting both Isay Katsman and Aaron Lou with graduate research fellowships. We would also like to acknowledge Prof. David Bindel for his useful insights on the numerics of SPD matrices.
[1] Cem Anil, James Lucas, and Roger B. Grosse. Sorting out lipschitz function approximation. In ICML, 2019.
[2] Rajendra Bhatia. Geometry of positive matrices. In Positive Definite Matrices, pages 201-236. Princeton University Press, 2007. ISBN 9780691129181. URL http://www.jstor.org/stable/j.ctt7rxv2.9.
[3] Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton. Latent variable modelling with hyperbolic normalizing flows. In Proceedings of the 37th International Conference on Machine Learning, pages 1045-1055, 2020.
[4] Martin R. Bridson and Andre Haefliger. Metric spaces of non-positive curvature. 1999.
[5] Daniel A. Brooks, Olivier Schwander, Frédéric Barbaresco, Jean-Yves Schneider, and Matthieu Cord. Riemannian batch normalization for spd neural networks. In NeurIPS, 2019.
[6] Mario Lezcano Casado and David Martínez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. ArXiv, abs/1901.08428, 2019.
[7] Rudrasis Chakraborty, Jose J. Bouza, Jonathan H. Manton, and Baba C. Vemuri. Manifoldnet: A deep neural network for manifold-valued data with applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:799-810, 2018.
[8] Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. In Advances in neural information processing systems, pages 4868-4879, 2019.
[9] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, volume 31, pages 6571-6583, 2018.
[10] Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Fully hyperbolic neural networks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2021.
[11] Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. In International Conference on Learning Representations, 2018.
[12] Calin Cruceru, Gary B'ecigneul, and Octavian-Eugen Ganea. Computationally tractable riemannian manifolds for graph embeddings. In AAAI, 2021.
[13] Abhinav Dhall, Roland Gocke, Simon Lucey, and Tom Gedeon. Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pages 2106-2112, 2011.
[14] Laurent El Ghaoui. Hyper-textbook: Optimization models and applications, 2021. URL https://inst.eecs.berkeley.edu/~ee127/sp21/livebook/l_vecs_hyp.html.
[15] Luca Falorsi and Patrick Forre. Neural ordinary differential equations on manifolds. arXiv preprint arXiv:2006.06663, 2020.
[16] Jean Gallier and Jocelyn Quaintance. Differential Geometry and Lie Groups: A Computational Perspective, volume 12. Springer, 2020.
[17] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. In Advances in neural information processing systems, pages 5345-5355, 2018.
[18] Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 409-419, 2018.
[19] Ian J. Goodfellow, Yoshua Bengio, and Aaron C. Courville. Deep learning. Nature, 521: 436-444, 2015.
[20] Caglar Güçehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter W. Battaglia, Victor Bapst, David Raposo, Adam Santoro, and Nando de Freitas. Hyperbolic attention networks. ArXiv, abs/1805.09786, 2019.
[21] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34, 2017.
[22] William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, 2017.
[23] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[24] Ernst Heintze and H I Hof. Geometry of horospheres. Journal of Differential Geometry, 12: 481-491, 1977.
[25] Sigurdur Helgason. Radon-fourier transforms on symmetric spaces and related group representations. Bulletin of the American Mathematical Society, 71:757-763, 1965.
[26] Zhiwu Huang and Luc Van Gool. A riemannian network forspd matrix learning. In AAAI, 2017.
[27] Mohamed E. Hussein, Marwan Torki, Mohammad Abdelaziz Gowayyed, and Motaz Ahmad El-Saban. Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations. In IJCAI, 2013.
[28] H. Karcher. Riemannian center of mass and mollifier smoothing. Communications on Pure and Applied Mathematics, 30(5):509-541, 1977. doi: https://doi.org/10.1002/cpa.3160300502. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/cpa.3160300502.
[29] Isay Katsman, Aaron Lou, Derek Lim, Qingxuan Jiang, Ser-Nam Lim, and Christopher De Sa. Equivariant manifold flows. ArXiv, abs/2107.08596, 2021.
[30] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[31] Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ArXiv, abs/1609.02907, 2017.
[32] John M. Lee. Riemannian Manifolds: An Introduction to Curvature. 1997.
[33] John M Lee. Introduction to Smooth Manifolds. Graduate Texts in Mathematics. Springer New York, 2013.
[34] F. Javier López, Béatrice Pozzetti, Steve J. Trettel, Michael Strube, and Anna Wienhard. Vector-valued distance and gyrocalculus on the space of symmetric positive definite matrices. ArXiv, abs/2110.13475, 2021.
[35] Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. Differentiating through the fréchet mean. In International Conference on Machine Learning, 2020.