Riemannian Transformation Layers for General Geometries

Authors: Ziheng Chen, Yue Song, Xiaojun Wu, Nicu Sebe
Primary area: learning on graphs and other geometries & topologies
Keywords: Manifold Learning, Representation Learning, Riemannian Manifolds

Abstract:
Recently, deep neural networks on manifold-valued representations have garnered significant attention in various machine learning applications. Several studies have attempted to generalize traditional Euclidean transformation layers, such as Fully Connected (FC) and convolutional layers, to non-Euclidean geometries. However, the previous approaches typically focus on a select few manifolds and rely on the specific properties of the target manifold. In this work, we propose a theoretical framework for constructing Riemannian FC and convolutional layers over general geometries, providing broader applicability. Utilizing this framework, we design convolutional networks across five distinct geometries of the Symmetric Positive Definite (SPD) manifold, as well as networks under two Grassmannian perspectives. Extensive experiments demonstrate that the proposed Riemannian convolutional networks significantly outperform existing SPD and Grassmannian networks.

Decision: reject
Reviewer scores: 3, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729026277537

thread_type:
official_review

<thread_object>
creator: reviewer_pika

<content>
summary:
The paper focuses on designing deep neural networks operating on general Riemannian manifolds, and in particular generalizing fully connected and convolutional layers to Riemannian geometries. Previous work in this domain has often been limited to specific manifolds or has relied heavily on the properties of certain geometric structures, limiting flexibility. The paper proposes a more general and flexible framework for constructing FC and convolutional layers that operate on general geometries, allowing broader applicability across different types of Riemannian manifolds. The paper demonstrates their framework primarily over the Symmetric Positive Definite (SPD) and Grassmannian manifolds.

strengths:
The groundwork for defining neural network architectures over general geometries is both theoretically interesting and provides a practical blueprint for future work designing such architectures. For manifolds where we cannot afford to (or where it may not make sense to) leave the manifold, such purely intrinsic formulations are crucial. Although components like the exponential map (let alone the logarithmic map) can often be intractable to compute, this work sets a valuable precedent for future endeavors in designing neural network architectures over specific manifolds in practice. The presented experiments also offer a broad array of baselines, with notable improvements over prior work, particularly on HDM05. Additionally, the framework's flexibility to vary the latent geometry could have significant implications for understanding and comparing geometric representations in neural networks.

weaknesses:
(roughly ordered by importance)

- **Theorem statements should be more self-contained**. For example, when reading Theorem 4.1, it appears to be merely defining the Riemannian FC layer, which leads to confusion about what is actually being proven. The lack of clarity here makes the theoretical contributions less accessible, reducing the impact of the formalism.
  
- **Flexibility claim for Grassmannian networks**. The main example manifold used to demonstrate the framework is the Grassmannian, where the authors claim that their layer uniquely allows the flexibility typical of MLP and CNN layers (i.e., modifying output dimensions like subspace dimension, ambient dimension, and channel count). While this is technically true, it is unclear why this flexibility cannot be achieved with simple modifications to previous architectures (e.g. basic lifting/copying maps and truncation), and there is no empirical evidence directly demonstrating the practical benefit of this added flexibility. This weakens the argument that the proposed method offers a clear advantage over prior approaches.
  
- **Marginal improvements over baselines**. While there is a non-negligible improvement on HDM05, the gains over other baselines are mostly marginal. Given that all of the listed manifolds have been extensively studied in their own right and with respect to neural network architectures (as well-documented by the authors), it remains unclear why practitioners should adopt the authors' framework rather than leveraging the extensive existing literature that exploits the structure of these highly structured manifolds. Furthermore, since this method relies on Riemannian logarithmic and exponential maps (which are often intractable or computationally prohibitive for general Riemannian manifolds) there is insufficient justification for the viability of this framework for designing neural network architectures over general manifolds. It would be nice to see an example manifold where NN architectures haven't been developed yet, or extensive ablations (with significant performance improvement) on one of the demonstrated manifolds to demonstrate the practical use of the flexibility in this framework.

questions:
- Could you clarify why this flexibility on the Grassmannian cannot be achieved through simple modifications of existing architectures (or why they would be ineffective)? Furthermore, do you have any empirical evidence demonstrating that this added flexibility provides a significant practical benefit compared to previous approaches?

- Are there any manifolds where (a) there is a practical benefit for learning features on this manifold in the broader ICLR community, (b) your framework provides a concrete guideline to designing a NN architecture on this manifold, and (c) NN architectures have not been studied before on this manifold?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729502055975

thread_type:
official_review

<thread_object>
creator: reviewer_qfyx

<content>
summary:
The authors introduce a framework for constructing fully connected and convolutional layers in contexts where one wants to construct a deep network with manifold-supported data and/or outputs. The framework for deriving such layers is general and only requires that the input and output spaces are Riemannian manifolds. For experiments, the authors specialize to several different geometries on the symmetric positive definite cone and the Grassmanian manifold, at which point they show improvements over previous constructions of such layers.

strengths:
- The main constructions of the layers are relatively intuitive, make sense, and natural to arrive at when starting at the analogous objects in Euclidean spaces.
- The cookbook looks sufficiently general as to be useful to others in the community as a starting point for formulating their own architectures. 
- The experimental results on different geometries are comprehensive relative to similar previous work, and show improvement over prior work.
- The paper is nicely written; the main intuitions are conveyed through clear writing and useful examples.

weaknesses:
- The Riemannian convolution may not be efficient. Part of why (Euclidean) convolutions are popular is that they can be efficiently computed by a GPU. Is there any possibility that the convolution may share this property (say for the SPD/Grassmanian examples)?
- More generally, it seems difficult efficiently compute/approximate a forward pass of a Riemannian feed-forward/conv layer on more general manifolds, since the forward pass uses exponential maps and logarithms. It seems that to get efficiently computable solutions one needs to do more ad-hoc/specialized work on top of the presented framework.

questions:
- Is there any intuition about why the layers formulated in the paper perform better than previous ad-hoc approaches? What practical benefits are gained by approaching this problem more generally?
- Is there any remedy when the exponential map is only locally defined? There seems to be an example in the appendix but is there any more general solution?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730154563000

thread_type:
official_review

<thread_object>
creator: reviewer_48wz

<content>
summary:
The paper proposes a technique for building fully-connected and convolution neural networks on Riemannian manifolds. The proposed network architecture builds on linear layers derived from Riemannian MLR [Chen et al., 2024c], and illustrates its manifestations on SPD and Grassmannian manifolds. To validate their method, the authors present experiments on matrix classification tasks.

strengths:
The proposed method can work generally on Riemannian manifolds, and does not rely on special structure such as gyrovector calculus. On the quantitative benchmarks, the method outperforms prior work on matrix classification.

weaknesses:
**Lack of Analysis**
Although the method seems to numerically out-perform prior work on matrix classification tasks, the paper should provide some explanation of why this may be true. The chosen baselines are known to be very sensitive to initialization and hyperparameters chosen, which make the numerical results difficult to interpret at face value. I would have liked if paper analyzed what exactly the model is learning. For example, it is surprising to see that Log-Euclidean metric seems to outperform the other metrics, even though it endows the SPD manifold with a flat structure. Does this mean that a flat structure is best for SPD matrix classification?  

Is there a theoretical explanation of why the proposed method may outperform prior work which use gyrovector calculus? 

**Other Riemannian Manifolds**
It would strengthen the paper to add experiments on other manifolds such as hyperbolic and spherical spaces. While the authors acknowledge their method’s inspiration from HNN++ [Shimizu et al., 2020], there are no experiments on hyperbolic space. It would have been interesting to see how the proposed method compares to the gyrovector calculus used in HNN++. One way to extend Riemannian projection layers to hyperbolic space is with horocycle projections. Prior work [A] have observed that horocycles can be used as an efficient feature map just like the Riemannian hyperplane projections proposed in this work.  

**Computational Cost**
The differences in computational cost seems significant with respect to the metrics used, due to the requirement of computing log and exponential maps. What is the compute time required for each method in Table 3? Do the methods vary much in parameters used and time required? For the proposed method, do the benefits of the added computational costs outweigh the drawbacks?

**Presentation**
It is not clear to me what the purpose of Section 7 is.

[A] Yu & De Sa., 2023. Random Laplacian Features for Learning with Hyperbolic Space. ICLR 2023.

questions:
- How does the computational cost compare between the different metrics used for the SPD tasks?
- How do you ensure that the input features are in a radius close enough to each layer’s base point so that the Log map is well-defined?
- Eq. 11 defines an inner product for a manifold via a linear isometry. Is there an example of this for SPD and Grassmannian spaces? Is there a canonical way to compute this isometry for general Riemannian manifolds?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730424756260

thread_type:
official_review

<thread_object>
creator: reviewer_gkrx

<content>
summary:
This work proposes a generalized Riemannian neural network by constructing a new linear layer and deriving several other constructions from it.

strengths:
This work has some positive aspects

1. The theory is well motivated, and the derivations do not appear to be wrong to me.
2. The experimental results show a marked improvement over prior work in terms of empirical performance.

weaknesses:
There are a few major weaknesses that prevent me from giving an accept rating for this paper.

1. The paper only compares on the SPD and Grassman manifold. Note that this paper is paper is proposing a "general framework" for constructing riemannian neural networks, so there should be an acknowledgement of other manifolds like hyperbolic space/sphere, which seem to be even more popular for experiments.
2. In my eyes, the paper does a bit of a disservice to prior work. For example, it is known from as early as Ganea that the gyro-multiplication/activation constructions are simple $\exp_0(f(\log_0(x))$ (where f is a standard linear or activation layer). If I remember correctly, this very premise was applied to more general geometries in the Skopek et al mixed curvature VAEs paper. If I also remember correctly (I believe I had derived this several years ago), the original SPDNet paper is effectively performing the same construction. It seems like a stretch to say the proposed FC layer is novel; rather, it would be more apt to say that the construction identifies a common thread in prior work while providing some references/examples of how this was done previously.
3. Building off of the previous points, beyond collecting some prior constructions under a common framework, the true contribution is the convolutional layer for practical purposes. The convolutional layer is not really a convolutional layer, but rather a linear layer between product manifolds $\mathcal{M} \to \mathcal{N}^d$. This does not seem morally right since convolution is defined in a separate way, and one can effectively view this as a hyperparameter/architecture change, which does not seem to be a good enough contribution.
4. One MAJOR GAP I see in the current construction is that there does not seem to be a mention of how to deal with activations/biases, which I looked for in the manuscript. If this is truly not talked about (which again, might be wrong since there are 32 pages of dense material where I can easily miss something in), then this seems to be a major problem. There needs to be a discussion of why the proposed construction does not simplify to $\exp_0(\mathrm{neuralnet}(\log_0(x))$, as is done in Ganea and Shizuma when they the introduce gyro-addition, which does not satisfy this form and breaks the $\exp-\log$ chain.

questions:
Why is parallel transport denoted PP instead of something like PT?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732920960910

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response (Part 1)

comment:
We thank all the reviewers for their constructive suggestions and valuable feedback 👍.  Below, we address the common questions (CQs).

**Summary of Changes and Remarks:**
- During the rebuttal, we implemented two additional hyperbolic FC layers, which have been added to App. J. As a result, our revised paper now includes **nine FC layers spanning three manifolds under nine different geometries**: five for the SPD manifold, two for the Grassmannian, and two for the hyperbolic space. 
- Sec. 4.1 (Riemannian FC layer) and Sec. 4.2 (convolutional layer) have been rewritten for improved clarity. 
- All references to the main paper cited in the rebuttal correspond to the numbering in the revised version of the manuscript.

***
## **CQ#1: Experiments on the Hyperbolic FC Layers (Reviewers $\textcolor{blue}{gKrX}$, $\textcolor{green}{48Wz}$, and $\textcolor{purple}{qFyX}$)**

The Riemannian computations over the hyperbolic space are much simpler than matrix manifolds, as they do not involve any matrix decomposition. Therefore, our Riemannian FC layer in Thm. 4.2 can manifest in a plug-in manner under different hyperbolic geometries. We focus on the Poincaré ball and hyperboloid models. We compare our hyperbolic FC layers against the Möbius transformation layer [a, Sec. 3.2] and Poincaré FC layer [b, Sec. 3.2]. The Möbius transformation is defined via the tangent space at the zero vector, while the Poincaré FC layer is specifically tailored for the Poincaré ball model.

**Expressions:** We denote our Riemannian FC layers under the Poincaré and hyperboloid models as RiemFC-P and RiemFC-H, respectively. The detailed expressions are provided in Thms. J.1 and J.2 in the revised manuscript.

**Implementation:** Following the official implementation in [c], we conducted experiments on the Cora, Disease, and Airport datasets for the link prediction task. Specifically, the baseline encoder consists of two transformation layers: the first maps the input feature dimension to 16, and the second maps 16 to 16. These transformation layers can be instantiated as our hyperbolic FC layers, Möbius transformation layers, or Poincaré FC layers.

**Table A**: Comparison of different transformation layers on the link prediction task. The hyperbolicity is denoted as $\delta$ (lower is more hyperbolic).

|  Method |    Geometry   |    Disease    |   Airport  |    Cora    |
|:--------:|:-------------:|:-------------:|:----------:|:----------:|
|          |               |    $\delta=0$ |$\delta=1$  |$\delta=11$ |
|    Möbius   | Poincaré Ball |  75.1 ± 0.3 | 90.8 ± 0.2 | 89.0 ± 0.1 |
| Poincaré FC | Poincaré Ball |  77.8 ± 1.4 | **94.0 ± 0.4** | 88.1 ± 0.3 |
|   RiemFC-P  | Poincaré Ball |  **79.2 ± 1.2** | 93.1 ± 0.7 | 89.2 ± 0.6 |
|   RiemFC-H  |  Hyperboloid  |  71.2 ± 0.6 | 84.3 ± 1.7 | **92.8 ± 0.4** |

Tab. A presents the 5-fold average testing AUC results across three datasets. We have the following following key observations:

- **Effectiveness**: Our RiemFC achieves superior or comparable performance to the prior Möbius and Poincaré transformations.
- **Hyperbolicity \& Poincaré ball**: On datasets with high hyperbolicity, RiemFC and Poincaré FC transformations consistently outperform the Möbius one. Conversely, on the Cora dataset with the lowest hyperbolicity, all three Poincaré transformations perform similarly. This suggests that for highly hyperbolic data, intrinsic Riemannian transformations are more effective, as tangent Möbius transformations may distort the geometry.
- **Hyperbolicity \& flexible metrics**: On the dataset with the lowest hyperbolicity, hyperboloid-based RiemFC outperforms all other Poincaré-based layers, highlighting the importance of the underlying metric in Riemannian networks. Unlike the prior Poincaré FC layer, which is designed specifically for the Poincaré ball model, our Riemannian FC layer in Thm. 4.2 can adapt to various metrics in a plug-and-play manner. This adaptability enhances the representation power of HNNs, making them more versatile for diverse applications.

***Remark:** The above discussion has been added to App. J in our revised manuscript. Please refer to it for more details.*
</content>
</thread_object>

</thread>
<thread>
creation date:
1733174393086

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gentle Reminder (24h Left)

comment:
Dear Reviewers,

We sincerely appreciate your dedicated time and effort in reviewing our paper. There are 24 hours left in the reviewer-author discussion period. As some reviewers have not acknowledged reading our response, we are uncertain if all concerns have been adequately addressed. If you have additional questions, we are more than willing to clarify further.

Your feedback is crucial to us, and we look forward to further discussions. 😄😄

Best regards,

Authors of paper 469
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work studies extensions of fully connected and convolutional layers to Riemannian geometries for use in neural networks. The authors propose a unified framework for the Riemannian fully connected (FC) layer and develop specific formulations for various Riemannian manifolds, including SPD, Grassmannian, and Hyperbolic spaces. Experiments on several tasks, such as radar and MoCAP classification, suggest improvements over prior art.  

The paper received three negative scores and one positive score. The primary concern raised by reviewers is the form of the solution, which is widely known—reformulating operations using a single tangent space (either at the origin of the manifold or, as done here, at a prototype point on the input manifold, followed by interpreting the result to lie on the identity tangent space of the output manifold). The idea of resortinbg to a chain of tangent spaces has been extensively utilized, and claims such as "faithfully respecting the underlying geometry" seem ambitious due to the potential for heavy distortion caused by linearization.  

The AC agrees with the reviewers' concerns. While the work has merits, the final formulation is based on a well-known idea. As such, the AC, sadly, recommends **rejection**. We hope this review helps the authors identify areas for improvement. Specifically, offering intuition about the impact of changing dimensionality on manifolds (even through toy problems), developing a more elegant approach to performing convolution, and explaining the implications of using different metrics (e.g., on the SPD manifold) within the context of the algorithm would provide very valuable insights.