Neural networks on Symmetric Spaces of Noncompact Type

Authors: Xuan Son Nguyen, Shuo Yang, Aymeric Histace
Primary area: learning on graphs and other geometries & topologies
Keywords: geometric deep learning, symmetric spaces, hyperbolic spaces, SPD manifolds

Abstract:
Recent works have demonstrated promising performances of neural networks on hyperbolic spaces and symmetric positive definite (SPD) manifolds. These spaces belong to a family of Riemannian manifolds referred to as symmetric spaces of noncompact type. In this paper, we propose a novel approach for developing neural networks on such spaces. Our approach relies on a unified formulation of the distance from a point to a hyperplane on the considered spaces. We show that some existing formulations of the point-to-hyperplane distance can be recovered by our approach under specific settings. Furthermore, we derive a closed-form expression for the point-to-hyperplane distance in higher-rank symmetric spaces of noncompact type equipped with G-invariant Riemannian metrics. The derived distance then serves as a tool to design fully-connected (FC) layers and an attention mechanism for neural networks on the considered spaces. Our approach is validated on challenging benchmarks for image classification, electroencephalogram (EEG) signal classification, image generation, and natural language inference.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730286687498

thread_type:
official_review

<thread_object>
creator: reviewer_yrjk

<content>
summary:
The paper presents a new formulation for the distance between a point and a hyperplane in symmetric spaces, i.e., spaces that generalize the commonly used hyperbolic spaces and the manifold of SPD matrices. Based on this distance, new FC and attention components of neural networks on such symmetric spaces are proposed. These neural networks are then demonstrated in the context of hyperbolic spaces and the SPD manifold, showing an advantage in performance compared to other existing methods.

strengths:
- **Solid Mathematical Foundation**: The paper introduces new definitions grounded in a well-developed mathematical framework.
- **New Neural Network Architecture**: The authors propose a new neural network design, including fully connected (FC) layers and an attention mechanism, that operate on symmetric spaces, extending the current capabilities of neural networks in non-Euclidean geometries.

weaknesses:
- **Presentation and Organization**: The paper's structure is fragmented, making it challenging to follow and fully appreciate its main contributions. Specifically:
    - **Sections 2 and 3** contain too many short, disconnected paragraphs or subsections that do not establish a cohesive narrative or logical organization.
    - **Section 4** would benefit from being divided into two distinct parts: one focused on the mathematical framework and the other on the proposed neural network. Since the neural network is the primary outcome, separating these would clarify the focus.
    - Additional minor structural suggestions:
        - **Section 4.3**: Consider renaming the section titled "Examples" to make its purpose clearer.
        - **Figure 2(b)**: Rotating this figure by 90 degrees could improve readability.

- **Experimental Results**: Some limitations in the experimental study reduce the impact of the findings:
    - **Section 5.1**: There are many other datasets with a clearer hierarchical structure that typically benefit more from hyperbolic space representation. For example, gene expression and word-document datasets are often considered as benchmarks of embedding in hyperbolic space. Therefore, the choice of CIFAR-10 and CIFAR-100 as benchmarks is questionable and should be explained, especially considering the marginal improvements reported in Table 2.
    - **Section 5.2**: There are multiple differences between the proposed methods and the baselines, with several moving parts. It remains unclear which component contributes most to the observed (marginal) improvement. Consider conducting an ablation study to isolate the effects of the different components. This would help clarify which aspects are most responsible for the performance improvements.
    - **Runtime Analysis**: Since the accuracy improvements are relatively small, including runtime metrics alongside accuracy would provide a fuller perspective on the method's practical benefits.

- **Limitations Statement**: The limitation statement at the end is very specific and could be extended to cover broader aspects of the proposed framework, e.g., scalability and practical applicability.

questions:
- Could the authors provide clarification on the choice of CIFAR-10 and CIFAR-100 datasets in light of the results and the available alternative datasets with stronger hierarchical structures?
- Please explain why more hierarchical datasets were not included.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730565421892

thread_type:
official_review

<thread_object>
creator: reviewer_75bj

<content>
summary:
The paper presents a new method for defining neural networks on symmetric spaces of noncompact type. The method is derived by deriving consider how hyperplanes are defined in such spaces and how point-to-hyperplane distance can be computed. This follows from expressing inner products via Busemann functions. This is then used for defining neural networks be generalizing the observation that affine functions in Euclidean space can be expressed as a function of a point-to-hyperplane distance. By replacing quantities in Euclidean space with their symmetric space counter parts, linear layers are defined to define neural networks on these spaces.

strengths:
- The approach provide a novel generalization of defining neural networks in the more general symmetric space of noncompact type. It is very appealing that the approach can be utilized on several types manifolds (Section 4.3)
- The paper provides is mostly written well to explain the technical background of the material (caveat below).
- Experimental results seem promising.

weaknesses:
- The connection between the proposed approach and previous ones are not exactly clear. Mostly in how / why the are different (see Questions)
- I think the narrative of eventually defining the FC layers in section 4.5 (and the attention mechanism) could be improved. Particularly, the I feel like the connection of expressing affine functions via point-to-hyperplane distances should be further elaborated (L396-404)

questions:
Questions / Remarks:

1. What are the specific connection between the proposed formulation versus the previous approaches presented in Table 1. I may be incorrect, but my understanding is that Ganea et al., 2018b is specialized for Hyperbolic spaces; and the b-distance approach is the Busemann function specialized to Hyperbolic spaces via Section 4.3 / Corollary 4.3. Is there a deeper reason why the point-to-hyperplane distances would not reduce to be the same? A reason for this question is that my initial perception was that the proposed look into symmetric spaces generalized the hyperbolic space, and thus when specializing to hyperbolic spaces one should obtain the same distance function. In summary, it would be great if you could provide a detailed summary / comparison for why the distances obtained in Ganea et al. 2018b differs from those obtained through your Busemann function approach.

2. Unsure if it is the original citation for the technique of affine maps as point-to-hyperplane distance functions, but Ganea et al., 2018b cites "Hyperplane margin classifiers on the multinomial manifold" by Lebanon & Lafferty. To this end, it would be useful to add this citation to Section 2.1 and perhaps further elaborate on the historical development of using point-to-hyperplane distances for affine maps.

Typos / Minor Mistakes:
 - Definition of hyperbolic distance is incorrect on Line 106. There is a type in the inner term's denominator. It should be "$(1 - \Vert x \Vert^2)$, the squared is in the wrong position.
 - wFM used on Line 470 before defined (in appendix Line 1192).
 - Missing "." punctuation on Line 425.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696102553

thread_type:
official_review

<thread_object>
creator: reviewer_dl7j

<content>
summary:
This paper introduces a framework for calculating point-to-hyperplane distances within symmetric spaces of noncompact type. Building upon this theoretical foundation, the authors propose novel manifold learning blocks tailored for neural networks, particularly designing fully connected (FC) layers and an attention mechanism applicable to these spaces. The paper demonstrates the effectiveness of this approach through numerical experiments, particularly on EEG classification tasks.

strengths:
1. The work presents a well-constructed theoretical basis by generalizing point-to-hyperplane distance formulations on symmetric spaces of noncompact type, encompassing both hyperbolic and SPD manifolds. This unified approach is a notable advancement that addresses the limitations in existing methodologies which often focus on narrower manifold types (e.g., Nguyen & Yang, 2023). The paper’s theoretical contribution strengthens its foundation, offering a comprehensive framework applicable across various symmetric spaces, potentially enhancing applications in machine learning on non-Euclidean geometries .
2. The experimental results, particularly on EEG datasets, demonstrate the approach’s capability. Despite minor performance gains, the proposed model achieves competitive accuracy, and in some cases, it outperforms existing methods such as EEG-TCNet, Graph-CSPNet, and MBEEGSE. This highlights the framework’s potential for real-world applications in EEG signal processing.

weaknesses:
Sections 4.5.1 and 4.5.2, which are the core practical contributions of this work, are difficult to follow. While the theoretical sections are clearly presented, the implementation of the proposed FC layers and attention mechanism in symmetric spaces feels briefly discussed and lacks an intuitive explanation. A more thorough discussion, with a step-by-step breakdown or additional illustrative examples, would greatly improve accessibility and clarity.

questions:
1. In Line 143, Add references following “Iwasawa decomposition of G” to provide readers with foundational context.
2. In Line 233, Correct “\(\Vert\ldot\Vert\)” to “\Vert\cdot\Vert” for accuracy in notation.
3. In Corollary 4.3, There seems to be an extraneous dot in the formula. 
4. In Definition 4.5, Shouldn’t the definition of “addition” assume an abelian group structure for coherence with traditional addition in symmetric spaces?
5. In Definition 4.7, The formula for “ g = k \exp(\mu) ” in line 342 could benefit from an intuitive explanation.
6. In Proposition 4.12, The construction of the FC layer from this proposition is not immediately intuitive. Adding a diagram or further expanding on its practical implications would help readers grasp the proposed structure better.
7.  Is there complexity analysis for the two proposed blocks. such as in the supp.?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730994098702

thread_type:
official_review

<thread_object>
creator: reviewer_dvj8

<content>
summary:
This paper proposes neural network structures on symmetric spaces of noncompact types, such as hyperbolic spaces or symmetric positive-definite (SPD) matrix manifolds. For this purpose, an expression for the distance between a point and a hyperplane is derived and utilized to generalize the Euclidean fully connected (FC) layer and attention layer. The proposed neural network is applied to various problems to demonstrate its performance advantages.

strengths:
- The proposed ideas of generalizing FC and attention layer to symmetric spaces are novel and seem reasonable. 
- They are also general enough to incorporate both hyperbolic spaces and SPD manifolds, which are non-Euclidean spaces of interest and frequent use in ML.
- Most mathematical derivations seem rigorous (I could not follow all the details).
- The paper made a great effort to show the empirical benefits of the proposed neural network by considering diverse benchmarks.

weaknesses:
- Understanding this paper requires a solid background in the geometry of symmetric spaces, and Section 3.2, in particular, is highly abstract and challenging to follow. This complexity may reduce the paper’s accessibility for a broader machine learning audience. Maybe incorporating the materials about decomposition equations from Appendix G.1 to G.3 in the manuscript, along with an explanation of their significance, would improve readability.
- The methods to forward propagate the FC layer and attention layer are explained, but it is not clearly defined what the inputs, outputs, dimensionalities, and trainable parameters of each layer are. Including equations or diagrams that summarize these details would enhance clarity. Additionally, there is limited discussion on the backward propagation process in each layer, particularly regarding gradient calculations. It would be helpful to specify whether gradient computation is feasible and, if so, describe the method.
- The rationale for considering both PEM and G-invariant metrics, as well as the differences between using each metric is not sufficiently discussed. A brief comparison of PEM and G-invariant metrics, with an overview of their respective strengths and limitations within the proposed neural network architecture, would help readers understand the motivation for including both metrics and their potential effects on network performance.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732188095207

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank the reviewers for their insightful comments and suggestions for improving our work. A common weakness pointed out 
by the reviewers is the clarity of the implementation of the FC layer and attention module. 
Those building blocks will be discussed in detail below. 
We propose to add those discussions to the Appendix. 

**FC layer**

*Input:* $x \in X$.

*Trainable parameters:* $k_j \in K$, $a_j \in \mathfrak{a},\\| a_j \\|=1$, $p_j \in X,j=1,\ldots,m$, $n \in N$. 

*Output:* $y \in X$.

In the case of SPD manifolds, we note that: 
- $K = O_m$ (the group of $m \times m$ orthogonal matrices). 
- $A$ is the subgroup of $m \times m$ diagonal matrices with positive diagonal entries.
- $N$ is the subgroup of $m \times m$ upper-triangular matrices with diagonal entries $1$. 

The dimensions of input, output and trainable parameters then can be inferred accordingly ($a_j \in \mathbb{R}^m,j=1,\ldots,m$). 

The computations performed by an FC layer are as follows.

*Step 1:* Compute $\ominus p_j \oplus x = h_j^{-1}gK$ where $p_j=h_jK,x=gK,h_j,g \in G,j=1,\ldots,m$. 

*Step 2:* Compute $g_j \in G,j=1,\ldots,m$ such that $k_j^{-1}[\ominus p_j \oplus x] = g_jK$. 

*Step 3:* Compute $H(g_j^{-1}),j=1,\ldots,m$ from the Iwasawa decomposition of $g_j^{-1}$, i.e., we need to determine the map 
$H: G \rightarrow \mathfrak{a}$ such that $g_j^{-1} = K \exp(H(g_j^{-1})) N$. 

*Step 4:* Compute $B_{\xi_j}(\ominus p_j \oplus x),j=1,\ldots,m$ as
\begin{equation*}
B_{\xi_j}(\ominus p_j \oplus x) = \langle a_j, H(g_j^{-1}) \rangle.
\end{equation*}

*Step 5:* Compute the output of the FC layer as
\begin{equation*}
y = n \exp \left ( [-B_{\xi_1}(\ominus p_1 \oplus x), -B_{\xi_2}(\ominus p_2 \oplus x), \ldots, -B_{\xi_m}(\ominus p_m \oplus x) ] \right ) K.
\end{equation*}

**Attention module**

The pipeline of the attention module is given in Fig. 2(a). 

*Input:* A sequence of points $x_j \in X,j=1,\ldots,l$. 

*Trainable parameters:* Those include trainable parameters of the three FC layers and $c_1,c_2 \in \mathbb{R}$ (used by the attention function). 

*Output:* A sequence of points $y_j \in X,j=1,\ldots,l$. 

In the case of SPD manifolds, the dimensions of input, output and trainable parameters follow from those in FC layers (see above). 

The computations performed by this module are as follows.  

*Step 1:* Apply the three FC layers to the sequence of input points $x_j$ and obtain three sequences of points $(q_j)\_{j=1}^l$ (queries), 
$(z_j)\_{j=1}^l$ (keys), and $(v_j)\_{j=1}^l$ (values): 
\begin{equation*}
(q_j)\_{j=1}^l = f_{lin}^q((x_j)\_{j=1}^l), \hspace{3mm} (z_j)\_{j=1}^l = f_{lin}^z((x_j)\_{j=1}^l), \hspace{3mm} (v_j)\_{j=1}^l = f_{lin}^v((x_j)\_{j=1}^l),  
\end{equation*}
where $f_{lin}^q(.)$, $f_{lin}^z(.)$, and $f_{lin}^v(.)$ are the linear transformations performed by the three FC layers. 

*Step 2:* Compute the similarities between queries and keys $\bar{\pi}\_{j'j},j',j=1,\ldots,l$ from the Riemannian distance function as
\begin{equation*}
\bar{\pi}\_{j'j} = -c_1d(q_{j'},z_j) - c_2.
\end{equation*}

*Step 3:* Compute the attention weights $\pi\_{j'j},j',j=1,\ldots,l$ as
\begin{equation*}
(\pi_\{j'j})\_{j=1}^l = \operatorname{softmax}\big( (\bar{\pi}\_{j'j})\_{j=1}^l \big).
\end{equation*}

*Step 4:* Perform the midpoint operation to get the sequence of output points $y_{j'},j'=1,\ldots,l$ as
\begin{equation*}
y_{j'} = f_{mid}\big( \\{ v_j,\pi\_{j'j} \\}\_{j=1}^l \big),
\end{equation*}
where $\pi\_{j'j}$ is the attention weight associated with $v_j$.
</content>
</thread_object>

</thread>
<thread>
creation date:
1744321131174

thread_type:
public_comment

<thread_object>
creator: ~bruno_aristimunha1

<content>
title:
The paper repository is empty.

comment:
Sorry, I couldn't find the code. Is the code in some other link?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper constructs novel neural network layers based on point-to-hyperplane distances in symmetric spaces of noncompact type. All reviews acknowledged the novelty and soundness of the authors' generalization, and are positive in accepting the paper.  In the discussion phase, the authors performed additional experiments on more datasets and clarified the complexity and runtime.

The reviewers also highlighted that the underlying mathematical concepts could be better presented with more intuitions to be accessible to the border audience. In preparing the final version, the authors are recommended to address these comments carefully and implement the proposed revisions in the rebuttal. Please include the computational complexity and runtime analysis, and revise the limitation statement accordingly.