Title: Towards a General Attention Framework on Gyrovector Spaces for Matrix Manifolds
Venue: NeurIPS 2025
Accepted: yes
Keywords: Manifold Learning

Gyrovector Space

Attention Mechanism

Riemannian Manifold

Signal Classification
Abstract: Deep neural networks operating on non-Euclidean geometries have recently demonstrated impressive performance across various machine-learning applications. Several studies have extended the attention mechanism to different manifolds. However, most existing non-Euclidean attention models are tailored to specific geometries, limiting their applicability. On the other hand, recent studies show that several matrix manifolds, such as Symmetric Positive Definite (SPD), Symmetric Positive Semi-Definite (SPSD), and Grassmannian manifolds, admit gyrovector structures, which extend vector addition and scalar product into manifolds. Leveraging these properties, we propose a Gyro Attention (GyroAtt) framework over general gyrovector spaces, applicable to various matrix geometries. Empirically, we manifest GyroAtt on three gyro structures on the SPD manifold, three on the SPSD manifold, and one on the Grassmannian manifold. Extensive experiments on four electroencephalography (EEG) datasets demonstrate the effectiveness of our framework.


Peer reviews:
comments: SUMMARY:
This paper proposes a general attention mechanism tailored for matrix manifolds by leveraging operations within gyrovector spaces. It introduces a homomorphism as a fundamental component of the proposed attention framework, specifically designed for Symmetric Positive Definite (SPD), Grassmannian, and Symmetric Positive Semi-Definite (SPSD) manifolds. The effectiveness of the proposed method is empirically validated through experiments on EEG signal classification tasks.

STRENGTHS AND WEAKNESSES:
### Strengths

- The paper is well-written and easy to follow.
- The newly proposed homomorphisms appear to be novel contributions.

### Weaknesses

- The proposed method exhibits poor performance, with overlapping confidence intervals between GyroAtt and baseline methods undermining the claimed superiority.
- Although the primary contribution is the introduction of homomorphisms, the paper lacks both theoretical justification and empirical validation demonstrating that these homomorphisms offer advantages over alternative transformations, such as multi-layer perceptrons (MLPs) with matrix manifold outputs.
- While the paper claims a unified perspective on attention mechanisms across matrix manifolds, it fails to derive meaningful insights from this unified framework. Moreover, due to the practical complexity involved in implementing homomorphisms for arbitrary matrix manifolds, the unified view has limited applicability. Consequently, the contribution effectively reduces to separately proposing attention mechanisms for three distinct manifolds rather than establishing a genuinely unified methodology.

QUESTIONS:
See the weaknesses section

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes GyroAttr, an attention framework that unifies operations across diverse matrix manifolds through a common gyrovector space formulation. GyroAttr is applied to three types of manifolds: the SPD manifold, the Grassmannian manifold, and the SPSD manifold. Experimental results show that the proposed method gets improvements in related tasks.\n\nSTRENGTHS AND WEAKNESSES:\nPros\n1.\tThe experimental results show improvement\n2.\tThe network architecture is described in sufficient detail, which is helpful for reproducibility.\n3.\tThe effectiveness of GyroAttr on multiple types of manifolds shows the generality of the proposed framework\nCons\n1.\tThe motivation of this paper (lines 100–108) is not well supported. For example, the authors claim that \"these approaches typically rely on geometry-specific operations, such as the BiMap layer [32] for SPD manifolds or the FrMap layer [33] for Grassmannian manifolds, which hinders their applicability to other manifolds and limits generalization.\" Although Section 3.1 revisits the network structures of previous methods, it is important to note that deep learning models are often treated as a kind of black-box system. Therefore, differences in network structure are insufficient to conclude that previous methods exist significant limitations and the proposed method effectively addresses them. \n2.\tAlthough the reported experimental results suggest that the proposed method outperforms existing approaches, the inconsistency in evaluation metrics raises concerns, especially since the authors claim in line 212: “Building on prior works [54, 40], we evaluate on four benchmarking datasets.” However, upon reviewing the cited works, it appears that there are no inter-session or inter-subject evaluation settings in Matt. In the SPD setting, the evaluation metrics used in this paper also appear to differ significantly.\n\nQUESTIONS:\n1.\tSome case studies or visualizations would help clarify the limitations of previous work. I suggest the authors either include some experiments to illustrate the issue or explain whether this is a commonly recognized challenge in the field with appropriate citations.\n2.\tIs it possible to align the experimental setup with that of any previously published work? Alternatively, could the authors clarify why the evaluation metrics used in this paper are more convincing?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper proposes GyroAtt, a framework for generalizing attention in (Euclidean) deep neural networks to a class of matrix manifold spaces (including the SPD, SPSD and Grassmannian manifolds). Methods using the framework are empirically validated on 4 real EEG datasets.

STRENGTHS AND WEAKNESSES:
*Quality*:
- (+) mathematically rigorous
- (+) extensive empirical evaluation
- (+) coherent and insightful framework for learning across different geometries

*Clarity*:
- (+) quite well-written and thorough

*Significance*:
- (+) potential for good impact within the field of BCI, and I imagine (though am less familiar with) many other applications
- (+) also interesting more abstractly for the mathematical deep learning community
- (-) seems to be no plans for sharing implementations of GyroAtt, which I consider a substantial weakness

*Originality*:
- (+) novel generalization/unification of existing approaches

QUESTIONS:
*Key points*:
1. Are there other obvious applications of the GyroAtt framework outside of EEG-related data? Elaborating a bit on the "various applications" on Line 16 would help make the motivation and potential impact more clear.

2. Line 1051 states plans to release code for accessing the data. Are there no plans to release implementations of GyroAtt?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new framework for computing transformer-like attention on general gyrovector spaces for data with underlying matrix geometry.  Their Gyro Attention (GyroAtt) framework is applied in three geometric settings, namely on the SPD manifold, on the SPSD manifold, and on the Grassmannian manifold. The real-world application under consideration is EEG signals classification using four separate datasets. Comparisons are performed against existing deep learning architectures designed for matrix geometry.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS:\n\n1. The Gyro-Attentional framework is an interesting and theoretically sound geometric extension of attention onto matrix geometries and manifold valued data, such as covariance matrices from EEG (or fMRI). \n2. The approach is principled and well motivated. The paper explains the relevant theory clearly with sufficient explanation and offers a unified framework applicable under different geometric assumptions on the manifold structure.\n3. Improved performance is observed for the EEG datasets for signal classification as compared against existing deep neural networks for matrix manifold data.\n\nWEAKNESSES:\n\n1. The authors gloss over how computationally expensive the optimisation is for their method, especially compared to the alternatives proposed. For example, the initialisation requires them to run multiple SVD like decompositions, running Karcher flow at every epoch incurs additional cost. \n\n2. There are very few details available on the baseline methods chosen, which makes it hard to evaluate their performance gains hard to contextualise.  Do the authors compare against graph neural networks (an alternate strategy for classification, but with potential for saving on computation with sparsity).\n\n3. How many model parameters do each have relative to the proposed framework? It is unclear whether this can be a contributing factor for the observed performance improvements.\n\nQUESTIONS:\n1. Since the datasets are on the small size,  are all the comparisons reported statistically significant in terms of improvements offered?\n2. What is the wall clock time for optimising each attentional framework\n3. How many model parameters do each considered attentional spaces have; how about the baselines being compared against?\n4. The authors should consider including a glossary for abbreviations as it is currently difficult to go back and forth within the manuscript each time one encounters an abbreviation.\n5. In section C3., why do the authors not choose to run a common principal components algorithm for initialisation instead of disjoint decompositions?\n\nLIMITATIONS:\nCurrently, within the datasets under consideration, the matrix sizes are <50x50. In the context of the questions on computation time and time to convergence, it is not clear whether Gyro-Attn would scale to larger matrices (such as those from fMRI brain-connectomes) for a broader range of applications", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}