Geometric Hyena Networks for Large-scale Equivariant Learning
Authors: Artem Moskalev
Mangal Prakash
Junjie Xu
Tianyu Cui
Rui Liao
Tommaso Mansi
Venue: ICML 2025 spotlightposter (2025)
Keywords: large molecule
equivariance
global context
long context
long convolution
scalability
memory efficient
mechanistic interpretability
architecrture
Primary area: deep_learning->graph_neural_networks
TL;DR: Geometric Hyena Networks is the first equivariant long-convolutional model that efficiently captures global geometric context at sub-quadratic complexity

Abstract: Processing global geometric context while preserving equivariance is crucial when modeling biological, chemical, and physical systems. Yet, this is challenging due to the computational demands of equivariance and global context at scale. Standard methods such as equivariant self-attention suffer from quadratic complexity, while local methods such as distance-based message passing sacrifice global information. Inspired by the recent success of state-space and long-convolutional models, we introduce Geometric Hyena, the first equivariant long-convolutional model for geometric systems. Geometric Hyena captures global geometric context at sub-quadratic complexity while maintaining equivariance to rotations and translations. Evaluated on all-atom property prediction of large RNA molecules and full protein molecular dynamics, Geometric Hyena outperforms existing equivariant models while requiring significantly less memory and compute that equivariant self-attention. Notably, our model processes the geometric context of $30k$ tokens $20 \times$ faster than the equivariant transformer and allows $72 \times$ longer context within the same budget.

Geometric Hyena Networks for Large-scale Equivariant Learning
Artem Moskalev'
Mangal Prakash' Junjie Xu 123
Tianyu Cui
Rui Liao
Tommaso Mansi
Abstract
Runtime
Memory
Processing global geometric context while pre- serving equívariance is crucial when modeling bi- ological, chemical, and physical systems. Yet, this is challenging due to the computational demands of equivariance and global context at scale. Stan- dard methods such as equivariant self-attention suffer from quadratic complexity, while local methods such as distance-based message passing sacrifice global information. Inspired by the re- cent success of state-space and conn--oovolutional models, we introduce Geometric Hyena, the first equivariant long-come lng-convolutional model for geomet- ric systems. Geometric Hyena captures global geometric context at qub-quadratic complexity while maintaining equivariance to rotations and translations. Evaluated on all-atom property pre- diction of large RNA molecules and full protein molecular dynamics, Geometric Hyena outper- forms existing equivariant models while requir- ing significantly less memory and compute that cquivariant self-attention. Notably, our model pro- cesses the geometric context of 30k tokens 20 faster than the equivariant transformer and allows 72x longer context within the same budget.
VN
Equiformer
Time
G-Tarsfomme
7.5
G-llyena
ac.
10000
20000
30000
10000
20000
30000
Sequence length
Sequence length
Figure I. Left: GPU forward runtime comparison. Geomet- ric Hyena scales sub-quadratically and achieves a considerable speedup compared to other equivariant models with global context. Right: Peak GPU memory consumption for G-Hyena is the most efficient for long sequences.
Processing global geometric context with equivariance is challenging due to the computational demands of processing high-dimensional data at scale. Existing methods either rely global all-to-all operators such self-attention (Liao & Smidt, 2023; de Haan et al., 2024; Brehmer et al., 2023), which scale poorly due to their quadratic complexity, or they restrict processing to local neighborhoods (Thomas et al., 2018; Köhler et al., 2020), losing valuable global information. This limitation a significant practical bot- tleneck, necessitating more efficient solutions for scalable equivariant modeling with global geometric context.
An efficient method for modeling global context should support easy parallelization during training while maintain- ing manageable computational costs at inference. One ap- proach involves recurrent operators (Orvicto et al., 2023; De et al., 2024), which provide bounded compute but lack easy parallelization. Another family of methods relies self- attention (Vaswani et al., 2017) allowing parallel processing at the cost of quadratic computational complexity. The most recent advances leverage state-space (Gu et al., 2021b; Fu et al., 2022; Gu & Dao, 2023) and long-convolutional (Romero et al., 2021: Poli et al., 2023) frameworks, enabling global context reasoning in sub-quadratic time with easy parallclization. Extending these models to accommodate equivariance remains unexplored direction.
1. Introduction
Modeling global geometric context with equivariance is crucial in many real-world tasks. The properties of a protein depend on the global interaction of its residues (Baker & Sali, 2001). Similarly, the global geometry of RNA dictates their functional properties (Sato al., 2021). In vision, modeling global geometric context is crucial when working with point clouds meshes (De Haan et al., 2020). In all these tasks, maintaining equivariance while capturing global context is essential for robust modeling and prediction.
'Jonnonn and Johnson Innovative Medicine "The Pennsylvania State University "This work done while the author was in- tem Johnson and Johnson. Correspondence to: Artem Moskaley cammoskalovartemee gmail.com>.
Inspired by the recent success of state-space and long- convolutional methods, we propose Geometric Hyena that efficiently models global geometric context in quadratcc time while preserving equivariance to rotations and trans- lations. We focus on a subset of geometric graphs where
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Geometric Hyena Networks for Large-scale Equivariant Learning
the canonical order can be established such as biomolecules.
2. Geometric Hyena Network
The focus on ordered geometric graphs differentiates Geo- metric Hyena from other equivariant frameworks and allows to leverage efficient sequence operators long convolutions. For equivariance, we introduce vector long convolution that utilizes vector products between equivariant queries and keys. The vector long convolution is implemented in the Fourier domain, achieving ( log N) computational com- plexity. We further show how to extend this vector long convolution to geometric long convolution by accommodat- ing interaction between invariant and equivariant subspaces to model more complex geometric relations.
The Geometric Hyena is designed for tasks that require mod- eling invariant and equivariant features in geometric graphs where canonical order can be established. The canonical ordering of a geometric graph implies unique and un- ambiguous enumeration of its nodes. For instance, such canonical ordering is naturally established by IUPAC rules (Damhus et al., 2005) for biomolecules.
An ordered geometric graph of N nodes can be written as a sequence (x,, (x2, f1). (xx,fx) where x; R$ represents vector features (e.g. coordinates or velocities), and f, E R represents scalar features (c.g. atom types or fingerprints). We call x; geometric or vector token, and f; is a scalar token. In addition, a geometric graph can have edge attributes eij E I between two nodes which are treated as scalar edge features. When working with geometric graphs, a neural network must respect symmetries of the input space and hence be equivariant with respect to geometric tokens and invariant with respect to scalar tokens.
We conduct experiments on four real-world datasets span- ning multiple large molecule property prediction and molec- ular dynamics simulation tasks. These include Open Vac- eine (Das et al., 2020) and Ribonanza-24 2k (He et al., 2024) datasets for stability and degradation prediction where our model outperforms other mfate-of-the-ae equivariant base- lines by up to 15%. Moreover, it achieves a 6% improvement on the mRNA switching-facee faactinnn-acoorooooo prediction task (Groher et al., 2018) and outperforms other equivariant methods by 9% on all-atom protein molecular dynamics prediction. In addition, introduce new geometric associative recall task for mechanistic interpretability (Olsson et al., 2022) of geomet- ric models. Experiments on this new geometric associative recall problem further validate the strong performance of our model and shed light on its scaling behavior. Overall, our results suggest that Geometric Hyena can outperform local and global equivariant methods while requiring less memory and compute for long geometric sequences. In particular, for sequence of 30k tokens, Geometric Hyena runs 20x faster than equivariant self-attention methods (see Figure 1). Notably, when the equivariant transformer runs out of memory on sequences over 37k tokens, our model can handle up to 2.7M million tokens on a single A10G GPU, providing up to 72x longer context length within the same computational budget.
Geometric Hyena consists of invariant and equivariant streams responsible for processing scalar and vector to- kens respectively. Formally, the Geometric Hyena model V/ : (I3 x IRd) + (IR3 x R satisfies equivariance propert::
(L, = ((L
(1)
where Lg 1R3 + R is representation of group action E SE(3). Thus, geometric tokens x; transform accord- ingly with the group action while scalar tokens f, remain invariant.
We make the overall information flow similar to the infor- mation flow of a transformer, as illustrated in Figure 2. The input is first mapped into keys, queries, and values via pro- jection layer. Next, the global context is aggregated via long convolution. Finally, the result of the long convolution is passed through the gating mechanism, and multiplied with values. We design each layer to be equivariant with respect to transformations of geometric tokens, and invariant for scalar tokens. This way, model consisting of compo- sition of equivariant layers is equivariant (Weiler & Cesa, 2019). We provide formal proof of equivariance for each module of Geometric Hyena in Appendix A.4.2.
To sum up, we make the following contributions:
We propose Geometric Hyena, the first equivariant long-convolutional architecture tailored for large geo- metric graphs with canonical ordering. Our model is designed to efficiently process global geometric con- text in sub-quadratic time.
We show that Geometric Hyena outperforms local and global equivariant methods, and can surpass equivari- ant aelf--attentonn on large-scale molecular property pre- diction and all-atom protein MD prediction tasks while requiring significantly less memory and compute.
Next, we will detail key architectural components of Geo- metric Hyena as shown in Figure 2.
2.1. Equivariant projection layer
We propose a new geometric associative recall task for the mechanistic interpretability of equivariant models on geometric data.
The equivariant projection layer & R x Rd R3 x Rd is a key component of our Geometric Hyena. It serves three primary functions: embedding scalar and vector tokens,
2

---

Geometric Hyena Networks for Large-scale Equivariant Learning
For global context aggregation of invariant scalar features, we employ long convolution (Romero et al., 2021; Poli et al., 2023) between query and key tokens. Queries serve as the input signal projection, while keys form a data-controlled implicit filter. To reduce computational complexity, the circular FFT-convolution is used. Let q' and kine be two signals length N composed of sequences of one-dimensional invariant queries (g)"*))(" and keys (I:"")) Ji=! respectively. Then, the global context for scalar tokens is aggregated by the FFT-convolution as:
Scalar long convolution
Add
SE(3) projection
Gating
Add
Geometric Long Convolution
SE(3)-Hyena
query, key
Operator
QKV projection
value
SE(3)
projection
{x
f
u on = q
q'nu @ k = F diag(Fk Fq
(2)
(a)
(b)
where F a discrete Fourier transform matrix, and diag(Fkfmr) is a diagonal matrix containing Fourier trans- form of the implicit filter k'nv, In the case when query's and key's dimension d > 1, the scalar FFT-convolution runs sep- arately for each dimension, rendering computational com- plexity of OdN log N) sub-quadratic in sequence length. This scalar long convolution is the core mechanism of the Hyena architecture (Poli et al., 2023).
Figure Geometric Hyena block. (a) Geometric Hyena block includes the SE(3)-Hyena operator and equivariant projections. (b) The SE(3)-Hyena operator includes query, key, value projection, geometric long convolution for global context aggregation, and gating.
mapping embedded tokens into query, key, and value triplets and serving as an output projection layer. Because have scalar and vector features, the projection layer should embed both of them into hidden scalar and vector features while maintaining equivariance. We discuss our specific choice of projection function in Section 3.
Vector long convolution To aggregate global context for geometric tokens, we introduce an equivariant vector long convolution. Unlike a scalar convolution that uses dot prod- ucts, vector convolution operates with vector cross product, denoted
between vector signals. For a vector signal consisting of N vector tokens qa40 E R and vector kernel k E RN3 we define the vector long-convolution as:
Projecting into queries, keys, and values Similar to transformers, we project tokens into queries, keys, and value triplets. Because our model includes equivariant and invari- ant streams, we need to obtain queries, keys, and values for invariant and equivariant features. To this end, we utilize the equivariant projection emit scalar and vector queries qene E Rdd, 0h R33 as qte q. @x f,) for i- th token, and similarly key and value projections Φx. Φv for scalar and vector keys kine kee and values v!n", egu respectively.
N Σ
eqv = (q
0x ke =
q
ogo
x
cqr
j-1
(3)
A naive implementation of convolution is Equation im- plies quadratic complexity. To obtain an efficient FFT-based ( log N) implementation, recall that we can rewrite the cross-product between two vectors a, b R component- wisc (a x b) [4] Ethpa [h]b[pp where Ethp is Levi- Civita symbol with 1, h. E {0, 1, and a[h] denotes h-th com- ponent of vector a. Substituting this identity into Equation 3 gives a scalar-comvolttion decomposition of the vector long- convolution:
2.2. Geometric long convolution
Geometric long convolution serves the global context aggregation module in the Geometric Hyena, analogous to the self-attention mechanism in transformer architectures. Its primary function is to encode global relations between queries and keys. The geometric long convolution comprises two components: scalar long convolution that accommo- dates global relations between scalar invariant tokens and equivariant vector long convolution that handles global rela- tions between vector tokens. The scalar and vector convolu- tions can be used separately for global context aggregation or they can be combined into the geometric long convolution enabling the interaction between invariant and equivariant subspaces.
(q Φ
Ethp qi 6qn [b] k (7m,(())
j=1 h.p
Σ
Ethp (9 (e
(4)
h.P
3

---

Geometric Hyena Networks for Large-scale Equivariant Learning
Thus, we can obtain 1-th output component of a resulting vector signal as signed sum of scalar convolutions be- tween the Cartesian components q and k<T [p]. To obtain the I-th output component, we only need the index pairs (h, p) for which Ethp * 0. Each 1 E 1,2) has exactly two such pairs, so the entire operation reduces to six scalar convolutions in total. Since the scalar convolution can be implemented with the FFT, decomposing the vector convolution to the series of sealar convolutions reduces its computational complexity from quadratic to O(N log N).
soft masking value between and for each token. Practi- cally, we implement the gating employing a projection layer (Section 2.1) with sigmoid activation. The gating emits a masking value my for i-th token in sequence. The masking is applied to the output of the geometric long convolution as m.u and m.u," 24e for scalar and vector tokens.
Finally, the resulting gated tokens are integrated with value tokens vine and vi using the element wise multiplication for scalar tokens and cross product for vector tokens. With this, the Geometric Hynea block (Figure 2) includes equiv- ariant projections, geometric long convolution, and gating. We provide a detailed ablation study on the contributions of each architecture ingredient in Appendix A.3.
Saariant-Equivaiann subspace interaction Scalar and vector long convolutions aggregate global context for scalar and vector tokens separately, limiting the geometric com- plexity of context relations that can be modeled. To address this limitation, we introduce geometric long convolution, which combines scalar and vector convolutions to enable scalar-vector interactions. Our approach expands possi- ble interactions between ccalar-vector tuples (a1, rι) and (a2, r2) E x 3 beyond basic scalar 0r10g and vector cross r1 X 12 products. In addition, we incorporate scalar- vector products an r2, 02hi and vector dot products rf r2 to represent invariant-oquiiaaiaat token interactions. With this, we define the mapping from input tokens to resulting scalar and vector tokens as 0g Xra1d2 + A2rf r2 and r3 = A3o1 F2 + \4 + As(ri x r2) where X; are train- able weights to learn the contribution of each product factor. Figure 5 of Appendix illustrates this information flow be- tween input and output tokens. Our approach is reminiscent of a convolution operation based on the Clebsch-Gorda tensor product between two [type-O, type-1] high-order ten- sors (Rowe& & Bahri, 2000). However, we maintain tokens in Cartesian space to avoid the high computational complex- ity associated with spherical representations. We derive the extension to higher-order steerable tensors in Appendix A.5.
3. Refining Geometric Hyena implementation
Equivariant projection with local and global context Inspired by the insights from planar graph data processing (Rampášek et al., 2022) where alternating global and local context processing significantly improves convergence, we found that the same holds also for geometric graphs. For projection with context, we employ one-layer Equivariant Graph Neural Network (EGNN) (Satorras al., 2021). This method maintains equivariance, allows interaction between invariant and equivariant subspaces, incorporates local con- text and easily accommodates edge features when available. Since EGNN operates over local context only, its complexity is linear with respect to number of nodes in a graph. We fur- ther improve the EGNN by including global context tokens that help summarizing the geometric graph. This is in line with the recent developments in planar GNNs, where the approach of using virtual nodes representing global context has shown to provide better convergence (Southern et al., 2024; Hwang et al., 2022).
Given N input tokens (x;; f.) G global context tokens [g., ) where g, E IR3 and h, E Rd, the equivariant projection layer follows the message passing framework with local and global messages computed as:
To implement geometric long convolution efficiently, we first map the d-dimensional invariant tokens f, to scalars via a linear layer: fi w f,, where w E TRd. This pro- duces feature tuples (ff,, x,) E R x 3 for each input token. Note that there are 2 interactions contributing to the scalar output and 3 interactions leading to vector output. These interactions, composed of scalar and cross products, can be handled using scalar and vector long convolutions as shown carlier, preserving sub-quadratic complexity. Additional implementation details are provided in Appendix A.4.1.
m = (f, Fjj ||x; - X,l12, Cis)
(5)
m
glob pg (f,, h,, log(1 + ||x: g (12)
(6)
where gg f1 Rd Rd R R - Rd are the func- tions to compute global and local messages implemented as one-layer perceptrons. Then, scalar and vector features are updated as:
2.3. Selective gating
Geometric Hyena incorporates selective gating mechanism to dynamically control information flow, similar to the soft- max operation in self-attention. This mechanism enables the model to emphasize relevant tokens while suppressing less informative ones. The gating is implemented through
*. = X1 + (V((1)] (E
Σ
(x:- - X, xx)x= Joc
(7)
the gating function 7 : 3 x Rdd + [0,1] that predicts
f, sp (f., ml + mloob)
(8)
4

---

Geometric Hyena Networks for Large-scale Equivariant Learning
where N, M
represents the top-k neighbors of the i-th token within radius and feature update functions 4f : IRd x R + _e and ioz R + R² are parameterized as one- layer perceptrons. We use log-distances for global messages to enhance stability, addressing large absolute distances that can occur with global tokens. For simplicity, we compute messages only between local tokens and from global-to- local tokens, omitting local-to-globul messages and updates to global tokens.
pendix A.4.1. Similar (Dao al., 2022; Poli et al., 2023), we sample sequences, and we test a one-layer model for the runtime and peak memory consumption. We record all runtimes on NVIDIA A10G GPU with CUDA 12.2.
The comparison is reported in Figure 1, showing forward pass time in milliseconds and peak GPU memory consump- tion in gigabytes. Geometric Hyena easily scales to longer sequences whereas VNT, Equiformer, and G-Transformer are from 20 to x20x slower for a sequence length of 30k tokens. Regarding memory usage, our model requires x less GPU memory than self-attention-biit
Global context tokens Global context tokens provide a snapshot of the overall structure of a geometric graph. We define global context tokens as a weighted average of scalar or vector tokens across all input tokens. Given input tokens (x;, f set of G global tokens {gy, h j==1 is com- puted as gj = C.TEE WiyX; for vector tokens and hy- =
methods for sequence length of 30k tokens. Moreover, we observed that when Vector Neuron Transformer and G- G-Transformer run out of memory on > 37k tokens, our model supports up to 2.7M tokens on a single GPU allowing for 72x longer ge- ometric context. This memory efficiency is attributed to the FFT long convolution that avoids materializing a quadratic self-attention matrix.
C "'E'" Winfi for scalar tokens where C2- = ΣM' Wij- The weighting factor Way is a learned scalar representing the contribution of the i-th input token to the j-th global context token. To detach the learning process from specific sequence length requirements, we employ a small SIREN network (Sitzmann et al., 2020) to predict way for each position.
4.2. Geometric associative recall
Associative recall is mechanistic interpretability task used to evaluate contextual learning capabilities of sequence mod- els (Olsson et al., 2022). In this task, a model must recall and copy bigrams it has seen before. For example, if the model encounters "Harry Potter" once, should predict "Potter" the next time it sees "Harry" (Gu & Dao, 2023). Recent studies demonstrate that strong associative recall correlates with better generalization real- down- stream tasks (Poli et al., 2024). Building on its widespread use for standard sequence models, we extend associative recall to geometric sequences of vectors in order to evaluate the contextual learning capabilities of equivariant models.
Key-Value normalization Key-Value normalization is crucial for maintaining numerical stability in Geometric Hyena. Let M be the maximum magnitude of an element in the query q, key k, or value v tokens. With this, the output magnitude after the long convolution can be bounded as ||ulla S ||a/|||| kll2 I 2 When further multiplied with the values, the final output magnitude is bounded as llylla s ||ul|//\v||2 S MM. This cubic growth with respect to input magnitudes can cause numerical instability and data type overflow. Consequently, training with the naive implementation requires extremely small learning rates and gradient clipping, resulting in slow convergence. To ad- dress this, we employ key-value normalization, ensuring unit norm for keys and values: ||k||2 = ||v||2 = 1. This bounds the output magnitude as ||>|| l|la S ||qlla S M re- moving the unstable cubic bound. In practice, we observed using key-value normalization to be critical step to ensure convergence and numerical stability of Geometric Hyena.
In geometric associative recall, each key-value bigram is linked by rotation matrix. Given a sequence of N vector tokens ending with query token, the model retrieves the as- sociated value based on its earlier occurrence (see Figure 4), Since rotating the entire sequence alters only the orientation of the predicted vector but not the key-value relationship. the task is -utation-quivaarian. Its complexity depends on the sequence length and the size of the vocabulary, where each vocabulary entry represents unique key-value bigram.
4. Experiments
We compare standard Hyena and Transformer models, as well as global-context equivariant models Vietels-Vector Neuron Transformer (VNT) (Deng et al., 2021; Assaad al., 2022) and Equiformer (Liao & Smidt, 2023) to proposed Geometrie Hyena. We also include G-Transformer baseline (Appendix 1.4.1), which mirrors the information flow of Geometric Hyena but uses equivari- ant self-attentin instead of long convolutions. We evaluate each model on 3D vector token sequences of lengths ranging from 27 to 210 and we also study how different hidden di-
Implementation details
4.1. Runtime and memory benchmarks
We benchmark the runtime and peak memory consumption for a forward pass of Geometric Hyena against other equiv- ariant models with global context: Vector Neuron Trans- former (Assaad et al., 2022), Equiformer (Liao & Smidt, 2023). Additionally, we compare with the G-Transformer baseline which follows the information flow of Geometric Hyena but relies on equivariant self- attention instead of long convolutions. The G-transformer baseline is detailed in Ap-
5

---

Geometric Hyena Networks for Large-scale Equivariant Learning
Geometric Associative Recall
but employed equivariant self-attention instead of geometric long convolution.
3.1
Figure 3 (bottom) reports results for varying hidden dimen- sion and vocabulary size. Geometric Hyena learns the re- trieval mechanism efficiently even with a small hidden di- mension, whereas other models require larger dimensions, and thus more parameters, to perform well. In particular, G-Transformer only excels higher dimensions, and VNT and Equiformer do not perform strongly even larger di- mensions. As vocabulary size grows, all models deteriorate, which is expected because there are more possible key-value associations and fewer occurrences of each bigram. Nev- ertheless, Geometric Hyena consistently outperforms the G- G-Transformer on all vocabulary sizes except for vocabulary size where they perform on par.
Transformer
2.3
Hyena
1,4
VNT
L2-dist.
0.6
Equiformer
G-Transformer
0.4
G- G-Hyena
0.2
27
2
2
Effect of hidden dimension
vocabu ary size
0.0
4.3. Large molecule property prediction
0.25
0.5
0:75
1.0
Hidden dimersion multiplier
Vocabu any
We evaluate Geometric Hyena on large-molecule prop- erty prediction task, focusing on RNA molecules repre- sented as atomic systems with equivariance to rotations and translations. Because of RNA's rapidly growing therapeutic importance, accurate in-silico modeling can substantially reduce expensive and time-consuming in-vitro and in-vivo experiments (Prakash et al., 2024; Yazdani-Jahroma et al., 2025). We focus on RNA molecules as they pose unique challenges: they have thousands of atoms, and the lack of robust backbone extraction methods often necessitates all-atom processing (Sponer et al., 2018). Moreover, their complex secondary and tertiary structures rely on both short- and long-range interactions (Xin et al., 2008). Consequently, an effective model must provide equivariance, incorporate both local and global context, and scale to high-dimensional RNA systems.
Figure Top:
The Læ-distance (4) between retrieved target vectors for the geometric associative recall task over various se- quence lengths. Bottom: The study of geometric associative recall performance of different models across varying hidden dimensions and vocabulary size.
mensions and vocabulary sizes affect the performance. We vary the hidden dimension by multiplying it by factors of [0.25, 0.5, 0.75, 1.J. and we vary the vocabulary size from to 6. For each experiment, we sample 2600 sequences for training. and 200 sequences each for validation and testing. We use Tæz-distance between predicted and ground truth vectors performance measure.
Our model uses two Geometric Hyena blocks and with hidden dimension of 80. For the other models, we choose hyperparameters so their depth and total parameter count match those of Geometric Hyena, ensuring a fair compari- son. All models are trained for 400 epochs using the Adam optimizer with cosine learning rate scheduler and linear warm-up of 10 epochs (see Appendix A.2 for further de- tails), We evaluate Equiformer only on sequences of lengths 2 and 28 as it runs out-of-memory for longer sequences.
Datasets We use two real-world RNA datasets used for vaccine development: Open Vaccine Covid-19 (Das et al., 2020) and Ribonanza-2k (He al., 2024). Open Vaccine consists of 3000 RNA sequences annotated with the stability profiles (reactivity, degradation at pH10, and degradation at Mg pH10) per nucleotide which a model is trained to predict. The Open Vaccine dataset contains sequences up to 7800 nodes in all-atom resolution. Ribonanza-28 2k includes 2260 sequences annotated with per nucleotide reactivity pro- files for DMS and 2A3 chemical modifiers (Low & Weeks, 2010). The Ribonanza-2k dataset contains RNA sequences of up to 11300 atoms. We obtain 3D all-atom structure representation using RhoFold (Shen et al., 2022), and we additionally evaluate models on eta-theta pseudotorsional backbone (Wadley et al., 2007) representation. Detailed dataset preparation is in Appendix A.2.
Results Figure 3 (top) shows the performance on se- quences of different lengths. Both Geometric Hyena and G- Transformer successfully learn to retrieve the correct value vector for given query. while VNT and Equiformer strug- gle but still outperform non-equivariant baselines which merely learn the dataset mean. We attribute the strong per- formance of Geometric Hyena to its alternating use of local and global context. This design allows it to first form local kay-v associations and then compare them across the entire sequence. This is also reflected in the strong results of G- Transformer, where we used similar building blocks
Implementation details We use atom and nucleotide iden- tities as invariant and 3D coordinates as equivariant features.
6

---

Geometric Hyena Networks for Large-scale Equivariant Learning
cosine learning rate scheduler and linear warm-up. The RMSE averaged over annotated nucleotides is used loss function. Appendix A.2 provides further details on the setup. Additionally, in Appendix A.3 we provide a detailed ablation study on the usefulness of different components of the Geometric Hyena architecture.
Table The RMSE (J) for two large RNA molecule stability and degradation prediction tasks over the all-atom and the backbone representations. Local context methods are red, and global context methods are in cyan. CCM denotes out-of-memory.
Open Vaccine C9vid-19 Ribonanza-2k
Model
(Dus el al., 2020)
(Ile el 2024)
Backbone representation
Results Table I reports the RMSE averaged over nu- cleotides on the Open Vaccine and Ribonanza-24 datasets, using three fixed random seeds. Methods that capture global context significantly outperform those relying on local con- text only, highlighting the importance of modeling global geometric structure. Among the global methods, Geometric Hyena achieves the best performance on both datasets, for both all-atom and backbone representations. On Open Vac- cine, it reduces RMSE by 12% and 15% for the backbone and all-atom representations, respectively, compared to the second- bootd model. On Ribonanza- 2k, it lowers RMSE by 1.5% and 8.4% for backbone and all-atom representations. An ablation study (Appendix A.3) shows that the largest im- provements come from alternating local and global context. We also hypothesize that long convolutions are natural fit for molecules with strong sequence prior (such RNA and proteins) as they inherently differentiate token orderings by breaking permutation symmetry.
SchNet
5.515
10900
TFN
0.522a0.006
0.927=0006
EGNN
0.529
0.94320000
FastEGNN
0.519x0.012
2-0002
LEFTNet
0.502+0.004
0.889=000
TMD-ET
0.500aa.006
0.781
Transformer
0,400aa.004
0,63720006
Hyena
0,447a0.057
1..1.40
VNT
0,401
0.659=000s
G- G-Transformer
0.41 2+0005
0.537-0007
Equiformer
0.409+0.008
0.649-0004
G-Hyena
0.363++..45
0.529-0005
All-atom representation
SchNet
0.512
1
TFN
0.510x0.004
0.910=001
EGNN
0.51 1 +0,005
0.928-oo22
FastEGNN
0.498+0.004
0.873-0010
LEFTNet
2.501
0. 0.88020008
TMD-ET
0,49420.009
0.855
Transformer
0.399
0.633
4.4. Predicting RNA molecule switching factor
Hyena
0.393
0.605=0017
VNT
0.39 20.013
00
We further test our model's ability to capture geometric dependencies by predicting the switching factor of RNA riboswitches (Groher et al., 2018). Specifically, we predict the dynamic range of riboswitches that measures the fold- change in gene expression between the ligand-bound and unbound states. This task requires reasoning over both short- and long-range geometric dependencies, as RNA molecules are represented as atomic systems where 3D spatial configu- rations significantly influence their regulatory behavior.
G- G-Transformer
0.391 1 x000771
0.59220043
Equiformer
OOM
DOM
G-Hyena
0.339
0.54620006
The models trained to regress stability and degrada- tion profiles per-nucleo The root mean squared error (RMSE) is used as evaluation metric averaged over nu- cleotides as in Das et al. (2020).
We compare Geometric Hyena against non-equivariant both trained with data
Dataset We utilize the Te-Ribo dataset (Groher et al., 2018) that contains 355 RNA molecules (up to 3850 atoms) with experimentally measured switching factors (dynamic range). Similar to Open Vaceine and Ribonanza-22 2k, we utilize RhoFold to obtain all-atom 3D representation, and we additionally experiment with eta-theta pseudotorsional backbone representation.
Hyena and an augmentation, well against other equivariant models with either local or global context. As local equivariant base- lines, we employ SchNet (Schütt et al., 2017), TFN (Thomas et al., 2018), EGNN (Satorras et al., 2021), and recent Torch- MD Equivariant Transformer (TMD-ET) (Pelaez et al., 2024), LEFTNet (Du et al., 2023) and FastEGNN (Zhang et al.. 2024). As global equivariant baselines, we employ Vector Neuron Transformer (VNT) (Assaad et al., 2022), Equiformer (Liao & Smidt, 2023) and the G-Transformer model (Appendix A.6). We use Geometric Hyena with 3 blocks and the hidden dimension of 80. For the baselines, we choose hyperparameters to equate the depth and number of parameters to the Geometric Hyena model. We train all models for 200 epochs with the Adam optimizer with
Implementation Details The experimental setup for RNA switching factor closely mirrors the pipeline used for RNA degradation and stability tasks. The invariant features in- clude atomic and nucleotide identities, while atomic coor- dinates are used to extract equivariant features. A model regresses the switching factor as a scalar output. And the root mean squared error (RMSE) is used as the evaluation metric. We employ the similar model and training config-
7

---

Geometric Hyena Networks for Large-scale Equivariant Learning
Table The RMSE (4) for predicting RNA ribo-switching factor measurement over the all-aton atom and the backbone representations.
Table 3. The MSE (L) of predicted and ground truth all all-atom and
backbone protein MD trajectories.
Te- Ribo
ProteinMD (Han al., 2022)
Model
Model
(Groher et al., 2018)
Backbone rept.
All-atom repr.
Backbone repr. All-atom repr.
SchNet
0.737-0002
0.691-0018
Linear
2.27-0001
2.90+0.01
TFN
0.733
0.71020.009
RF
2.2620.001
2.85
EGNN
0.728a0.001
0.729x00022
TEN
2.2610.002
2.8520.002
FasEEGNN
0.704g00005
0.727
EGNN
2.25
2.72+0.003
LeftNet
20006
0.750=0.04
FastEGNN
1.84
NaN
TMD-ET
0.750
0.75 10000
Transformer
75.83
79.68a24.2
Transformer
0.556
005
Hyena
48.9 48.94.9.03
55.34
Hyena
0.5
0.
G-Transformer
2.45
3.67 3.67+0.640
G- T-Tannffrrmer
0.554
0.553
Equiformer
OOM
OOM
Equiformer
0.550
CCM
G-Hyena
1880200099
2.4920.037
G-HHynna
0.548=0.088
0.548=0.001
Implementation Details A model predicts the spatial dis- placement of backbone atoms in the next MD frame 3D vector for each atom. The MSE between the predicted and ground-truth displacements is used as both the loss function and evaluation metric. Our model consists of 2 Geometric Hyena blocks with hidden dimension of 32 for backbone version, and Geometric Hyena blocks with a hidden dimension of 50 for all-atom version. Baseline meth- ods are configured to have comparable parameter counts and depth. The optimization parameters are set as in Experiment 4.3. We do not run VNT, LeftNet, and TMD-ET models as their publicly available implementations do not permit rector-valued prediction and hence are not applicable for this task.
urations as in the stability and degradation experiment 4.3 except for using smaller hidden dimension of 30 and larger weight decay of 0.05.
Results Table 2 reports the RMSE averaged over three ran- dom seeds for the Tc-Ribo dataset. As observed in the Open Vaccine and Ribonanza-24 2k datasets, methods incorporating global context consistently outperform baselines limited to local context. Among the equivariant approaches, Geomet- ric Hyena and Equiformer achieve the best performance, slightly edging out other global equivariant methods and showing significant improvements over local ones. Interest- ingly, we observed that standard non-equivariant Hyena and Transformer, when trained with data augmentation, per- form competitively even with global equivariant methods, suggesting potentially lower geometric complexity of the dataset.
Results Table 3 reports the MSE, averaged over three fixed random seeds, on the protein MD dataset. For MD trajectory prediction, Geometric Hyena reduces the final MSE by 2% for backbone and by 9% for all-atom represen- tation compared to the second-best performing model. For FastEGNN, we observed severe numerical instabilities in the all- all-atom setting, leading to NaN outputs.
4.5. Protein molecular dynamics
Next, we evaluate Geometric Hyena on equivariant pro- tein molecular dynamics (MD) prediction task. This task involves modeling the equilibrium dynamics of protein back- bone atoms over time, where interactions between atoms are determined by both local and long-range geometric rela- tionships.
5. Conclusions
We introduce Geometric Hyena being, to the best of our knowledge, the first equivariant long-convolutional archi- tecture for efficient modeling of global geometric context at sub-quadratic complexity. Through experiments on the novel geometric associative recall, large RNA molecule property prediction, and full protein molecular dynamics, we demonstrate that equivariant long-convolutional models can outperform other local and global equivariant methods while requiring a fraction of the computational and memory cost of self-attention. By efficiently capturing global geo- metric context at scale, our method opens the door to a wide
The protein molecular dynamics dataset is pro- cessed from MDAnalysis (Han et al., 2022), derived from an AdK equilibrium MD trajectory (Seyler & Beckstein, 2017). The dataset consists of 4186 proteins tertiary structures with corresponding trajectories. A time span of the MD is set to At = 15 as in Zhang et al. (2024). Experiments are conducted on both backbone and all-atom versions of the dataset, where the backbone version contains 855 atoms per molecule, and the all-atom version comprises 3341 atoms.
Dataset
8

---

Geometric Hyena Networks for Large-scale Equivariant Learning
range of future applications across biological, chemical, and physical domains.
Cen, J., Li, A., Lin, N., Ren, Y., Wang, Z., and Huang. W. Are high-degree representations really unnecessary in equivariant graph neural networks? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: htt.httsss//pppeeeiiiiie openrevieee net/ forum? MddMOnoNN.GYN..
Limitations and Future work This work introduces new approach for modeling global geometric context with equivariance, tested on geometric graphs from biological or chemical systems with canonical ordering (Jochum & Gasteiger, 1977). Unlike self-attention, the FFT-based convolution at the core of our method is not permutation cquivariant (except for cyclic shifts). This property can actually be beneficial for important areas such as biomolec- ular modeling (e.g.-min e-g.-modeling RNA or proteins), where the natural ordering provides important inductive bias. In settings like point cloud processing, though, establishing canonical order is challenging, and the lack of permu- tation equivariance can become critical. Addressing this limitation by incorporating explicit permutation equivari- ance or learning it from data (Moskalev et al., 2023) could extend the benefits of our approach to even broader range of applications.
Damhus, T., Hartshorn, R., and Hutton, A. Nomenclature of inorganic chemistry: Iupac recommendations 2005. Chem. Int, 27:25 260 2005.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory -eenory-efficient exact attention with io-awareness. Advances Neural Information Process- ing Systems, 35: 35:16344-16359, 2022.
Das, R., Wayment-Steele, H., Soon Kim, D., Choe, C., Tunguz, B., Reade, W., and Demkin, M. Open- vaccine: Covid-19 mrna vaccine degradation pre- diction, 2020. URL https://kaggle, com/ competitions/stariiit
covid
De, S., Smith, S. L., Fernando, A., Botev, A., Cristian- Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., et al. Griffin: Mixing gated linear recur- rences with local attention for efficient language models. arXiv preprint 24t.:::2002..922... 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
De Haan, P., Weiler, M., Cohen, T., and Welling, M. Gauge equivariant mesh cnns: Anisotropic convolutions on geo- metric graphs. arXiv preprint arXiv:2003.05425, 2020.
References
Assaad, S., Downey, C., Al-Rfou, R., Nayakanti, N., and
Haan, P., Cohen, T., and Brehmer, Euclidean, pro- jective, conformal: Choosing geometric algebra for equivariant transformers. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, volume 27, 2024. URL https://arxiv org/ abaa /000//331.4444. 04744.
Sapp. B. Vn-transformer: Rotation-equivariant attention for vector neurons. 2022.
Baker, D. and Sali, A. Protein structure prediction and structural genomics, Science, 294:93 96, 2001. URL http:s///app.ssmattiissssssss org/CorppsID:: 7193705
Deng, C., Litany, O., Duan, Y., Poulenard, A., Tagliasacchi, A., and Guibas, L. Vector neurons: general frame- work for -equivaaui networks. arXiv preprint arXiv:2104. /2229, 2021.
Bekkers, E. J., Vadgama, S., Hesselink, R. D., van der Lin- den, P. A., and Romero, D. W. Fast, expressive se (n) equivariant networks through weight-sharing in position- orientation space. arXiv preprint arXiv:2310.02970, 2023.
Dorst, L., Fontijne, D., and Mann, S. Geometric algebra for computer science (revised edition): An -breett- approach to geometry. Morgan Kaufmann, 2009.
Brandstetter, J., Hesselink, R., van der Pol, E.,, Bekkers, E. and Welling, M. Geometric and physical quantities im- prove (3) equivariant message passing. In International Conference on Learning Representations, 2021.
Du, Y., Wang, L., Feng, D., Wang, G., Ji, S., Gomes, C. P., Ma, Z.-M., et al. A new perspective on building efficient and expressive 3d equivariant graph neural networks. Ad- vances in Neural Information Processing Systems, 36, 2023.
Brehmer, J., Haan, P. D., Behrends, S., and Cohen, T. Geometric algebra transformer. In sevethh Conference on Neural Information Processing Systems, 2023. URL https:/ // http::///peeeeeeeeeeeem net forum? 1CC44.CC
Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Ré, C. Hungry hungry hippos: Towards lan- guage modeling with state space models. arXiv preprint arXiv:22 14052, 2022.

---

Geometric Hyena Networks for Large-scale Equivariant Learning
Fuchs, F,, Worrall, D., Fischer, V., and Welling, M. Se (3) ts-ttassfoo::::: 3d aoto-translation equivariant attention networks. Advances in neural information processing systems, 33:1970-1981, 2020.
Kim, J., Nguyen, D., Min, S., Cho, S., Lee, M., Lee, H., and Hong, S. Pure transformers powerful graph learners. Advances in Neural Information Processing Systems, 35: 14582-14595, 2022.
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for quantum chem- istry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Kingma, D. P. and Ba, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. arXiv preprint arXiv: 1609. 2016.
Groher, A.-C., Jager, S., Schneider, C., Groher, F,, Hamacher, K., and Suess, B. Tuning the performance of synthetic riboswitches using machine learning. ACS synthetic biology, 8(1):34-44, 2018.
Köhler, J., Klein, L., and Noé, F. Equivariant flows: exact likelihood generative learning for symmetric densities. In International conference on machine learning, pp. 5361- 5370. PMLR, 2020.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. 2023.
Kreuzer, D.,, Beaini, D., Hamilton, W., Létourneau, V., and Tossou, P. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34:21618-21622 2021.
Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00300 2021a.
Li, S., Singh, H., and Grover, A. Mamba-nd: Selective state space modeling for multi-dimensional data, arXiv preprint arXiv:2402.05892 2024.
Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and Ré, C. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34: 572-585, 2021b.
Liao, Y.-.. and Smidt, T. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In- ternational Conference on Learning Representations, wwt.:orumtf
2023, URL
Han, J., Huang, W., Xu, T., and Rong, Y. Equivariant graph hierarchy-basee based neural networks. Advances in Neural Information Processing Systems, 35:9176-9187, 2022.
id-KwmP FARgÖTD.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gra- dient descent with warm restarts. arXiv preprint 1608. 2016.
He, S., Huang, R., Townley, J., Kretsch, R. C., Karagianes, T. G., Cox, D. B., Blair, H., Penzar, D., Vyaltsev, V., Aristova, E., et al. Ribonanza: deep learning of ma structure through dual crowdsoureing. bioRxiv, 2024.
Low, J. T. and Weeks, K. M. Shape-directed ma secondary structure prediction. Methods, 52(2):150-158 2010.
Moskalev, A., Sepliarskaia, A., Bekkers, E. J., and Smeul- ders, A. W. On genuine invariance learning without weight-tying. In Topological, Algebraic and Geometric Learning Workshops 2023, pp. 218-227. PMLR, 2023.
Hwang, E., Thost, V., Dasgupta, S.S., and Ma, T. Revisiting virtual nodes in graph neural networks for link prediction, 2022. URL https::////. / http///ppeeeeeeeeeeeeeeeeee net/ forum?
id=ETiaOyNwJW.
Nguyen, E., Poli, M., Faizi, M., Thomas, A., Wornow, M., Birch-Sykes, C., Massaroli, S., Patel, A., Rabideau, C., Bengio, Y., al. Hyenadna: Long-range genomic se- quence modeling single nucleotide resolution. Ad- vances in neural information processing systems, 36. 2024.
Jing, B., Eismann, S., Soni, P. N., and Dror, R. O. Equivari- ant graph neural networks for 3d macromolecular struc- ture. arXiv preprint (0Xiv:2106008844 2021a.
Jing, B., Eismann, S., Suriana, P., Townshend, R. J. L., and Dror, R. Learning from protein structure with ge- ometric vector perceptrons. In International Confer- ence on Learning Representations, 2021b. URL. https: / //openreview. net forum? - weiimmidddddidddioe IYLJVV56644
Olsson, C.,, Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. In-context learning and induction heads. arXiv preprint arXiv: 1895, 2022.
Jochum, C. and Gasteiger, J. Canonical numbering and con- stitutional symmetry. Journal of Chemical Information and Computer Sciences, 17(2): 13-1177, 1977.
Orvicto, A., Smith, S. L., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., and De, S. Resurrecting recurrent neural
10

---

Geometric Hyena Networks for Large-scale Equivariant Learning
networks for long sequences. In International Conference Machine Learning, pp. (6670-26698,, PMLR, 2023.
Satorras, V. G., Hoogeboom, E., and Welling, M. E (n) equivariant graph neural networks. In International con- ference machine learning, pp. 9323-9332. PMLR, 2021.
Pelaez, R. P., Simeon, G., Galvelis, R., Mirarchi, A., East- man, P., Doerr, S., Thölke, P., Markland, T. E., and Fabri- tiss, G. D. Torchmd-net 2.0: Fast neural network poten- tials for molecular simulations, 2024.
Schiff, Y., Kao, C.-H., Gokaslan, A., Dao, T., Gu, A., and Kuleshov, V. Caduceus: Bi-directional equivari- ant long-range dna sequence modeling, arXiv preprint MoXiv:2403.022344 2024.
Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Ré, C. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp. 28043 28078. PMLR, 2023.
Schütt, K., Kindermans, P.-.,, Sauceda Felix, H. E., Chmicla, S., Tkatchenko, A., and Müller, K.-R. Schnet: A Nontinuous-filter convolutional neural network for model- ing quantum interactions. Advances in neural information processing systems, 30, 2017.
Poli, M., Thomas, A. W., Nguyen, E., Ponnusamy, P., Deiseroth, B., Kersting, K., Suzuki, T., Hie, B., Ermon, S., Ré, C., Zhang, C., and Massaroli, S. Mechanistic design and scaling of hybrid architectures. In ICML, wtw.eewwwee///ruurnneee
Seyler, S. and Beckstein, O. Molecular dynamics tra- jectory for benchmarking mdanalysis, 2017. URL: https://figshare camntaricces/Moleeulaadddmmmm tra- jectory for hatp:aaanninnnnDaaaaasaasss//11/ doi, 10: m9, 2017.
2024. URL
GdyGDp7Gyd9nf.
Prakash, M., Moskalev, A., DiMaggio, P. A., Combs, S., Mansi, T., Scheer, J., and Liao, R. Bridging biomolecular modalities for knowledge transfer in bio-language models. bioRxiv, pp. 2024-10, 2024.
Shen, T., Hu, Z., Peng. Z., Chen, J., Xiong, P., Hong, L., Zheng, L., Wang, Y., King, I., Wang. S., et al. E2efold-3d: End-to-end deep learning method for accurate de novo ma 3d structure prediction. arXiv preprint arXiv:2207.01588, 2022.
Rampäšek, L., Galkin, M., Dwivedi, V. Luu, A. T., Wolf, G., and Beaini, D. Recipe for a general, powerful, seal- able graph transformer. Advances in Neural Information Processing Systems, 35: 15:14501114515, 2022.
Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wetzstein, G. Implicit neural representations with peri- odic activation functions. Advances in neural information processing systems, 33:7462-7473. 2020.
Romero, D. W., Kuzina, A., Bekkers, E. Tomezak, J. M., and Hoogendoom, M. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.02611, 2021.
Southern, J., Di Giovanni. F., Bronstein, M., and Lutzeyer, J. F. Understanding virtual nodes: Oversmoothing, oversquashing, and node heterogeneity. arXiv preprint arXiv:2405.1333 13526, 2024.
Rowe, D. and Bahri, C. Clebsch-gordal coefficients of su (3) in su (2) and so (3) bases. Journal of Mathematical Physics, (1(9):554466555 6565, 2000.
Sponer, J., Bussi, G., Krepl, M., Banas, Bottaro, S.,
Ruhe, D., Brandstetter, J., and Forré, P. Clifford group equivariant neural networks. In Thirty-seventh Con- ference on Neural Information Processing Systems, net/ net/forum? forum?
Cunha, R. A., Gil-Ley, A., Pinamonti, G., Poblete. S., Jurecka, P., et al. Rna structural dynamics as captured by molecular simulations: comprehensive overview. Chemical reviews, 118(8):4177-4335 2018
2023a. URL wtttts:://ppereeeeeeeeeeeeeeeeee 1b=n@GbZMGGUD.
Thomas, N., Smidt, T., Kearnes, S., Yang. L., Li, L., Kohlhoff, K., and Riley, P. Tensor field networks: Rotation-and translation-equivariann neural networks for 3d point clouds. arXiv preprint wtX.:::8822.02/999 2018.
Ruhe, D., Gupta, J. K., De Keninck, S., Welling, M., and Brandstetter, J. Geometric clifford algebra networks. In International Conference Machine Learning, pp- PMLR, 2023b.
Vadgama, S., Islam, M. M., Buracas, D., Shewmake, C., Moskalev, A., and Bekkers, E. Probing equivariance and symmetry breaking in convolutional networks. arXiv preprint arXiv:2501.01999 2025.
Rusch, T. K., Bronstein, M. M., and Mishra, S. A survey oversmoothing in graph neural networks, arXiv preprint arXiv:2303.10999 2023.
Sato, K., Akiyama, M., and Sakakibara, Y. Rna secondary structure prediction using deep learning with thermody- namic integration. Nature communications, 12(1):941, 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.

---

Geometric Hyena Networks for Large-scale Equivariant Learning
Wadley, L. M., Keating, K. S., Duarte, C. M., and Pyle, A. M. Evaluating and learning from ra pseudotorsional space: quantitative validation of a reduced representation for ma structure. Journal of molecular biology, 372(4): 942-957, 2007.
graph. Advances in Neural Information Processing Sys- lems,
2021.
Yazdani-Jahromi M., Prakash, M., Mansi, T., Moskalev, A., and Liao, HELM: Hierarchical encoding for mRNA language modeling. In The Thirteenth In- lernational Conference on Learning Representations, 2025. URL https: http:::ttttttreeeeeeetee http///pppeeeeeeeeeeeeeeeeeeeeet net /ormm? i 1d=MMHcnUOn1C.
Wang, X., Wang, S., Ding. Y., Li, Y., Wu, W., Rong, Y., Kong, W., Huang, J., Li, S., Yang. H., et al. State space model for new-generation network alternative to trans- formers: A survey. arXiv preprint arXiv:2404.09516, 2024a.
Zhang, X., Wang, L.,, Helwig, J., Luo, Y., Fu, C., Xie, Y., Liu, M., Lin, Y., Xu, Z., Yan, K., et al. Artificial intelli- gence for science in quantum, atomistic, and continuum systems. arXiv preprint
Wang, Y., Cheng, C.. Li, S., Ren, Y., Shao, B., Liu, G., Heng, P.-A., and Zheng, N. Neural 3Sm: A long- range interaction modeling enhancer for geometric GNNs. In The Thirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024b. URL https: //openrevien fortmformm id=ncqauwS115.
2023.
Zhang, Y., Cen, J., Han, J., Zhang, Z.. ZHOU, J., and Huang, W. Improving equivariant graph neural net- works on large geometric graphs via virtual nodes learn ing. In Forty-first International Conference on Machine Learning, 2024, URL https: /pwpnrevieneeeeeeeeeeeeeeeeeeeee .net/ forum? ? 1d=widkNKUYk..
Vayment-Steele, H. K., Kladwang. W., Watkins, A. M., Kim, D. S., Tunguz, B., Reade, W., Demkin, M., Ro- mano, J., Wellington-Oguri, R., Nicol, J. J., et al. Deep learning models for predicting ma degradation via dual crowdsourcing. Nature Machine Intelligence, 1(12):1174 1184, 2022.
Zhdanov, M., Hoffmann, N., and Cesa, G. Implicit convo- lutional kernels for steerable cnns. Advances in Neural Information Processing Systems, 36, 2024a.
Weiler, M. and Cesa, G. General (2)-eqvrvr steerable cnns. Advances in neural information processing systems, 32, 2019.
Zhdanov, M., Ruhe, D., Weiler, M., Lucic, A., Brandstetter, J., and Forré, P. Clifford-steerabbe convolutional neural networks, 2024b.
Wu, W., Qi, Z., and Fuxin, L. Pointconv: Deep convolu- tional networks 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 9621-96233 2019.
Zhdanov, M., Welling, M., and van de Meent, J.-W. Erwin: A tree-based hierarchical transformer for large-scale phys- ical systems. In International Conference Machine Learning (ICML), 2025.
Wu, Z., Pan, S., Chen, F. Long, G., Zhang, C., and Philip,
Zhu, L.,, Liao, B., Zhang, Q. Wang. X., Liu, W., and Wang, X. Vision mamba: Efficient visual representation learn- ing with bidirectional state space model. arXiv preprint arXiv:2401.09417 2024.
S. Y. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning sys- tems, 32(1):4 24, 2020.
Xin, Y., Laing, C., Leontis, N. B., and Schlick, T. Annota- tion of tertiary interactions in ma structures reveals varia- tions and correlations. Rna, 14(12): 14(12)22465-247) 2008.
Xu, J., Moskalev, A., Mansi, T., Prakash, M., and Liao, R. Beyond sequence: Impact of geometric context for ma property prediction. arXiv preprint arXiv:2410.11933 2024.
Xu, J., Moskalev, A., Mansi, T., Prakash, M., and Liao, R. HARMONY: A multi-representation framework for RNA property prediction. In ICLR 2025 Workshop on Al for Nucleic Acids, 2025. URL https:////// https::/tteereeeeeeeee oppereview. net/ forum htt.ummnnnnnnnnnuuuuu3.a
Yang, J., Liu, Z., Xiao, S., Li, C., Lian, D., Agrawal, S., Singh, A., Sun, G., and Xie, X. Graphformers: Gnn- nested transformers for representation learning on textual
12

---

Geometric Hyena Networks for Large-scale Equivariant Learning
A. Appendix
A.1. Related work
Equivariance Equivariance to group transformations, particularly rotations and translations in 3D, is crucial for modeling physical systems (Zhang et al., 2023). Schütt et al. (2017) condition continuous convolutional filters on relative distances to build model invariant to rotations and translations. Thomas et al. (2018); Fuchs et al. (2020); Brandstetter et al. (2021), I); Liao& & Smidt (2023); Bekkers et al. (2023) utilize spherical harmonics as a steerable basis which enables equivariance between higher-order representations. Since computing spherical harmonics can be expensive, Jing al. (2021b;a); Satorras et al. (2021); Deng et al. (2021) focus on directly updating vector-valued features to maintain equivariance, while Zhdanov et al. (2024a) employ another equívariant network to implicitly parameterize steerable kernels. Another recent line of work (Ruhe et al., 2023a;b; Brehmer et al., 2023; Zhdanov et al., 2024b) employs geometric algebra representation which natively provides a flexible framework for processing symmetries in the data (Dorst et al., 2009).
While these works focus on how to build equivariance into a neural network, in this paper we focus on efficient equivariance to model global geometric contexts at scale.
Modeling geometric context Various strategies are employed to process context information in geometric data. Convolu- tional methods aggregate context linearly within a local neighborhood, guided by either a graph topology (Kipf & Welling, 2016; Vadgama et al., 2025) or spatial relations in geometric graphs (Schütt et al., 2017; Wu et al., 2019; Thomas et al., 2018). Message-passinng framework (Gilmer et al., 2017) generalizes convolutions, facilitating the exchange of nonlinear messages between nodes with learnable message functions. These approaches are favored for their simplicity, balanced computational demands, and expressiveness (Wu et al., 2020). However, they are limited to local interactions and are known to suffer from oversmoothing (Rusch et al., 2023). This hinders building deep message- passing networks capable of encompassing a global geometric context in a receptive field, a constraint particularly critical for modeling large geometric systems (Xu al., 2024; 2025). To address these limitations, recent line of work focuses on architectural modifications in form of virtual nodes (Zhang al., 2024) or using node-to-mesh message passing (Wang et al., 2024b) to enhance the capability of model to handle long-range interactions. Being effective in certain scenarios, these methods are constrained by their use of low-rank approximations for global interactions, limiting their expressivity when the interactions do not adhere to low-rank behavior. Another line of work adopts self-attention mechanisms for graph (Yang et al., 2021; Kreuzer et al., 2021; Kim et al., 2022; Rampášek et al., 2022) and geometric graph (Fuchs et al., 2020; Liao & Smidt, 2023; Brehmer et al., 2023) data, outperforming convolutional and pesssag--passigg approaches. Yet, the quadratic computational cost of self-attention poses significant challenges when modeling large-scale physical systems. Concurrent work by Zhdanov et al. (2025) employs ball-tree approximations for global interactions to design equivariant self-attention, mitigating the computational overhead of large-scale systems. However, this approach may constrain the expressivity of the modeled interactions, particularly when capturing complex global dependencies beyond the approximation's 's scope.
In this work, we aim to develop a method for global geometric context processing with sub-quadratic computational complexity.
State-space and long-convolutional models The quadratic computational complexity of self-attention has driven the exploration of alternatives for modeling long context. Structured state-space models (Gu et al., 2021b) have emerged as a promising alternative, integrating recurrent and convolutional mechanisms within a single framework. These models enable parallelized training in convolutional mode and maintain linear complexity with respect sequence length in recurrent mode. Models like S4 (Gu et al., 2021a), H3 (Fu et al., 2022), and Mamba (Gu & Dao, 2023; Li et al., 2024) have consistently matched or exceeded transformer performance in diverse tasks such as genomics (Schiff et al., 2024), long-range language (Wang et al., 2024a), and vision tasks (Zhu et al., 2024). Concurrently, another line of work integrates cong-convolutional framework with implicit filters (Sitzmann et al., 2020; Romero et al., 2021; Zhdanov et al., 2024a) to capture global sequence context. The implicit filter formulation allows for cata-controlled filtering similar to transformers, while FFF-based long convolution enables global context aggregation sub-quadratic time (Poli et al., 2023). Such models have shown competitive performance comparable to state-space and transformer architectures in time-series modeling (Romero et al., 2021), genomics (Nguyen et al., 2024), and vision tasks (Poli al., 2023).
Although state-space and long-convolutional methods dramatically reduced the computational costs associated with processing long sequences, their application to geometric data with equivariance remains unexplored. In this work, we take inspiration from the recently proposed Hyena operator (Poli et al., 2023) to incorporate SE(3) equivariance. To the best of our knowledge, this is the first equivariant nongcconvolutional model that processes global geometric contexts with sub-qua sub-quadratic memory and time requirements.
13

---

Geometric Hyena Networks for Large-scale Equivariant Learning
R2
R2
R
?
Key,
Value,
Key,
Value,
Key,
Value,
Query
Next Token Prediction
Figure Geometric associative recall task. geometric sequence consists of key and value vector tokens, where consecutive key-value pairs form bigrams. Geometric associative recall requires retrieving the value vector corresponding to query, where the query matches one the keys in sequence.
A.2. Additional experimental results and details
A.2.1. GEOMETRIC ASSOCIATIVE RECALL
Data generation Each associative recall sequence is generated by sampling bigrams (key-value pairs) from the bigram vocabulary. The size of vocabulary vocab_s i ze denotes total number of unique bigrams. The key and value in bigram relate through rotation matrix. Following (Olsson et al., 2022), the vocabulary is generated sequence. The vocabularies are generated as follows: one key-value bigram consists of two vectors with orientations sampled as random unit 3D vectors from isotropic normal distribution, and with magnitudes randomly sampled from a uniform distribution a range of [1, vocab_size], When generating sequences from a vocabulary, the last token serves as a query as illustrated in Figure 4. additionally constraint the generation so the target key-value pair present in a sequence at least once.
fly for each training
Models The vector tokens are used the input to the equivariant branch, while for the invariant branch, we use positional encoding features (Vaswani al., 2017) of dimension 16 as the input. The Geometric Hyena model concludes with a global pooling of equivariant features into one vector prediction. With two Geometric Hyena blocks and hidden dimension of 80, our model ends with 480k trainable parameters. We configure other baseline models to match the depth and and parameter count Geometric Hyena. The local neighborhood of each token in EGNN projection layer is defined as its one immediate preceding and one subsequent tokens.
Training We train all models for 400 epochs with a batch size of 8. We employ Adam optimizer (Kingma & Ba, 2014) with an initial learning rate of 0.001 and cosine learning rate scheduler (Loshchilov Hutter, 2016) with 10 epochs of linear warmup. The weight decay set to 0.00001. Euclidean distance used the loss function.
A.2.2. LARGE RNA MOLECULE PROPERTY PREDICTION
Open Vaccine and Ribonanza-2k datasets The Open Vaccine COVID-19 dataset (Das et al., 2020) provides RNA nucleotide sequences with signal-to-noise ratio (SNR) and per-nucleotide stability profile annotations, and annotated secondary structure. We first filter out sequences with an SNR lower than 1. The resulting dataset contains mRNA sequences up to 130 nucleotides which translates up to 7800 atoms. The Ribonanza dataset (He et al., 2024) contains annotated per-nucceotide degradation profiles of approximately 168k RNAs across 17 sub-datasets. From these, we select the sub-dataset with the label distribution closest a normal distribution, referred as Ribonanza-2k, The number of atoms for individual RNA sequence goes up to 11300. Analogous the Open Vaccine dataset Ribonanza-22 2k comes with annotated secondary structures. The Te-Ribo dataset (Groher et al., 2018) consists of 355 mRNA sequences up to 73 nucleotides and 3850 atoms, and a one regression label per molecule. For all datasets, we employ the state-of-the-at 3D structure prediction tool RhoFold (Shen et al., 2022) to get tertiary structures for RNA molecules. Compared to other existing 3D structure prediction tools which can take hours to run for even single sequences, RhoFold is reported to be more accurate and fast to run and has been used in prior works as the tool of choice (He et al., 2024).
14

---

Geometric Hyena Networks for Large-scale Equivariant Learning
Table EGNN with sequence prior Open Vaceine dataset
Open Vaccine
Model
Backbone repr.
All-atom repr.
Seq-EGNN EGNN-Seq
0.527 10.
0.506 0.490
0.48941.000
Table 5. Ablation study results. Checkmarks indicate the presence of a component. Lower RMSE values indicate better performance. The
best result is highlighted bold.
Component
RMSE (all-atom)
Local
Global context
KV
Geometric convolution
Open Vaccine
Gating
Ribonanza- 2k
context
norm
Covid-19
QK
0.339
0.544
K
V
0.345
0.547
v
0.449
0.551
QK
V
0,450
0.614
QK
0.421
0.545
V
QK
v
NaN
NaN
OK
0.349
0.557
Each RNA molecule represented as a geometric graph where nodes correspond individual atoms, and edges are defined by the point cloud representation, capturing the spatial arrangement of these atoms. The node features have a dimension of 14, encoding the atomic number integer and including one-hot encoding for the elements (H.C.N.O.P). and also nucleotide identities and nucleotide loop type, estimated by RhoFold, as input features.
Models and training When predicting per-nucleotide properties from atom-level representation, we employ nucleotide sum pooling which sums atom features within each nucleotide in the last layer before making a prediction. For global readout, we directly use sum pooling on atom features. For the Open Vaccine and Ribonanza-2k datasets, we follow (Wayment-Steele et al., 2022) and train all models in a multi-task manner, predicting all properties simultaneously using average root mean squared error (RMSE) the loss function. Training runs for 200 epochs with batch size of 16 for Open Vaccine and Te-Ribo and 10 for Ribonanza-2k. We employ the Adam optimizer (Kingma & Ba, 2014) with initial learning rate of 0.001, using cosine learning rate schedule with 10-epoch of linear warm-up. Weight decay is set to 0.0005 for Open Vaccine and Ribonanza-22 2k datasets, and to 0.05 for Tc-ribo.
A.2.3. EXTENDING LOCAL CONTEXT WITH SEQUENCE-STRUCTUREE PRIORS
Given the sequence-lin like structure of RNA molecules in the property prediction task, we further explore incorporating sequence-structured priors to enhance local methods. While this direction presents significant challenges and extends beyond the scope of this paper, we conducted a preliminary experiment inspire future research. Specifically, we used Hyena layer to extract sequence features followed by EGNN (Seq-EGNN) and also tested the reverse setup, where EGNN first extracts geometric features and a sequence model processes them afterward (EGNN-Seg). The experiment is conducted in Open Vaccine dataset.
The results, presented in Table 4, indicate that incorporating sequence priors has minimal impact when EGNN is applied to sequence features. However, applying the sequence model to EGNN-derived features shows slight improvement, highlighting the potential for further exploration this direction.
A.3. Ablation study
We provide the ablation study on individual architectural components of Geometric Hyena in Table We test the impact of various modules for the model used in RNA property prediction Experiment 4.3. We follow the same experimental setup
15

---

Geometric Hyena Networks for Large-scale Equivariant Learning
with the models in the main experiment but train the models common random seed to reduce training and evaluation time.
As can be seen from Table 5, the optimal configuration for Geometric Hyena includes using both local and global context with the gating applied on top of the features resulting from geometric long convolution (QK). Note that using geometric long convolution with scalar-vector interaction improves the performance of the model compared to using scalar and vector long convolutions separately Also, we observe that using key-value normalization (KV-norm) is critical to ensure stable convergence of the model.
A.4. Additional details on architectural components Geometric Hyena
A.4.1. GEOMETRIC LONG CONVOLUTION BY SCALAR AND VECTOR LONG CONVOLUTIONS
Here we provide details on how the geometric long convolution can implemented using scalar and vector long convolutions. Recall that the geometric long convolution combines scalar and vector convolutions to enable scalar-vector interactions. Given input scalar-vector tuples r1) and (a2, r2) I R$ the information flow from inputs the output tuple (og, 13) can be schematically represented as in Figure 5. In particular, the elements of the output tuple computed as:
03 λians A
A3a + Aqaori
+ \o(ri r2)
To implement this convolutional manner, we decompose the geometric long convolution into series of scalar and vector long convolutions. For clarity, omit the token index in the following equations, treating and scalar and vector-valued signals of N tokens, respectively.
Scalar output computation The scalar output 03 involves two terms: (i) scalar- scalar product: X12102 and (ii) vector dot product: Aerf r2. The scalar-scalar term can be directly computed using the scalar long convolution as defined in Equation 2: @ 02 F diag(Fa>) ( Faj. The vector dot product can be decomposed into three scalar convolutions, one for each dimension: (xf Ed_ [ri[d] @ r2(d) where ri[d] and r2[d] [d] represent the d-th component of vectors ri and r2, respectively. With this, the scalar part output of the geometric long convolution can be written as:
α
O
r
r
3 X2
Q3 = X1 2 02) +
Σ (ma [d] @ rd(
(9)
d=1
Vector output computation The vector output r3 involves three terms: (i) scalar- vector product: Az01T2, (ii) scalar-vector product: AqarT1, and (iii) vector cross product: Ar x r2). Both seator-vector product terms can be computed using scalar convolutions for each vector component as aj @ @r2 and ar @ri i [d] for d-th component of the resulting vector. The vector cross product part can be computed using the vector long convolution defined in Equation of the main text. With this, the vector part output of the geometric long convolution can be written
O
r
Figure 5. Scalar-vector interactions in ge- ometric long convolution. Blue lines rep- resent interactions leading to scalar out- put ass and red lines are interactions lead- ing to vector output
r3 = A (ai r2(d)) + ( rrid) + Ao(ri 8x r2)
(10)
where _ denotes concatenation along the d-th component.
16

---

Geometric Hyena Networks for Large-scale Equivariant Learning
A.4.2. PROOF OF EQUIVARIANCE
Projection function Our framework permits any equivariant network as projection function, making equivariance of the projection function a prerequisite. We use modified EGNN (Satorras al., 2021) with global messages m Po (f., hy, (1 + ||x; gyll2)). This maintains E(n-eqquivariance global context tokens g, are equivariant. The proof follows EGNN's, as f, and h, are invariant, and [[x, t) (Rg, + t)ll2 [ g,)ll2 = ||x; -g,ll2-
Global context tokens We next proof the equivariance of global context token mechanism. Recall that given input tokens {x., f J i=1. set of G geometric tokens (g), (Bj,, ) is obtained gi C > 1 Wiy for vector tokens and h3 c - Σ -=1 wiyl; for scalar tokens where Cj ii= Wij with wij being learnable weights. Firstly, hy maintain invariance as function quantities f and wij which by design only depend on the ordinal position of the token in sequence. For the equivariance part:
N Wijl [Rx; + 1) = C
NV
N Σ
7
Σ
WiRRX,
wiy
Rx, + CJ
Wist
1=1
N
N
N
= =R(C
Σ
wixx;)
+
Way)
wiy)t
1=1
1=1
-gg +
Vector long convolution Given vector signal consisting of N vector tokens qaav E R and a vector kernel kevv E Rx3 recall that i-th element of the result if the vector long-convolution is defined as Lj=1 equ x kj-; Since a cross product is already equivariant to rotations, the whole vector convolution is also equivariant to rotations provided the queries rh and the keys k are rotated accordingly (which is guaranteed when the projection function is equivariant). Formally:
Rq:
Rk
Σ 1=1
R(q oTT x xk)T))
R
Σ
q cq
x k
This provides equivariance to the SO(3). For the SE(3) equivariance, can first center the input tokens relative their center of mass, apply the convolution, then uncenter the result.
Geometric long convolution The geometric long convolution combines scalar and vector convolutions for scalar-vector interactions. We prove its equivariance by examining each term under transformations. For the scalar output: (i) a @ a2
invariant as it involves only scalar quantities; (ii) (n [d] @ r2(d]) is rotation-invarian as it reassembles a dot product which is invariant to rotations. For the vector output: (i) Terms - (or @ (d] and L (a2 @ &rridd [d]) transform equivariantly under rotation R because R(a @ r) = @ (Rr) for any scalar a and vector as convolution distributes over vector components; (ii) ri @x F2 is equivariant as previously proven for vector long convolution. This establishes SO(3) equivariance. For SE(3) equivariance, we center input tokens to their center of mass before convolution and uncenter afterwards, making the operation translation-equivrrin
(ni
Selective gating The gating function g IR3 Rd [o, 1] is composed of an equivariant projection layer, a linear layer, and sigmoid activation. Since the projection layer is equivariant (as proven earlier), and both the linear layer and sigmoid operate only on top of invariant features, the gating function itself is invariant SE(3) transformations. For scalar tokens, the gating operation m. preserves invariance as it's a product of invariant terms. For vector tokens, under rotation R. we have mn,(R = R(muuç") since m, is scalar. For translations, centering and uncentering steps ensure translation equivariance, obtaining the -uuuvv-qqivariance.
normalization We prove that the key-value normalization preserves SO(3) equivariance but not SE(3) equivariance. We normalize keys and values to unit norm: |k||2
||v||2 1. For sealar keys and values, this normalization
17

---

Geometric Hyena Networks for Large-scale Equivariant Learning
is invariant rotations it operates on top of invariant quantities. For vector keys and values, under rotation R, The normalized vectors transform k/|lk||2 + Rk/ Rk|la
have | | and ||Rv||2 = ||vl|2
R(k/ [ and similarly for v. Thus, the normalization commutes with rotations, providing the SO(3) equivariance. For translations, centering and uncentering steps ensure translation equivariance, obtaining the SE())-quivarriace.
A.5. Extending Geometric Hyena to higher-order representation
To further encourage the research towards equivariant long-convolutional models for geometric systems, we outline blueprint of the potential extension of Geometric Hyena to accommodate higher-order representations. We consider two possibilities: (i) employing spherical harmonics-basee steerable representation with the tensor product, and (ii) interaction with higher-order features by scalarization trick (Satorras et al., 2021; Cen et al., 2024).
Steerable long convolution with higher-order representations Following the steps used in this work to derive the cross product-based vector long convolution (Section 2.2), one can employ steerable representations and tensor products to derive a long convolution for higher-order steerable vectors. Instead of vectors R$ and the cross product, we use steerable vectors and the tensor product. The long convolution can then be constructed using the tensor product and reduced to a series of scalar convolutions by factoring out the Clebsch-Gordan (CG) coefficients.
Let (1)) and (12) be sequences of steerable features of degrees l and l2 respectively, each with length N. These features transform according to the spherical harmonics representations of SO(3). We can define steerable long convolution as:
(q" @cg k/) k
Σ
(1m) (11) (lam))
qim, k (12) j-1,m
(11)
mi= m2=-B
where (.lm) (Gim)))amm denotes Clebsch-Gorde coefficients from type 1 l2 to
To efficiently compute this convolution, similarly to geometric long convolution, we can decompose into scalar convolutions factoring out the Gordanh-Gordan coefficients, reducing the computational complexity from O(N log N):
((lm) (1am))
(11) (qm)
@
k
(12) m2
(12)
m2
This is a direct extension of our method, since the cross product used in our vector long convolution corresponds to the case where | = 1 = 12 = 1, and the Clebsch-Gorde coefficients reduce to the Levi-Civita symbol.
The advantage of the fully steerable approach is that allows interactions between features of arbitrary degrees, capturing complex geometric relationships and maintaining strict equivariance under rotations. However, this approach might be computationally expensive due to the large number of long convolutions and Clebsch- coefficients involved, especially for higher degree representation. Potentially, this computational overhead hinders the usage of fully steerable methods for large-scale geometric systems, where computational efficiency is crucial. In this paper, we purposefully chose to focus on simpler case with cross product due to its computational efficiency. By restricting to type-1 I representations, we reduce the computational complexity and make the method more practical for large-scale applications, while still capturing essential geometric properties through equivariant operations.
Faster higher-order representations by scalarization Alternatively, to efficiently model interactions between high and low order features without the heavy computational cost of fully steerable methods, we can employ the scalarization trick. This approach extends the geometric long convolution framework by introducing higher-order components into the feature representations while simplifying computations by selectively considering interactions.
In this method, each feature tuple is expanded to include higher-order part. Specifically, using notation from Section 2.2, we extend feature representation from (a, r) to (a, v) containing scale, vector, and higher-order representation respectively. When performing the convolution between two such feature tuples, we drop the higher-order to higher-order interaction terms. With this, we can focus on interactions that involve higher-order to low-order order and low-order to low-order interactions only, effectively scalarizing the processing of higher-order features.
18

---

Geometric Hyena Networks for Large-scale Equivariant Learning
By excluding higher-order to higher-order interactions, we significantly reduce computational complexity. The higher-order features still contribute to the model through their interactions with scalar parts, allowing the network capture important geometric information without the full computational overhead associated with processing all higher-order interactions.
A.6. Details on the G-Transformer baseline
Since Geometric Hyena utilizes long convolutions based on cross products, design simple equivariant vector self- attention mechanism align the global geometric context aggregation for Hyena and transformer models for more direct comparison. To align the information flow, we use similar architecture to Geometric Hyena (i.e. input/output projections, QK V-roojeeciio as in Figure 2) but instead of long convolutions, we employ cross product-based equivariant self-attention to aggregate global context. Thus, the only difference between Geometric Hyena and the SE(3)- Transformer baseline is the global context aggregation mechanism.
Equivariant dot product vector self-attention Consider sequences of N vector query, key, and value tokens denoted as q. k, v E RNx3 We first construct the dot product matrix C RNXN with Cij q ky. The entries of the dot product matrix are invariant to rotations of queries and keys since q,'k, k, q. R'RRR,, Then, softmax is applied row-wise to compute the self-attention matrix S = softmax( V/v C). The self-attention matrix is used to aggregate vector values as
ش D N V. The rotation equivariance preserved as Siy invariant scalar so Σラ= St, R E Siy Ru,, Equivariance to translations can be achieved by initially centering the data (subtracting the center of mass) and then re-centering the resulting tokens.
u; =
Equivariant cross product vector self-attention We also experiment with cross product based equivariant self-attention. We start by constructing a query-key cross product tensor C E I NxN33 where each element Ciy = qi X kj. or using Levi-Civita notation as in Eq. 4, Ciyl1] = q: [A]kk((p) [p]. To integrate a softmax selection mechanism, we first compute a matrix 7 E N containing the L2 norms all cross products, specifically (C)) = ||q: ky 121 Applying softmax
to T7( C) then determines the vector pairs to select from the cross product tensor. Lastly, the values v are cross-multiplied
with the -ilttex-iltered cross product tensor. Overall, the equivariant vector self-attention reads as:
S = softmax(
(C)) oC
(13)
N
Su XV,
(14)
N
where the softmax applied row-wise, and @ stands for element-wise product. Consequently, S E NxN3 represents a tensor that encapsulates a soft selection of cross products between qi and ky. Since the tensor C is constructed using cross products, it naturally maintains equivariance to rotations queries and keys. Furthermore, the softmax is applied to the L2 norms of the cross products making of rotation-invarian scalar and rotation-equivariint vector quantities, rendering it rotation-equiivaiant. The Eq. 14 further preserves rotation-equivariann due to the inherent equivariance of the cross product. Equivariance to translations can be achieved by initially centering the data and then re-centering the resulting tokens.
Consequently, the self-attention tensor S is a product
Early experiments revealed that the equivariant dot product vector self-attention either performs on par or better than the cross product version while being computationally faster and less memory intensive. We hence proceeded with the equivariant dot product self-attention as a global context module for Graansfformer.
1. PYTORCH IMPLEMENTATION
Equivariant vector long convolution is an essential building block in Geometric Hyena. It can be used as is to aggregate global geometric context across vector tokens or can be combined with standard scalar long convolution into geometric long convolution. Here, we provide simple Pytorch implementation for the rotation-equivariant (without centering) vector long convolution in Code 1.
19

---

Geometric Hyena Networks for Large-scale Equivariant Learning
class VectorL.ongConv (an. Module):
def ..init.- -I self): super super(VectorLong) (VectorLongConv, self)...initt...
# cross-prod fnctorized:
expand_vec.mat = torch. FloatTensor (Il1. 0. 01.
[1.
[0, 1, 0].
[0, I, 0].
[0. 0. 1l.
[0. 0. vie) (1........11...6.33
self register.buuffer ("expand _vee_mat", expand_vec.mat, persistent=False)
14
cross -prod tensor factorized:
eross_expand.xec.mal torch, Floar ([[0
1, 0].
0.-1].
[-1. 0. 01.
0, 11.
0].
21
[0 .-1. 011) (1..
register. buffer ("fctes crcosss. eew.andeeaddd.d.... mat" crossexxaand.ve. persixtent=False)
23
24
# cross-prod tensor factorized:
torch. FloatTensor (110, .0..0.0..111.
26
[0,100.0.1.0],
[1.0.1.0...0] wwww (1..1,11.133.
28
self. reg.steer..buferrree. cross.suum..... mat" cruss.sum.mal. persixiemt=False)
def cony (oe_ff
k):
31
= x. shapel-1 -2]
ffi.x torch. fft.rfff. aEN, dim=-2)
ffi_k = torch. fft. rfft rfft (k, n-N, dim=-2)
conv_ak
xffi-xffftkk. norm="backward", dim=-2)
35
return conv.xk
37
def (eel. k):
38
# batch, channel, sequence length R.C,N,- = x.
40
42
# expand aux matrices
expand_vee vee_mat mat self.expane wxw.addvveedecccc.oo seedd_ee_mattt expand 6.3,6 ccoss_oxpand.vee.m... cross.sum,mat self.cross cross.sum.mal (B..
self cross.cxpand.ves.mm (B....))
45
47
# expand inputs and matmul expanded_x torch (xtmand
unsqueeze(-I)) squcezo(-1)
49
cross expanded torch, matmul ( expand vec mat, k. (nnspueeztt-l)))sqqeeee
# fft conv
52
ffl.conv.xk welf.eefe conv, ffft (expanded.a. cxoss_eppanded.k) reduced_fft_conv_ torch.
ffi_cony_xk unsqueeze(-1)). (quueeee)(
54
reduced
Code J. Pytorch implementation of the equivariant vector long convolution.
20


Reviews:
[{"review_id": "mbtLifIoXw", "reviewer": "Reviewer_YW9h", "summary": "This paper presents a novel equivariant SO(3) neural network for processing geometric graphs with sequence structure and invariant node features.  The network a geometric version of Hyena and implements equivariant long convolution in fourier space, allowing for global information flow with subquadratic complexity.  The model is evaluated over a synthetic recall task and several real world RNA property prediction tasks and protein molecular dynamics prediction.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive feedback on contributions of our paper, highlighting the novelty (\"the extension is non-trivial and the method is clearly novel\") of our method and its experimental validation (\"the authors consider a strong set of experiments to test their methods\").\n\n## Non-equivariant baselines:\nWe agree that the extended comparison with non-equivariant data augmentation baselines will further strengthen the experiments. We train and test Hyena and Transformer models with data augmentation on Open Vaccine, Ribonanza-2k, Tc-ribo and Protein-MD datasets.\n\n|Results (*backbone*) |Open Vaccine|Ribonanza-2k|Tc-ribo|Protein-MD|\n|-----|------------------------|------------------------|-------------------|----------------|\n|Hyena|0.447±0.037|0.810±0.124|0.560±0.002|48.94±9.03|\n|Transformer|0.400±0.004|0.637±0.006|0.556±0.001|75.83±6.35|\n|G-Hyena|**0.363±0.045**|**0.529±0.005**|**0.517±0.025**|**1.80±0.009**|\n\n|Results (*all-atom*)|Open Vaccine|Ribonanza-2k|Tc-ribo|Protein-MD|\n|-----|------------------------|------------------------|-------------------|----------------|\n|Hyena|0.393±0.002|0.605±0.017|0.569±0.001|55.34±5.51|\n|Transformer|0.399±0.004|0.633±0.007|**0.553±0.002**|79.68±24.2|\n|G-Hyena|**0.339±0.004**|**0.546±0.006**|**0.552±0.003**|**2.49±0.037**|\n\nResults suggest that on Open Vaccine and Ribonanza-2k data, non-equivariant models lag significantly behind Geometric Hyena (with G-Hyena delivering up to 34% error reduction), and on Tc-Ribo all-atom Transformer performs on par with. Yet, non-equivariant models struggle to generalize on the Protein-MD task where non-equivariant Hyena and Transformer perform poorly compared to Geometric Hyena.\n\n## Local methods with sequence-structured priors:\nExtending local methods with sequence-structured prior presents a non-trivial direction for future work that extends beyond the scope of our paper. However, to foster future research, we conducted an extra experiment where we employed a Hyena layer to extract sequence features and then ran EGNN on top (Seq-EGNN). Alternatively, we can first run EGNN to extract geometric features and then run a sequence model on top (EGNN-Seq).\n\n|Results|Open Vaccine (*backbone*)|Open Vaccine (*all-atom*)|\n|-----|-------------------------------|--------------------------------|\n|EGNN|0.529±0.006|0.511±0.005|\n|Seq-EGNN|0.527±0.006|0.506±0.009|\n|EGNN-Seq|0.489±0.002|0.490±0.004|\n|G-Hyena|**0.363±0.045**|**0.339±0.004**|\n\nOur initial results suggest that adding sequence prior has little effect when EGNN is applied on top of sequence features. However, when the sequence model is applied on top of EGNN features, we observe slight improvement, suggesting potential for further research in this direction.\n\n## Future benchmarking on materials:\nIn our work, we focus specifically on large bio-molecules. We agree that extending experimental comparison to a broader domain, including materials, presents an important direction for future work, and we plan to do so in the future. We would appreciate if the reviewer could point us to relevant material benchmarks that include large molecules for our future work.\n\n## Higher-order Geometric Hyena:\nRecent works [1,2] observed that higher-order representations provide diminishing performance improvement while significantly increasing computational requirements and memory footprint. This is also supported by the excellent performance of scalarization-based equivariant GNNs [3]. Based on this evidence, we decided not to proceed with a higher-order version of Geometric Hyena since our focus is on computational and memory efficiency.\n\n## Other comments and questions:\n- Indeed, our model operates on a sequence of $N$ scalar vector feature tuples $(x_1, f_1), (x_2, f_2), \\dots, (x_N, f_N)$ rather than unordered set, we thank the reviewer for pointing this out. With this, a more precise formulation of the domain and co-domain would be $\\Psi: (\\mathbb{R}^{3} \\times \\mathbb{R}^{d})^{N} \\rightarrow (\\mathbb{R}^{3} \\times \\mathbb{R}^{d})^{N}$.\n- Projection terminology is used for alignment with the terminology in the Transformer and Hyena papers.\n- In our implementation, we use standard discrete FFT with circular convolution. There is no explicit frequency truncation and all available frequencies are used up to the Nyquist limit.\n- 190-192L: Yes, similar to the scalarization trick [3], this results in a lossy compression of information.\n- In Protein-MD each protein has 855 and 3341 atoms in backbone and all-atom version. Atom identities are used as feature for all methods.\n- All clarifications, discussion, and new results will be added to the revised version of the paper.\n \n[1] Brandstetter et al. Geometric and Physical Quantities Improve E(3) Equivariant Message Passing. ICLR 2022. \\\n[2] Wand et al. Rethinking the Benefits of Steerable Features in 3D Equivariant Graph Neural Networks. ICLR 2024. \\\n[3] Satorras et al. E(n) Equivariant Graph Neural Networks. ICML 2021.", "claims_and_evidence": "The primary claim of the paper is that geometric hyena can outperform both local equivariant methods and global equivariant attention methods in terms of accuracy and compute time and memory efficiency.    This is well supported both in the design of the method and in the strong empirical results.", "methods_and_evaluation_criteria": "- The method builds on Hyena by making it equivariant.  The extension is non-trivial and the method is clearly novel.  For the long convolution layer, elementwise products of scalar are replaced with cross products of vectors.  The authors show this operation can also be performed in the fourier domain in subquadratic time.  vector-scalar interactions are also supported.  \n- One limitation of this work is that the input can have only scalar node features (and vector positions).  However, the authors have a section in the appendix which extends their framework to higher order SO(3) features and a suggestion on how to implement it efficiently. \n- The use of E(N)-GNN for the projection is reasonable since it is fairly efficient and also operators on geometric graphs with scalar node features.  The authors improve E(N)-GNN by adding global context tokens. \n- A geometric selective gating mechanism is proposed to emphasize certain tokens, similar to softmax. \n- The authors include an ablation study to highlight the importance of the key contributions.  In particular, they compare to the G-transformer which is similar to their method but uses self-attention instead of long convolution.\n- A potential limitation of the method, which the author comment directly on, is that long convolution is not permutation equivariant.  Thus for point cloud tasks without a prior sequence structure, a sequence would need to be imposed.  This would likely require significant data augmentation. The authors also note, this could be considered an advantage in tasks like the one they select in which there is a useful sequence structure in the data. \n- The normalization of the key and values is considered and implemented to prevent numerical instability.", "theoretical_claims": "The authors include the appropriate theoretical support for their method, proving the equivariance of geometric Hyena in the appendix.", "experimental_designs_or_analyses": "- The authors consider a strong set of experiments to test their methods.  The real world RNA and protein experiments demonstrate the method can scale and is practical for interesting real world applications.\n- The results are strikingly impressive, with geometric Hyena showing substantial accuracy improvements with much better efficiency and scaling. \n- The experiments are carefully chosen to showcase the strength of the method, the inputs have only scalar node features and there is a preferred sequence to use for the points.\n- The baselines are well-chosen, representing a selection of well-known, popular, and often SoTA equivariant methods including global attentional methods and local convolutional methods.   These include Equiformer, VectorNeurons Transformer, SchNet, Tensor Field Network, E-GNN, and more. \n- I believe non-equivariant methods are included in the recall experiment, but not in the RNA experiments.  It would be nice to see how they compare, for example, with ample data augmentation.  Non-equivariant methods can often perform well too, especially when controlling for memory/compute budget which a focus of the current work.\n- Many of the local equivariant methods are targeted more for smaller scale material science tasks, so while they are good to include as baselines, it may not be surprising they do worse on these larger scale tasks with sequence structure.  Is there some way to encode the sequence structure for these models?  Also, although it is not the goal, I wonder how well would Geometric Hyena perform on the types of materials benchmarks those baselines target.", "relation_to_broader_scientific_literature": "It is a bit unusual to not include a background section or related works section on prior works.  That said, I think the narrative and explanation is clear and that past work and current contributions are clear as written.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "### Strengths\n- Compute and memory efficiency is a large problem under focused on in the equivalence literature.  The focus of this paper, the design of the method, and strong results are very encouraging.  Figure 1 is quite striking.", "other_comments_or_suggestions": "- 073R Shouldn’t the input and output spaces for $\\Psi$ be the N-fold product of these spaces? \n- 111-113R: I found this a touch confusing.  Are they sequences of sets of scalars?  Maybe if the index i is across a sequence, you could use parenthesis instead of curly braces to indicate the order matters. \n- Why is the equivariant projection layer called a “projection”?  Isn’t it just a mapping?", "questions_for_authors": "- Is the Fourier transform truncated at some maximum frequency? If so, does it matter where? \n- 190L-192L: Does this scalar mapping result in a lot of lost information?\n- 220L-223L: Are there any trade-offs with omitting some message pathways this way? \n- How big is the sequence length for the protein task?  What is the input feature for the model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qiySNFH93P", "reviewer": "Reviewer_VW9r", "summary": "The authors introduce a novel equivariant model designed for modeling long geometric sequences. To ensure compatibility with higher-order geometric tensors in global convolution, they propose an equivariant global convolution specifically for vectors. Additionally, they demonstrate that these computations can be performed efficiently using the Fast Fourier Transform (FFT), building on the concept of scalar-valued global convolution. The Geometric Hyena model achieves superior results across various RNA sequencing datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Application focus of Geometric Hyena:\nRegarding the generalizability of our method to arbitrary geometric graphs, we refer the reviewer to the Limitations section (lines 424–439R), where we explicitly discuss the limitations of G-Hyena for point clouds and emphasize that our method is best suited for problems where a canonical ordering can be established. Also, note that Reviewer YW9h acknowledges in their review that we have noted this limitation explicitly. With this, we disagree that we overclaimed the contribution of our method to arbitrary geometric graphs. That said, we will revise the text to make our target application domain more explicit. Specifically, we will update the phrasing to: \"The Geometric Hyena is designed for tasks that require modeling invariant and equivariant features in geometric graphs where canonical order can be established\". This will be clarified in the revised version of the paper.\n\nSecond, we would like to point out that the class of geometric graphs with canonical ordering is quite broad and includes large bio-molecules such as proteins, antibodies, peptides, enzymes, and RNA - systems that are highly relevant to biosciences and drug discovery. In this context, we also remind the reviewer that our submission is under the Application-Driven Machine Learning track, which explicitly includes biosciences as a focus area in ICML call for papers (icml.cc/Conferences/2025/CallForPapers). In addition, large bio-molecules represent a well-established and rapidly growing application area by its own in the machine learning community, with numerous top-tier papers focused on this domain [1,2,3,4,5]. Viewed in this light, the reviewer’s remark that our work may “have a significant contribution to RNA-Seq analysis” also reflects its broader relevance to machine learning for biosciences. Finally, we note that our method applies beyond RNA, as demonstrated in the protein molecular dynamics task in Section 4.5.\n## Equiformer and VNT:\nOur method is designed for large bio-molecules, such as RNA or proteins, which can consist of hundreds or thousands of atoms. In contrast, Equiformer and VNT are developed for small geometric systems. For example, the Ribonanza-2k dataset we use has bio-molecules with up to 11300 atoms, whereas the MD17 dataset used to evaluate Equiformer has molecules with only up to 25 atoms. The high dimensionality of our data introduces unique computational challenges that our method is specifically designed to address that do not arise in low-dimensional settings and that existing equivariant models are not equipped to handle at scale. As a result, it is also infeasible to directly reuse the hyperparameters from Equiformer paper (while VNT paper does not report exact optimal hyperparameters). Still following reviewer's suggestion, we tried running Equiformer with the smallest default configuration from the original paper and it runs out of memory on our data even with a batch size of 1 highlighting further the importance of our model for large-scale bio-molecular graphs. Finally, our initial reason for running the models to match the parameter count of G-Hyena was to ensure that we could fairly compare the benefits of these models by isolating the impact of model scale on performance which is a standard practice [6,7].\n## Associative recall:\nWe appreciate the reviewer’s attention to the clarity of presentation of the geometric associative recall task. To aid understanding, we illustrated how “each bigram (key and value) is connected by a rotation matrix” in Figure 4 of the supplementary material. To further improve clarity, we propose updating the caption of Figure 4 to: “A geometric sequence consists of key and value vector tokens, where consecutive key-value pairs form bigrams. Geometric associative recall requires retrieving the value vector corresponding to a query, where the query matches one of the keys in the sequence.”\n## Vector long convolution:\nVector long convolution is a novel component introduced in our work that we demonstrate how to implement efficiently in sub-quadratic time. It is not a standard definition.\n\nWe thank the reviewer for their constructive feedback! We believe we have addressed all questions and comments. In light of this, we kindly ask the reviewer to consider raising the score.\n\n[1] Groth et al. Kermut: Composite kernel regression for protein variant effects. ICLR 2024.\\\n[2] Jing et al. AlphaFold Meets Flow Matching for Generating Protein Ensembles. ICML 2024.\\\n[3] Nori et al. RNAFlow: RNA Structure & Sequence Design via Inverse Folding-Based Flow Matching. ICML 2024.\\\n[4] Gong et al. Evolution-Inspired Loss Functions for Protein Representation Learning. ICML 2024.\\\n[5] Tan et al., Deciphering RNA Secondary Structure Prediction: A Probabilistic K-Rook Matching Perspective. ICML 2024.\\\n[6] He et al. Deep Residual Learning for Image Recognition. CVPR 2015.\\\n[7] Tan et al. Rethinking Model Scaling for Convolutional Neural Networks. ICML 2019.", "claims_and_evidence": "**The Geometric Hyena is designed for tasks that require modeling invariant and equivariant features in geometric graphs.** -- Geometric graphs can appear in different formats. They can be a point cloud with features associated with each point, a mesh, or a big molecule. One issue of using the Hyena or Mamba style model in these cases is converting the geometric graphs into a sequence.\nFor example, to mamba or state space model for computer vision, researchers explored various ways to convert the $2D$ image into a $1D$ sequence.\nHowever, this case is completely ignored in this case. \nThe authors might have mostly dealt with RNA data - which, unlike generic geometric graphs, is a definite sequence.\n\nHowever, unlike RNA seq, most of the geometric graphs do not come with an ordering.\n\nThis means the proposed model is not for all \"geometric graphs\", it is for a small subset of \"geometric graphs\" where we have some ordering on the nodes. Thus the work do not comply with the main claim.", "methods_and_evaluation_criteria": "As claimed to be a generic model for Geometric data, authors should consider all the datasets from EQUIFORMER and VN-Transformer.\n\nAlso, as mentioned in line 307 -- \"For the other models, we choose hyperparameters so their depth and total parameter count match those of Geometric Hyena, ensuring a fair comparison\" --, It is not clear why it is not possible to use the hyper-parameters proposed in the original works.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments done are valid but not enough to demonstrate the significance.", "relation_to_broader_scientific_literature": "The work may have a significant contribution to RNA-Seq analysis. However, as discussed earlier, in general cases, geometric graphs do not have order, thus severely limiting it's application.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. The idea of \" Geometric associative recall\" should be explained further. For example, the statement \"In geometric associative recall, each bigram (key and value) is connected by a rotation matrix\" is not well understood if not explained in more detail.", "other_comments_or_suggestions": "1. Type line 241  ”Harry Potter” --> ``Harry Potter''", "questions_for_authors": "1. Regarding \"Vector long convolution,\" is this proposed by authors, or is it a standard definition?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uvt7XzYd4v", "reviewer": "Reviewer_XkYv", "summary": "This paper introduces an SE(3)-equivariant extension of the Hyena model (Poli et al. 2023) which employs long convolutions evaluated in the Fourier domain. This Geometric Hyena model is used for property predictions of large biological molecules. The authors show that their model outperforms transformer models in terms of runtime and memory as well as performance in several numerical experiments.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Computational and performance benefits:\n\nWe appreciate the reviewer acknowledging the runtime and memory benefits of our method, as computational efficiency is the main purpose of our work. \n\nSecond, we believe there may be a misunderstanding regarding performance improvements. G-Hyena significantly outperforms the G-Transformer model—with non-overlapping error bars on Tc-Ribo in both the backbone and all-atom settings (-7\\% error reduction in both cases). It also shows substantial gains on the Protein-MD task in both backbone and all-atom scenarios (-26\\% and -36\\% error reduction, respectively). Since the primary architectural difference between G-Hyena and G-Transformer is the geometric long convolution, we attribute these performance improvements to that component. In addition, the associative recall experiment (Figure 3, bottom left) shows that G-Hyena scales more favorably with respect to the hidden dimension compared to G-Transformer. This improved scaling behavior is practically relevant, as it potentially enables more efficient model size reduction and further lowers memory usage. Taken together, we consider both the computational and performance advantages of our model to be significant in the context of the conducted experiments.\n\n## Non-equivariant state space model:\nWe agree that the extended comparison with the non-equivariant Hyena model will further strengthen the experiments. In addition to Hyena in the associative recall experiments, we test a standard Hyena model trained with data augmentation on Open Vaccine, Ribonanza-2k, Tc-ribo, and Protein-MD datasets. \n\n|Results (*backbone*) |Open Vaccine|Ribonanza-2k|Tc-ribo|Protein-MD|\n|-----|------------------------|------------------------|-------------------|----------------|\n|Hyena|0.447±0.037|0.810±0.124|0.560±0.002|48.94±9.03|\n|G-Hyena|**0.363±0.045**|**0.529±0.005**|**0.517±0.025**|**1.80±0.009**|\n\n\n|Results (*all-atom*)|Open Vaccine|Ribonanza-2k|Tc-ribo|Protein-MD|\n|-----|------------------------|------------------------|-------------------|----------------|\n|Hyena|0.393±0.002|0.605±0.017|0.569±0.001|55.34±5.51|\n|G-Hyena|**0.339±0.004**|**0.546±0.006**|**0.552±0.003**|**2.49±0.037**|\n\nResults suggest that standard non-equivariant Hyena trained with extensive data augmentation still significantly lags behind properly equivariant Geometric Hyena (with G-Hyena performance gain ranging from -3\\% error reduction on Tc-Ribo all-atom to -53\\% error reduction Ribonanza-2k backbone). Furthermore, the non-equivariant Hyena model struggles to generalize on the Protein-MD dataset where it underperforms significantly compared to equivariant methods. The results will be included in the revised version of the paper.\n\n## Presentation of the paper and clarifications:\n- We will refine the text of the paper to correct grammatical errors and typos as soon as openreview allows revision. We will further streamline the writing in \"invariant-equivariant subspace interaction\" section and remove redundancies in the beginnings of sections 4.1 and 4.2. \n- In Eq. (1), $\\hat{\\mathbf{x}}$ and $\\hat{\\mathbf{f}}$ refer to the output of Geometric Hyena. \n- Indeed, the more precise domain and co-domain formulation would be $\\Psi: (\\mathbb{R}^{3} \\times \\mathbb{R}^{d})^{N} \\rightarrow (\\mathbb{R}^{3} \\times \\mathbb{R}^{d})^{N}$ as our model operates on a sequence of $N$ scalar vector feature tuples $(x_1, f_1), (x_2, f_2), \\dots, (x_N, f_N)$ with each feature tuple being $\\mathbb{R}^{3} \\times \\mathbb{R}^{d}$.\n- The notation $F^H$ in Eq. (2) refers to the Hermitian transpose of the FFT matrix. \n- The gating function indeed acts per token.\n- We appreciate the reviewer's focus on clarity and precision of the presentation of the paper! We will fix typos, add clarification and adjust notation accordingly in the revised version.\n\n## Related work section:\nDue to space constraints, a comprehensive review of related work could not be included in the main text. Instead, we focused on discussing the most essential prior work in the introduction, while providing a more detailed review in Supplementary Section A. This structure is not uncommon with multiple top-tier conference ML papers (e.g., [1,2,3]) adopting a similar format. If the reviewer prefers an explicit related work section in the main text, we are happy to include a shortened version there while retaining the extended related work in the supplementary material.\n\nWe thank the reviewer for their constructive feedback! We believe we have addressed all questions and comments. In light of this, we kindly ask the reviewer to consider raising the score.\n\n[1] Gu et al. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. COLM 2024. \\\n[2] Yi et al. Bridge the Modality and Capability Gaps in Vision-Language Model Selection. NeurIPS 2024. \\\n[3] Razin et a. Implicit Regularization in Deep Learning May Not Be Explainable by Norms. NeurIPS 2020.", "claims_and_evidence": "The claimed runtime- and memory benefits over transformer models are convincing. The claimed performance benefits are largely convincing. However, as the authors point out themselves, the performance benefit in the RNA property prediction task and on the geometric associative recall task is likely due to the alternating local and global context and not the long convolutions (the error bars for G-Transformer and G-Hyena overlap).", "methods_and_evaluation_criteria": "The chosen problems largely seem to be appropriate to evaluate the model. However, I am missing a baseline of a non-equivariant state space model (e.g. the non-equivariant Hyena model), since the claimed benefit of the proposed model is the combination of the efficiency state-space models with the performance of equivariant models. In light of recent doubts about the importance of equivariance (e.g. [arXiv.2311.17932], [arXiv: 2410.24169], AlphFold3) it would be interesting to see how the Geometric Hyena compares to efficient non-equivariant models.", "theoretical_claims": "The equivariance of the proposed layers is obvious.", "experimental_designs_or_analyses": "The experimental designs seem appropriate.", "relation_to_broader_scientific_literature": "Equivariant architectures are a standard tool in quantum mechanical property prediction. Since state-space models are a recent development in other domains, it is interesting to consider an equivariant version of a popular state-space model.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\nStrong experimental results and thorough ablations.\n\nWeaknesses:\nThe presentation of the paper is seriously lacking. There are numerous typos and grammatical errors in the manuscript (e.g. \"Heyna\" instead of \"Hyena\" in the tables). The related works section is an important part of the paper and has to be part of the main text, it does not belong in the supplementary material. For removing such an important section from the main text, the manuscript seems wordy at times, e.g. the \"invariant-equivariant subspace interaction\" section can be shortened considerably and the beginnings of sections 4.1 and 4.2 -> \"implementation details\" are almost the same. Also the equations given lack precision. For instance, the domain and co-domain of $\\Psi$ given above (1) do not match the sets in (1) and the hatted quantities are not introduced. The notation $F^H$ in (2) is not introduced. Similarly, the gating function $\\gamma:\\mathbb{R}^3\\times\\mathbb{R}^d\\rightarrow[0,1]$ seems to act per token as it is introduced, is that correct?", "other_comments_or_suggestions": "For some typos, see above.", "questions_for_authors": "No further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces "Geometric Hyena Networks", which is a method for applying long-convolutions while preserving Euclidan equivariance. This method leads to improved results in terms of accuracy and speed in comparison with standard equivariant methods. The reviewers were mostly supportive of the paper, and hence I  recommend accepting the paper.
Decision: Accept (spotlight poster)