# Hyena Model (deep learning)

The Hyena model is a neural network architecture that was developed to address the scalability issues associated with traditional self‐attention mechanisms. It is designed to efficiently handle very long sequences by replacing the quadratic-complexity self‐attention with a subquadratic operator that interleaves implicit long convolutions with data-controlled gating.

## Motivation and context

Traditional Transformer models rely on self-attention to allow each token in a sequence to interact with every other token. Although this mechanism is highly effective for capturing dependencies, its computational cost scales quadratically ($O(L^{2})$) with the sequence length L. This quadratic scaling creates significant challenges when processing long sequences, such as entire documents, long time series, or high-resolution images.
The need for more efficient models that can process long-range dependencies has led researchers to explore alternatives that reduce computational and memory requirements. The Hyena model was introduced as a drop-in replacement for self-attention, aiming to maintain the global receptive field and expressive power of attention while scaling subquadratically with sequence length.

## Architecture

At the core of the Hyena model is the concept of implicit long convolutions. Traditional convolutions use fixed kernels that are explicitly defined and stored, resulting in a parameter count that scales linearly with the kernel size. In contrast, Hyena generates convolutional filters implicitly using a parameterized function—typically implemented as a small feed-forward network. This allows the model to synthesize long filters on the fly, effectively decoupling the filter length from the number of parameters.
In addition to implicit convolutions, the Hyena operator incorporates data-controlled multiplicative gating. In this mechanism, each token is modulated by gating signals that are derived from learned linear projections of the input. The gating operation is performed element-wise and serves to dynamically adjust the influence of the convolutional output, effectively tailoring the operator to the specific input context.
The overall Hyena operator is defined as a recurrence that alternates between implicit long convolutions and element-wise gating. For an order-N Hyena operator, the recurrence is expressed as follows:
* $z_{1}[t]=v[t]$, where $v$ is one of the linear projections of the input.
* For $n=1,\dots ,N$:
** $z_{n+1}[t]=x_{n}[t]\cdot {\Bigl(h_{n}\ast z_{n})[t]{\Bigr )}$, where $x_{n}$ represents a gating projection and $h_{n}$ is an implicitly parameterized long convolution filter.
* The final output is given by $y[t]=z_{N+1}[t]$.
, where
* $z_{n}[t]$ is the intermediate state at recurrence step $n$ and time position $t$.
* $v[t]$ is a linear projection of the input at time position $t$, analogous to the "value" in self-attention.
* $x_{n}[t]$ is the gating projection at recurrence step $n$.
* $h_{n}$ is the implicit long convolution filter for step $n$.
* The operator $*$ denotes convolution, so $(h_{n}*z_{n})[t]$ is the result of convolving filter $h_{n}$ with the signal $z_{n}$ at time $t$.
* The dot "$\cdot$" indicates element-wise multiplication.
Mathematical Formulation
The implicit convolution filters in Hyena are typically parameterized as functions of time. For each filter $h_{n}$, the response at time is given by:
$h_{n}[t]={\text{Window}}(t)\cdot ({\text{FFN}}\circ {\text{PositionalEncoding}})(t)$
, where $\circ$ is the composition operator, meaning that the positional encoding is first applied to $t$ and then processed by the FFN.
Here, the window function serves to modulate the filter (for example, by imposing an exponential decay), and the feed-forward network (FFN) together with positional encodings generate the filter values. This implicit parameterization is a key design choice that allows Hyena to capture long-range dependencies without a proportional increase in parameter count.

## Efficiency and scalability

By replacing the quadratic self-attention mechanism with a sequence of FFT-based convolutions and element-wise multiplications, the Hyena operator achieves an overall time complexity of $O(NL\log L)$, where $N$ is the number of recurrence steps. This subquadratic scaling is particularly advantageous for long sequences, allowing the model to process inputs that are orders of magnitude longer than those feasible with conventional attention.
The operations in the Hyena model—both the implicit convolutions and the gating functions—are highly parallelizable and amenable to optimization on modern hardware accelerators. Techniques such as fast Fourier transforms (FFT) further enhance the efficiency, making the model well-suited for large-scale applications where both speed and memory efficiency are critical.

## Comparison with transformer models

While Transformer models use self-attention to achieve a global receptive field, this comes at the cost of quadratic complexity with respect to the sequence length. In contrast, the Hyena model achieves a similar global context through its recurrence of long convolutions and gating, but with much lower computational cost. This makes Hyena a promising alternative in settings where long-range dependencies need to be modeled efficiently.
| Aspect | Hyena Model | Transformer |
| Computational Complexity | $O(NLlogL)$ (subquadratic) | $O(L^{2})$ (quadratic) |
| Memory Footprint | Lower; uses FFT-based convolutions and implicit filters | Higher; requires storing the full self-attention matrix |
| Global Context | Yes; achieved via interleaved implicit convolutions and gating | Yes; achieved through dense pairwise interactions in self-attention |
| Scalability to Long Sequences | Highly efficient; can process sequences of millions of tokens (e.g., genomic data) | Limited by quadratic scaling; effective only up to a few thousand tokens |
| Parameter Scaling | Decoupled from sequence length due to implicit parameterization of filters | Fixed parameter count independent of sequence length, but scaling becomes costly |
| Speed on Long Sequences | Significantly faster (e.g., 160× faster at 1M tokens in certain cases) | Slower due to quadratic cost in computation and memory |
| Hardware Utilization | High; operations like FFT and element-wise gating are highly parallelizable | Optimized for dense matrix operations, but efficiency drops with very long sequences |