Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences

## Abstract

Abstract Language models for biological and chemical sequences
enable crucial applications such as drug discovery, protein engineering, and precision medicine.
Currently, these language models are predominantly based on Transformer architectures.
While Transformers have yielded impressive results, their quadratic runtime dependency on the sequence length complicates their use for long genomic sequences and in-context learning on proteins and chemical sequences.
Recently, the recurrent xLSTM architecture has been shown to perform favorably compared to Transformers and modern state-space model ( SSM ) architectures
in the natural language domain.
Similar to SSMs , xLSTMs have a linear runtime dependency on the sequence length and allow for constant-memory decoding at inference time,
which makes them prime candidates for modeling long-range dependencies in biological and chemical sequences.
In this work, we tailor xLSTM towards these domains and propose a suite of architectural variants called Bio-xLSTM.
Extensive experiments in three large domains, genomics, proteins, and chemistry, were performed to assess xLSTM’s ability to model biological and chemical sequences.
The results show that models based on Bio-xLSTM
a) can serve as proficient generative models for
DNA, protein, and chemical sequences,
b) learn rich representations for those modalities, and
c) can perform in-context learning for proteins and small molecules.

## 1 Introduction

Figure: Figure 1: Overview of Bio-xLSTM. Top left: xLSTM for natural language processing tasks. Top right: Considered modeling approaches for biological sequences: masked language modeling, equivariance to reverse complementary sequence, and in-context learning. Bottom left: DNA-xLSTM models are trained on genomic DNA sequences and then fine-tuned on downstream tasks. Bottom center: Prot-xLSTM models are trained in a causal modeling setting with a fill-in-the-middle objective and use homologous proteins for in-context learning. Bottom right: Chem-xLSTM models are trained to generate small molecules. For an in-context learning setting, Chem-xLSTM models use molecules with known properties.
Refer to caption: x1.png

Accurate computational models for biological sequences are essential for translating data into actionable insights in modern biology. Biological sequences like DNA, RNA, and proteins
are central to molecular biology, genomics, and drug discovery.
Major projects like the Human Genome Project (Lander et al.,, )
and the 1000 Genomes Project (1000 Genomes Project Consortium,, ) have driven large-scale data collection efforts.
Modeling these sequences is key to advancing life sciences (Benegas et al.,, ; Karollus et al.,, ),
interacting with biological systems (Hopf et al.,, ; Riesselman et al.,, ; Yang et al.,, )
or predicting phenotypes from genetic variants
(Ashley,, ; Brandes et al.,, ; Acosta et al.,, ).
Similar efforts exist for protein sequences (The UniProt Consortium,, ) and
small molecules (Kim et al.,, ; Zdrazil et al.,, ),
used for tasks like protein engineering (Arnold,, ; Yang et al.,, ),
predicting 3D structures (Jumper et al.,, ), and drug discovery (Zhavoronkov et al.,, ).
\Acpllm (Brown et al.,, ; Bubeck et al.,, ) have emerged as prime candidates
for modeling biological sequences and serving as foundation models for
molecular biology and chemistry (Ji et al.,, ; Schiff et al.,, ; Nguyen et al.,, ; Rives et al.,, ; Lin et al.,, ).

Large language models for biological sequences must handle long sequences and incorporate context.
The rise of large language models (Radford et al.,, ; Brown et al.,, ; Bubeck et al.,, )
has revolutionized numerous fields, including life sciences.
Most LLMs are based on the Transformer architecture (Vaswani et al.,, ),
which excels at predicting the next or missing token using self-attention.
However, the self-attention mechanism scales quadratically with sequence length,
making long-sequence modeling computationally expensive.
As a result, most biological sequence models use short contexts (Rives et al.,, ; Ji et al.,, ; Dalla-Torre et al.,, ).
However, biological sequences require long context windows for accurate modeling because
of their important long-range interactions due to 3D folding (Anfinsen,, ),
or gene regulation in DNA (Bouwman and de Laat,, ).
The human genome spans around three billion base-pairs,
far exceeding the context limits of Transformer-based models.
Furthermore, long contexts also benefit models that exploit homologous proteins (Truong Jr and Bepler,, ; Sgarbossa et al.,, ) and molecular context for small molecules (Papadatos et al.,, ; Schimunek et al.,, ).
The emergence of SSMs, like S4 (Gu et al.,, ), Hyena (Poli et al.,, ), and Mamba (Gu and Dao,, ), enables handling longer contexts in biological domains (Nguyen et al.,, ; Schiff et al.,, ; Sgarbossa et al.,, ).
However, the recently proposed xLSTM architecture (Beck et al.,, ),
a recurrent neural network,
has outperformed SSM architectures in natural language processing (Beck et al.,, ).
For further related work, see Appendix Section A.

The recently proposed xLSTM is a powerful architecture
for sequence modeling and a promising candidate for biological and chemical sequences.
The xLSTM architecture (Beck et al.,, ) introduces enhanced
memory structures and exponential gates that boost its
performance, particularly in natural language modeling.
Despite these enhancements over traditional LSTM, xLSTM retains the efficiency of a recurrent neural network
and can handle varying sequence lengths effectively (Beck et al.,, ),
while maintaining expressivity (Merrill et al.,, )
and scalability (Katharopoulos et al.,, ; Choromanski et al.,, ).
These features make xLSTM ideal for modeling:
i) DNA sequences, which are inherently long and for which long-range interactions between distant parts of the sequence have been observed,
ii) protein sequences, where modeling
strongly benefits from contextual information
of evolutionary-related proteins (Rives et al.,, ), and
iii) small molecules represented as chemical sequences, such as Simplified Molecular Input Line Entry System (SMILES) (Weininger,, ),
for which in-context learning (ICL) abilities are an option to generate new molecules with desired properties or from a particular molecular domain (Segler et al.,, ; Schimunek et al.,, ).
However, it remains unclear
how to best tailor xLSTM for biological and chemical sequences
and how xLSTM compares to other domain-specific LLMs architectures.

We introduce:
a) DNA-xLSTM, an architectural variant tailored for DNA sequences with reverse-complement equivariant blocks, and evaluate its performance on long-context generative modeling, representation learning, and downstream tasks.
b) Prot-xLSTM, a variant for homology-aware protein language modeling with in-context learning for both generative modeling and in-painting, which we benchmark on generative modeling, conditioned protein design and protein variant fitness prediction tasks.
c) Chem-xLSTM, an architectural variant for SMILES representations
of small molecules for which we demonstrate ICL capabilities.
An overview of Bio-xLSTM is shown in Fig. 1.

## 2 Background and Notation

xLSTM (Beck et al.,, ) consists of two types of layers:
sLSTM (see Section 2.1)
and mLSTM (see Section 2.2)
which are the main components within
block structures (see Section 2.3)
of its multi-layer architectures.
We consider a series of
input vectors $\bm{x}_{t}\in\mathbb{R}^{D}$
given at a certain time step $t\in\{1,\ldots,T\}$.
$\bm{X}=\bm{X}_{1:T}=(\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{T})\in\mathbb{R}^{D\times T}$
denotes the matrix of stacked input vectors from all time steps.
Both sLSTM and mLSTM are recurrent neural networks, which either map a
state $(\bm{h}_{t-1},\bm{c}_{t-1},\bm{n}_{t-1})$ to
a successor state $(\bm{h}_{t},\bm{c}_{t},\bm{n}_{t})$
given an input $\bm{x}_{t-1}$ (sLSTM) or
a state $(\bm{h}_{t-1},\bm{C}_{t-1},\bm{n}_{t-1})$ to a
successor state $(\bm{h}_{t},\bm{C}_{t},\bm{n}_{t})$ given
an input $\bm{x}_{t-1}$ (mLSTM).
Here, $\bm{h}_{t}\in\mathbb{R}^{d}$ denotes a hidden state,
$\bm{c}_{t}\in\mathbb{R}^{d}$ and $\bm{C}_{t}\in\mathbb{R}^{d\times d}$
denote cell states
responsible for long-term memory
and, $\bm{n}_{t}\in\mathbb{R}^{d}$ denotes a normalizer state.
sLSTM and mLSTM utilize several adjustable
weight matrices and bias vectors (detailed equations below)
and employ input-, output-, and forget-gates,
activated by exponential ($\exp$) or the sigmoid functions ($\sigma$).
For cell inputs in sLSTM, the hyperbolic tangent function
($\tanh$, abbreviated as $\varphi$) is used as an activation function.

### 2.1 sLSTM

The forward pass of sLSTM in the vectorized version is defined as follows:

$$ $\displaystyle\bm{c}_{t}\$ $\displaystyle=\ \mathbf{f}_{t}\odot\bm{c}_{t-1}\ +\ \mathbf{i}_{t}\odot\bm{z}_{t}$ cell state (1) $\displaystyle\bm{n}_{t}\$ $\displaystyle=\ \mathbf{f}_{t}\odot\bm{n}_{t-1}\ +\ \mathbf{i}_{t}$ normalizer state (2) $\displaystyle\bm{h}_{t}\$ $\displaystyle=\ \mathbf{o}_{t}\odot\tilde{\bm{h}}_{t}\ ,$ $\displaystyle\tilde{\bm{h}}_{t}\$ $\displaystyle=\ \bm{c}_{t}\odot\bm{n}_{t}^{-1}$ hidden state (3) $\displaystyle\bm{z}_{t}\$ $\displaystyle=\ \varphi\left(\tilde{\bm{z}}_{t}\right)\ ,$ $\displaystyle\tilde{\bm{z}}_{t}\$ $\displaystyle=\ \bm{W}_{\bm{z}}\ \bm{x}_{t}\ +\ \bm{R}_{\bm{z}}\ \bm{h}_{t-1}\ +\ \bm{b}_{\bm{z}}\ \$ cell input (4) $\displaystyle\mathbf{i}_{t}\$ $\displaystyle=\ \exp\left(\tilde{\mathbf{i}}_{t}\right)\ ,$ $\displaystyle\tilde{\mathbf{i}}_{t}\$ $\displaystyle=\ \bm{W}_{\mathbf{i}}\ \bm{x}_{t}\ +\ \bm{R}_{\mathbf{i}}\ \bm{h}_{t-1}\ +\ \bm{b}_{\mathbf{i}}\ \$ input gate (5) $\displaystyle\mathbf{f}_{t}\$ $\displaystyle=\ \exp\left(\tilde{\mathbf{f}}_{t}\right)\ \text{OR}\ \sigma\left(\tilde{\mathbf{f}}_{t}\right)\ ,$ $\displaystyle\tilde{\mathbf{f}}_{t}\$ $\displaystyle=\ \bm{W}_{\mathbf{f}}\ \bm{x}_{t}\ +\ \bm{R}_{\mathbf{f}}\ \bm{h}_{t-1}\ +\ \bm{b}_{\mathbf{f}}\ \$ forget gate (6) $\displaystyle\mathbf{o}_{t}\$ $\displaystyle=\ \sigma\left(\tilde{\mathbf{o}}_{t}\right)\ ,$ $\displaystyle\tilde{\mathbf{o}}_{t}\$ $\displaystyle=\ \bm{W}_{\mathbf{o}}\ \bm{x}_{t}\ +\ \bm{R}_{\mathbf{o}}\ \bm{h}_{t-1}\ +\ \bm{b}_{\mathbf{o}}\ \$ $\displaystyle\text{output gate},$ (7) $$

where $\mathbf{i}_{t},\mathbf{o}_{t},\mathbf{f}_{t}\in\mathbb{R}^{d}$ are the input,
output and forget gate, respectively,
$\bm{W}_{\bm{z}},\bm{W}_{\mathbf{i}},\bm{W}_{\mathbf{f}},\bm{W}_{\mathbf{o}}\in\mathbb{R}^{d\times D}$,
$\bm{R}_{\bm{z}},\bm{R}_{\mathbf{i}},\bm{R}_{\mathbf{f}},\bm{R}_{\mathbf{o}}\in\mathbb{R}^{d\times d}$,
and $\bm{b}_{\bm{z}},\bm{b}_{\mathbf{i}},\bm{b}_{\mathbf{f}},\bm{b}_{\mathbf{o}}\in\mathbb{R}^{d}$
are trainable weight matrices and biases.

### 2.2 mLSTM

The forward pass of the mLSTM is defined as follows:

$$ $\displaystyle\bm{C}_{t}\$ $\displaystyle=\ \mathrm{f}_{t}\ \bm{C}_{t-1}\ +\ \mathrm{i}_{t}\ \bm{v}_{t}\ \bm{k}_{t}^{\top}$ cell state (8) $\displaystyle\bm{n}_{t}\$ $\displaystyle=\ \mathrm{f}_{t}\ \bm{n}_{t-1}\ +\ \mathrm{i}_{t}\ \bm{k}_{t}$ normalizer state (9) $\displaystyle\bm{h}_{t}\$ $\displaystyle=\ \mathbf{o}_{t}\ \odot\ \tilde{\bm{h}}_{t}\ ,\qquad\qquad\tilde{\bm{h}}_{t}\ =\ \bm{C}_{t}\bm{q}_{t}\ /\ \max\left\{{{\left|\bm{n}_{t}^{\top}\bm{q}_{t}\right|}},1\right\}$ hidden state (10) $\displaystyle\bm{q}_{t}\$ $\displaystyle=\ \bm{W}_{q}\ \bm{x}_{t}\ +\ \bm{b}_{q}$ query input (11) $\displaystyle\bm{k}_{t}\$ $\displaystyle=\ \frac{1}{\sqrt{d}}\bm{W}_{k}\ \bm{x}_{t}\ +\ \bm{b}_{k}$ key input (12) $\displaystyle\bm{v}_{t}\$ $\displaystyle=\ \bm{W}_{v}\ \bm{x}_{t}\ +\ \bm{b}_{v}$ value input (13) $\displaystyle\mathrm{i}_{t}\$ $\displaystyle=\ \exp\left(\tilde{\mathrm{i}}_{t}\right)\ ,\qquad\qquad\ \ \ \ \,\tilde{\mathrm{i}}_{t}\ =\ \bm{w}^{\top}_{\mathrm{i}}\ \bm{x}_{t}\ +\ b_{\mathrm{i}}\ \ \$ input gate (14) $\displaystyle\mathrm{f}_{t}\$ $\displaystyle=\ \sigma\!\left(\tilde{\mathrm{f}}_{t}\right)\ \text{OR}\ \exp\!\!\left(\tilde{\mathrm{f}}_{t}\right),\ \ \>\>\tilde{\mathrm{f}}_{t}\ =\ \bm{w}^{\top}_{\mathrm{f}}\ \bm{x}_{t}\ +\ b_{\mathrm{f}}$ forget gate (15) $\displaystyle\mathbf{o}_{t}\$ $\displaystyle=\ \sigma\left(\tilde{\mathbf{o}}_{t}\right)\ ,\qquad\qquad\quad\ \ \ \,\tilde{\mathbf{o}}_{t}\ =\ \bm{W}_{\mathbf{o}}\ \bm{x}_{t}\ +\ \bm{b}_{\mathbf{o}}$ output gate (16) $$

where $\mathrm{i}_{t},\mathrm{o}_{t},\mathrm{f}_{t}\in\mathbb{R}$ are the input,
output and forget gate, respectively,
$\bm{q}_{t},\bm{k}_{t},\bm{v}_{t}\in\mathbb{R}^{d}$ are query, key and
value inputs with trainable weight matrices $\bm{W}_{q},\bm{W}_{k},\bm{W}_{v}\in\mathbb{R}^{d\times D}$,
$\bm{w}_{\mathrm{i}},\bm{w}_{\mathrm{f}}\in\mathbb{R}^{D}$
are input and forget gate weights and
the respective $b_{\mathrm{i}},b_{\mathrm{f}}\in\mathbb{R}$ biases.
All other quantities are identical to sLSTM.

### 2.3 Block Structures

The sLSTM and mLSTM layers are integrated into
larger residual backbones (Srivastava et al.,, ; He et al.,, ),
which incorporate layer normalization (Ba et al.,, ),
pre- or post-up projection layers (Vaswani et al.,, ; Dao,, ),
with short causal convolutions
and group normalization (Wu and He,, ).
Figure A1 depicts sLSTM and mLSTM blocks,
as well as, a bidirectional mLSTM configuration with weight-tied blocks.
For more details, refer to (Beck et al.,, , Sec.2.4).
For Bio-xLSTM we retain these basic building blocks but adjust them to the respective domains.
The entire architecture, including all layers, normalization, blocks,
and other components, defines a mapping from an input sequence of length $t$ to an output sequence. This mapping is denoted as $\mathrm{xLSTM}:\mathbb{R}^{D\times t}\mapsto\mathbb{R}^{D\times t}$,
where $\mathrm{xLSTM}$ transforms the stacked inputs up to time step $t$, i.e., $\bm{X}_{1:t}\coloneqq(\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{t})\in\mathbb{R}^{D\times t}$, to the corresponding stacked outputs of sequence length $t$, i.e., $\bm{Y}_{1:t}\coloneqq(\bm{y}_{1},\bm{y}_{2},\ldots,\bm{y}_{t})\in\mathbb{R}^{D\times t}$ (^1^11Here $\bm{x}_{i}$ and $\bm{y}_{i}$ represent the inputs to and outputs from a particular model from an instance of an xLSTM architecture, rather than the inputs and outputs of a specific sLSTM or mLSTM layer).
The $i$-th sequence element is denoted with the subscript $i$, e.g. the $i$-th element from $\bm{X}_{1:t}$ would be $(\bm{X}_{1:t})_{i}$.
Similarly to the mapping $\mathrm{xLSTM}$, we also define mappings for the sequence-wise input-/output behaviour of layers themselves for an sLSTM layer ($\mathrm{sLSTM}:\mathbb{R}^{D\times t}\mapsto\mathbb{R}^{D\times t}$) or an mLSTM layer ($\mathrm{mLSTM}:\mathbb{R}^{D\times t}\mapsto\mathbb{R}^{D\times t}$).
If the specific parameters used for the mapping are unclear, we will denote this by including a second argument in the function, separated by a semicolon.
For details on the block structures, see Appendix Section B.1.

### 2.4 Modes of Operation: Parallel, Chunkwise, and Recurrent

The recurrent forms of sLSTM and mLSTM, introduced in Sections 2.1 and 2.2, provide efficient, constant-memory decoding during inference. This eliminates the need for expensive key-value caching, which represents a major challenge for Transformer models in long-range settings. Like Transformers, mLSTM allows for parallelization across the sequence length which significantly speeds up training. Additionally, similar to linear attention variants (Katharopoulos et al.,, ; Yang et al.,, ), mLSTM supports chunkwise parallel processing, blending recurrent and parallel modes
(Beck et al.,, ). This approach is especially advantageous for long-sequence training and prompt encoding. For further details, refer to Appendix B.2.

## 3 Bio-xLSTM: Longe-Range Modeling of Biological and Chemical Sequences

Bio-xLSTM introduces three xLSTM-based architectural variants tailored
specifically to DNA (Section 3.4),
proteins (Section 3.5) and
small molecules (Section 3.6).
For these application domains, we extend xLSTM from causal language modeling (CLM)
(Section 3.1) to new modeling approaches such as fill-in the middle (FIM) , in-context learning (ICL)
and masked language modeling (MLM) (Section 3.2).

### 3.1 Causal Language Modeling and Next-Token Prediction

clm uses the

$$ CLM loss: $\displaystyle\mathcal{L}^{\mathrm{CLM}}=\mathbf{\mathbb{E}}_{\bm{X}\sim p_{\bm{X}}}\ \mathbf{\mathbb{E}}_{t\sim[[1,T-1]]}\ \mathrm{CE}\left(\bm{x}_{t+1},\mathrm{xLSTM}(\bm{X}_{1:t})_{t}\right),$ (17) $$

where $\mathrm{CE}$ is the cross-entropy loss (with logits),
$p_{\bm{X}}$ is the data distribution, and $[[1,T-1]]$ is the discrete
uniform distribution from $1$ to $T-1$. The objective
measures how well a particular sequence token $\bm{x}_{t+1}$
can be predicted based on the previous tokens $\bm{X}_{1:t}$
by the model
$\mathrm{xLSTM}:\mathbb{R}^{D\times t}\mapsto\mathbb{R}^{D\times t}$.
Therefore, this type of modeling is sometimes
also called * next token prediction (NTP)*, *uni-directional modeling* or
* autoregressive (AR) modeling* and the loss is also called *NTP loss*.

fim
(Bavarian et al.,, )
is a modeling paradigm
that integrates aspects of both CLM and MLM.
In this approach, parts of the sequence are replaced with mask tokens,
which are then appended to the end of the sequence.
This allows the model to utilize the entire context to predict the
masked tokens while maintaining an AR training framework.
This strategy, allows the model to perform both a) generative modeling and b) inpainting with CLM.

icl
is a capability of language models to learn and perform tasks by leveraging additional information provided as the contextual input without updating their parameters (Brown et al.,, ; Min et al.,, ).
This approach allows models to learn from analogy, drawing insights from patterns in the context to adapt their behavior.
When the contextual input sequence is denoted as $\bm{Z}\in\mathbb{R}^{D\times S}$ and the conventional input sequence is denoted as $\bm{X}\in\mathbb{R}^{D\times T}$, then with an ICL model, we obtain $\bm{Y}^{\prime}=\mathrm{xLSTM}([\bm{Z},\bm{X}])_{(S+1):(S+T)}$, where $[\bm{Z},\bm{X}]$ indicates concatenation of contextual and conventional inputs, and, the subscript ${(S+1):(S+T)}$ indicates that the last output tokens (those corresponding to the $\bm{X}$ tokens) are selected.
For natural language processing tasks, the context $\bm{Z}$ often contains the solution to a similar problem or exemplary solutions that guide the input and inform the model.
For biological and chemical sequences, the context $\bm{Z}$ could represent similar genetic regions, homologous proteins, or molecules with desired properties, enabling the model to make informed predictions or generate outputs based on these analogies.

### 3.2 Masked Language Modeling (MLM)

Bio-xLSTM extends xLSTM to masked modeling of biological sequences,
for which the typical de-masking
or de-noising objective (Vincent et al.,, ; Devlin et al.,, )
is used, concretely the

$$ MLM loss: $\displaystyle\mathcal{L}^{\mathrm{MLM}}=\mathbf{\mathbb{E}}_{\bm{X}\sim p_{\bm{X}}}\ \mathbf{\mathbb{E}}_{t\sim[[1,T]]}\ \mathbf{\mathbb{E}}_{\bm{M}\sim p_{\bm{M}}}\ \mathrm{CE}\left(\bm{x}_{t},\mathrm{xLSTM}(\bm{X}\odot\bm{M})_{t}\right),$ (18) $$

where $\bm{M}\in\{0,1\}^{D\times T}$ is a random matrix with binary entries which are
usually drawn from a Bernoulli distribution $p_{\bm{M}}$, and
$\odot$ is element-wise multiplication. The objective
measures how well the original sequence $\bm{X}$
can be reconstructed from a noisy version $\bm{X}\odot\bm{M}$
by the model
$\mathrm{xLSTM}:\mathbb{R}^{D\times T}\mapsto\mathbb{R}^{D\times T}$.
This modeling paradigm has also been called *bidirectional modeling*.
It has been highly successful in learning
representations of proteins at evolutionary scale (Rives et al.,, ),
which has powered many subsequent applications such as protein
engineering and machine-learning guided directed evolution
(Yang et al.,, ). For details on how xLSTM is extended to the MLM setting, we refer to Appendix Section B.3.

### 3.3 \Acf rc equivariance

We develop an xLSTM block that is equivariant
to the reverse complement (RC) of an input sequence, a property particularly relevant to DNA-based applications.
In double-helix DNA structures, both strands are semantically equivalent,
with one strand being the RC of the other.
The RC strand is oriented in the opposite direction of the *forward* strand,
with base pairs converted from A to T and C to G.
Shrikumar et al., () show that a data-driven approach to
learn the equivalence between RC sequences can fail.
Therefore, Schiff et al., ()
propose to enforce RC-equivariance by design,
making use of two different inductive
biases, post-hoc conjoining (PH) (Zhou et al.,, ) and parameter sharing (PS), in the architecture.
In PH architectures, the backbone is trained to handle
both DNA sequences and their RCs by applying RC augmentations during pre-training.
For downstream tasks, PH architectures are applied
to both the original sequence and its RC, and
their outputs are summed to reach overall RC invariance.
In contrast, PS architectures
integrate RC-equivariant xLSTM blocks with equivariant
word embeddings and language model heads similar to Schiff et al., ().
For additional details, see Appendix Section C.4.

### 3.4 DNA-xLSTM

For the DNA domain, we propose the DNA-xLSTM architecture
to enhance sequence modeling capabilities, particularly for varying context lengths.
We introduce three model configurations based on DNA-xLSTM:
two sLSTM-based configurations trained with a context window of 1,024 tokens (DNA-xLSTM-500k and DNA-xLSTM-2M),
and an mLSTM-based configuration trained with a context window of 32,768 tokens (DNA-xLSTM-4M).
The short-context configuration, DNA-xLSTM-500k, has an embedding dimension of 128, 5 sLSTM blocks,
an up-projection ratio of 1.25:1 to match the baseline model parameter count,
and a total parameter count of 500k, while DNA-xLSTM-2M has an embedding dimension of 256,
6 sLSTM blocks, a 1:1 up-projection ratio, and 2M parameters.
The long-context configuration, DNA-xLSTM-4M, has an embedding
dimension of 256, 9 mLSTM blocks, a 2:1 up-projection ratio, and
is augmented with Rotary Position Encodings (RoPE) () to
handle long-range dependencies effectively, with a total of 4M parameters.
All three configurations are trained with both CLM and MLM.
Furthermore, we introduce RC-equivariant versions, xLSTM-PH and xLSTM-PS, which
use the original sequence and its reverse complement (RC).
We trained models with these configurations on the human genome and benchmarked them
against state-of-the-art DNA models, such as
Transformers, DNA-Mamba (Caduceus) (Schiff et al.,, ), and HyenaDNA (Nguyen et al.,, ),
showing competitive or better performance on pre-training and downstream classification tasks
(see Section 4.1).

### 3.5 Prot-xLSTM

For the protein domain, we propose the architectural variant
Prot-xLSTM to address the complexities of protein sequence data,
particularly in capturing long-range dependencies to enable homology-conditioned modeling.
We introduce two configurations: Prot-xLSTM-26M and Prot-xLSTM-102M, with 26M and 102M parameters, respectively. Both
configurations consist of 16 mLSTM blocks, with embedding dimensions of 512 for Prot-xLSTM-26M and 1,024 for Prot-
xLSTM-102M and maintaining a consistent 2:1 projection ratio across both configurations,
and are trained with increasing context sizes ranging from 2,048 to 262,144 tokens.
To effectively manage the wide range of protein sequence lengths and context sizes,
RoPEs are implemented for Prot-xLSTM.
The according Prot-xLSTM models are trained with CLM using a FIM
strategy on non-aligned homologous sequences,
enabling them to perform ICL at inference time in two modes: a) generative and b) inpainting.
Both approaches can be used for protein design,
with the latter also suited for residue-based predictions, such as mutant fitness estimation.
Prot-xLSTM shows better performance than similarly configured Mamba- and Transformer-based models
and shows promising results for homology-conditioned sequence generation (see Section 4.2).

### 3.6 Chem-xLSTM

For the chemical sequence domain, we introduce Chem-xLSTM to enhance
generative modeling of SMILES strings (Weininger,, ),
enabling domain-conditioned generation of small molecules without fine-tuning.
We introduce two models: an unconditional generative model
trained with a context length of 100 tokens (Chem-xLSTM-15M)
and a domain-conditioned model trained with a 4,096-token context
for in-context learning tasks (Chem-xLSTM-15M-icl).
The latter can generate molecules within a specific domain without fine-tuning only based
on examples provided as context, a highly sought-after capability in drug discovery.
Both models are configured to have 15M parameters, consist of 9 mLSTM blocks with an
embedding dimension of 512 and a 1.3:1 projection ratio.
The models have been benchmarked against other generative models
for SMILES and at their ICL capabilities (see Section 4.3).

## 4 Experiments and Results

### 4.1 DNA Sequences

For the DNA-xLSTM experiments, we followed the experimental protocol outlined in Schiff et al., () and Nguyen et al., () for both pre-training and downstream adaptation.

Figure: Figure 2: Pre-training of 2M-parameter DNA models on the human reference genome (GRCh38). Models are trained at single-nucleotide resolution with a context length of 1024 bases. Left: causal language modeling. Learning curves display NTP loss ($\downarrow$) on a test set, plotted against the number of tokens processed. Right: masked language modeling. Learning curves showing MLM loss ($\downarrow$) on a test set across the number of tokens seen for various models. In both tasks, the xLSTM-based models consistently achieve the lowest loss values across all update steps.
Refer to caption: x2.png

Pre-training. The training data for both the CLM and MLM tasks was sourced from the human reference genome (Church et al.,, ), with context lengths set to 1,024 and 32k tokens. Our baseline models included HyenaDNA (Nguyen et al.,, ) and Caduceus (Schiff et al.,, ), which is based on the Mamba architecture . Additionally, we trained Transformer++ baselines, building on the Llama architecture (Touvron et al.,, ). Similar to Caduceus, we experimented with both PH- and PS-equivariant xLSTM configurations, benchmarking them against the corresponding Mamba baselines. All models that did not use PS-equivariance were trained with RC augmentation. Hyperparameters were selected using a separate validation set.
Figure 2 presents the test losses for 2M parameter CLM and MLM models trained with RC augmentation, i.e. non-PS models, and a context size of 1,024 tokens. In the CLM setting, DNA-xLSTM-2M achieved the best performance, surpassing Transformer++, Mamba, and HyenaDNA. The performance gap became even more pronounced on the MLM task, where DNA-xLSTM-2M outperformed both Transformer-based models and Mamba.
Additionally, we extended DNA-xLSTM-2M to the PS equivariant setting and trained smaller RC-equivariant DNA-xLSTM-500k models. The resulting PH and PS models were subsequently used for downstream fine-tuning. In Appendix Section C, we present additional pre-training results including comparisons for large-context and PS equivariant models. We found that xLSTM-DNA matches or outperforms strong baselines in all pre-training settings.

Downstream fine-tuning.
To evaluate the learned representations, we fine-tuned the pre-trained DNA-xLSTM-2M and DNA-xLSTM-500K (both PH and PS) on two genomic classification benchmarks: the Genomic benchmark (Grešová et al.,, ) and
the Nucleotide Transformers Tasks (Dalla-Torre et al.,, ),
which span 18 datasets from five studies.
DNA-xLSTM-2M-PH and DNA-xLSTM-2M-PS models pre-trained with context size 1,024 were compared against HyenaDNA and Mamba-PS and Mamba-PH.
DNA-xLSTM performed best (see Table 1),
outperforming baseline models in the under 2M parameter range on 12 out of 18 tasks, and was comparable to
the much larger Nucleotide Transformer (500M parameters), winning 8 tasks.
The comparisons with larger Transformer models and xLSTM-DNA-500k performance on the Genomics benchmark are presented in Appendix Section C.

**Table 1: Downstream adaption of DNA models. The performance of 2M parameter models fine-tuned on Nucleotide Transformer classification tasks on the test set is shown. PS or PH indicate models trained to be RC equivariant. Performance is averaged over 10 random seeds and error bars indicate the difference between maximum and minimum values across the 10 runs. The best values are highlighted in green. DNA-xLSTM outperforms both Mamba and Hyena on 12 out of 18 tasks. Scores for Mamba- and Hyena-based models were obtained from Schiff et al., ().**
| Task | Metric | HyenaDNA | Mamba-PS^a | Mamba-PH^a | xLSTM-PS | xLSTM-PH |
| --- | --- | --- | --- | --- | --- | --- |
| Histone Markers |  |  |  |  |  |  |
| H3 | MCC $\uparrow$ | $0.779^{\pm 0.037}$ | $0.799^{\pm 0.029}$ | $0.815^{\pm 0.048}$ | $0.796^{\pm 0.014}$ | $0.824^{\pm 0.010}$ |
| H3K14AC | MCC $\uparrow$ | $0.612^{\pm 0.065}$ | $0.541^{\pm 0.212}$ | $0.631^{\pm 0.026}$ | $0.570^{\pm 0.008}$ | $0.598^{\pm 0.017}$ |
| H3K36ME3 | MCC $\uparrow$ | $0.613^{\pm 0.041}$ | $0.609^{\pm 0.109}$ | $0.601^{\pm 0.129}$ | $0.588^{\pm 0.019}$ | $0.625^{\pm 0.010}$ |
| H3K4ME1 | MCC $\uparrow$ | $0.512^{\pm 0.024}$ | $0.488^{\pm 0.102}$ | $0.523^{\pm 0.039}$ | $0.490^{\pm 0.012}$ | $0.526^{\pm 0.001}$ |
| H3K4ME2 | MCC $\uparrow$ | $0.455^{\pm 0.095}$ | $0.388^{\pm 0.101}$ | $0.487^{\pm 0.170}$ | $0.489^{\pm 0.024}$ | $0.504^{\pm 0.012}$ |
| H3K4ME3 | MCC $\uparrow$ | $0.549^{\pm 0.056}$ | $0.440^{\pm 0.202}$ | $0.544^{\pm 0.045}$ | $0.520^{\pm 0.019}$ | $0.537^{\pm 0.012}$ |
| H3K79ME3 | MCC $\uparrow$ | $0.672^{\pm 0.048}$ | $0.676^{\pm 0.026}$ | $0.697^{\pm 0.077}$ | $0.662^{\pm 0.011}$ | $0.697^{\pm 0.007}$ |
| H3K9AC | MCC $\uparrow$ | $0.581^{\pm 0.061}$ | $0.604^{\pm 0.048}$ | $0.622^{\pm 0.030}$ | $0.622^{\pm 0.013}$ | $0.627^{\pm 0.008}$ |
| H4 | MCC $\uparrow$ | $0.763^{\pm 0.044}$ | $0.789^{\pm 0.020}$ | $0.811^{\pm 0.022}$ | $0.793^{\pm 0.011}$ | $0.813^{\pm 0.008}$ |
| H4AC | MCC $\uparrow$ | $0.564^{\pm 0.038}$ | $0.525^{\pm 0.240}$ | $0.621^{\pm 0.054}$ | $0.558^{\pm 0.018}$ | $0.583^{\pm 0.014}$ |
| Regulatory Annotation |  |  |  |  |  |  |
| Enhancer | MCC $\uparrow$ | $0.517^{\pm 0.117}$ | $0.491^{\pm 0.066}$ | $0.546^{\pm 0.073}$ | $0.375^{\pm 0.030}$ | $0.545^{\pm 0.024}$ |
| Enhancer Types | MCC $\uparrow$ | $0.386^{\pm 0.185}$ | $0.416^{\pm 0.095}$ | $0.439^{\pm 0.054}$ | $0.444^{\pm 0.046}$ | $0.466^{\pm 0.011}$ |
| Promoter: All | F1 $\uparrow$ | $0.960^{\pm 0.005}$ | $0.967^{\pm 0.004}$ | $0.970^{\pm 0.004}$ | $0.962^{\pm 0.002}$ | $0.967^{\pm 0.001}$ |
| NonTATA | F1 $\uparrow$ | $0.959^{\pm 0.011}$ | $0.968^{\pm 0.006}$ | $0.968^{\pm 0.010}$ | $0.963^{\pm 0.002}$ | $0.970^{\pm 0.001}$ |
| TATA | F1 $\uparrow$ | $0.944^{\pm 0.040}$ | $0.957^{\pm 0.015}$ | $0.953^{\pm 0.016}$ | $0.948^{\pm 0.006}$ | $0.952^{\pm 0.005}$ |
| Splice Site Annotation |  |  |  |  |  |  |
| All | Accuracy $\uparrow$ | $0.956^{\pm 0.011}$ | $0.927^{\pm 0.021}$ | $0.940^{\pm 0.027}$ | $0.965^{\pm 0.006}$ | $0.974^{\pm 0.004}$ |
| Acceptor | F1 $\uparrow$ | $0.958^{\pm 0.010}$ | $0.936^{\pm 0.077}$ | $0.937^{\pm 0.033}$ | $0.970^{\pm 0.005}$ | $0.953^{\pm 0.008}$ |
| Donor | F1 $\uparrow$ | $0.949^{\pm 0.024}$ | $0.874^{\pm 0.289}$ | $0.948^{\pm 0.025}$ | $0.962^{\pm 0.004}$ | $0.951^{\pm 0.005}$ |

### 4.2 Protein Sequences

We followed the experimental protocols from Sgarbossa et al., () for protein sequences.

Homology-aware training.
Training data was sourced from the filtered OpenProteinSet
(Ahdritz et al.,, ), consisting of 270k UniClust30
clusters (508M sequences, 110B residues). Using the ProtMamba
pipeline, we constructed homology-aware, alignment-free inputs by concatenating unaligned homologous sequences and mask patches for training with the FIM strategy.
We trained two xLSTM-based models: Prot-xLSTM-26M and Prot-xLSTM-102M. For comparison, we also trained a smaller ProtMamba (ProtMamba-28M) and Transformer-based (Prot-Transformer++-26M) (Touvron et al.,, ) model and used the ProtMamba Long Foundation (ProtMamba-107M) provided by Sgarbossa et al., (). The initial training followed a context length scheduling strategy, with models gradually increasing context from $2^{11}$ to $2^{17}$ tokens. Finally, Prot-xLSTM-102M was further trained with $T=2^{18}$.(^2^22The protein downstream tasks were evaluated using the model trained with $T$ up to $2^{17}$ tokens.)
We evaluated the models using negative log-likelihood and perplexity, calculated for different parts of the concatenated-FIM sequences. As shown in Fig. 3 and Tab. 2, Prot-xLSTM outperformed the other architectures. Its advantage becomes even more pronounced with longer contexts, which Prot-Transformer++ cannot handle, and where Prot-xLSTM significantly outperforms ProtMamba. Furthermore, Prot-xLSTM-102M outperforms ProtMamba-107M, despite being trained on only a quarter of the total training tokens used for ProtMamba-107M.
Further details are provided in Appendix Section  D.1.

Figure: Figure 3: Generative pre-training of protein language models. The learning curves show the validation loss of homology-aware protein language models during training. Left: Small models trained for 20B tokens with a context size of $2^{11}$ and fine-tuned for 10B with a context of $2^{17}$ tokens. Transformer++ can only be run for a small context size. Right: Prot-xLSTM-102M model trained with increasing context sizes from $2^{11}$ to $2^{18}$. Vertical gray dashed lines mark the points where context size was increased. The arrow at 60B tokens indicates the model used for downstream tasks. The orange dashed line corresponds to the validation loss of ProtMamba-107M trained up to a context size of $2^{17}$ for a total of 195B tokens. Prot-xLSTM consistently outperforms other models and sets a new state-of-the-art at homology-aware generation.
Refer to caption: x4.png

**Table 2: Performance comparison of protein language models at homology-conditioned generation. Test set perplexity ($\downarrow$) of different models with a context size of $2^{17}$ is shown across different token subsets. The average and 95% confidence interval values are computed across the test set clusters. Prot-xLSTM outperforms ProtMamba, especially when using a long context.**
|  | Prot-xLSTM-26M | ProtMamba-28M | Prot-xLSTM-102M^a | ProtMamba-107M^b |
| --- | --- | --- | --- | --- |
| All tokens | $8.73^{\pm{0.31}}$ | $10.15^{\pm{0.32}}$ | $6.83^{\pm 0.25}$ ($6.65^{\pm 0.25}$) | $7.47^{\pm 0.26}$ |
| First seq tokens | $15.40^{\pm{0.26}}$ | $15.28^{\pm{0.26}}$ | $13.36^{\pm 0.35}$ ($13.41^{\pm 0.35}$) | $13.04^{\pm 0.36}$ |
| Last seq tokens | $9.19^{\pm{0.30}}$ | $11.08^{\pm{0.27}}$ | $7.32^{\pm 0.29}$ ($7.14^{\pm 0.29}$) | $8.37^{\pm 0.29}$ |
| FIM tokens | $6.77^{\pm{0.25}}$ | $7.96^{\pm{0.27}}$ | $5.52^{\pm 0.20}$ ($5.50^{\pm 0.20}$) | $6.47^{\pm 0.23}$ |

Homology-conditioned protein generation. We generated 2,500 protein sequences each for 19 clusters using different parameters and score them using multiple metrics. Hamming distance, HMMER score, and structural scores correlate well with sequence perplexity, with an average absolute Pearson correlation of 0.57 across clusters for the large Prot-xLSTM model (Table A10). Table 3 shows Kolmogorov-Smirnov test statistics, which quantifies how well the score distributions of the generated proteins match those of the real proteins. For each cluster, we compared scores between 100 random real proteins and the 100 generated proteins with the lowest perplexity. For further details see Appendix Section D.2.

Table: Table 3: Homology-conditioned protein generation. Average Kolmogorov-Smirnov statistic ($\downarrow$) between scores of natural and generated sequences with 95% confidence intervals across 19 homology clusters. For three of five metrics, score distributions of Prot-xLSTM-generated sequences are closest to natural sequences.

Protein variant fitness prediction. We assessed Prot-xLSTM’s predictive power for mutational effects by leveraging its inpainting capability from the FIM training objective and using homologous sequences as context on the ProteinGym DMS zero-shot substitutions benchmark (Notin et al.,, ). Table A11 presents the performance comparison of Prot-xLSTM, other well-known protein models, and the current top performers on the ProteinGym leaderboard. In summary, Prot-xLSTM outperforms larger unconditional protein language models like ESM-2 (Lin et al.,, ) and ProGen2 (Nijkamp et al.,, ), and matches or surpasses ProtMamba’s performance. However, it falls short compared to models that directly use the MSA, such as TranceptEVE (), or use structural information, like ProSST (Li et al.,, ). Further details and results are provided in Appendix Section D.3.

### 4.3 Chemical Sequences

Unconditional molecule generation aims to produce valid small organic molecules without imposing specific constraints, such as being from a particular molecular domain.
Following the setup of Özçelik et al., (), we trained models to generate
SMILES strings using a CLM approach on a dataset derived from ChEMBL with a context length of 100 tokens.
We compared our Chem-xLSTM architecture with several architectures, including LSTM, GPT, S4, and Mamba, where all models contain approximately 15 million parameters.
The evaluation focused on two primary metrics: perplexity and Fréchet ChemNet Distance (FCD) (Preuer et al.,, ).
Chem-xLSTM achieved the lowest FCD of 0.13 and a competitive perplexity score of 1.68, indicating its strong ability to generate realistic chemical structures (see Table 4).
All models produced valid, unique, and novel
molecules, showcasing their effectiveness in this task. Further details and results are provided in Appendix Section E.1.

**Table 4: Unconditional generation of molecules with 15M parameter models. 102,400 SMILES sequences have been generated and evaluated. Error bars represent standard deviations across bootstrap resampling.Green cells highlight the best values per row. Chem-xLSTM yields the best FCD and SMILES-GPT the best perplexity.**
|  | SMILES-LSTM^a | SMILES-GPT^b | SMILES-S4^c | Chem-Mamba^d | Chem-xLSTM |
| --- | --- | --- | --- | --- | --- |
| FCD ↓ | $0.46^{\pm{<0.01}}$ | $0.15^{\pm{<0.01}}$ | $0.28^{\pm{<0.01}}$ | $0.21^{\pm{<0.01}}$ | $0.13^{\pm{<0.01}}$ |
| Perplexity ↓ | $1.88^{\pm{3.8}}$ | $1.65^{\pm{0.6}}$ | $1.73^{\pm{2.4}}$ | $1.74^{\pm{0.5}}$ | $1.68^{\pm{1.0}}$ |

For conditional molecule generation, the objective
is to generate molecules belonging to a specific
molecular domain or possessing desired properties.
Here, we focused on generating molecules from a particular domain using the in-context learning abilities of LLMs.
To achieve this, we assembled a dataset, referred to as the *molecular domains* dataset, which comprises a diverse range of molecular domains:
natural products, click-chemistry, proteolysis-targeting chimera (PROTACs), DNA-encoded chemical libraries, approved and failed drugs, and bioactive compounds from various bioassays.
Molecules from the same domain, are concatenated as a long sequence, and augmented through permutation during training.
We split the dataset into training, validation, and test domains, following an 8:1:1 ratio (Figure 4, left). The validation and test sets contained molecules from unseen domains, enabling us to evaluate the models’ conditional generation capabilities through in-context learning.
We trained Chem-xLSTM, Mamba, Transformer++, and S4-based models with the CLM approach on the *molecular domains* with an increased context length of 4,096 tokens. The context length for S4 models was restricted to 2,048 due to memory constraints. We evaluated the models based on NTP loss across unseen domains.
The trained model
Chem-xLSTM-15M-icl shows promising results in this
conditional setting, outperforming the other benchmarked model classes (Figure 4, right).
This demonstrates Chem-xLSTM’s capability to generate
molecules from an unseen chemical domain when provided with only a few exemplary molecules without fine-tuning. Further details and results are provided in Appendix Section E.2.

Figure: Figure 4: Conditional generation of molecules via ICL and 15M parameter models. Left: Visualization of different molecular domains contained in the *molecular domains* dataset. A t-SNE down-projection of molecules from different domains is shown. Clusters on the exterior represent highly specific molecular domains. The validation and test sets contain molecules from highly specific, unseen molecular domains. Right: Generative training of chemistry language models on the *molecular domains* dataset. Learning curves showing mean CLM loss ($\downarrow$) on a validation set across the training epochs. Shaded areas represent the standard-deviation over runs. The Chem-xLSTM achieved the lowest loss at conditional generation of molecules using ICL.
Refer to caption: x5.png

### 4.4 Compute demand and resources.

The experiments were conducted on multiple GPU servers with A100 GPUs. Model training was performed in both single-node and multi-node setups, utilizing 1–8 A100 GPUs per node. Prot-xLSTM-102M training with a context length of $2^{17}$ was completed on a node with 8 H200 GPUs. The largest models were trained across up to four nodes using distributed data parallelism. Some experiments leveraged compute resources provided by EuroHPC Joint Undertaking clusters, including Karolina at IT4Innovations, Leonardo at CINECA, and MeluXina at LuxProvide. The total amount of GPU hours required for the experiments in this paper is approximately 50k.

## 5 Discussion

In this work, we demonstrated the potential of Bio-xLSTM variants as prime candidates to model biological and chemical sequences. We have provided clarity in two key areas: a) how to tailor xLSTM for biological and chemical sequences, and b) comparing xLSTM-based models to other domain-specific LLMs, showcasing their robust performance in DNA, protein, and chemical sequence modeling tasks.
Despite certain limitations, DNA-xLSTM showed strong performance in DNA sequence modeling, excelling in both masked and causal language tasks across different context sizes. In protein modeling, Prot-xLSTM proved particularly effective at handling long-range dependencies, positioning it as a promising tool for generating homologous proteins. In small molecule modeling, Chem-xLSTM achieved the best FCD scores for unconditional generation and demonstrated strong ICL capabilities.
Our findings underscore the potential of xLSTM as a prime candidate for foundational models in molecular biology. The models we have introduced, trained, and made available can be used for example to generate rich learned representations for DNA sequences and homology- and chemical domain-conditioned generation of proteins and molecules without the need for fine-tuning.

While Bio-xLSTM has shown strong performance across DNA, protein, and chemical sequence modeling, it has several limitations.
The manual hyperparameter selection process, which was due to limited
computational resources, may prevent optimal model configurations. We
will explore the hyperparameter space further in the future,
which might yield even better models.
For DNA, the reliance on character-level tokenization might also
restrict the performance and scaling to larger context sizes. Also for
proteins, amino acid level tokenization without explicit structural
information might limit it’s performance.
The DNA-xLSTM, Prot-xLSTM, and Chem-xLSTM models are currently
constrained by the training dataset and their generalizability across
organisms and chemical domains needs further exploration.
Across all three domains, the training datasets contain
biases – whether it is population biases in the genomic data, sequence distribution biases in protein datasets,
or chemical exploration biases in molecular datasets.
These biases could influence the model’s predictions and limit its generalizability in real-world applications.
In line with many works, we consider the perplexity metric, for example, next token perplexity, or the related cross-entropy losses as a proxy for performance on downstream tasks. However, this metric might not capture the capacities of biological and chemical language models appropriately.
Future work could address these limitations by expanding the training datasets and downstream evaluations of Bio-xLSTM. Finally, assessing Bio-xLSTM’s performance in parameter regimes beyond the billion scale remains an open question.

## Ethics Statement

The development of our large language model for biological
sequences, including DNA, proteins, and small molecules,
has the potential to significantly advance biomedical
research and therapeutics. In creating this model, we have
taken care to train exclusively on publicly available
data, such as the human reference genome, OpenProteinSet,
and publicly available small molecule databases. Note
that no human subjects are involved in the studies, since
the human reference genome does not represent a particular
individual but a type of average human genome.
As common with machine learning methods, potential
danger lies in the possibility that users rely too much on
our new approach and use it without reflecting
on the outcomes. However, the full pipeline, in which our
method would be used, includes wet lab tests
after its application, to verify and investigate the
results, which decreases the danger of misuse or overly
relying on the predictions. To further mitigate the risk of misuse,
we provide model limitations and mention potential biases.
Users are encouraged to approach the model’s predictions critically and
consider them as one component of a broader decision-making process.

## Code and Data Availability

To ensure reproducibility and facilitate future research, we provide three standalone code repositories for [DNA-xLSTM](https://github.com/ml-jku/DNA-xLSTM), [Prot-xLSTM](https://github.com/ml-jku/Prot-xLSTM), and [Chem-xLSTM](https://github.com/ml-jku/Chem-xLSTM), each containing the respective implementations, training scripts, evaluation procedures, and pre-processed datasets.

## Acknowledgements

The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), FWF AIRI FG 9-N (10.55776/FG9), AI4GreenHeatingGrids (FFG-899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID: 951847), Stars4Waters (HORIZON-CL6-2021-CLIMATE-01-01). We thank NXAI GmbH, Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo), Software Competence Center Hagenberg GmbH, Borealis AG, TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation.

## Appendix A Related Work

In all three areas, genomics, proteomics, and chemistry, we observe a similar trend that until around 2018 the language models were based on LSTMs (Hochreiter and Schmidhuber,, ),
then a large amount of models were based on Transformers (Vaswani et al.,, ), with different modeling paradigms and styles, and from 2023 onwards SSMs appeared.

Language models for genomic sequence data.
DNABERT (Ji et al.,, ) and its successor DNABERT-2 (Zhou et al.,, ) are
Transformer-based models that leverage bidirectional encoder representations and masked language modeling
to capture nucleotide context, achieving high performance in tasks like promoter and splice site prediction.
LOGO (Yang et al.,, ), another Transformer-based model,
applies self-supervised learning to the human genome
for sequence labeling and variant prioritization,
while VIBE (Gwak and Rho,, ) employs a hierarchical BERT model
to enhance the detection of eukaryotic viruses in metagenomic data.
Models like LookingGlass (Hoarfrost et al.,, ), based on recurrent neural network (RNN), and GPN (Benegas et al.,, ), which uses convolutional neural networks (CNNs), are examples of non-Transformer-based approaches,
with LookingGlass focusing on microbial genomes and GPN on plant genomes.
More recent developments include the nucleotide transformer (NT) (Dalla-Torre et al.,, ),
a Transformer model trained on the human genome and data from the 1000 Genomes Project,
and SpeciesLM (Karollus et al.,, ), which trains Transformer-based models on 1500 fungal genomes.
The latest advances, represented by Caduceus (Schiff et al.,, ) based on Mamba (Gu and Dao,, )
and HyenaDNA (Nguyen et al.,, ),
introduce SSMs that allow
generative modeling and representation learning for long DNA sequences.

Language models for protein sequence data.
Until around 2019, the field was dominated by RNNs and LSTM-based models trained with CLM.
Notable examples include UniRep (Alley et al.,, ),
which employed multiplicative-LSTM to capture rich protein representations, and SSA (Bepler and Berger,, ), which used bidirectional RNNs for structural similarity prediction.
Since then the field has shifted towards Transformer-based models and MLM, driven by their success in natural language processing. Early adopters of this shift included the TAPE benchmark for protein downstream tasks (Rao et al.,, ), which evaluated both an LSTM and a Transformer architecture trained with CLM and MLM, respectively.
Elnaggar et al., () further expanded the use of Transformers with large-scale MLM, setting new benchmarks in protein sequence analysis with Prot-T5.
ESM (Rives et al.,, ) applied MLM to a Transformer on a massive scale, capturing evolutionary patterns across diverse protein sequences.
Other significant Transformer-based models include MSA-Transformer (Rao et al.,, ), which applied MLM to multiple-sequence alignments (MSA), and ProGen (Madani et al.,, ), which used CLM and Transformers for controlled protein sequence generation.
Additionally, models like ProtGPT2 (Ferruz et al.,, ) and ProteinBERT (Brandes et al.,, ) utilized the power of Transformer architectures in generating novel protein sequences and functional predictions.
Furthermore, () introduced a "structure-aware vocabulary" which they use as input for Transformer-based models.
The recently proposed PoET (Truong Jr and Bepler,, ) is an autoregressive Transformer model trained on non-aligned homologous sequences, providing a novel approach for conditional protein design and protein fitness prediction.
Building on the concept of non-aligned homologous sequences, ProtMamba (Sgarbossa et al.,, ) leverages emerging SSMs to manage long-context conditioning on proteins, effectively utilizing autoregressive and FIM strategies.
For a more comprehensive review of these advancements,
including their applications in functional protein design,
see Notin et al., () and Hu et al., ().

Language models for chemical sequence data.
The first language model for chemical sequences was
an LSTM-based, autoregressive method developed by Segler et al., (), which demonstrated that the SMILES syntax (Weininger,, ) and generation of realistic organic molecules can be learned.
Honda et al., () introduced a Transformer model for this task, showing that this leads to informative representations of molecules.
The Molecular Transformer (Schwaller et al.,, ) consists of a Transformer-based encoder and decoder, trained on chemical reaction data to translate between reactants and products.
More recently, SSMs have been used for generative modeling of SMILES strings (Özçelik et al.,, ).
Subsequent models such as MolGPT (Bagal et al.,, ) and cMolGPT (Wang et al.,, ) utilized the GPT architecture to generate SMILES strings, with MolGPT conditioning on chemical properties and scaffolds, and cMolGPT focusing on biomolecular targets. Transformer-based approaches have also been employed to optimize the properties of small molecules in a reinforcement-learning setting (Mazuz et al.,, ).
Encoder-style language models for chemistry,
such as SmilesLSTM (Mayr et al.,, ), ChemNet (Preuer et al.,, ), and CNN-based models (Jastrzębski et al.,, ), initially used activity and property prediction as pre-training or training objectives.
Later, these encoder-style language models were trained with the masking language modeling objective, as seen in ChemBERTA (Chithrananda et al.,, ), Chemberta-2 (Ahmad et al.,, ), SMILES-BERT (),
MolFormer (Ross et al.,, ) and rxnfp-BERT (Schwaller et al.,, ).
Some models have also adopted contrastive objectives (Seidl et al.,, ).
Large language models for
molecules have also been shown to learn complex
molecular distributions (Flam-Shepherd et al.,, ).
For a more thorough and comprehensive overview, we
refer to the surveys by Bran and Schwaller, ()
and Zhang et al., ()

## Appendix B xLSTM Architecture Details

### B.1 xLSTM and Bio-xLSTM Blocks

Beck et al., () suggested xLSTM blocks, which are residual (Srivastava et al.,, ; He et al.,, ) block modules, into which the sLSTM and mLSTM layers can be integrated. The two basic blocks can in principle be characterized by either applying post-sLSTM/mLSTM up- and down-projections (similar to Vaswani et al., ()) or by applying pre-sLSTM/mLSTM up-projections and post-sLSTM/mLSTM down-projections (similar to Dao, ()). An sLSTM block integrates the sLSTM layer into the up- and down-projection block, while the mLSTM block integrates the mLSTM layer into the pre-up-projection and post-down-projection block. The two basic xLSTM blocks also make use of neural network modules like layer normalization (Ba et al.,, ), short causal convolutions,
and, group normalization (Wu and He,, ). For the exact architecture of the blocks, we refer to Beck et al., (, Sec.2.4).
An xLSTM architecture is constructed by residually stacking the suggested xLSTM blocks. For that, the most commonly used pre-LayerNorm residual backbone is used.

Figure: Figure A1: xLSTM and Bio-xLSTM blocks. Left: mLSTM block. LN (Layer Normalization) and GN (Group Normalization) refer to normalization modules, while L Skip represents learnable skip connections and Conv denotes causal 1D convolutions. The mLSTM block utilizes a gated pre-up-projection structure, akin to modern State-Space Models, with gates activated by the Swish function. Middle: sLSTM block. The sLSTM block features a GELU-gated post-up-projection structure, similar to Transformer architectures. Right: Bidirectional mLSTM block. For bidirectional processing, the xLSTM applies each block to the input sequence twice before combining the outputs: once left-to-right and once right-to-left.
Refer to caption: x7.png

For Bio-xLSTM we keep the basic xLSTM building blocks and the basic xLSTM architecture template, but adjust them to the respective domains. Figure A1 depicts sLSTM and mLSTM blocks, as well as, a bidirectional mLSTM configuration with weight-tied layers.

### B.2 Modes of Operation: Parallel, Chunkwise or Recurrent

Similar to linear attention variants (Katharopoulos et al.,, ; Yang et al.,, ), the mLSTM has three possible formulations: parallel, recurrent or chunkwise.
The presentation in section 2.2 (and Beck et al.,, ) focuses on the recurrent form:

$$ $\displaystyle\bm{C}_{t}$ $\displaystyle=\sigma(\tilde{f}_{t})\,\bm{C}_{t-1}+\exp(\tilde{i}_{t})\odot\bm{v}_{t}\bm{k}_{t}^{\mathsf{T}}$ $\displaystyle\bm{n}_{t}$ $\displaystyle=\sigma(\tilde{f}_{t})\,\bm{n}_{t-1}+\exp(\tilde{i}_{t})\,\bm{k}_{t}$ $\displaystyle\bm{h}_{t}$ $\displaystyle=\sigma(\tilde{\bm{o}}_{t})\odot\frac{\bm{C}_{t}\bm{q}_{t}}{\max\bigl(|\bm{n}_{t}\bm{q}_{t}|,1\bigr)}.$ $$

This form is especially useful for inference when samples arrive one time-step at a time.

The omission of the recurrent connections in mLSTM allows for a parallel formulation (Beck et al.,, , Appendix):

$$ $\displaystyle\tilde{\bm{F}}$ $\displaystyle=\begin{bmatrix}0&0&0&\ldots&0\\ \ln\sigma(\tilde{f}_{2})&0&0&\ldots&0\\ \ln\sigma(\tilde{f}_{2})+\ln\sigma(\tilde{f}_{3})&\ln\sigma(\tilde{f}_{3})&0&\ldots&0\\ \vdots&\vdots&\vdots&\ddots&\vdots\\ \sum_{t=2}^{T}\ln\sigma(\tilde{f}_{t})&\sum_{t=3}^{T}\ln\sigma(\tilde{f}_{t})&\sum_{t=4}^{T}\ln\sigma(\tilde{f}_{t})&\ldots&0\end{bmatrix}$ $\displaystyle\bm{D}$ $\displaystyle=\exp\bigl(\tilde{\bm{F}}+\bm{1}\otimes\tilde{\bm{i}}\bigr)\odot\bm{M}$ $\displaystyle\bm{H}$ $\displaystyle=\sigma(\tilde{\bm{O}})\odot\frac{\bm{D}\odot\bm{Q}\bm{K}^{\mathsf{T}}}{\max\bigl(|\bm{D}\odot\bm{Q}\bm{K}^{\mathsf{T}}|\cdot\bm{1},\bm{1}\bigr)}\bm{V},$ $$

where $\bm{Q},\bm{K},\bm{V},\tilde{\bm{O}}\in\mathbb{R}^{T\times d}$, $\tilde{\bm{i}}\in\mathbb{R}^{T}$ and $\bm{M}\in\{0,1\}^{T\times T}$ is a causal (i.e. lower-triangular) masking matrix.
The $\otimes$ refers to an outer product, while $\odot$ is a Hadamard (i.e. element-wise) product.
The fraction, $\max$ and other non-linear functions are also applied element-wise.
This parallel form enables an efficient training regime, similar to Transformers.

The chunkwise formulation is a
hybrid of the recurrent and parallel forms:

$$ $\displaystyle\tilde{\bm{F}}$ $\displaystyle=\begin{bmatrix}0&0&0&\ldots&0\\ \ln\sigma(\tilde{f}_{t-C+2})&0&0&\ldots&0\\ \ln\sigma(\tilde{f}_{t-C+2})+\ln\sigma(\tilde{f}_{t-C+3})&\ln\sigma(\tilde{f}_{t-C+3})&0&\ldots&0\\ \vdots&\vdots&\vdots&\ddots&\vdots\\ \sum_{\tau=2}^{C}\ln\sigma(\tilde{f}_{t-C+\tau})&\sum_{\tau=3}^{C}\ln\sigma(\tilde{f}_{t-C+\tau})&\sum_{\tau=4}^{C}\ln\sigma(\tilde{f}_{t-C+\tau})&\ldots&0\end{bmatrix}$ $\displaystyle\bm{D}$ $\displaystyle=\exp\bigl(\tilde{\bm{F}}+\bm{1}\otimes\tilde{\bm{i}}\bigr)\odot\bm{M}$ $\displaystyle\bm{f}$ $\displaystyle=\bigg(\sigma(\tilde{f}_{t-C+1}),\sigma(\tilde{f}_{t-C+1})\,\sigma(\tilde{f}_{t-C+2}),\ldots,\prod_{\tau=1}^{C}\sigma(\tilde{f}_{t-C+\tau})\bigg)$ $\displaystyle\bm{H}$ $\displaystyle=\sigma(\tilde{\bm{O}})\odot\frac{(\bm{D}\odot\bm{Q}\bm{K}^{\mathsf{T}})\,\bm{V}+\operatorname{diag}(\bm{f})\,\bm{Q}\bm{C}_{t-C}^{\mathsf{T}}}{\max\bigl(|(\bm{D}\odot\bm{Q}\bm{K}^{\mathsf{T}})\cdot\bm{1}+\operatorname{diag}(\bm{f})\,\bm{Q}\bm{n}_{t-C}|,\bm{1}\bigr)}$ $\displaystyle\bm{C}_{t}$ $\displaystyle=\bigg(\prod_{\tau=1}^{C}\sigma(\tilde{f}_{t-C+\tau})\bigg)\,\bm{C}_{t-C}+\bm{V}^{\mathsf{T}}\operatorname{diag}(\bm{d}_{C})\,\bm{K}$ $\displaystyle\bm{n}_{t}$ $\displaystyle=\bigg(\prod_{\tau=1}^{C}\sigma(\tilde{f}_{t-C+\tau})\bigg)\,\bm{n}_{t-C}+\bm{K}^{\mathsf{T}}\bm{d}_{C},$ $$

with $\bm{Q},\bm{K},\bm{V},\tilde{\bm{O}}\in\mathbb{R}^{C\times d}$ and $\tilde{\bm{i}}\in\mathbb{R}^{C}$ the pre-activations from $t-C+1$ to $t$. Furthermore, $\bm{M}\in\{0,1\}^{C\times C}$, is a local causal (i.e. lower-triangular) masking matrix, $\bm{d}_{C}$ denotes the last row of $\bm{D}$, $\operatorname{diag}$ transforms a vector into a diagonal matrix, and $C$ is the chunk size. The chunk-wise formulation allows for
implementing hardware-aware
efficient kernels (Beck et al.,, ).
For $C=1$, we recover the recurrent form, whereas for $C=T$, we obtain the parallel form.

### B.3 Efficient Bidirectional Modeling for Weight-Tied Layers of Bio-xLSTM

Bidirectional modeling is often required to learn
representations of biological and chemical sequences, for
example with the MLM paradigm. The default approach for bidirectional modeling would be
to use an mLSTM layer on the usual sequence $\bm{X}_{1:T}=(\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{T})$
and then applying a weight-tied layer on the reversed sequence $\bm{X}_{T:1}=(\bm{x}_{T},\bm{x}_{T-1},\ldots,\bm{x}_{1})$
and subsequently summing those outputs:

$$ $\displaystyle\bm{H}^{+}$ $\displaystyle=\mathrm{mLSTM}(\bm{X}_{1:T};\bm{w})$ (19) $\displaystyle\bm{H}^{-}$ $\displaystyle=\mathrm{mLSTM}(\bm{X}_{T:1};\bm{w})$ (20) $\displaystyle\bm{H}$ $\displaystyle=\bm{H}^{+}+\bm{H}^{-}_{T:1},$ (21) $$

where $\bm{H}^{-}_{T:1}$ indicates that the sequence is reversed again,
and $\bm{w}$ are the parameters of the LSTM-layer
$\mathrm{mLSTM}(\bm{X}_{1:T};\bm{w})$ which are assumed to be
the same for both directions, i.e. weight-tied. This approach is schematically depicted in Figure A1 (Right).
However, this approach is inefficient
with respect to memory and
operations because it has to calculate and store
all internal quantities, such as the cell states,
twice for the backward pass.
A variant of this approach is to perform
the forward direction in one block (Eq. 19)
and the reverse direction
in a consecutive block (Eq. 20)
of the architecture (Alkin et al.,, ).

We propose an efficient bidirectional modeling approach.
Because of the parallelism of mLSTM and its gates depending only on the current time step, the weighted cumulative sum required for the cell state (Eq. 8), can be done backwards to obtain the representations for the reversed sequence

$$ $\displaystyle\bm{C}^{+}_{t}$ $\displaystyle=\mathrm{f}_{t}\bm{C}^{+}_{t-1}+\mathrm{i}_{t}\bm{v}_{t}\ \bm{k}_{t}^{\top}$ $\displaystyle\bm{C}^{-}_{t}$ $\displaystyle=\mathrm{f}_{t}\bm{C}^{-}_{t+1}+\mathrm{i}_{t}\bm{v}_{t}\ \bm{k}_{t}^{\top}$ (22) $\displaystyle\bm{n}^{+}_{t}$ $\displaystyle=\mathrm{f}_{t}\bm{n}^{+}_{t-1}+\mathrm{i}_{t}\bm{k}_{t}$ $\displaystyle\bm{n}^{-}_{t}$ $\displaystyle=\mathrm{f}_{t}\bm{n}^{-}_{t+1}+\mathrm{i}_{t}\bm{k}_{t}.$ (23) $$

The resulting $\bm{h}_{t}=\bm{h}_{t}^{+}+\bm{h}_{t}^{-}$ is a
bidirectional representation of the input sequence, whereby this
variant is more efficient with respect to memory usage because of
shared quantities. Note that the two variants, the default approach, and the efficient approach, are mathematically equivalent.

## Appendix C DNA-xLSTM: Details and Additional Results

In this section, we provide further details regarding the
architecture, training setup, and evaluation metrics for the DNA-xLSTM models.

### C.1 Pre-Training

Experimental setup. We followed the experimental protocol established in Schiff et al., () and Nguyen et al., (). The human reference genome (Church et al.,, ) was used as the training dataset for two main tasks: a) causal language modeling (CLM) and b) masked language modeling (MLM). We employed context lengths of 1,024 and 32,000 tokens for these tasks.

To ensure a fair comparison with previous methods, such as Schiff et al., (), we used character- or base pair-level tokenization, training models with parameter sizes ranging from 500k to 4M. This experimental setup enabled us to evaluate model performance for both a) generative modeling of DNA sequences and b) learning rich DNA sequence representations—core tasks in this domain.

Methods and hyperparameters. In our pre-training experiments, we compared several architectures: a Transformer variant based on the Llama architecture, referred to as Transformer++ (Touvron et al.,, ), DNA-xLSTM, HyenaDNA (Nguyen et al.,, ), and DNA-Mamba (also known as Caduceus) (Schiff et al.,, ). Each architecture was trained under both CLM and MLM settings. Additionally, we assessed two types of reverse-complement (RC) equivariant models when applicable: DNA-Mamba-PH and DNA-Mamba-PS, as well as DNA-xLSTM-PH and DNA-xLSTM-PS. For non-equivariant models, reverse-complement augmentation was applied, following the approach described in Schiff et al., (). Further details on RC-equivariant modeling can be found in Section C.4. The hyperparameters for DNA-xLSTM and Transformer++ were optimized using a validation set, with the final configurations reported in Appendix Tables A4 and A3.

Metrics. We report cross-entropy loss on a held-out test set for both CLM and MLM pre-training experiments.

Results.
Our experiments show that the sLSTM-based DNA-xLSTM-2M model, trained with a context size of 1,024 and reverse-complement augmentation, outperforms DNA-Mamba (Schiff et al.,, ), HyenaDNA (Nguyen et al.,, ), and Transformer++ across both CLM and MLM tasks. Notably, the performance gap between DNA-xLSTM and the baseline models increases in the MLM setting. See Figure 2.

We further enhanced DNA-xLSTM-500k and DNA-xLSTM-2M models by incorporating reverse-complement equivariance via parameter sharing. For smaller models, we achieved MLM losses comparable to DNA-Mamba-PS, with a significant improvement over DNA-Mamba-PS as model size scaled to 2M parameters (Figure A3). Additionally, we pre-trained a long-range DNA-xLSTM model based on mLSTM, with a context size of 32k, using both CLM and MLM objectives. This model achieved the lowest cross-entropy loss in both tasks, outperforming Transformers and HyenaDNA, while performing comparably to Mamba (Figure A2).

Figure: Figure A2: Pre-training of 4M-parameter DNA models on the human reference genome (GRCh38). The models are trained on the human reference genome at single-nucleotide resolution with a context length of 32k bases. Left: causal language modeling. Learning curves display CLM loss ($\downarrow$) on a held-out test set, plotted against the number of tokens processed. Right: masked language modeling. Learning curves for bidirectional models trained with the MLM objective ($\downarrow$). The DNA-xLSTM-4M model outperforms both Transformer++ and Hyena-DNA models of similar size, and matches the performance of Caduceus-4M.
Refer to caption: x8.png

### C.2 Downstream Tasks

Experimental setup. Two sets of downstream tasks were used
for evaluating the learned representations:
the Genomic benchmark (Grešová et al.,, ) and
the Nucleotide Transformers Tasks (Dalla-Torre et al.,, ),
which is a collection of 18 datasets derived from
five peer-reviewed studies (Phaml et al.,, ; Oubounyt et al.,, ; ; Scalzitti et al.,, ; Geng et al.,, ). These classification tasks were selected to determine how rich the learned representations of the architectures are. To extract representations from the pre-trained xLSTM-DNA models, we perform average pooling on the activations from the final xLSTM block. For each downstream dataset, these representations served as inputs to a task-specific classification head that were jointly fine-tuned with the pre-trained model parameters.

Methods and hyperparameters.
For Nucleaotide Transformer tasks, we compared HyenaDNA, DNA-Mamba, and xLSTM-based
models pre-trained with 2M parameters. For Genomic benchmark tasks, we compare the smaller xLSTM-500k against Mamba. In both settings, models were pre-trained with a context size of 1,024.

Metrics. For the Nucleotide Transformer downstream tasks different metrics are used depending on the type of task: MCC was used for histone markers and enhancer annotation, F1-score was used for promoter annotation and splice site acceptor/donor, and accuracy was used for the splice site.
The downstream performance on the Genomic benchmark was evaluated using the Top-1 accuracy.

Results. On the extensive set of downstream tasks, DNA-xLSTM is the best model with fewer than 2M parameters outperforming other small models on 12 of 18 tasks. In a comparison with much larger models,
DNA-xLSTM and is on par with the 500M parameter
model Nucleotide Transformer (NT-v2) winning 8 of
18 tasks (see Table A1). On the Genomic benchmark, DNA-xLSTM is overall on par with Mamba-DNA and shows especially strong results with posthoc conjoining, winning 5 of 8 tasks compared to Mamba-DNA-PH. Results are reported in Table A2.

**Table A1: Downstream adaption of DNA models (extended version). The test set performance of DNA models with 2M parameters and models with over 100M parameters, fine-tuned on Nucleotide Transformer classification tasks, is shown. Models marked with PS or PH are trained to be RC equivariant. The used metric is provided in the Metric column and best values are highlighted in green Results are averaged over 10 random seeds, with error bars representing the difference between the maximum and minimum values across the runs. The best scores are highlighted in green. xLSTM-DNA-PH with 2M parameters outperforms similarly sized Hyena- and Mamba-based models, while achieving comparable results to the much larger Nucleotide Transformer. Scores for all models except xLSTM were obtained from Schiff et al., ().**
| Task | Metric | > 100M Param. Models | 2M Param. Models |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Enformer (252M) | DNABERT-2 (117M) | NT-v2 (500M) | HyenaDNA | Mamba-PS | Mamba-PH | xLSTM-PS | xLSTM-PH |
| Histone Markers |  |  |  |  |  |  |  |  |  |
| H3 | MCC $\uparrow$ | $0.719^{\pm 0.048}$ | $0.785^{\pm 0.033}$ | $0.784^{\pm 0.047}$ | $0.779^{\pm 0.037}$ | $0.799^{\pm 0.029}$ | $0.815^{\pm 0.048}$ | $0.796^{\pm 0.014}$ | $0.824^{\pm 0.010}$ |
| H3K14AC | MCC $\uparrow$ | $0.288^{\pm 0.077}$ | $0.516^{\pm 0.028}$ | $0.551^{\pm 0.021}$ | $0.612^{\pm 0.065}$ | $0.541^{\pm 0.212}$ | $0.631^{\pm 0.026}$ | $0.570^{\pm 0.008}$ | $0.598^{\pm 0.017}$ |
| H3K36ME3 | MCC $\uparrow$ | $0.344^{\pm 0.055}$ | $0.591^{\pm 0.020}$ | $0.625^{\pm 0.030}$ | $0.613^{\pm 0.041}$ | $0.609^{\pm 0.109}$ | $0.601^{\pm 0.129}$ | $0.588^{\pm 0.019}$ | $0.625^{\pm 0.010}$ |
| H3K4ME1 | MCC $\uparrow$ | $0.291^{\pm 0.061}$ | $0.511^{\pm 0.028}$ | $0.550^{\pm 0.021}$ | $0.512^{\pm 0.024}$ | $0.488^{\pm 0.102}$ | $0.523^{\pm 0.039}$ | $0.490^{\pm 0.012}$ | $0.526^{\pm 0.001}$ |
| H3K4ME2 | MCC $\uparrow$ | $0.211^{\pm 0.069}$ | $0.336^{\pm 0.040}$ | $0.319^{\pm 0.045}$ | $0.455^{\pm 0.095}$ | $0.388^{\pm 0.101}$ | $0.487^{\pm 0.170}$ | $0.489^{\pm 0.024}$ | $0.504^{\pm 0.012}$ |
| H3K4ME3 | MCC $\uparrow$ | $0.158^{\pm 0.072}$ | $0.352^{\pm 0.077}$ | $0.410^{\pm 0.033}$ | $0.549^{\pm 0.056}$ | $0.440^{\pm 0.202}$ | $0.544^{\pm 0.045}$ | $0.520^{\pm 0.019}$ | $0.537^{\pm 0.012}$ |
| H3K79ME3 | MCC $\uparrow$ | $0.496^{\pm 0.042}$ | $0.613^{\pm 0.030}$ | $0.626^{\pm 0.046}$ | $0.672^{\pm 0.048}$ | $0.676^{\pm 0.026}$ | $0.697^{\pm 0.077}$ | $0.662^{\pm 0.011}$ | $0.697^{\pm 0.007}$ |
| H3K9AC | MCC $\uparrow$ | $0.420^{\pm 0.063}$ | $0.542^{\pm 0.029}$ | $0.562^{\pm 0.040}$ | $0.581^{\pm 0.061}$ | $0.604^{\pm 0.048}$ | $0.622^{\pm 0.030}$ | $0.622^{\pm 0.013}$ | $0.627^{\pm 0.008}$ |
| H4 | MCC $\uparrow$ | $0.732^{\pm 0.076}$ | $0.796^{\pm 0.027}$ | $0.799^{\pm 0.025}$ | $0.763^{\pm 0.044}$ | $0.789^{\pm 0.020}$ | $0.811^{\pm 0.022}$ | $0.793^{\pm 0.011}$ | $0.813^{\pm 0.008}$ |
| H4AC | MCC $\uparrow$ | $0.273^{\pm 0.063}$ | $0.463^{\pm 0.041}$ | $0.495^{\pm 0.032}$ | $0.564^{\pm 0.038}$ | $0.525^{\pm 0.240}$ | $0.621^{\pm 0.054}$ | $0.558^{\pm 0.018}$ | $0.583^{\pm 0.014}$ |
| Regulatory Annotation |  |  |  |  |  |  |  |  |  |
| Enhancer | MCC $\uparrow$ | $0.451^{\pm 0.108}$ | $0.516^{\pm 0.098}$ | $0.548^{\pm 0.144}$ | $0.517^{\pm 0.117}$ | $0.491^{\pm 0.066}$ | $0.546^{\pm 0.073}$ | $0.375^{\pm 0.030}$ | $0.545^{\pm 0.024}$ |
| Enhancer Types | MCC $\uparrow$ | $0.309^{\pm 0.134}$ | $0.423^{\pm 0.051}$ | $0.424^{\pm 0.132}$ | $0.386^{\pm 0.185}$ | $0.416^{\pm 0.095}$ | $0.439^{\pm 0.054}$ | $0.444^{\pm 0.046}$ | $0.466^{\pm 0.011}$ |
| Promoter: All | F1 $\uparrow$ | $0.954^{\pm 0.006}$ | $0.971^{\pm 0.006}$ | $0.976^{\pm 0.006}$ | $0.960^{\pm 0.005}$ | $0.967^{\pm 0.004}$ | $0.970^{\pm 0.004}$ | $0.962^{\pm 0.002}$ | $0.967^{\pm 0.001}$ |
| NonTATA | F1 $\uparrow$ | $0.955^{\pm 0.010}$ | $0.972^{\pm 0.005}$ | $0.976^{\pm 0.006}$ | $0.959^{\pm 0.011}$ | $0.968^{\pm 0.006}$ | $0.968^{\pm 0.010}$ | $0.963^{\pm 0.002}$ | $0.970^{\pm 0.001}$ |
| TATA | F1 $\uparrow$ | $0.960^{\pm 0.023}$ | $0.955^{\pm 0.021}$ | $0.966^{\pm 0.013}$ | $0.944^{\pm 0.040}$ | $0.957^{\pm 0.015}$ | $0.953^{\pm 0.016}$ | $0.948^{\pm 0.006}$ | $0.952^{\pm 0.005}$ |
| Splice Site Annotation |  |  |  |  |  |  |  |  |  |
| All | Accuracy $\uparrow$ | $0.848^{\pm 0.019}$ | $0.939^{\pm 0.009}$ | $0.983^{\pm 0.008}$ | $0.956^{\pm 0.011}$ | $0.927^{\pm 0.021}$ | $0.940^{\pm 0.027}$ | $0.965^{\pm 0.006}$ | $0.974^{\pm 0.004}$ |
| Acceptor | F1 $\uparrow$ | $0.914^{\pm 0.028}$ | $0.975^{\pm 0.006}$ | $0.981^{\pm 0.011}$ | $0.958^{\pm 0.010}$ | $0.936^{\pm 0.077}$ | $0.937^{\pm 0.033}$ | $0.970^{\pm 0.005}$ | $0.953^{\pm 0.008}$ |
| Donor | F1 $\uparrow$ | $0.906^{\pm 0.027}$ | $0.963^{\pm 0.006}$ | $0.985^{\pm 0.022}$ | $0.949^{\pm 0.024}$ | $0.948^{\pm 0.025}$ | $0.874^{\pm 0.289}$ | $0.962^{\pm 0.004}$ | $0.951^{\pm 0.005}$ |

Table: Table A2: Downstream adaption of DNA language models on the Genomics Benchmarks. The Top-1 accuracy ($\uparrow$) for RC-equivariant PS and PH xLSTM and Mamba-based Caduceus models, both with 500k parameters, are shown. Error bars represent the range of scores across five random seeds. xLSTM achieves comparable overall performance to Mamba and demonstrates superior accuracy when both models employ post-hoc conjoining. Scores for all models except xLSTM were obtained from Schiff et al., ().

### C.3 Architecture and Hyperparameters

The hyperparameters and composition of the DNA-xLSTM and DNA-Transformer++ models for pre-training with context size 1k and 32k are reported in Tables A4 and A3.
The hyperparameters were selected on a separate validation set using manual hyperparameter selection due to limited computational resources.

**Table A3: Pre-training hyperparameters for DNA-Transformer++ models with 2M and 4M parameters. Comma-separated values represent hyperparameter sweeps, with the chosen values indicated in bold.**
| Hyperparameters | DNA-Transformer++-2M | DNA-Transformer++-4M |
| --- | --- | --- |
| Embedding Dimension | 256 | 256 |
| Number of Blocks | 4 | 6 |
| Number of Heads | 8 | 8 |
| Up-Projection Ratio | 1.25:1 | 2:1 |
| Norm Bias and Linear Bias | false | false |
| Context Length | 1,024 | 32,768 |
| Position Embeddings | RoPE | RoPE |
| Learning Rate | 6e-3, 8e-3, 1e-2 | 6e-3, 8e-3, 1e-2 |

### C.4 Reverse-Complement Invariance

We develop an xLSTM version that is invariant
to the RC of an input sequence which
is relevant for DNA applications following Schiff et al., ().
In double-helix DNA structures, both strands are semantically equivalent,
as one strand is the RC of the other.
Given a strand, $\square$, its RC, $\overline{\square}$, is
oriented in the opposite direction with a base conversion from A to
T and C to G (Schiff et al.,, ).
Shrikumar et al., () show that a data-driven approach to learning the equivalence between reverse-complement sequences can fail, which is why Schiff et al., ()
propose to enforce RC-equivariance by design,
making use of two different inductive
biases in the model architecture: PH (Zhou et al.,, ) and PS.
For PH models,
sequence-to-sequence models — in our case realized by the xLSTM —
learn to handle both DNA sequences and their RC during pre-training by applying RC augmentation to the inputs.
RC augmentation refers to the process of randomly replacing input sequences by their RCs.
For downstream tasks PH models are applied once to the original sequence and once to the RC and eventually outputs are summed:

$$ $\displaystyle\bm{Y}$ $\displaystyle=\mathrm{xLSTM}(\bm{X})+\mathrm{xLSTM}(\overline{\bm{X}}).$ (24) $$

For PS models — we assume models are realized by xLSTM architectures and therefore a $\mathrm{block}$ refers to a single mLSTM or sLSTM block — both the DNA sequence and its RC are provided simultaneously to each block in the architecture (for both pre-training and downstream task fine-tuning). Precisely, a joint representation, originating from combining a sequence representation and its RC representation, is split into $\bm{X}\in\mathbb{R}^{D\times t}$ and $\overline{\bm{X}}\in\mathbb{R}^{D\times t}$ and fed into the mLSTM or sLSTM block:

$$ $\displaystyle\left[\bm{H},\overline{\bm{H}}\right]$ $\displaystyle=\left[\mathrm{block}(\bm{X}),\mathrm{RC}(\mathrm{block}(\mathrm{RC}(\overline{\bm{X}})))\right].$ (25) $$

Notably, for each block the reverse-complement input is built by the $\mathrm{RC}$-function which flips both dimensions of $\overline{\bm{X}}$ and $\left[\cdot,\cdot\right]$ indicates concatenation along the first dimension. Eventually, logits for the input sequence and its reverse complement are combined. For more details, we refer to Schiff et al., ().

Figure: Figure A3: Pre-training of RC-equivariant xLSTM-DNA-PS and Caduceus-PS models with 500k and 2M parameters trained on the human reference genome. Models were trained on 1k context windows using the MLM objective. Left: MLM losses ($\downarrow$) for models with 500k parameters. Right: MLM losses ($\downarrow$) for models in the 2M parameter range. DNA-xLSTM-PS outperforms Caduceus-PS in both settings, with the performance gap widening at larger scales.
Refer to caption: x10.png

### C.5 Implementation Details

For both CLM and MLM pre-training we perform 10,000 update steps holding the number of tokens per step constant at $2^{20}$. CLM models are trained using autoregressive next-token prediction. For MLM pre-training, we follow the methodology presented by Devlin et al., (), where 15% of the input tokens are masked and the model is tasked to predict the corrupted tokens. Concretely, 80% of the masked tokens are replaced by a special [MASK] token, 10% are replaced by random tokens sampled from the vocabulary and 10% remain unchanged. For MLM settings, we use weight-tied bidirectionality as a default (see Section B.3). For long-context bidirectional modeling, we use unidirectional xLSTM cells and alternate the modeling direction at each block.

To fine-tune pre-trained models on downstream tasks, we follow the framework from Schiff et al., (). Pre-trained models are augmented with a task-specific classification head, which is trained on average-pooled activations from a model’s final block. During fine-tuning, all model parameters are unfrozen. For the Genomic benchmark, we perform five randomly seeded train-validation splits, fine-tune models for 10 epochs, and use early-stopping on validation performance. Final test results are reported as the mean performance $\pm$ max/min over the 5 seeds on a held-out test set. For the Nucleotide Transformer tasks, we use 20 epochs and 10 seeds. For both the Genomic benchmarks and the Nucleotide Transformer tasks, we performed a hyperparameter search for both PH and PS models over batch sizes $\{64,128,256,512\},$ and learning rates $\{\text{4e-4},\text{6e-4},\text{8e-4},\text{1e-3},\text{2e-3}\}$. The best results for each Nucleotide Transformer task can be found in Table A5.

**Table A4: Pre-training hyperparameters of DNA-xLSTM Models from 500k to 4M parameters. Comma-separated values represent hyperparameter sweeps, with the chosen values indicated in bold.**
| Hyperparameters | DNA-xLSTM-500k | DNA-xLSTM-2M | DNA-xLSTM-4M |
| --- | --- | --- | --- |
| Embedding Dimension | 128 | 256 | 256 |
| Number of Blocks | 5 | 6 | 9 |
| Conv 1D Kernel Size | 4 | 4 | 4 |
| Number of Heads | 4 | 4 | 4 |
| Up-Projection Ratio | 1.25:1 | 1:1 | 2:1 |
| Bidirectionality | alternating, blockwise | alternating, blockwise | alternating, native, blockwise |
| Norm Bias and Linear Bias | true, false | true, false | true |
| QKV Projection Blocksize | - | - | 4 |
| m/sLSTM ratio | [0:1], [1:0] | [0:1], [1:0] | [0:1], [1:0] |
| Context Length | 1,024 | 1,024 | 32,768 |
| Position Embeddings | None | None | RoPE |
| Optimizer | AdamW $\beta=(0.9,0.95)$ | AdamW $\beta=(0.9,0.95)$ | AdamW $\beta=(0.9,0.95)$ |
| Learning Rate | 6e-3, 8e-3, 1e-2 | 6e-3, 8e-3, 1e-2 | 6e-3, 8e-3, 1e-2 |
| Learning Rate Schedule | Cosine Decay | Cosine Decay | Cosine Decay |
| Learning Rate Warmup Steps | 1,000 | 1,000 | 1,000 |
| Weight Decay | 0.1 | 0.1 | 0.1 |
| Dropout | 0 | 0 | 0 |
| Batch Size | 1,024 | 1,024 | 32 |
| Update Steps | 10,000 | 10,000 | 10,000 |

**Table A5: Hyperparameter selection for DNA-xLSTM-PS and DNA-xLSTM-Ph on Nucleotide Transformer tasks. Fine-tuning hyperparameters were chosen based on best scores averaged over ten train-validation splits.**
|  | DNA-xLSTM-Ph | DNA-xLSTM-PS |  |  |
| --- | --- | --- | --- | --- |
|  | Learning Rate | Batch Size | Learning Rate | Batch Size |
| Histone Markers |  |  |  |  |
| H3 | 8e-4 | 128 | 4e-4 | 64 |
| H3K14AC | 6e-4 | 128 | 4e-4 | 64 |
| H3K36ME3 | 6e-4 | 64 | 4e-4 | 64 |
| H3K4ME1 | 8e-4 | 128 | 1e-3 | 128 |
| H3K4ME2 | 6e-4 | 64 | 2e-3 | 512 |
| H3K4ME3 | 8e-4 | 128 | 1e-3 | 512 |
| H3K79ME3 | 1e-3 | 128 | 4e-4 | 64 |
| H3K9AC | 4e-4 | 64 | 1e-3 | 128 |
| H4 | 8e-4 | 64 | 6e-4 | 64 |
| H4AC | 4e-4 | 64 | 1e-3 | 128 |
| Regulatory Annotation |  |  |  |  |
| Enhancers | 2e-3 | 512 | 2e-3 | 512 |
| Enhancers Types | 2e-3 | 512 | 2e-3 | 512 |
| Promoter All | 4e-4 | 64 | 1e-3 | 128 |
| Promoter No TATA | 1e-3 | 128 | 1e-3 | 128 |
| Promoter TATA | 3e-3 | 128 | 1e-3 | 128 |
| Splice Site Annotation |  |  |  |  |
| Splice Sites All | 8e-4 | 64 | 2e-3 | 128 |
| Splice Sites Acceptor | 2e-3 | 128 | 2e-3 | 128 |
| Splice Sites Donors | 3e-3 | 128 | 2e-3 | 128 |

## Appendix D Prot-xLSTM: Details and Additional Results

This section provides further details regarding the
architecture, training setup, and evaluation metrics for the Prot-xLSTM models. Additionally, we present supplementary results that complement the main findings discussed in Section 4.2. We adopted the experimental protocols outlined in Sgarbossa et al., () to train and evaluate our Prot-xLSTM models. We conducted three key experiments to assess the models’ capabilities: a) protein language modeling (Section D.1), b) homology-conditioned protein design (Section D.2), and c) protein variant fitness prediction (Section D.3).

### D.1 Homology-Aware Training

Data.
The protein language model training data was derived from the filtered OpenProteinSet (Ahdritz et al.,, ), comprising 270k UniClust30 MSA clusters that included a total of 508M sequences and 110B residues. We used the ProtMamba pipeline to construct the training data, which is illustrated in Figure 1 of Sgarbossa et al., () and involved two key steps: (i) creating homology-aware but alignment-free training inputs by concatenating unaligned homologous sequences, and (ii) masking patches of tokens in each sequence and concatenating the unmasked patches at the end of each sequence to train the model autoregressively with the FIM strategy. We also use the train, validation (192 clusters), and test (500 clusters) split provided by ProtMamba.

Methods and hyperparameters.
We trained two versions of the model: Prot-xLSTM-26M and Prot-xLSTM-102M. The larger model was designed to match the architecture and scale of the original ProtMamba model (ProtMamba-107M) in terms of layer count and embedding size. We optimized the Prot-xLSTM architecture, including block types and positional encodings, on the smaller Prot-xLSTM-26M model, and then applied these optimized architectural choices to the larger Prot-xLSTM-102M model. For comparison, we also trained a smaller ProtMamba model (ProtMamba-28M with an embedding dimension of 512) and implemented a LLaMA-based model (Prot-Transformer++-26M) (Touvron et al.,, ). Both models incorporate Absolute Positional Encodings (AbsPE) as implemented in ProtMamba for xLSTM blocks, with RoPE applied specifically to the mLSTM blocks. As the sLSTM blocks (or Mamba) lack a $QK$-formulation, RoPE cannot be directly applied to them. The results of the hyperparameter search are provided in
The results of the hyperparameter search are reported in Table A7, and the composition of the Prot-xLSTM and Prot-Transformer++ models are reported in Table A6.

**Table A6: Hyperparameter space considered for the Prot-xLSTM and Prot-Transformer++ at different sizes. The selected values are marked in bold.**
| Hyperparameter | Prot-xLSTM-26M | Prot-xLSTM-102M | Prot-Transformer++-26M |
| --- | --- | --- | --- |
| Embedding dimension | 512 | 1024 | 512 |
| Context length | 2^11,2^17^a | 2^11-17^a | 2^11 |
| Number of blocks | 16 | 16 | 6 |
| m/sLSTM ratio | [0:1], [1:0], [1:7]^b | [1:0] | - |
| Conv 1D kernel size | 4 | 4 | - |
| QKV projection blocksize | 4 | 4 | - |
| Number of heads | 4 | 4 | 8 |
| Up projection dimension | 1024 | 2048 | 2176 |
| Norm bias and linear bias | False | False | False |
| Position embeddings | -, AbPE, RoPE | RoPE | RoPE |

**Table A7: Prot-xLSTM hyperparameter search: Training loss comparison across different protein language model architectures after 4B training tokens.**
| Model type | #p (M) | Positional Encodings | Train loss |
| --- | --- | --- | --- |
| Mamba | 27.7 | AbPE | 2.623 |
| Transformer++ | 26.4 | RoPE | 2.568 |
| sLSTM | 25.8 | - | 2.694 |
| 26.3 | AbPE | 2.688 |  |
| mLSTM | 25.9 | - | 2.569 |
| 26.4 | AbPE | 2.545 |  |
| 25.9 | RoPE | 2.524 |  |
| 102 | RoPE | 2.482 |  |
| xLSTM | 25.9 | - | 2.554 |
| 26.4 | AbPE | 2.551 |  |

Training details.
We trained our models using the ProtMamba pipeline with CLM with the FIM strategy.
The pipeline efficiently handles long, concatenated sequences by extending the context length up to $T=2^{18}$, supported by a context-length scheduling strategy. For the Prot-xLSTM-102M model, we adhered to the ProtMamba protocol, gradually increasing the context length from $2^{11}$ to $T=2^{18}$, doubling $T$ at each stage when the loss plateaued. In contrast, for the smaller models (Prot-xLSTM-26M and ProtMamba-28M), as recommended in previous work (Devlin et al.,, ; Press et al.,, ), we initially trained with $T=2^{11}$ for 20B tokens, then switched to $T=2^{17}$ for an additional 10B tokens. Due to the quadratic scaling of Transformer architectures, Prot-Transformer++-26M was only trained with $T=2^{11}$, as it could not handle the computational demands of $T=2^{17}$. Given the substantial computational resources required, we did not fine-tune the training parameters. Instead, we used the default settings established by ProtMamba, which are reported in Table A8.

Table: Table A8: Hyperparameters for training protein sequence models.

Metrics. During training, we evaluated the next-token prediction capabilities of the models using negative log-likelihood and token perplexity.
The perplexity was calculated for four subset of tokens of the concatenated-FIM sequence: the first protein sequence, the last protein sequence the FIM token and the entire concatenated sequence.
Once the models were trained we evaluated their performance on the independent test set.

### D.2 ICL: Homology-Conditioned Protein Generation

Experimental setup. To evaluate the capacity of Prot-xLSTM to autoregressively generate novel protein sequences given a context of known homologs, we follow the protocol outlined in Section 3.4 of Sgarbossa et al., (). For a subset of 19 homology clusters from the test set, we generate sequences with contexts consisting of 10, 100, 500, 1000, and $N$ (total number of sequences in the cluster) sequences. For each context length, we generate 100 sequences each with the following parameter combinations of generation temperature ($\tau$), $\text{top-}k$, which restricts the output selection to the $k$ most probable tokens, and $\text{top-}p$, which limits the output to tokens reaching a cumulative probability $p$: $(\tau,\text{top-}k,\text{top-}p)\in\{(0.8,10,0.9),(0.9,10,0.95),(1,10,0.95),(1,10,1),(1,15,1)\}$ (Ferruz et al.,, ). This results in a total of 2,500 sequences per cluster.

Methods compared. We compare both Prot-xLSTM models to ProtMamba models with a similar number of parameters.

Metrics. We evaluate the novelty of generated sequences by calculating the Hamming distance to the closest natural sequence in the cluster using pairwise Smith-Waterman alignment. Additionally, we measure sequence similarity to homologs with the HMMER score from a Hidden Markov Model (HMM) trained on the cluster’s MSA. The generated sequences are also folded using ESMFold (Lin et al.,, ) and assessed by pTM and average pLDDT confidence scores. To compare these scores with natural sequences, we compute Kolmogorov-Smirnov test statistics between the scores of 100 natural sequences and the 100 generated sequences with the lowest perplexity.

Results. Figure A4 displays the distribution of scores for 100 randomly sampled natural sequences from each cluster as well as the 100 sequences with the lowest perplexity generated by Prot-xLSTM and ProtMamba models for 10 randomly selected clusters. The averages across all 19 evaluated test clusters are shown in Table A9. Sequences generated by Prot-xLSTM-102M were on average longer, more similar to other proteins in the cluster (measured by Hamming distance), and got a higher HMMER score and higher folding confidence scores compared to ProtMamba-generated sequences. Notably, these observations mostly also hold when compared to natural sequences.

Figure: Figure A4: Scores of natural and generated proteins. Boxplots of score distributions (sequence length, Hamming distance to the closest natural neighbor, HMMER score, pLDDT, and pTM) for 10 randomly selected clusters evaluated for 100 randomly chosen natural sequences and 100 generated sequences with lowest perplexity values for large Prot-xLSTM and ProtMamba models.
Refer to caption: x12.png

Table: Table A9: Score comparison of natural and generated proteins. Average scores (sequence length, Hamming distance to the closest natural neighbor, HMMER score, pLDDT, and pTM) across 19 test clusters for sequences generated with Prot-xLSTM and ProtMamba models. Error bars indicate 95% confidence intervals across clusters.

Table A10 demonstrates that Hamming distance, HMMER score, pTM, and pLDDT correlate well with sequence perplexity for both, Prot-xLSTM and ProtMamba, with an average Pearson correlation coefficient of 0.57 and 0.58, respectively, for the large models.

Table: Table A10: Score distribution comparison of natural and generated proteins. Average Pearson correlation between model perplexity and sequence scores (sequence length, Hamming distance to the closest natural neighbor, HMMER score, pLDDT, and pTM) for sequences generated with Prot-xLSTM and ProtMamba models. Error bars indicate 95% confidence intervals across 19 test clusters.

### D.3 Protein Variant Fitness Prediction

Experimental setup.
We evaluate Prot-xLSTM’s ability to predict mutational effects by leveraging its inpainting capabilities from the FIM training objective. This assessment follows the protocol described in Section 3.2 of Sgarbossa et al., () for the ProteinGym DMS substitution benchmark (Notin et al.,, ), which consists of 217 datasets of single and multiple substitutions in protein sequences, allowing comparison with state-of-the-art methods for protein variant fitness prediction.
Briefly, for each wild-type sequence, three sets of 200 homologs were obtained by subsampling MSAs following the ColabFold protocol (Mirdita et al.,, ) to be used as context. The context sequences are ordered from the least similar to the most similar one. The wild-type sequence is then concatenated with the context, the mutated residue is masked, and this residue is predicted using the FIM method. Fitness is evaluated as the difference in likelihood between the concatenated sequence with the wild-type and the mutated amino acid and averaged over the triplicate. For multiple mutations, fitness is approximated as the sum of the likelihoods of single mutations.

Results. ProteinGym’s main metric is the average Spearman correlation between the fitness predictions and the experimental DMS results.
Table A11 summarizes reports ProteinGym’s main metric, the average Spearman correlation between the fitness predictions and the experimental DMS results, for Prot-xLSTM and several other well-known protein models and the current top of the

**Table A11: ProteinGym zero-shot DMS substitution benchmark. The average Spearman correlation ($\uparrow$) between predicted fitness scores and experimental measures over 217 DMS assays is shown. While even small Prot-xLSTM models already yield high scores, models which use additional structure tokens, such as SaProt and ProSST, perform best.**
| Model Type | Model | Reference | #Params | Spearman $\rho$ |
| --- | --- | --- | --- | --- |
| Alignment-based | Site-Independant | Hopf et al., () | - | 0.359 |
|  | EVE | Frazer et al., () | -^a | 0.432 |
|  | GEMME | Laine et al., () | - | 0.455 |
| Protein language model | Tranception L (w/o R) |  | 700M | 0.374 |
| (PLM) | VespaG | Marquet et al., () | 3B | 0.458 |
|  | ProGen2 XL | Nijkamp et al., () | 6B | 0.391 |
|  | ESM-2 | Lin et al., () | 15B | 0.401 |
| Alignment + PLM | MSA-Transformer | Rao et al., () | 100M | 0.432 |
|  | Tranception L (w/ R) |  | 700M | 0.434 |
|  | TranceptEVE L |  | >700M^a | 0.456 |
| Homology-aware PLM | Prot-xLSTM | Ours | 26M | 0.411^b |
|  | ProtMamba | Sgarbossa et al., () | 28M | 0.360^b |
|  | Prot-xLSTM | Ours | 102M | 0.416^b |
|  | ProtMamba | Sgarbossa et al., () | 107M | 0.415^b |
|  | PoET | Truong Jr and Bepler, () | 201M | 0.470 |
| Inverse folding | ESM-IF1 | Hsu et al., () | 142M | 0.422 |
| Structure + PLM | SaProt |  | 35M | 0.407 |
|  | ProSST | Li et al., () | 110M | 0.507 |
|  | SaProt |  | 650M | 0.457 |

## Appendix E Chem-xLSTM: Details and Additional Results

For chemical sequences, we perform two sets of experiments: a) unconditional molecule generation
where we follow the experimental protocol of Özçelik et al., ().
Additionally, we propose a new and more challenging task:
b) conditional generation with ICL, in which we generate new compounds conditional based on provided in-context compounds.

### E.1 Unconditional Molecule Generation

*Unconditional molecule generation* is the task of generating valid small molecules without imposing constraints on their characteristics or properties.
Generative models aim to learn a general distribution by processing many examples of desirable results.
To this end, models are trained on large training sets of arbitrary small molecules without particular conditions or constraints (Segler et al.,, ; Gómez-Bombarelli et al.,, ; Özçelik et al.,, ).
Following this approach, we compared the ability of xLSTM and several other models to generate valid and diverse molecules.

Experimental setup.
For comparability, we aligned our experiments with the setting and dataset of Özçelik et al., ().
This means that all models are trained to generate molecules as SMILES strings (Weininger,, ) using a CLM paradigm.
The dataset used in (Özçelik et al.,, ) is derived from ChEMBL with a random split in 1.9M training, 100k validation, and 23k test molecules, which have been encoded as SMILES.
Before training, all SMILES strings were tokenized using a regular expression, containing all elements. This results in atoms being represented as one token as well as additional SMILES symbols.

Methods and hyperparameters.
We compared xLSTM with several other model classes.
The first baseline is the default LSTM (Hochreiter and Schmidhuber,, ) in PyTorch, which includes a forget gate (Gers et al.,, ).
This can be considered the direct predecessor of the xLSTM architecture.
We also included a variant GPT-2 (Radford et al.,, ) model based on the Transformer architecture (Vaswani et al.,, ) with causal masking.
Finally, we included two SSMs in our comparison.
On one side, we considered an S4 model with the implementation from Gu et al., (), following (Özçelik et al.,, ).
On the other side, we incorporated a Mamba model, using the official repository provided with (Gu and Dao,, ).
For our Chem-xLSTM, we used an xLSTM using only mLSTM blocks (Beck et al.,, ).
The 15M-parameter model consists of 9 layers with a hidden dimension of 512 and 8 heads.
We trained the model for up to 100 epochs with a batch size of 1,024, a context length of 100, a dropout rate of 0.25, and a learning rate of 0.005.
All models were trained using the Adam optimizer (Kingma and Ba,, ) using $\beta=(0.9,0.999)$, $\epsilon={1e^{-8}}$, and a learning-rate schedule with warm-up and cosine decay.
We selected the best model based on the minimum validation loss observed at the end of each epoch.
The hyperparameters were manually tuned to match the model parameter count for a fair comparison.
Detailed training information and learning curves can be found in Appendix Section E.4.

Metrics.
We evaluated each model with the next token perplexity next token, and the FCD (Preuer et al.,, ).
The FCD has been introduced as an alternative to the FID, which is used to evaluate image generation, for molecule generation.
Auxiliary metrics that measure the syntactic correctness, novelty, diversity, or synthetic accessibility are reported in Appendix Section E.

Results.
Our proposed Chem-xLSTM model achieved the best results, with the lowest FCD (0.13) and a perplexity (1.68) that is competitive with that of GPT-based models.
This indicates that Chem-XLSTM is able to generate realistic chemical structures that match the target distribution well.

All models in our comparison were able to produce valid, unique, and novel molecules.
Even though these models have not been optimized for these properties.
This is evidenced by the auxiliary metrics surpassing practical thresholds (see Table A13).

### E.2 Conditional Molecule Generation with In-Context Learning

Conditional molecule generation with in-context learning (ICL) leverages contextual information to guide the design of novel molecules tailored for specific domains. By incorporating a sequence of molecules as the input, models can conditionally generate new compounds of the same distribution, without the need for fine-tuning.

Experimental setup.
Similar to the unconditional setup, the input consists of SMILES strings.
In the conditional setup, we additionally model sets of molecules
from the same molecular domain as a sequence.
Molecules from one molecular domain are serialized and concatenated,
separated with the "." token.
During training, the order of the molecules is permuted to improve generalization and robustness.
We construct a novel dataset derived from a variety of molecular domains:

- •
We consider natural-products as domain and utilize the Coconut (Nainala et al.,, ) as source dataset.
- •
Kinase inhibitors,
withdrawn,
malaria,
tool compounds,
pathogen,
NIH mechanistic,
lopac,
natural product-based probes and drugs,
zinc tool,
axon medchem,
adooq bioactive,
novartis chemogenetic,
drug matrix,
PROTACs,
covalentIn db,
DrugBank compounds,
reframe,
cayman bioactive all from the Probes & Drugs portal (Skuta et al.,, ),
- •
product molecules from the reaction dataset USPTO-50k (Lowe,, ) split into 10 reaction classes.
- •
The domains bio,
diversity, green, yellow,
orange, and red,
from ZINClick (Levré et al.,, ).
- •
Active molecules from
the domains BACE, BBBP,
Clintox, HIV, SIDER,
Tox21, Tox21-10k, and Toxcast
from MoleculeNet (Wu et al.,, ).
- •
Active molecules from 95 bioassays from FS-MOL (Stanley et al.,, ) considered each as separate domain.
- •
Active molecules from 109 bioassays from PubChem (Kim et al.,, ) considered each as separate domain.
- •
A subset of active molecules from the BELKA challenge (Quigley et al.,, ) is modeled as a domain.

For the domains that are defined by the active molecules from a particular
bioassay, we selected assays with at least 300 active molecules
and only use the active compounds. For the dataset each of the total 249 domains is
limited to 100,000 compounds, where compounds are selected at random.
The final dataset is split at 8:1:1 into train-, validation- and test-domains, sorted by their character length in descending order.

Methods and hyperparameters.
We benchmark and orient our choices for the model classes as well as hyperparameters based on the unconditional molecule generation results, We consider a context length of 4,096 and adjust batch sizes as well as accumulation steps to accommodate GPU memory constraints.
For the S4 model, we were only able to fit a context length of 2,048.

Metrics.
To evaluate conditional molecule generation we evaluate NTP loss. This metric quantifies how well the model predicts the next token in a sequence, thus assessing whether a model is able
to generate molecules from an unseen, and potentially very special,
molecular domain given only a few molecules from that domain.

### E.3 Architecture and Hyperparameter Selection

Considered and selected hyperparameters for Chem-xLSTM
are given in A12.

Table: Table A12: Hyperparameter space considered for the Chem-xLSTM at different sizes. The selected values are marked in bold.

### E.4 Implementation Details

Unlike Özçelik et al., (), we do not backpropagate the loss for [PAD] tokens, nor do we interpret them for decoding. We observed that not ignoring [EOS] and [PAD] token leads to more diversity but is not the standard way of decoding in e.g. NLP. Padding tokens are not typically generated during decoding. They are primarily a pre-processing step to handle batches of data efficiently. In our implementation, we end decoding the SMILES string with the [EOS] token. Further, we do not use SMILES augmentation, which could further
improve the performance of all architectures.

### E.5 Additional Results

Practical thresholds are defined based on several key metrics. First, a high percentage of generated SMILES strings must correspond to chemically valid molecules, with a threshold typically set above 90% to ensure reliability. Additionally, a practical threshold for uniqueness might require that over 80% of the generated molecules are unique, ensuring diversity in the explored chemical space. For novelty, at least 50-70% of the generated molecules should be novel compared to known chemical databases, indicating the model’s ability to explore new regions of chemical space. Finally, all models exhibit favorable synthetic accessibility (SA) scores, typically ranging between 2.5 and 5, ensuring that the generated molecules are feasible for synthesis. Further metrics and details are provided in the appendix.

**Table A13: Diversity and correctness metrics for the 15M parameter models for small molecules (SMILES). The table reports the percentage of valid, unique, and novel molecules, the synthetic accessibility (SA), and the diversity metric by the percentage of unique Murcko scaffolds divided by the total number of generated molecules.**
| Model | valid % | unique % | novel % | SA ↓ | diverse % |
| --- | --- | --- | --- | --- | --- |
| SMILES-LSTM (Segler et al.,, ) | ${90.11^{\pm{10.7}}}$ | ${56.72^{\pm{3.4}}}$ | ${56.66^{\pm{3.6}}}$ | ${2.85^{\pm{0.0}}}$ | ${44.71^{\pm{1.1}}}$ |
| SMILES-GPT (Adilov,, ) | ${99.05^{\pm{0.5}}}$ | ${62.09^{\pm{12.1}}}$ | ${61.81^{\pm{12.0}}}$ | ${2.90^{\pm{0.0}}}$ | ${48.82^{\pm{9.7}}}$ |
| SMILES-S4 (Özçelik et al.,, ) | ${97.48^{\pm{0.0}}}$ | ${61.47^{\pm{0.0}}}$ | ${61.34^{\pm{0.0}}}$ | ${2.86^{\pm{0.0}}}$ | ${48.49^{\pm{0.0}}}$ |
| Chem-Mamba^a | ${91.41^{\pm{8.9}}}$ | ${57.75^{\pm{3.2}}}$ | ${57.63^{\pm{3.8}}}$ | ${2.84^{\pm{0.0}}}$ | ${45.65^{\pm{7.2}}}$ |
| Chem-xLSTM (ours) | ${97.08^{\pm{0.7}}}$ | ${61.09^{\pm{8.9}}}$ | ${60.84^{\pm{9.6}}}$ | ${2.83^{\pm{0.0}}}$ | ${45.97^{\pm{5.5}}}$ |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - Do the authors have any explanations why Bio-xLSTM is strong on the histone tasks, but outperformed by NT-v2 500M on the regulatory annotation and splice site annotation tasks (Table A1)? Is there a general difference between the tasks that makes it harder to model it with xLSTMs?

- Did the authors generally observe any patterns where the xLSTM approach is beneficial, and where it might be less effective?
soundness: 4
strengths: - The usage of xLSTMs in the biological domain is a reasonable approach, particularly in the field of DNA where long context appears to be of importance.
- The paper is well-written, the evaluations and benchmarking is strong and includes reasonable baselines/competitors.
- The overall performance of Bio-xLSTM is strong and Bio-xLSTM is a valuable model for future research in the field.
summary: The paper introduces Bio-xLSTM, a set of models that is tailored towards modeling of biological sequences. Specifically, the authors apply xLSTMs to different tasks of DNA, protein and small molecular modeling, analyzing the capabilities of xLSTMs in these domains compared to transformer and state-space models. Bio-xLSTM shows remarkable performance across all three domains and achieves state-of-the-art performance at homology aware protein generation.
weaknesses: - I’m missing an evaluation of the diversity of the generated proteins and small molecules. 
- Regarding small molecule generation, I do not really see the benefit of large contexts. Specifically, the unconditional molecule generation uses a context size of 100 tokens. Maybe the authors could explain why the usage of xLSTMs should be beneficial in this domain (I see the point of ICL here, but for unconditional generation there doesn't seem to be any advantage, or?).

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: “While Transformers have yielded impressive results, their quadratic runtime dependency” - In theory this is true, but it practice, there are more efficient implementations, such as Longformer (Beltagy 202), Linformer (Wang 2020), etc.

For the DNA-xLSTM tasks, to what degree do the PH or PS architecture additions cause a benefit vs just a xLSTM model? Same with the Mamba models (Table 1).

What are the spikes in the validation loss in Figure 3?

For Table 3, I’m always curious about an HMM baseline. How would a HMM of that cluster perform? It’s most certainly going to have many many fewer parameters.
soundness: 3
strengths: Generally I think the paper is well written, and the relevant baselines and comparisons across all domains are there.

I think the DNA-xLSTM tasks and benchmarks presented in Table 1 are strong.

Generally, I think the models (parameter sizes, configurations, training data) are comparable across baselines.
summary: The authors adapt the xLSTM model architecture to the DNA, protein, and chemical informatics space. They compare with SoA models and benchmarks.
weaknesses: Generally, I feel like this paper is a bit of a grab-bag of computational biology. While the xLSTM architecture is consistent, the additional model additions are quite varied and bespoke.

I’m not sure the large blocks of sLSTM and mLSTM math contribute much to this paper. Do you ever refer back to these equations later in the work?

Typo in header 3 - “BIO-XLSTM: LONGE-RANGE MODELING OF BIOLOGICAL AND CHEMICAL SEQUENCES”

“Hamming distance, HMMER score, and structural scores correlate well with sequence perplexity, with an average absolute Pearson correlation of 0.57 across clusters for the large Prot-xLSTM model” I would not say those correlate well. The R**2 is approx 0.325, which is not a lot of variance explained. Moreover, are these distributions normal? Should you be using spearman?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The main motivation for the choice of LSTM is their favorable scaling. There is no scaling analysis in the paper, that clarifies the factual scaling and the observed prefactors. Even with fundamentally improved scaling, prefactors can be prohibitively unfavorable.

The authors provide performance measures in the form of averages and error bars over an ensemble of runs, which is great. At this point, however, it is more meaningful to test if the distributions of performance results are distinguishable instead of comparing averages. The authors do not have to do this exhaustively, but at least for the cases when their models are claimed to be outperforming the alternatives.
soundness: 3
strengths: The paper explores alternatives to the modeling sequential molecular representations in chemistry that are expected to improve scaling and memory requirements. This is a great motivation, considering a) footprint of generative computing and b) its accessibility.

The authors discuss how to tailor the described architecture in relevant tasks.
summary: The contribution describes application of an LSTM architecture to the modeling of sequential representation of molecules (biomolecules). The motivation for this choice of the architecture is to overcome quadratic scaling of transformers and offer linear scaling and constant memory requirements decoding. Three models are introduced as components of bio-xLSTM suite, specialized in modeling sequential representations of small molecules, proteins, and DNA. The reported results demonstrate performance improvement in certain tasks compared to the alternatives.
weaknesses: The general weakness is that the paper contributes to an oversaturated field but does not offer any breakthrough. The case for LSTM is made by the appeal to their compute requirements, which are not discussed in terms of factual requirements of this study concerning memory bottlenecks, prefactors, scaling, etc. If all the reported results are produced on the same computing resource and performance improvement with LSTM  is marginal, why bother?

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Could you elaborate on the specific motivations behind choosing xLSTM over other SSM models? What unique advantages or improvements does xLSTM offer in the context of biological sequence design presented in this paper?
2. The performance improvements demonstrated in the paper may not be solely attributable to the xLSTM architecture itself. To convincingly argue that the enhancements are a result of the architecture and not just parameter tuning, the authors could provide a more detailed ablation study.
soundness: 2
strengths: The paper introduces the novel xLSTM architecture, demonstrating its versatility in biological sequence modeling.It incorporates homology awareness in Prot-xLSTM and reverse-complement equivariance in DNA-xLSTM, addressing key challenges in protein and DNA sequence analysis.
summary: This paper presents the Bio-xLSTM framework, which encompasses three specialized models for distinct biological sequences: DNA-xLSTM, Prot-xLSTM, and Chem-xLSTM. DNA-xLSTM introduces reverse-complement equivariant blocks, essential for capturing the symmetry in DNA sequences. Prot-xLSTM is a homology-aware protein language model that employs in-context learning, addressing the variability in protein sequence lengths and context sizes. Chem-xLSTM is designed for SMILES representations of small molecules. Experiments demonstrate that the Bio-xLSTM framework is proficient in generative modeling and representation learning for DNA, proteins, and small molecules.
weaknesses: 1. **Lack of Motivation**: The paper fails to provide a compelling justification for the use of xLSTM over existing SSM models like Mamba. The authors do not clearly articulate why xLSTM is a better choice for the tasks at hand, which leaves readers without a clear understanding of the advantages it offers over established models.

2. **Lack of Novelty**: The concept of reverse-complement equivariance and the application of post-hoc conjoining (PH) and parameter sharing (PS) within the model architecture have been previously discussed in the literature, as seen in the Caduceus model (Schiff et al., 2024). This reduces the perceived novelty of the current work and may be perceived as a weak contribution to the field.

3. **Results Lack Convincing Evidence**: The manuscript's results section lacks the depth required to fully convince the reader of the proposed xLSTM models' effectiveness. An ablation study would significantly strengthen the paper by demonstrating the impact of the xLSTM components on performance, which is currently not provided.

4. **Redundant Writing**: The paper repetitively introduces the xLSTM architecture and training stategies, which has already been described in the original paper. This redundancy is unnecessary and detracts from the focus on the paper's new contributions and findings.

Meta-review: This paper introduces Bio-xLSTM, a set of models leveraging the xLSTM architecture for modeling biological and chemical sequences across three domains: DNA, proteins, and small molecules. The architecture addresses limitations of Transformers in terms of quadratic scaling and memory efficiency, offering linear runtime dependency and constant-memory decoding. The authors adapt the architecture with domain-specific enhancements, such as reverse-complement equivariance for DNA and homology awareness for proteins. Evaluations demonstrate competitive performance in generative modeling and representation learning tasks.

Decision: Accept (Poster)