Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences
Authors: Niklas Schmidinger
Lisa Schneckenreiter
Philipp Seidl
Johannes Schimunek
Pieter-Jan Hoedt
Johannes Brandstetter
Andreas Mayr
Sohvi Luukkonen
Sepp Hochreiter
Günter Klambauer
Venue: ICLR 2025 Poster (2025)
Keywords: xLSTM
large language model
foundation model
DNA
protein
small molecule
SMILES
in-context learning
masked language modeling
causal language modeling
fill-in the middle
equivariance
reverse complementary sequence
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Bio-xLSTM introduces a suite of xLSTM-based language models tailored for biological and chemical sequences, demonstrating proficiency in generative modeling, representation learning, and in-context learning across DNA, proteins, and small molecules.

Abstract: Language models for biological and chemical sequences enable crucial applications such as drug discovery, protein engineering, and precision medicine. Currently, these language models are predominantly based on Transformer architectures. While Transformers have yielded impressive results, their quadratic runtime dependency on sequence length complicates their use for long genomic sequences and in-context learning on proteins and chemical sequences. Recently, the recurrent xLSTM architecture has been shown to perform favorably compared to Transformers and modern state-space models (SSMs) in the natural language domain. Similar to SSMs, xLSTMs have linear runtime dependency and allow for constant-memory decoding at inference time, which makes them prime candidates for modeling long-range dependencies in biological and chemical sequences. In this work, we tailor xLSTM towards these domains and we propose a suite of language models called Bio-xLSTM. Extensive experiments in three large domains, genomics, proteins, and chemistry, were performed to assess xLSTM’s ability to model biological and chemical sequences. The results show that Bio-xLSTM is a highly proficient generative model for DNA, protein, and chemical sequences, learns rich representations, and can perform in-context learning for proteins and small molecules.

Published as a conference paper at ICLR 2025
BIO-XLSTM: GENERATIVE MODELING, REPRESENTA- TION AND IN-CONTEXT LEARNING OF BIOLOGICAL AND CHEMICAL SEQUENCES
Niklas Schmidinger'
Lisa Schneckenreiter'
Philipp Seidl'
Johannes Schimunek 1
Pieter-Jan Hoedt
Johannes Brandstetter 1.2
Andreas Mayr!
Sohvi Luukkonen'
Sepp Hochreiter Günter Klambauer 1.2
1 ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University, Linz, Austria 2 NXAI GmbH, Linz, Austria
ABSTRACT
Language models for biological and chemical sequences enable crucial applications such drug discovery, protein engineering, and precision medicine. Currently, these language models are predominantly based on Transformer architectures. While Transformers have yielded impressive results, their quadratic runtime depen- dency on the sequence length complicates their use for long genomic sequences and in-context learning on proteins and chemical sequences. Recently, the current xLSTM architecture has been shown to perform favorably compared to Transformers and modern state-space model (SSM) architectures in the natural language domain. Similar to SSMs, XLSTMs have linear runtime dependency on the sequence length and allow for constant- memoryy decoding at inference time, which makes them prime candidates for modeling long-range dependencies in biological and chemical sequences. In this work, we tailor xLSTM towards these domains and propose suite of architectural variants called Bio-XLSTM. Extensive experiments in three large domains, genomics, proteins, and chemistry, were per- formed to assess xLSTM's ability to model biological and chemical sequences. The results show that models based on Bio-XLSTM a) can serve as proficient generative models for DNA, protein, and chemical sequences, b) learn rich representations for those modalities, and c) can perform in-context learning for proteins and small molecules.
INTRODUCTION
Accurate computational models for biological sequences essential for translating data into actionable insights in modern biology. Biological sequences like DNA, RNA, and proteins are central to molecular biology, genomics, and drug discovery. Major projects like the Human Genome Project (Lander et al., 2001) and the 1000 Genomes Project (1000 Genomes Project Consortium, 2010) have driven large-scale data collection efforts. Modeling these sequences is key to advancing life sciences (Benegas et al., 2023; Karollus et al., 2024), interacting with biological systems (Hopf al., 2017: Riesselman al., 2018: Yang et al., 2019) or predicting phenotypes from genetic variants (Ashley, 2016; Brandes et al., 2023; Acosta et al., 2022). Similar efforts exist for protein sequences (The UniProt Consortium, 2023) and small molecules (Kim et al., 2023; Zdrazil al., 2023), used for tasks like protein engincering (Arnold, 2018; Yang et al., 2019), predicting 3D structures (Jumper et al., 2021), and drug discovery (Zhavoronkov et al., 2019). Large language models (LLMs) (Brown al., 2020; Bubeck al., 2023) have emerged as prime candidates for modeling biological sequences and serving as foundation models for molecular biology and chemistry (Ji et al., 2021; Schiff et al., 2024: Nguyen et al., 2023; Rives et al., 2021; Lin et al., 2023).
Large language models for biological sequences must handle long sequences and incorporate context. The rise of LLMs (Radford al., 2018; Brown al., 2020; Bubeck et al., 2023) has

---

Published as a conference paper at ICLR 2025
XLSTM
Bio-XLSTM: modeling approaches
GLITM
KLSTM
P
P
Corne sign mecleding
M
① 2 ② e
C
=Exiv WISIPEDIA
NESTM
3ESTMM
DNA-XLSTM
Prot-XLSTM
Chem-XLSTM
Blocks
Data
Blocks
Blocks
Data
M
Human Ganoma Glic38
OpenPoten
ChEMBL
Figure 1: Overview of Bio-xLSTM. Top left: xLSTM for natural language processing tasks. Top right: Considered modeling approaches for biological sequences: masked language modeling, equivariance to reverse complementary sequence, and in-context learning. Bottom left: DNA- XLSTM models are trained on genomic DNA sequences and then fine-tuned on downstream tasks. Bottom center: Prot-XLSTM models are trained in causal modeling setting with fil-iie-themmidlle objective and use homologous proteins for in-context learning. Bottom right: Chem- Xeem- models trained to generate small molecules. For an in-context learning setting, Chem-xLSTh, models use molecules with known properties as the context.
revolutionized numerous fields, including life sciences. Most LLMs are based on the Transformer architecture (Vaswani et al., 2017), which excels at predicting the next or missing token using self- attention. However, the self-attention mechanism scales quadratically with sequence length, making long-sequence modeling computationally expensive. As a result, most biological sequence models short contexts (Rives et al., 2021; Ji et al., 2021; Dalla-Torre et al., 2024). However, biological sequences require long context windows for accurate modeling because of their important long-range interactions due to 3D folding (Anfinsen, 1973), or gene regulation in DNA (Bouwman and de Laat, 2015). The human genome spans around three billion base-pairs (bps), far exceeding the context limits of Transformer-basee homologous proteins (Truong Jr and Bepler, 2023; Sgarbossa et al., 2024) and molecular context for small molecules (Papadatos et al., 2010; Schimunek et al., 2023). The emergence of state-space models (SSMs), like S4 (Gu et al., 2022), Hyena (Poli et al., 2023), and Mamba (Gu and Dao, 2024), enables handling longer contexts in biological domains (Nguyen et al., 2023; Schiff et al., 2024; Sgarbossa et al., 2024). However, the recently proposed xLSTM architecture (Beck et al., 2024), recurrent neural network, has outperformed SSM architcctures in natural language processing. For further related work, see Appendix Section A.
models. Furthermore, long contexts also benefit models that exploit
The recently proposed xLSTM is a powerful architecture for sequence modeling and a promis- ing candidate for biological and chemical sequences. The xLSTM architecture (Beck al., 2024) introduces enhanced memory structures and exponential gates that boost its performance, particularly in natural language modeling. Despite these enhancements over traditional LSTM, XLSTM retains the efficiency of a recurrent neural network and can handle varying sequence lengths effectively (Beck al., 2024), while maintaining expressivity (Merrill et al., 2024) and scalability (Katharopoulos et al., 2020; Choromanski et al., 2021). These features make xLSTM ideal for modeling: i) DNA sequences, which are inherently long and for which long-range interactions between distant parts of the sequence have been observed, ii) protein sequences, where modeling strongly benefits from contextual infor- mation of evolutionary-relan chemical sequences, such as Simplified Molecular Input Line Entry System (SMILES) (Weininger, 1988), for which in-context learning (ICL) abilities are an option to generate new molecules with desired properties or from a particular molecular domain (Segler et al., 2018; Schimunek et al., 2023). However, it remains unclear how to best tailor xLSTM for biological and chemical sequences and how XTM compares to other domain-specific L.MSs architectures.
proteins (Rives et al., 2021), and iii) small molecules represented as
2

---

Published as a conference paper at ICLR 2025
We introduce: a) DNA-xLSTM, an architectural variant tailored for DNA sequences with reverse- complement equivariant blocks, and evaluate its performance on long-contex. generative modeling. representation learning, and downstream tasks. b) Prot-xLSTMM. variant for homology-an aware protein language modeling with ICL. for both generative modeling and inpainting, which we benchmark on generative modeling, conditioned protein design and protein variant fitness prediction tasks. e) Chem-XLSTM, an architectural variant for SMILES representations of small molecules for which we demonstrate ICL capabilities. An overview of Bio-xLSTM is shown in Figure
aware
2 BACKGROUND AND NOTATION
XLSTM (Beck et al., 2024) consists of two distinct layer types: sLSTM (Section 2.1) and mLSTM (Section 2.2). These layers the components within the block structures (Section 2.3) of its multi-layer architectures. We consider a series of input vectors I; E IR given at a certain time step t (1..... T). X X 1:T (I1, L2 ZT) E RxT denotes the matrix of stacked input vectors from all time steps. Both sLSTM and mLSTM are recurrent neural networks, which either map a state (,,-1, C--1 1 to successor state (h,, Ct2 ne) given an input 2;-1 (sLSTM) or state (h:-1, Ct-1, 14-11 to successor state (h,, Cr, mt) given an input #;-1 (mLSTMM. Here, he E R. denotes hidden state, C₁ E Id and C₁ E [Rd denote cell states responsible for long-term memory and, E Rd denotes normalizer state. sLSTM and mLSTM utilize several adjustable weight matrices and bias vectors (detailed equations below) and employ input- output-, and forget- gates, activated by exponential (cxp) the sigmoid functions
(a).. For cell inputs in sLSTM, the hyperbolic tangent function (tanh, abbreviated as (p) is used as an activation function.
2.1 SLSTM
The forward pass of sLSTM in the vectorized version is defined follows:
Ct ① C
i, 2t
cell state
(1)
ne f, ① 1 + i,
normalizer state
(2)
he O2 hr,
h C, ni
hidden state
Zt (z()
& Wz at + R2 h4-1 + b
cell input
(4)
i = exp
i Wi + R hi-1 + bi
input gate
(5)
f. exp
f
OR
fr
be f, Wr + Rs h: +
forget gate
Or (ō:)
o, W. az + R. h:-1 + bo
output gate,
where i, oc, E Rd are the input, output and forget gates, respectively, W., Wi, Wr, W. E d R2, R2, R₁, Ro dadd and b,, bj, br. b. R. are trainable weight matrices and biases, respectively.
2.2 MLSTM
The forward pass of the mLSTM is defined as follows:
C₁ = f, Ci-1 +
i, Uz kj
cell state
(8)
n2 f, n it k:
normalizer state
(9)
h = Ot @ h
h = C.qe / max {In] qel .:1.1}
hidden state
(10)
q: = Wg St + by
query input
(11)
k, Vd
Wi + be
key input
(12)
= W. at + b
value input
(13)
i, exp (1))
i w. 2t by
input gate
(14)
f OR exp (1)
I w 0 + bi
forget gate
(15)
(ōt)
o W. 2t bo
output gate
(16)
3

---

Published as a conference paper at ICLR 2025
where is R and f R are now scalar input and forget gates, q.kk, is R are query, key and value inputs with trainable weight matrices Wg. Wa. W, E RixD, Wi, wp E RD are input and forget gate weights and the respective b, br E R biases. All other quantities are identical to sLSTM.
2.3 BLOCK STRUCTURES
The sLSTM and mLSTM layers are integrated into larger residual backbones (Srivastava et al., 2015; He et al., 2016), which incorporate layer normalization (Ba et al., 2016), pre- post-up projection layers (Vaswani et al., 2017; Dao, 2024), with short causal convolutions and group normalization (Wu and He. 2020). Figure Al depicts sLSTM and mLSTM blocks, as well as, a bidirectional mLSTM configuration with weight-tied blocks. For more details, refer to (Beck et al., 2024, Sec.2.4). For Bio-xL.STM we retain these basic building blocks but adjust them to the respective domains. The entire architecture, including all layers, normalization, blocks, and other components, defines a mapping from an input sequence of length an output sequence. This mapping is denoted as xLSTM: Dxx + R Dxt where xLSTM transforms the stacked inputs up to time step 6, i.e., XI.t == [11, 22 Tt. E IR Dxt, to the corresponding stacked outputs of sequence length 4, i.e., Y1. = U2.............................. yı)
Dxt The i-th sequence element is denoted with the subscript e.g.. the i-th element from X1: would be (X1:)) Similarly to the mapping xLSTM, we also define input-/output behavior of layers themselves for sLSTM layer (sLSTM: Dxx + IR Dxt) or an ML.TM layer (mL.TM Dxt + IR Dxt). If the specific parameters used for the mapping are unclear, we will denote this by including second argument in the function, separated by a semicolon. See Appendix Section B.1 1 for details on the block structures.
R
mappings for the sequence-wis
2.4 MODES OF OPERATION: PARALLEI, CHUNKWISE, AND RECURRENT
The recurrent forms of sLSTM and mLSTM, introduced in Sections 2.1 and 2.2. provide efficient, constant-memory decoding during inference. This climinates the need for expensive key-value caching, which represents a major challenge for Transformer models in long-range settings. Like Transformers, mLSTM allows for parallelization across the sequence length, significantly speeding up training. Additionally, similar to linear attention variants (Katharopoulos et al., 2020; Yang al., 2024), mLSTM supports chunkwise parallel processing, blending recurrent and parallel modes (Beck et al., 2025). This approach is especially advantageous for long-sequence training and prompt encoding. For further details, refer to Appendix Section B.2.
3 BIO-XLSTM: LONG-RANGE MODELING OF BIOLOGICAL AND CHEMICAL SEQUENCES
Bio-XLSTM introduces three xLSTM-based architectural variants tailored specifically to DNA (Section 3.4), proteins (Section 3.5) and small molecules (Section 3.6). For these application domains, we extend xLSTM from causal language modeling (CLM) (Section 3.1) new modeling approaches such as fill-in the middle (FIM), in- in-context learning (ICL) and masked language modeling (MLM) (Section 3.2).
3.1 CAUSAL LANGUAGE MODELING AND NEXT-TOKEN PREDICTION
Causal language modeling (CLM), which is the xLSTM's ' default paradigm, uses the
CLM loss: CCLM Ex-px Be-[ur--11]] CE (æ++1, XLSMMX
(17)
where CE is the cross-entropy loss (with logits), px is the data distribution, and [[1,-- - 1]] is the discrete uniform distribution from 1 to T 1. The objective measures how well a particular sequence token #++11 can be predicted based the previous tokens X1:t by the model xLSTM : Rxxt Dxt Therefore, this type of modeling sometimes also called next token prediction (NTP), uni-directional modeling or autoregressive (AR) modeling and the loss is also called NTP loss.
Fill-in the middle (FIM) is a modeling paradigm that integrates aspects of both CLM and MLM (Bavarian et al., 2022). In this approach, parts of the sequence are replaced by mask tokens, and for
Here æ; and y. respectively, represent the inputs to and outputs from a particular model from instance of an xI TTM architecture, rather than the inputs and outputs of specific SLM or ml.S SL. layer
4

---

Published as a conference paper at ICLR 2025
each replaced segment, both the mask token and the corresponding masked sequence are appended to the end of the sequence. This allows the model to utilize the entire context to predict the masked tokens while maintaining an AR training framework. Consequently, the model can perform both i) generative modeling and ii) inpainting with CLM.
In-context learning (ICL) is a capability of language models learn and perform tasks by leveraging additional information provided as the contextual input without updating their parameters (Brown al., 2020; Min et al., 2022). This approach allows models to learn from analogy, drawing insights from patterns in the context to adapt their behavior. When the contextual input sequence is denoted as Z E RR PxS and the conventional input sequence is denoted as X RxT then with an ICL model, we obtain Y' = xLSTM( Z, ())((++):sss 1):$5)) where [Z. X] indicates concatenation of contextual and conventional inputs, and, the subscript (S - 1): : (S + T)) indicates that the last output tokens (those corresponding to the X tokens) are selected. For natural language processing tasks, the context Z often contains the solution to similar problem or exemplary solutions that guide the input and inform the model. For biological and chemical sequences, the context Z could represent similar genetic regions. homologous proteins, or molecules with desired properties, enabling the model to make informed predictions or generate outputs based on these analogies.
3.2 MASKED LANGUAGE MODELING (MLM)
Bio-XLSTM extends xLSTM to masked modeling of biological sequences, for which the typical de-masking or de-noising objective (Vincent et al., 2010; Devlin et al., 2019) is used, concretely the
MLM loss: C Ex~px [e-[[1.]]] EM-pM CE (zx, XLSTM(X @ M):), (18)
where M E 10. is random matrix with binary entries which are usually drawn from Bernoulli distribution pM, and @ is element-wise multiplication. The objective measures how well the original sequence X can be reconstructed from noisy version X M by the model xLSTM R DxT + RxT, This modeling approach, often referred to as bidirectional modeling, has proven instrumental in enabling the successful learning of protein representations at an evolutionary scale (Rives et al., 2021), which has powered many subsequent applications, such as machine-learning guided directed evolution for protein engineering (Yang al., 2019). For details on how extended XLSTMM to the MLM setting, we refer to Appendix Section B.3.
3.3
RC EQUIVARIANCE
We introduce an xLSTM block equivariant to the reverse complement (RC) of an input sequence. a key property for DNA-based applications. In double-helix DNA structures, both strands are semantically equivalent, with one strand being the RC of the other. The RC strand is oriented in the opposite direction of the forward strand, with base pairs converted from A to T and C to G. Shrikumar et al. (2017) show that data-driven approaches to learning RC sequence equivalence can fail. To address this, Schiff et al. (2024) enforced RC-equivariance by design using two inductive biases: post-hoc conjoining (PH) (Zhou et al., 2022) and parameter sharing (PS). In PH architectures, the backbone is trained to handle both DNA sequences and their RCs by applying RC augmentations during pre-training. For downstream tasks, PH architectures are applied to both the original sequence and its RC, and their outputs are summed reach overall RC invariance. In contrast, PS architectures integrate RC-equivariant XTM blocks with equivariant word embeddings and language model heads similar Schiff al. (2024). For additional details, see Appendix Section C.4.
3.4 DNA-XLSTM
For the DNA domain, we propose the DNA-xLSTM architecture to enhance sequence modeling capabilities, particularly for varying context lengths. We introduce three model configurations based on DNA-xLSTM: two sLSTM- configurations trained with context window of 1,024 tokens (DNA-XLSTM-500K and DNA-XLSTM-2MD, and an mLSTM-based configuration trained with context window of 32,768 tokens (DNA-XLSTM-4M) The short-context configuration, DNA- XLSTM-550K, has an embedding dimension of 128, 5 sLSTM blocks, an up-projection ratio of 1.25 to match the baseline model parameter count, and total parameter count of 500k, while DNA-XLSTM-2N has an embedding dimension of 256, sLSTM blocks, a 1.0 up- up-projection ratio, and 2M parameters. The long-context configuration, DNA-XI has embedding
5

---

Published as a conference paper at ICLR 2025
dimension of 256. 9 mLSTM blocks, 2.0 up-projection ratio, and is augmented with Rotary Position Encodings (RoPE) (Su et al., 2024a) to handle long-range dependencies effectively, with total of 4M parameters. All three configurations introduce RC-equivariant versions, x XL.STM-PH and xl ALSTM-PS, which use the original sequence and its reverse complement (RC). We trained models with these configurations on the human genome and benchmarked them against state-of-the-2 DNA models, such as Transformers, DNA-Mamba (Caduceus) (Schiff et al., 2024), and HyenaDNA (Nguyen al., 2023), showing competitive or better performance on pre-training and downstream classification tasks (Section 4.1).
trained with both CLM and MLM. Furthermore,
3.5 PROT-XLSTM
For the protein domain, we propose the architectural variant Prot-XLSTM address the complexities of protein sequence data, particularly in capturing long-range dependencies to enable homology- conditioned modeling. We introduce two configurations: Prot-XLSTM-26M and Prot-xLSTM-- 102M, with 26M and 102M parameters, respectively. Both configurations consist of 16 mLSTM blocks, with embedding dimensions of 512 for Prot-XLSTM-26M and 1,024 for Prot-XLSTTM-102M and maintaining consistent 2.0 up-projection ratio, and are augmented with RoPFs, across both configurations. The Prot-XLSTM models are trained using CLM with a FIM strategy on non- aligned homologous sequences with increasing context sizes ranging from 2,048 to 262,144 tokens. This enables them to perform ICL at inference time two modes: i) generative mode, which is inherently incompatible with MLM, and ii) inpainting, which is not supported by "vanilla" CLM. Both approaches can be used for protein design, with the latter also suited for residue-based predictions, such mutant fitness estimation. Prot-xLSTM shows better performance than similarly configured Mamba- and Transformer- based models and shows promising results for conditoo-e sequence generation (Section 4.2).
3.6 CHEM-XLSTM
For the chemical sequence domain, we introduce Chem-XLSTM to enhance generative modeling of SMILES strings (Weininger, 1988), enabling condiiinnondiiooeed generation of small molecules without fine-tuning, We introduce two models: an unconditional generative model trained with a context length of 100 tokens (Chem-xl.STM-155))) and domain--onditionn model trained with a 4,096-token context for in-context learning tasks SThemxX.STM-15MM molecules within a specific domain without fine-tuning only based on examples provided as context, a highly sought-after capability in drug discovery. Both models are configured to have 15M parameters, consisting of 9 mLSTM blocks with an embedding dimension of 512 and a 1.3 up-projection ratio. The models have been benchmarked against other generative models for SMILES and at their ICL capabilities (Section 4.3).
The latter can generate
4 EXPERIMENTS AND RESULTS
4.1 DNA SEQUENCES
For the DNA-XLSTM experiments, we followed the experimental protocol outlined in Schiff et al. (2024) and Nguyen al. (2023) for both pre-training and downstream adaptation.
Pre-training. The training data for DNA-XLSTM models was sourced from the human reference genome (Church et al., 2011), which also served as the basis for our Mamba-based baseline models HyenaDNA (Nguyen et al., 2023) and Caduceus (Schiff et al., 2024) - as well as for our Transformer++ baselines built on the Llama architecture (Touvron et al., 2023). Initially, we pretrained two variants using both CLM and MLM objectives for each: the sLSTMM-asee DNA-xLSTM- 2M with a context length of 1,024 (Figure 2) and the mLSTM-based DNA-XLSTM-4M with a context length of 32k (Figure A3). Both models were trained with PH-equivariance (using RC augmentation without PS -suivaaarii.c. Under the CLM objective, DNA-XLSTM-2M achieved superior performance compared to Transformer++, Mamba, and HyenaDNA. This performance advantage was even more pronounced on the MLM task, where DNA-XLSTM-2M outperformed both the Transformer- and Mamba-based models.

---

Published as a conference paper at ICLR 2025
1.15
DNA-XLSTM
1.03
DNAXLSTM
1.14
Mamba
1.02
Mamba
Validation Loss
1:13
Transtormers
Hyena
Validation Loss
1.01
Transformer++
1.12
1.00
1.11
0.99
1.10
0.98
1.09
0.97
1.08
0.96
Billion Tokens
Billion Tokens
Figure 2: Pre-training of 2M-parameter DNA models on the human reference genome (GRCh38). Models are trained single-nucleotide resolution with a context length of 1024 bases. Left: causal language modeling, Learning curves display NTP loss (J) on test set, plotted against the number of tokens processed. Right: masked language modeling. Learning curves showing MLM loss (L) on test set across the number of tokens seen for various models. In both tasks, the LLSTM-based models consistently achieve the lowest loss values across all update steps.
Similar to Caduceus, we also experimented with PS-equivariance and trained DNA-XLSTM-2M using parameter sharing for RC-equivariance under an MLM objective (Figure A4). Additionally, to match the model size used by Caduceus in one of the downstream tasks, we trained both PH- and PS-equivariant versions of DNA-XLSTM-50000 with an MLM objective and a context length of Ik. These resulting PH and PS models were subsequently used for downstream fine-tuning. In Appendix Section C, we present additional pre-training results, including comparisons of large-context and PS-equivariant models. Our findings indicate that XLSTM-DNA matches or outperforms strong baselines across all pre-training settings.
Downstream fine-tuning. To evaluate the learned representations, we fine-tuned our MLM- pretrained, short-context -NA-XLSTM models (both PH and PS variants) on two genomic classi- fication benchmarks. Specifically, the DNA-xLSTM-SUN Benchmark (Grešová et al., 2023), while the DNA-XISTM-DD cleotide Transformers Tasks (Dalla-Torre et al., 2024), which together span 18 datasets from five studies. In Table 1, we compare the performance of DNA-XLSTTM-2M-PP and DNA-x DNA-XLSTM-2M-P on the Nucleotide Transformers Tasks against HyenaDNA as well as Mamba-PS and Mamba-PH. DNA-xLSTM achieved the best performance, outperforming baseline models in the sub-2M parame- ter range on 12 of 18 tasks and performing comparably to the much larger Nucleotide Transformer (500M parameters) by winning 8 tasks (Table A1). Additional comparisons with larger Transformer models and the performance on the Genomic Benchmark are provided in Appendix Section C.
model was fine-tuned on the Genomic
model was fine-tuned on the Nu-
4.2 PROTEIN SEQUENCES
We followed the experimental protocols from Sgarbossa al. (2024) for protein sequences.
Homology-aware training. Training data was sourced from the filtered OpenProteinSet (Ahdritz al., 2023), consisting of 270k UniClust30 clusters (508M sequences, 110B residues). Using the ProtMamba pipeline, we constructed homology-awand alignment-free inputs by concatenating unaligned homologous sequences and mask patches for training with the FIM strategy. We trained two XLSTM- based models: ProtxXLSTM-26M and Prot-XLSTM-
For comparison, we also trained a smaller ProtMamba (ProtMamba-281) 28MM) and Transformer- based (Prot- Proo-TTransfomme+++2660 (Touvron et al., 2023) model and used the ProtMambu Long Foundation (ProtMamba- 107M) provided by Sgarbossa et al. (2024). The initial training followed a context length scheduling strategy. with models gradually increasing context from 2 to 2 tokens. Finally, Prot-xLSTM- 102M was further trained with T = 218 We evaluated the models using negative log-likelihood and perplexity, calculated for different parts of the concatenated-FIM. sequences. As shown in Figure and Table 2, Prot-XLSTTM outperformed the other architectures. Its advantage becomes even more pronounced with longer contexts, which mot-Traasformer++ cannot handle, and where ProM-XLSTM significantly
The protein downstream tasks were evaluated using the model trained with up to 2 tokens.
7

---

Published as a conference paper at ICLR 2025
Table Downstream adaption of DNA models. The performance of 2M parameter models fine-tuned on Nucleotide Transformer classification tasks on the test set is shown. PS or PH indicate models trained be RC equivariant. Performance is averaged over random seeds, and error bars indicate the difference between maximum and minimum values. The best values are highlighted in green. Bold numbers indicate statistically significant improvement over previous methods. DNA-xLSTM outperforms both Mamba and Hyena 12 out of 18 tasks. Scores for Mamba- and Hyena-base models were obtained from Schiff et al. (2024).
Task
Metric
HyenaDNA
Mamba-PS*
Mamba-PHI
xLSTM-PS
XLSTM-PH
Histone Markers
H3
MCC
0.779=0.057
0.7992
0.815=0.045
0.796=0.001
0.824
H3K14AC
MCC
0.612=0.006
0.54 20.212
0.63
0.570
0.59g
H3K36ME3
MCC
0.613
0.613-40.041 40.041
0.60g 0.609-4o.100 10.
0.601 1:0.129
0.588-40.019
0.625=400010 $0.010
0.625
НЗK4ME1
0.512=0.024
0.48g
0.523=0.00 039
0.4002
#00000
HЗK4ME2
MCC
-4.5200006
0.388 101
0.487=0.170
0.489 20.024
0.504
HЗK4ME3
MCC
0.549 +00056
0.440 20.202
0.544 1.544
0.520 +0.019
0.537
HЗК79ME3
MCC
0.672
0.676
0.697
0.662
0.697
HЗK9AC
MCC
0..8140.061
0.60g
0.622
0.62220:013
0.627
H4
MCC
0.763400004
2..00000200
0.811 20 022
0.81340.008
H4AC
MCC
0.564tt0.08
0.525 0.525+0.224 24.240
0.621 #0.064
0.558
t0.014
Regulatory Annotation
Enhancer
MCC
0.517#0.317
40.066
0.546=0:077
0.375=
#0:022
Enhancer Types
MCC T
0.386=0.118 185
0.416=#0.095
0.439=0.064
0.444
0.466=00011
Promoter: All
FI f
0.960 0.000 0.960=+0.00
0.967
0.970*#0.00
0.962+00.00
0.962 400000
0.967 00007
NoTTATA
FI 1
0.959 0.959+0.001
0.968 0.968
+0.006
0.968 0.968*0.010
0.963
0.970 #0.001
TATA
F1 f
0.944200000
0.957
0.953=0.011
0.948
0.952
Splice Site Annotation
All
Accuracy f
0.956
0.927
0,940
0.965 100000
0.974 1.00004
Acceptor
F1 f
0.958=0.00
0.936
0.937=0.08$
0.970 0.970#00005
0.953 0.953+00000 t0.008
Donor
f
0.94g=0.024
0.874#0.280
0.948=0:025
0.962 000004
0.951 t00.00
This method is also called Caduceus (Schiff al., 2024).
22
2.8
2.8
Validation Loss
2.6
2.6
2.4
2.4
2.2
2.2
ProtxXLSMM226M
2.0
ProtMamba-28M
2.0
Prat-XLSTM-100M
Prot Prot-Transformer + 26M
ProtMamba -07M
Billion Tokens
Billion Tokens
Figure 3: Generative pre-training of protein language models. The learning curves show the validation loss of homology-awe aware protein language models during training. Left: Small models trained for 20B tokens with context size of 2 and fine-tuned for 10B with context of 2 tokens. Transformert+ can only be run for small context size. Right: Prot-XLSSTM-102M model trained with inereasing context sizes from 2 to 218 Vertical gray dashed lines mark the points where context size was increased. The arrow at 60B tokens indicates the model used for downstream tasks. The orange dashed line corresponds the validation loss of ProtMamba- 107M trained up to context size of 27 for a total of 195B tokens. Prot-xLSTM consistently outperforms other models and sets new state-of the-art homology-awe aware generation.
outperforms ProtMamba. Furthermore, Prot-xLSTM- 102M outperforms ProtMamba- MMMMMmma-110M.... despite being trained on only a quarter of the total training tokens used for ProtMamba- 107M. Further details provided in the Appendix Section D.I.
8

---

Published as a conference paper at ICLR 2025
Table Performance comparison of protein language models at saamology-coniitoonee generation. Test set perplexity (4) of different models with context size of 2 is shown across different token subsets. The average and 95% confidence interval values are computed across the test set clusters. Prot-XLSTM outperforms ProtMamba, especially when using a long context. For each token subset, the top-performing model that significantly outperforms the others, as indicated by a Wilcoxon test (p < 0.05, Figure A6), is marked bold.
Prot- -TS--XL
ProtMamba-28M
Prot-xLSTM: 102M
ProtMamba- 107M
All tokens
9.41=0.22
11.23+0.18
7.35#0011
(7.1720.200
8,2740.18
First seq tokens
15.40 20.28
15.28 15.28+00.26 H0.26
13.36 0.33
133.22
13. 13.04 40.30
Last seq tokens
9.19 00.00
11.08 10027
7.32 40:29
7.32
(7.14 20
8.37 F0.29 F0.29
FIM tokens
7.23 =0.19
8.66 8.66 2.118
5.87 (5.85 10.10)
7.00 10.16
# Trained for 60B tokens with T up to 247,
Trained for 115B tokens with T up to 2 parentheses.
b "Trained for 195B tokens with T up to 2
Homology-conditionee protein generation. We generated 2,500 protein sequences for each of the 19 clusters using different parameters and evaluated them across multiple metrics. The sequence similarity to natural homologs (min daH and HMMER scores) and structural metries (pLDDT and PTM) correlate with sequence perplexity, with average absolute Spearman correlation of 0.57 across clusters for the large Prot-XLSTM model (Table A9). The distribution of these scores, such as HMMER and pLDDT, of real proteins also matches well with generated proteins. When evaluating the alignment with real-world protein score distributions, Prot-xI outperforms ProtMamba across all five metrics for the small models. For the large models, Prot-xI.STM, surpasses ProtMamba on three metries (Table All). Additional details and results can be found in Appendix Section D.2.
Protein variant fitmess prediction. We assessed Prot-XLSTM's predictive power for mutational effects by leveraging its inpainting capability from the FIM training objective and using homologous sequences as context on the ProteinGym DMS zero-shot substitutions benchmark (Notin al., 2023). Table A12 presents the performance comparison of Prot-XLSTM, other well-known protein models, and the current top performers on the ProteinGym leaderboard. In summary, Prot-XI.STM outperforms larger unconditional protein language models like ESM-2 (Lin et al., 2023) and ProGen22 (Nijkamp et al., 2023), and matches or surpasses ProtMamba's performance. However, it falls short compared models that directly use the MSA, such as TranceptE (Notin et al., 2022b), or use structural information, like ProSST (Li et al., 2024). For further details, see Appendix Section D.3.
4.3 CHEMICAL SEQUENCES
Unconditional molecule generation aims to produce valid small organic molecules without imposing specific constraints, such as being from a particular molecular domain. Following the setup of Özşelik et al. (2024), trained models to generate SMILES strings using CLM approach derived from ChEMBL with a context length of 100 tokens. We compared our Chem- 2hem-XLSTM architecture with several architectures, including LSTM, GPT, $4, and Mamba, where all models contain approximately 15 million parameters. The evaluation focused on two primary metrics: perplexity and Fréchet ChemNet Distance (FCD) (Preuer al., 2018). Chem-XLSTM achieved the lowest FCD of 0.13 and a competitive perplexity score of 1.68, indicating its strong ability to generate realistic chemical structures (see Table 3). All models produced valid, unique, and novel molecules, showcasing their effectiveness in this task. For further details and results, see Appendix Section E.
dataset
For conditional molecule generation, the objective is to generate molecules belonging to a specific molecular domain or possessing desired properties. Here, we focused on generating molecules from a particular domain using the in-context learning abilities of L.MMs. To achieve this, we assembled a dataset, referred to as the molecular domains dataset, which comprises a diverse range of molecular domains: natural products, click-chemistry, proteolysis-targeting chimera (PROTACs). DNA-encoded chemical libraries, approved and failed drugs, and bioactive compounds from various bioassays. Molecules from the same domain are concatenated as a long sequence and augmented through permutation during training. We split the dataset into training, validation, and test domains, following an 8:1:1 ratio (Figure 4, left). The validation and test sets contained molecules from unseen domains, enabling us evaluate the models' conditional generation capabilities through in-context learning.

---

Published as a conference paper at ICLR 2025
Table 3: Unconditional generation of molecules with 15M parameter models. 102,400 SMILES sequences have been generated and evaluated. Error bars represent standard deviations across bootstrap resampling. Green cells highlight the best values per row. Chem-xLSTM, yields the best FCD and SMILES-GPT the best perplexity. Both values are statistically significant, as indicated in bold, using a Wilcoxon test.
SMILES-LSTM' SMILES-GPP
SMILES-S4
Chem- Mamba"
Chem-xLSTM
FCD |
0,46+cuu
0.15 +00000
0.28
+c2.0
0.13 +13+<00 <0.01
Perplexity 4
1.88
1.65 +0.6
1.73+2.4
1.74
1.68
SSegler al. (2018)
Adilov (2021)
Özşellik al. (2024) "Adapted to SMILES in this work.
We trained
Mamba, Transformer++, and $4-based models with the CLM approach on the molecular domains with an increased context length of 4,096 tokens. The context length for S4 models was restricted 2,048 due memory constraints. We evaluated the models based on NTP loss across unseen domains. The trained model Cheem-XSSTM-15M-- shows promising results in this conditional setting, outperforming the other benchmarked model classes (Figure 4, right). This demonstrates Chem-XLSTM' capability to generate molecules from an unseen chemical domain when provided with only a few exemplary molecules without fine-tuning. Further details and results are provided in Appendix Section E.2.
VEPTO-S
valid
Transtomere+
Mamba
Validation Loss
XLSTM
44
0.8
0.6
0.5
20
60
80
100
120
140
Epoch
Figure 4: Conditional generation of molecules via ICL and 15M parameter models. Left: Visu- alization of different molecular domains contained in the molecular domains dataset. A t-SNE down-projection of molecules from different domains is shown. Clusters on the exterior represent highly specific molecular domains. The validation and test sets contain molecules from highly specific, unseen molecular domains. Right: Generative training of chemistry language models on the molecular domains dataset. Learning curves showing mean CLM loss (J) on a validation set across runs. The Chem-XLSTM
the training epochs. Shaded areas represent the standard-deviatio achieved the lowest loss conditional molecule generation.
5 LIMITATIONS AND CONCLUSIONS
Limitations. Manual hyperparameter selection may not yield optimal configurations of the models, and the reliance on character-level tokenization for DNA could restrict performance with larger context sizes. Additionally, the generalizability of our models across different organisms and chemical domains is uncertain due to biases the training datasets. Metries like perplexity, commonly used as performance proxies, may not fully capture the true capacity of the models (Appendix G). Conclusion. Despite these limitations, Bio-xLSTM demonstrated effectiveness DNA sequence modeling, performing best in masked and causal language tasks across context sizes. Our findings underpin that Bio-XLSTM is a prime candidate for foundation models molecular biology and the models presented in this work already be used effectively to generate DNA sequences, protein inpainting or suggesting mutations in proteins, or for conditional molecular generation without fine-tuning.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
The development of our large language models for biological sequences, including DNA, proteins, and small molecules, has the potential to significantly advance biomedical research and therapeutics. In creating these models, we have taken care to train exclusively on publicly available data, such as the human reference genome, OpenProteinnet, and publicly available small molecule databases. is important emphasize that the models presented in this work are general base models without any specific safety tuning. These models still require human expert knowledge to produce harmful outcomes and do not possess complex dnstruction-folloviin opposed to general-puryy purpose language models. This significantly limits their potential for misuse by non-experts in the field. For future larger-scale model development, we recognize that additional safety measures may be necessary. One option is the exclusion of certain training samples, such as those related to human viruses, as well as the implementation of safety classifiers or guardrails that can identify and flag potentially harmful outputs. At the same time, we believe that openly sharing these base models has significant advantages for the scientific community and can aid in preparedness for addressing potential misuse scenarios. As common with machine learning methods, potential danger lies in the possibility that users rely too much on our new approach and it without reflecting on the outcomes. However, the full pipeline, in which our method would be used, includes wet lab tests after its application, to verify and investigate the results, which decreases the danger of misuse overly relying on the predictions. Users are encouraged to approach the model's predictions critically and consider them as one component of broader ceciision-making process that incorporates expert knowledge and empirical validation.
experiment planning abilities as
REPRODUCIBILITY STATEMENT
To ensure reproducibility and facilitate future research, we provide three standalone code repositories for DNA-xLSTM, Prot-xLSTM, and Chem-XLSTM, each containing the respective implementations, training scripts, evaluation procedures, and pre-proceste rre-processed datasets.
ACKNOWLEDGEMENTS
We acknowledge EuroHPC Joint Undertaking for awarding us access to Karolina at IT44Inovvaiions Czech Republic; MeluXina at LuxProvide, Luxembourg; and Leonardo at CINECA, Italy.
The ELLIS Unit Linz, the LIT AI Lab, and the Institute for Machine Learning are supported by the Federal State Upper Austria. This research was partly funded by the Austrian Science Fund (FWF) [10.55776/COF1)) 2]. We thank the projects INCONTROL-RI (FFG-88 16664) PRI- MAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG- 892171), FWF AIRI FG 9-N (10. (10.55776/FG9), Al4GreennHcatingrrri,, (FFG- 899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019) LLIMATE-01011 We thank NXAI GmbH, Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL). FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo), Software Competence Center Hagenberg GmbH, Borealis AG, TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation.
ID: 951847), Stars4Waters (HORIZON-CL6-
REFERENCES
1000 Genomes Project Consortium (2010). A map of human genome variation from population scale sequencing. Nature, 467(191)):0061
Acosta, J. N., Falcone, G. J., Rajpurkar, P., and Topol, E. J. (2022). Multimodal biomedical AL. Nature Medicine, 1779):17331178.
Adilov, S. (2021). Generative pre-training from molecules. ChemRxiv preprint chemrxiy-2022 2hemrxiv-2021-5ffvddl
Ahdritz, G., Bouatta, N., Kadyan, S., Jarosch, L.. Berenberg, D., Fisk, I., Watkins, A., Ra, S., Bonneau, R., and AlQuraishi, M. (2023). Openproteinset: Training data for structural biology
11

---

Published as a conference paper at ICLR 2025
at scale. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 4597-4609 Curran Associates, Inc.
Ahmad, W., Simon, E., Chithrananda, S., Grand, G., and Ramsundar, B. (2022). ChemBERTa-2: Towards chemical foundation models. arXiv preprint arXiv:2209.01712.
Alkin, B., Beck, M., Pöppel, K., Hochreiter, S., and Brandstetter, J. (2024). Vision-LSTM: xLSTM as generic vision backbone. arXiv preprini arXiv:2406.04300:
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi M., and Church, G. M. (2019). Unified rational protein engineering with sequence-based deep representation learning. Nature methods, 16(12):1315-1322.
Anfinsen, C. B. (1973). Principles that govern the folding of protein chains. Science, 181(4096):223- 230.
Arnold, F. H. (2018). Directed evolution: bringing new chemistry to life. Angewandte Chemie, 57(16):4143.
Ashley, E. A. (2016). Towards precision medicine. Nature Reviews Genetics, 17(9)55075522.
Ba, L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint
arXiv:1607.06450.
Bagal, V., Aggarwal, R., Vinod, P., and Priyakumar, D. (2021). MolGPT:
molecular generation using transformer-decoddrr model. Journal of Chemical Information and Modeling, 62(9))2064- 2076.
Baum, L. E., Petrie, T.. Soules, G., and Weiss, N. (1970). A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains. The Annals of Mathematical Statistics, 41(1):164-1 171.
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., and Chen, M. (2022). Efficient training of language models fill in the middle, arXiv preprint 2225:22207.12255.
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, (2024). XLSTM: Extended long short-term memory. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C., editors, Advances in Neural Information Processing Systems, volume 37, pages 107547-107603. Curran Associates, Inc.
Beck, M., Pöppel. K., Lippe, P., and Hochreiter, S. (2025). Tiled flash linear attention: More efficient linear RNN and xLSTM kernels. Under review.
Benegas, G., Batra, S. S., and Song. Y. S. (2023). DNA language models are powerful pre- dictors of genome-wide variant effects. Proceedings of the National Academy of Sciences, 120(44)e2311221122.
Bepler, T. and Berger, B. (2019). Learning protein sequence embeddings using information from structure. In International Conference on Learning Representations (ICLR), volume 7.
Bouwman, B. A. and de Laat, W. (2015). Getting the genome in shape: the formation of loops, domains and compartments. Genome Biology, 16(1):154.
Bran, A. M. and Schwaller, (2024). Transformers and large language models for chemistry and drug discovery. In Drug Development Supported by Informatics, pages 143-163. Springer.
Brandes, N., Goldman, G., Wang, C. H., Ye, C. J., and Ntranos, V. (2023). Genome-wide prediction of disease variant effects with deep protein language model. Nature Genetics, 152-)
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., and Linial, M. (2022). ProteinBERT: a universal deep-learning model of protein sequence and function. Bioinformatics, 3802)2100-
12

---

Published as a conference paper at ICLR 2025
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam,
P., Sastry, G., Askell, A., Agarwal. S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu. Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877-1901. Curran Associates, Inc.
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundherg. S., et al. (2023). Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.1271:
Chandrasekhar, V., Rajan, K., Kanakam, S., Sharma, N.. Weißenborn, V., Schaub, J., and Steinbeck, C. (2024). COCONUT 2.0: a comprehensive overhaul and curation of the collection of open natural products database. Nucleic Acids Research, 53(D1))D634-D443
Chithrananda, S., Grand, G., and Ramsundar, B. (2020). ChemBERTa: large-scale self-supervised pretraining for molecular property prediction. arXiv preprint arXiv:2010.09885.
Choromanski, K. M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q.. Mohiuddin, A., Kaiser, L., Belanger, D. B., Colwell, L. J., and Weller, A. (2021). Rethinking attention with performers. In International Conference Learning Representations (ICLR), volume 9.
Church, D. M., Schneider, V. A., Graves, T., Auger, K., Cunningham, F., Bouk, N., Chen, H.-C., Agarwala, R., McLaren, W. M., Ritchie, G. R., et al. (2011). Modernizing reference genome assemblies. PLoS biology. 100111001
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, P., Sirelkhatim, H., et al. (2024). Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, pages 1-11.
Dao, T. (2024). Flashattention-2: Faster attention with better parallelism and work partitioning. In International Conference Learning Representations (ICLR), volume 12.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Eddy, S. R. (2020). HMMER: biosequence analysis using profile hidden markov models. http: //hmmer org.
Elnaggar, A., Heinzinger, M.. Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2021). ProtTrans: Toward understanding the language of life through supervupere learning. JEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):7112-7127
Ertl, P. and Schuffenhauer, A. (2009). Estimation of synthetic accessibility score of drug like molecules based on molecular complexity and fragment contributions. Journal of cheminformatics, 1:1-11.
Evans, R., O'Neill, M., Pritzel, A., Antropova, N., Senior, A., Green, T., Zidek, A., Bates, R., Blackwell, S., Yim, J., Ronneberger, O., Bodenstein, S., Zielinski, M., Bridgland, A., Potapenko, A., Cowie, A., Tunyasuvunakool, K., Jain, R., Clancy, E., Kohli, P., Jumper, J., and Hassabis, D. bioRxiv.
(2022). Protein complex prediction with alphafold- alphafold-multimer multimer.
Ferruz, N., Schmidt, S., and Höcker, B. (2022). ProtGPT2 is deep unsupervised language model for protein design. Nature Communications, 13:4348.
Flam-Shepherd, D., Zhu, K., and Aspuru-Guzik, A. (2022). Language models can learn complex molecular distributions. Nature Communications, 13(1):3293.
13

---

Published as a conference paper at ICLR 2025
Frazer, J., Notin, P., Dias, M., Gomez, A., Min, J. K., Brock, K., Gal, Y., and Marks, D. S. (2021). Disease variant prediction with deep generative models of evolutionary data. Nature, 599(7883);11- 95.
Geng, Q. Yang. R., and Zhang, L. (2022). deep learning framework for enhancer prediction using word embedding and sequence generation. Biophysical Chemistry, 286:106822
Gers, F. A., Schmidhuber, J., and Cummins, F. (1999). Learning to forget: continual prediction with LSTM. In 9th International Conference Artificial Neural Networks ICANN '99, pages 850-855. IЕT.
Jomeez-Bombareli,, R., Wei, N., Duvenaud, D., Hernández-Lobatoo J. M., Sánchez-Lengelingg, B., Sheberla, D., Sguilera-Iparraguing J., Hirzel, T. D., Adams, R. P., and Aspuru-Guzik A. (2018). Automatic chemical design using data-driven continuous representation of molecules. ACS Central Science, (22):6682272)
Grešová, K., Martinek, V., Čechák, D., Šimeček, P., and Alexiou, P. (2023). Genomic benchmarks: collection of datasets for genomic sequence classification. BMC Genomic Data, 24(1) 1425)
Gu, and Dao, (2024). Mamba:
Linear-time sequence modeling with selective state spaces, In First Conference Language Modeling.
Gu, A., Goel, K., and Ré, C. (2022). Efficiently modeling long sequences with structured state spaces.
In International Conference on Learning Representations (ICLR), volume 10.
Gwak, H.-J. and Rho, M. (2022). ViBE: hierarchical BERT model to identify eukaryotic viruses using metagenome sequencing data. Briefings in Bioinformatics, 23(4):bbac204.
Handa, K., Thomas, M. C., Kageyama, M., lijima, T., and Bender, A. (2023). On the difficulty of validating molecular generative models realistically: case study public and proprietary data. Journal of Cheminformatics, 15(1):112.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings CVPR, pages 770-778.
Hoarfrost, A., Aptekmann, A., Farfañuk, G., and Bromberg, Y. (2022). Deep learning of bacterial and archaeal universal language of life enables transfer learning and illuminates microbial dark matter. Nature Communications, 13(1):2606.
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 98): 1780.
Honda, S., Shi, S., and Ueda, H. R. (2019). SMILES transformer: Pre-trained molecular fingerprint for low data drug discovery. arXiv preprint arXiv:1911.04735.
Hopf. T. A., Ingraham, J. B., Poelwijk, F. J., Schärfe, C. P., Springer, M., Sander, C., and Marks, D. S. (2017). Mutation effects predicted from sequence co- no-variation. Nature Biotechnology. 3282):
Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B., Sercu, T., Lerer, A., and Rives, A. (2022). Learning inverse folding from millions of predicted structures. In International conference machine learning, pages 8946-8970. PMLR.
Hu, B., Xia, J., Zheng, J., Tan, C., Huang, Y., Xu, Y., and Li, S. Z. (2022). Protein language models and structure prediction: Connection and progression. arXiv preprint arXiv:2211.16742.
Jastrzebski, S., Lešniak, D., and Czarnecki, W. M. (2016). Learning to SMILE(S). arXiv preprint 1rXiv:1602.06288,
Ji, Y.. Zhou, Z., Liu, H., and Davuluri, R. V. (2021). DNABERT: pre-trained bidirectional en- coder representations from transformers model for DNA-language genome. Bioinformatics, 37(15):1112-12200
14

---

Published as a conference paper at ICLR 2025
Jumper, J., Evans, R., Pritzel, A., Green, T., Figumov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko. A., et al. (2021). Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-555
Karollus, A., Hingerl, J., Gankin, D., Grosshauser, M., Klemon, K., and Gagneur, J. (2024). Species- aware DNA language models capture regulatory elements and their evolution. Genome Biology, 25(1):83.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. (2020). Transformers are RNNs: Fast autore- gressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119, pages 5156-5165, PMLR.
Kim, S., Chen, J., Cheng, T., Gindulyte, A., He, J., He, S., Li, Q.. Shoemaker, B. A., Thiessen, P. A., Yu, B., al. (2023). Pubchem 2023 update. Nucleic Acids Research, 31(D1):D1333-D1333
Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), volume
Laine, E.. Karami, Y., and Carbone, A. (2019). GEMME: A Simple and Fast Global Epistatic Model Predicting Mutational Effects. Molecular Biology and Evolution, 36(11):2604-261
Lander, S., Linton, M., Birren, B., Nusbaum, C., Zody, M. C., Baldwin, J., Devon, K., Dewar, K., Doyle, M., Fitzhugh, W., et al. (2001). Initial sequencing and analysis of the human genome. Nature, 40966622)
Landrum, G. (2013). Rdkit documentation. Release, 1(1-79):4.
Levré, D., Arcisto, C.,, Mercalli, V., and Massarotti, A. (2018). Zinelick v. 18: expanding chemical space of 1, 3-triazoles. Journal of Chemical Information and Modeling, 59(5):1697-1700
Li, M., Tan, Y., Ma, X., Zhong, B., Yu, H., Zhou, Z., Ouyang, W., Zhou, B., Tan, P., and Hong. (2024). ProSST: Protein language modeling with quantized structure and disentangled attention. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., dos Santos Costa, A., Fazel-Zarandi,, M., Sercu, T., Candido, S., and Rives, A. (2023). Evolutionary- scale prediction of atomic-leve protein structure with language model. Science, 379(6637):1113- 1130.
Lowe, D. M. (2012). Extraction chemical structures and reactions from the literature. PhD thesis, University of Cambridge.
Madani, A., Krause, B., Greene, E. R., and et al. (2023). Large language models generate functional protein sequences across diverse families. Nature Biotechnology, 41:1099-1106 1106.
Marquet, C., Schlensok, J., Abakarova, M., Rost, B., and Laine, E. (2024). Expert-guided protein lan- guage models enable accurate and blazingly fast fitness prediction. Bioinformatics, bbbtae6,,.
Mayr, A., Klambauer, G., Unterthiner, T., Steijaert, M., Wegner, J. K., Ceulemans, H., Clevert, D.-A., and Hochreiter, S. (2018). Large-scale comparison of machine learning methods for drug target prediction on ChEMBL. Chemical science, 9(4))5441-5451.
Mazuz, E., Shtar, G., Shapira, B., and Rokach, L. (2023). Molecule generation using transformers and policy gradient reinforcement learning. Scientific Reports, 13(1):8799
Merrill, W., Petty. J., and Sabharwal, A. (2024). The illusion of state in state-space models. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235, pages 35492-35506. PMLR.
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work? In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 11048-11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
15

---

Published as a conference paper at ICLR 2025
Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L., Ovchinnikov, S., and Steinegger, M. (2022).
ColabFold: making protein folding accessible to all. Nature Methods, 19(6):679 682.
Nguyen, E., Poli, M., Faizi, M., Thomas, A., Wornow, M., Birch-Sykes, C., Massaroli, S., Patel, A., Rabideau, C., Bengio, Y., et al. (2023). HyenaDNA: Long-range genomic sequence modeling at single nucleotide resolution. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 43177-43201. Curran Associates, Inc.
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., and Madani, A. (2023). ProGen2: Exploring the boundaries of protein language models. Cell Systems, (111)))6-77....
Notin, P., Dias, M., Frazer, J., Marchena- Hurttad,, J., Gomez, A. N., Marks, D., and Gal, Y. (2022a). Tranception: Protein fitness prediction with autoregressive transformers and inference-time re- trieval. In Chaudhuri, K., Jegelka, S., Song. L.., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceed- ings of Machine Learning Research, pages 16990-17017. PMLR.
Notin, P., Kollasch, A., Ritter, D., van Niekerk, I.., Paul, S., Spinner, H., Rollins, N., Shaw, A., Orenbuch, R., Weitzman, R., Frazer, J., Dias, M.. Franceschi, D., Gal. Y., and Marks, D. (2023). ProteinGym: Large-scale benchmarks for protein fitness prediction and design. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages Associates, Inc.
Curran
Notin, P., Nickerk, L. V., Kollasch, A. W., Ritter, D., Gal, Y., and Marks, S. (2022b). TranceptEVE: Combining Family-specific and Family- Models of Protein Sequences for Improved Fitness Prediction.
Notin, P., Rollins, N., Gal, Y., Sander, C., and Marks, D. (2024). Machine learning for functional protein design. Nature Biotechnology, 2262):216-228.
Oubounyt, M., Louadi, Z., Tayara, H., and Chong, K. T. (2019). Deepromoter: robust promoter predictor using deep learning. Frontiers in Genetics, 10:286.
Özęelik, R., de Ruiter, S., Criscuolo, E., and Grisoni, F. (2024). Chemical language modeling with structured state space sequence models. Nature Communications, 15(1):6176.
Papadatos, G., Alkarouri, M., Gillet, V. J., Willett, P., Kadirkamanathan, V., Luscombe, C. N., Bravi, G., Richmond, N. J., Pickett, S. D., Hussain, J., et al. (2010). Lead optimization using matched molecular pairs: inclusion of contextual information for enhanced prediction of herg inhibition, solubility, and lipophilicity. Journal of Chemical Information and Modeling, 50(10):1872-1886
Phaml, T. H., Tran, D. H., Ho, T. B., Satou, K., and Valiente, G. (2005). Qualitatively predicting acetylation and methylation areas in DNA sequences. Genome Informatics, 16(2):3-11.
Poli, M., Massaroli, S., Nguyen, E.,, Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Ré, C.
(2023). Hyena hierarchy: Towards larger convolutional language models. In Proceedings of the
40th International Conference on Machine Learning (ICML), volume 202, pages 28043-28078. PMLR.
Press, O., Smith, N. A., and Lewis, M. (2021). Shortformer: Better language modeling using shorter inputs. In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, volume 1, pages 5493-5505. Association for Computational Linguistics.
Preuer, K., Renz, P., Unterthiner, T., Hochreiter, S., and Klambauer, G. (2018). Fréchet chemnet distance: a metric for generative models for molecules in drug discovery. Journal of Chemical Information and Modeling, 173))1736-1
Quigley, I. K., Blevins, A., Halverson, B. J., and Wilkinson, N. (2024). Belka: The big encoded library for chemical assessment. In NewrIPS 2024 Competition Track.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). Improving language understand- ing by generative pre-training. OpenAl Blog.
16

---

Published as a conference paper at ICLR 2025
Radford, A., Wu, J., Child, R.,, Luan, D., Amodei, D., and Sutskever, I. (2019). Language models unsupervised multitask learners. OpenAl Blog.
Rao, R... Bhattacharya, N., Thomas, N., Duan, Y., Chen, P., Canny, J., Abbeel, P., and Song, Y. (2019).
Evaluating protein transfer learning with TAPE. Advances in Neural Information Processing Systems (NeurIPS),
Rao, R. M., Liu, J., Verkuil, R., Meier, J., Canny, J., Abbeel, P., Sercu, T., and Rives, A. (2021). MSA transformer. In Proceedings of the 38th International Conference Machine Learning (ICML), volume 139, pages 8844-8856. PMLR.
Renz, Luukkonen, S., and Klambauer, G. (2024). Diverse Hits in De Novo Molecule Design: Diversity-Base Comparison of Goal-Directed Generators. Journal of Chemical Information and Modeling, 64(15) 64(15555766-5711
Riesselman, A. J., Ingraham, J. B., and Marks, D. S. (2018). Deep generative models of genetic variation capture the effects of mutations. Nature Methods, 16--022
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L.,, Ma, J., and Fergus, R. (2021). Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15).
Ross, Belgodere, B., Chenthamarakshan. V., Padhi, I., Mrouch, Y., and Das, (2022). Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, 4(22):125-1224.
Scalzitti, N., Kress, A., Orhand, R., Weber, T., Moulinier, L., Mlaamin--Grardoon A., Collet, P., Poch, O., and Thompson, D. (2021). Spliceator: multi-species splice site prediction using convolutional neural networks. BMC Bioinformatics, 22:1-26.
Schiff, Y., Kao, C. H., Gokaslan, A., Dao, T., Gu, A., and Kuleshov, (2024). Caduceus: Bi- directional equivariant long-range DNA sequence modeling. In Proceedings of the 41st Interna- tional Conference on Machine Learning (ICML), volume 235, pages 43632 43648. PMLR.
Schimunek, J., Seidl, P., Friedrich, L., Kuhn, D., Rippmann, F., Hochreiter, S., and Klambauer, G. (2023). Context-enriched molecule representations improve few-shot drug discovery. In International Conference Learning Representations (ICLR), volume 11.
Schwaller, P., Laino, T., Gaudin, Bolgar, P., Hunter, C. A., Bekas, C., and Lee, A. (2019). Molecular transformer: model for uncertainty-calibratee chemical reaction prediction. ACS Central Science, 579):1572-1583.
Schwaller, P., Probst, D., Vaucher, A. C., Nair, V. H., Kreutter, D., Laino, T., and Reymond, J.-L. (2021). Mapping the space of chemical reactions using attention-base neural networks. Nature Machine Intelligence, 1445
Segler, M. H., Kogej, T., Tyrchan, C., and Waller, M. (2018). Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Central Science, 4(1):120-131.
Seidl, P., Vall, A., Hochreiter, S., and Klambauer, G. (2023). Enhancing activity prediction models in drug discovery with the ability understand human language. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202, pages 30458-30490 PMLR.
Sgarbossa, D., Malbranke, C., and Bitbol, A.-F. (2024). ProtMamba: homology-awar but alignment- free protein state space model. bioRxiv, pages 2024-05.
Shrikumar, A., Greenside, P., and Kundaje, A. (2017). Reverse-complement parameter sharing improves deep learning models for genomics. BioRxiv, page 103663.
Skuta, C., Popr, M., Muller, T., Jindrich, J., Kahle, M., Sedlak, D., Svozil, D., and Bartunek, P. (2017). Probes drugs portal: an interactive, open data resource for chemical biology. Nature methods, 14(8):759-760.
17

---

Published as a conference paper at ICLR 2025
Srivastava, R. K., Greff, K., and Schmidhuber, J. (2015). Training very deep networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 28, pages 2377-2385. Curran Associates, Inc.
Stanley, M., Bronskill, F., Maziarz, K., Misztela, H., Lanini, J., Segler, M., Schneider, N., and Brockschmidt, M. (2021). FS-Mol: A few-shot learning dataset of molecules. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. (2024a). Roformer: Enhanced transformer with rotary position embedding. Neurocompuring, 568: 127063.
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., and Yuan, F. (2024b). SaProt: Protein language modeling with structure-awarc vocabulary. In International Conference on Learning Representations (ICLR), volume 12.
The UniProt Consortium (2023). Uniprot: the universal protein knowledgehase in 2023. Nucleic Acids Research, D1(D1):D533-D531
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro. E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. (2023). LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2300..333 139711.
Truong Jr, T. and Bepler, T. (2023). PoET: A generative model of protein families as sequences-of- sequences, In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 17379-774155. Curran Associates, Inc
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pages 5998-6008. Curran Associates, Inc.
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., and Manzagol, P-A. (2010). Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11(110):3371 13408.
Wang, R., Wang, Z., Wang, J., and Li, S. (2019a). Splicefinder: ab initio prediction of splice sites using convolutional neural network. BMC Bioinformatics, 20:1-13.
Wang, S., Guo, Y., Wang, Y., Sun, H.. and Huang. J. (2019b). SMILES-DEE -ERET:::-TT::-- large scale unsuper- vised pre-training for molecular property prediction. In Proceedings of the 10th ACM International Conference Bioinformatics, Computational Biology and Health Informatics, pages 429 436.
Wang. Y., Zhao, H., Sciabola, S., and Wang. W. (2023). cMoIGPT: conditional generative pre-trained transformer for target-specific de novo molecular generation. Molecules, 28(1 1):4430.
Weininger, D. (1988). SMILES, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of Chemical Information and Modeling, 28(1):31-366 1-36.
Wu, Y. and He, K. (2020). Group normalization. International Journal of Computer Vision, 1280)):42-755.
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., and Pande, V. (2018). Moleculenet: a benchmark for molecular machine learning. Chemical science, 9():51135530.
Xie. Y.. Xu, Z., Ma, J., and Mei, Q. (2023). How much space has been explored? measuring the chemical space covered by databases and machine-generater molecules. In International Conference on Learning Representations (ICLR), volume 11.
Yang, K. K., Wu, 7.,, and Arnold, H. (2019). Machine-leaminn-guueee guiidedggggdddddg directed evolution for protein engineering. Nature Methods, 16(8):687-64.
Yang, M., Huang, L., Huang, H., Tang. H., Zhang. N., Yang, H., Wu, J., and Mu, F. (2022). Integrating convolution and self- attention improves language model of human genome for interpreting non- coding regions base-resolution. Nucleic Acidy Research, 50(14)e811-e)
18

---

Published as a conference paper at ICLR 2025
Yang. S., Wang, B., Shen, Y., Panda, R., and Kim, Y. (2024). Gated linear attention transformers with hardware-efficiee training. In Proceedings of the 41st Interaational Conference on Machine Learning (ICML), volume 235, pages 56501-56522 PMLR.
Zdrazil, B., Felix, E., Hunter, F., Manners, E. J., Blackshaw, J., Corbett, S., de Veij, M., Ioannidis, H., Lopez, D. M., Mosquera, J., Magarinos, M., Bosc, N., Arcila, R., Kizilören, T., Gaulton, A., Bento, A.. Adasme, M., Monecke, P., Landrum, G., and Leach, A. (2023). The ChEMBL database in 2023: drug discovery platform spanning multiple bioactivity data types and time periods. Nucleic Acids Research, 52(D1):D1 180-D1192.
Zhang. Q., Ding, K., Lv, T., Wang, X., Yin, Q.. Zhang, Y., Yu, J., Wang, Y., Li, X., Xiang. Z., Zhuang, X.. Wang, Z., Qin, M., Zhang, M., Zhang, J., Cui, J., Xu, R., Chen, H., Fan, X., Xing, H., and Chen, H. (2025). Scientific large language models: A survey on biological chemical domains. ACM Comput. Surv., 57(6).
Zhavoronkov, A., Ivanenkov, Y. A., Aliper, A., Veselov, M. S., Aladinskiy, V. A., Aladinskaya, A. V. Terentiev, V. A.. Polykovskiy, D. A., Kuznetsov, M. D., Asadulaev, A., et al. (2019). Deep learning enables rapid identification of potent DDRI kinase inhibitors. Nature Biotechnology. 1739):003-11440
Zhou, H., Shrikumar, A., and Kundaje, A. (2022). Towards better understanding of reverse- complement equivariance for deep learning models in genomics. In Machine Learning in Compu- tational Biology, pages 1-33. PMLR.
Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R. V., and Liu, H. (2024). DNABERT-2: Efficient foundation model and benchmark for multi-species genomes. In International Conference on Learning Representations (ICLR), volume 12.
19

---

Contents
A Related Work
21
xLSTM Architecture Details
23
B.1
XLSTM and Bio-xLSTMM Blocks
23
B.2 Modes of Operation: Parallel, Chunkwise or Recurrent
23
B.3 Efficient Bidirectional Modeling for Weight-Tied Layers of Bio-XLSTM
24
C DNA-xLSTM: Details and Additional Results
26
C.1 Pre-Training
26
C.2 Downstream Tasks
27
C.3 Architecture and Hyperparameters
28
C.4 Reverse-Complement Invariance
28
C.5 Implementation Details
30
Prof-XLSTM: Details and Additional Results
32
D.I Homology-Aware Training
................................
32
D.2 ICL:
Protein Generation
................................
34
D.3 Protein Variant Fitness Prediction
................................
36
E Chem-xLSTM: Details and Additional Results
38
E.1
Unconditional Molecule Generation
................................
38
E.2 Conditional Molecule Generation with In-Context Learning
................................
39
E.3
Architecture and Hyperparameter Selection
40
E,4
Implementation Details
40
E.5
Additional Results..
40
Compute Demand and Resources
41
Extended Discussion
42
20

---

Published as a conference paper at ICLR 2025
A RELATED WORK
In all three areas, genomies, proteomics, and chemistry, observe similar trend that until around 2018 the language models were based on LSTMs (Hochreiter and Schmidhuber, 1997), then large amount of models were based on Transformers (Vaswani al., 2017), with different modeling paradigms and styles, and from 2023 onwards SSMs appeared.
Language models for genomic sequence data. DNABERT (Ji al., 2021) and its successor DNABERT-2 (Zhou et al., 2024) are Transformer-busee models that leverage bidirectional encoder representations and masked language modeling to capture nucleotide context, achieving high per- formance in tasks like promoter and splice site prediction. LOGO (Yang et al., 2022), another Transformer-bast model, applies self-supervised learning to the human genome for sequence la- beling and variant prioritization, while VIBE (Gwak and Rho, 2022) employs a hierarchical BERT model to enhance the detection of eukaryotic viruses in metagenomic data. Models like Looking- Glass (Hoarfrost et al., 2022), based on recurrent neural network (RNN), and GPN (Benegas et al., 2023), which uses convolutional neural networks (CNNs), are examples of non Transformer- based approaches, with LookingGlass focusing on microbial genomes and GPN on plant genomes. More recent developments include the nucleotide transformer (NT) (Dalla-Torr) et al., 2024), a Transformer model trained the human genome and data from the 1000 Genomes Project, and SpeciesI.M (Karollus et al., 2024), which trains Transformer-basse advances, represented by Caduceus (Schiff al., 2024) based on Mamba (Gu and Dao, 2024) and HyenaDNA (Nguyen et al., 2023), introduce SSMs that allow generative modeling and representation learning for long DNA sequences.
models on 1500 fungal genomes. The latest
Language models for protein sequence data. Until around 2019, the field was dominated by RNNs and LSSeM models trained with CL.M. Notable examples include UniRep (Alley et al., 2019), which employed multiplicative-LSTTM to capture rich protein representations, and SSA (Bepler and Berger, 2019), which used bidirectional RNNs for structural similarity prediction. Since then the field has shifted towards Transformer-base models and MLM, driven by their success in natural language processing. Early adopters of this shift included the TAPE benchmark for protein downstream tasks (Rao et al., 2019), which evaluated both an LSTM and Transformer architecture trained with CLM and MLM, respectively. Elnaggar et al. (2021) further expanded the use of Transformers with large-scale MLM, setting new benchmarks in protein sequence analysis with Prot-T5. ESM (Rives et al., 2021) applied ML.M to a Transformer on a massive scale, capturing evolutionary patterns across models include MrA-Transformer
diverse protein sequences. Other significant Transformer-basse (Rao et al., 2021), which applied MLM to multiple-sequence aligmments (MSA), and ProGen (Madani et al., 2023), which used CLM and Transformers for controlled protein sequence generation. Additionally, models like ProtGPT2 (Ferruz et al., 2022) and ProteinBERT (Brandes et al., 2022) utilized the power of Transformer architectures in generating novel protein sequences and functional predictions. Furthermore, (Su et al., 2024b) introduced a "structure-aware vocabulary" which they use as input for Transformer-based models. The recently proposed PoET (Truong Jr and Bepler, 2023) is an autoregressive Transformer model trained non-aligned homologous sequences, providing novel approach for conditional protein design and protein fitness prediction. Building on the concept of non-aligned homologous sequences, ProtMamba (Sgarbossa et al., 2024) leverages emerging SSMs to manage long-contexs conditioning on proteins, effectively utilizing autoregressive and FIM strategies. For more comprehensive review of these advancements, including their applications in functional protein design, see Notin et al. (2024) and Hu et al. (2022).
Language models for chemical sequence data. The first language model for chemical sequences was an LSTM-based, autoregressive method developed by Segler et al. (2018), which demonstrated that the SMILES syntax (Weininger, 1988) and generation of realistic organic molecules can be learned. Honda et al. (2019) introduced a Transformer model for this task, showing that this leads to informative representations of molecules. The Molecular Transformer (Schwaller et al., 2019) consists of a Transformer-basee encoder and decoder, trained on chemical reaction data to translate between reactants and products. More recently, SSMs have been used for generative modeling of SMILES strings (Özçelik et al., 2024). Subsequent models such as MolGPT (Bagal et al., 2021) and cMolGPT (Wang et al., 2023) utilized the GPT architecture to generate SMILES strings. with MolGPT conditioning on chemical properties and scaffolds, and cMolGPT focusing biomolecular targets. Transformer-busee approaches have also been employed to optimize the properties of small molecules in eeinforeement-eemmiie setting (Mazuz et al., 2023). Encoder-style language models
21

---

Published as a conference paper at ICLR 2025
for chemistry, such SmilesLSTM (Mayr al., 2018), ChemNet (Preuer al., 2018), and CNN- based models (Jastrzębski et al., 2016), initially used activity and property prediction as pre-training training objectives. Later, these encoder-style language models were trained with the masking language modeling objective, as seen in ChemBERTA (Chithrananda et al., 2020), Chemberta-2 (Ahmad al., 2022), AMILES--BRRT (Wang al., 2019b), MolFormer (Ross et al., 2022) and Ixnfp-BERT (Schwaller et al., 2021). Some models have also adopted contrastive objectives (Seidl al., 2023). Large language models for molecules have also been shown to learn complex molecular distributions (Flam- (Flam-Shepherd et al., 2022). For more thorough and comprehensive overview, we refer the surveys by Bran and Schwaller (2024) and Zhang et al. (2025)
22

---

Published as a conference paper at ICLR 2025
B XLSTM ARCHITECTURE DETAILS
B.I XLSTM AND BIO-XLSTM BLOCKS
Beck et al. (2024) suggested XLSTM blocks, which are residual (Srivastava et al., 2015; He et al., 2016) block modules, into which the sLSTM and mLSTM layers can be integrated. The basic blocks can in principle be characterized by either applying post-sLSTM/LLSTY up- and down- projections (similar Vaswani et al. (2017)) or by applying pre- Dre-LLSTM/mLSS and post-LLSTM/LLSTI
up-projections down-projections (similar to Dao (2024)). An sLSTM block integrates the sL.TM layer into the up- and down-projection block, while the ml ML.TMM block integrates the MLSTM layer into the pre-up-projection and post-down- cete-doo--rootttionn block. The two basic xLSTM blocks also make use of neural network modules like layer normalization (Ba et al., 2016), short causal convolutions, and, group normalization (Wu and He, 2020). For the exact architecture of the blocks, we refer to Beck et al. (2024, Sec.2.4). An xLSTM architecture is constructed by residually stacking the suggested xLSTM blocks. For that, the most commonly used ne-LLayrNormm residual backbone is used.
mLSTM Block
sLSTM Block
Bi-mLSTM Block
Flip
Down
Down
Down
ON
Up Gate
DN
Conv
Conv
Up + Gate
Up + Gate
Conv
LK
Flip
LN
Figure Al: xLSTM and Bio-XLSTM blocks. Left: mLSTM block. LN (Layer Normalization) and GN (Group Normalization) refer to normalization modules, while L Skip represents learnable skip connections and Conv denotes causal ID convolutions. The mLSTM block utilizes gated pre-up- projection structure, akin to modern State-Space Models, with gates activated by the Swish function. Middle: sLSTM block. The sLSTM block features GELU-gated post-up-projection structure, similar to Transformer architectures. Right: Bidirectional mLSTM block. For bidirectional processing, the XLSTM applies each block the input sequence twice before combining the outputs: once left-to-right and once right-to-left.
For Bio-xLSTM we keep the basic xLSTM building blocks and the basic xLSTM architecture template, but adjust them to the respective domains. Figure A1 depicts sLSTM and mLSTM blocks, as well as, a bidirectional mLSTM configuration with weight-tied layers.
B.2 MODES OF OPERATION: PARALLEL, CHUNKWISE OR RECURRENT
Similar to linear attention variants (Katharopoulos et al., 2020; Yang et al., 2024), the mLSTM has three possible formulations: parallel, recurrent or chunkwise. The presentation section 2.2 (and Beck et al., 2024) focuses on the recurrent form:
C, = a (f.)) C.-1 + exp(i:) o v,kkT
n, = a off) n:-1 + exp(ii) k
Ciqi
h = の (ō,) i max ())
This form is especially useful for inference when samples arrive one time-step at time.
23

---

Published as a conference paper at ICLR 2025
The omission of the recurrent connections in mLSTM allows for parallel formulation (Beck et al., 2024, Appendix):
Ino(fe) Ino(f2)) (fa
0
F
Inf(fa)
O
E3 (no
D exp F + OM M
(19)
DOQK'
H ( max
V.
D O QK 1,1)
where K, K, T iT and M E f0, 10 is a causal (i.e. lower-triangular) masking matrix. The & refers outer product, while O is Hadamard (i.e. element- clement-wise) product. The fraction, max and other non-linear functions are also applied element-wise. This parallel form enables an efficient training regime, similar to Transformers.
The chunkwise formulation is hybrid the recurrent and parallel forms:
0
(fr = (no(f-c(2) + (
0
0
0
(nt(fi-c4a)
O
DC hos D ex (1
(f
II ((-cc)))
(Do QK diag(F (lcggf)
H = (
max [I QKT) diag(f (f) Qn.-cl.1)
C
C. =
Il o ((-
Ci-c + V diag(dc)
II (f+-ct+) m.c + KTdc.
with Q, K, K,K.V. E RCxd and i E RC the pre-activations from L- to L. Furthermore, M {0, 10 a local causal (i.e. trianulaa)) masking matrix, de denotes the last row of D, diag transforms a vector into diagonal matrix, and C' is the chunk size. The chunkwise formulation allows for implementing hardware-aware efficient kernels (Beck et al., 2025). For C we recover the recurrent form, whereas for C = T, we obtain the parallel form.
B.3 EFFICIENT BIDIRECTIONAL MODELING FOR WEIGHT- TIED LAYERS OF BIO-XLSTM
Bidirectional modeling is often required to learn representations of biological and chemical sequences, for example with the MLM paradigm. The default approach for bidirectional modeling would to use an mLSTM layer on the usual sequence X1-7 = (æ1, .2,... 2T) and then applying weight- tied layer on reversed sequence XT.1 = (æT, -1 x1) and subsequently summing those outputs:
H mLSTM(X 1 w)
(20)
H mLSTM (T.T:))
(21)
H H H
(22)
24

---

Published as a conference paper at ICLR 2025
where Hy, indicates that the sequence is reversed again, and the parameters of the LSTM- layer mLSTM(X (1:r; w) which are assumed to be the same for both directions, i.c. weight-tied. This approach is schematically depicted in Figure Al (Right). However, this approach is inefficient with respect to memory and operations because has to calculate and store all internal quantities, such the cell states, twice for the backward pass. A variant of this approach is perform the forward direction in one block (Eq. 20) and the reverse direction in a consecutive block (Eq. 21) of the architecture (Alkin al., 2024).
We propose an efficient bidirectional modeling approach. Instead of running the network twice, it is possible to compute both H and T with a single forward pass. This effectively reduces computational demands and memory. The simultancous computation is especially interesting for the parallel formulation of the ml M.STM. Concretely, H+ can directly computed using equation (19) and
DILIT 2 Qr K
max ( [PT.I @ Qr K .1,1) H7.1 = o(ÔT.)
DTaTA Q+K+T
= O
V+.
max DraTa O 2r K
1,11)
Because D is upper triangular, D DT.IT: lower triangular and the sum can be written as:
D
H+ + H = = max D Q x<+1--1.11
v+,
where
D= D + ++D DTTTA = exp( FF + 1
(fi(
Ii''mo(f
Ino(f2)
0
F=
(no(fr-2)
Ino(fr-2) + +nnofr-11) Ino(fr-1)
Ino(fi)
(no(fr-1)
Ino((
En
Ino(fr)
Note that this principle can also be applied in the parallel part of the chunk- wise formulation (see B.2).
25

---

Published as a conference paper at ICLR 2025
C DNA-XLSTM: DETAILS AND ADDITIONAL RESULTS
In this section, we provide further details regarding the architecture, training setup, and evaluation metrics for the DNA-xLSTM models.
Short-range DNA Modeling
Bi-sLSTM Blocks
Language Lodeling
Ai
TAO08
XL
Up Side
Le Ore
DNA-sLSTM
hg38
twc
1k Context
Longer-range DNA Modeling
Bi-mLSTM
Masked L Language Modelling Loss
XL
DNA-mLSTM
hg30
AeDE BONTE
6A
TAO
Figure A2: DNA-xLSTM Architecture. The top panel shows ONA-sLSTM trained with masked language modeling for short-range contexts (1,024 nucleotides) using Bi-sLSTMM blocks. The bottom panel depicts DNA-mLSTM, designed for long-range contexts (32,768 nucleotides) with sequence- parallel mLSTM blocks. Both models are trained on the human reference genome (hg38).
C.I PRE-TRAINING
Experimental setup. We followed the experimental protocol established in Schiff al. (2024) and Nguyen et al. (2023). The human reference genome (Church et al., 201 1) was used as the training dataset for two main tasks: a) causal language modeling (CLM) and b) masked language modeling (MLM). We employed context lengths of 1,024 and 32,000 tokens for these tasks.
To ensure fair comparison with previous methods, such Schiff al. (2024), used character- or base pair-level tokenization, training models with parameter sizes ranging from 500k to 4M. This experimental setup enabled us evaluate model performance for both a) generative modeling of core DNA sequences and b) learning rich DNA sequence representations core tasks in this domain.
Methods and hyperparameters. In our pre-training experiments, we compared several architectures: Transformer variant based on the Llama architecture, referred to as Transformer++ (Touvron al., 2023), DNA-xLSTM HyenaDNA (Nguyen et al., 2023), and DNA-Mamba (also known as Caduceus) (Schiff et al., 2024). Each architecture was trained under both CL.M and MLM settings. Additionally, we assessed two types reverse-complement (RC) equivariant models when applicable: DNA-Mamba-PHs and DNA-Mamba-PPl
well DNA-XLSTM-PHH and DNA-XLSTM-PS. For non non-cquivariant models, reverse-compe cooose-complementt augmentation was applied, following the approach described in Schiff et al. (2024). Further details on RC-equivariant modeling be found in Section C.4. The typerparameters for DNA-xLSTMM and Transformer++ were optimized using a validation set, with the final configurations reported in Appendix Tables A4 and A3,
Metrics. We report cross-entropy loss on a held-out test set for both CLM and MLM pre-training experiments.
26

---

Published as a conference paper at ICLR 2025
DNAXLSTIX
DNAXLSTM
Mamba
1.04
Mamba
1.12
Validation Loss
Transforment+ Hyena
Validation Loss
1.02
Transformer++
1.10
1.00
1.08
0.98
0.96
1.06
1.04
Billion Tokens
Billion Tokens
Figure A3: Pre-training of 4M-parameter DNA models on the human reference genome (GRCh38 The models are trained on the human reference genome single-nucleotide resolution with a context length of 32k bases. Left: causal language modeling. Learning curves display CLM loss (L) held-out test set, plotted against the number of tokens processed. Right: masked language modeling. Learning curves for bidirectional models trained with the MLM objective (4).. The DNA-xL RNA-xLSTM-44M model outperforms both Transformer++ and Hyena- -NN models similar size, and matches the performance of Caduceus-4M.
Results. Our experiments show that the sLSTM-based DNA-XLSTM-22M model, trained with a context size of 1,024 and compleent augmentation, outperforms DNA-Mamba (Schiff et al., 2024), HyenaDNA (Nguyen et al., 2023), and Transformer++ across both CLM and MLM tasks. Notably, the performance gap between DNA-XLSTM and the baseline models increases in the MLM setting. See Figure 2.
We further enhanced DNA-XLSTM-5000 and DNA-XLSTMM22M models by incorporating reverse- complement equivariance via parameter sharing. For smaller models, we achieved MLM losses comparable to DNA-Mamba-PS, with significant improvement over DNA-Mamba-PS as model size scaled to 2M parameters (Figure A4). Additionally, we pre-trained a long-range DNA-XLSTM model based on mLSTM, with a context size of 32k, using both CLM and MLM objectives. This model achieved the lowest cross- cross-entropy loss in both tasks, outperforming Transformers and HyenaDNA, while performing comparably to Mamba (Figure A3).
C.2 DOWNSTREAM TASKS
Experimental setup. Two sets of downstream tasks used for evaluating the learned repre- sentations: the Genomic benchmark (Grešová et al., 2023) and the Nucleotide Transformers Tasks (Dalla-Torre et al., 2024), which is collection of 18 datasets derived from five peer-reviewed studies (Phaml et al., 2005; Oubounyt et al., 2019; Wang et al., 2019a; Scalzitti et al., 2021; ; Geng et al., 2022). These classification tasks were selected to determine how rich the learned representations of the architectures are. To extract representations from the pre-trained xLSTM-DX average pooling on the activations from the final xLSTM block. For each downstream dataset, these representations served as inputs to task-specific classification head that were jointly fine-tuned with the pre-trained model parameters.
models, we perform
Methods and hyperparameters. For Nucleaotide Transformer tasks, we compared HyenaDNA, DNA-Mamba, and XLSTM- models tre-trained with 2M parameters. For Genomic benchmark tasks, we compare the smaller xLSTM-500K against Mamba. In both settings, models were pre-trained with a context size 1.024.
Metrics. For the Nucleotide Transformer downstream tasks different metrics used depending the type of task: MCC was used for histone markers and enhancer annotation, Fl-score was used for promoter annotation and splice site acceptor/donor, and accuracy was used for the splice site. The downstream performance on the Genomic benchmark was evaluated using the Top-1 accuracy.
Results. On the extensive set of downstream tasks, DNA-xLSTM is the best model with fewer than 2M parameters outperforming other small models on 12 of 18 tasks. comparison with much larger models, DNA-xL and is par with the 500M parameter model Nucleotide Transformer
27

---

Published as a conference paper at ICLR 2025
Table A1: Downstream adaption of DNA models (extended version). The test set performance of DNA models with 2M parameters and models with over 100M parameters, fine-tuned on Nucleotide Transformer classification tasks, shown. Models marked with PS or PH are trained be RC equivariant. The used metric is provided in the Metric column and best values are highlighted in green Results are averaged over random seeds, with error bars representing the difference between the maximum and minimum values across the runs. The best scores are highlighted in green. ILSTM-DNA-- models, while achieving comparable results to the much larger Nucleotide Transformer. Scores for all models except XLSTM were obtained from Schiff et al. (2024).
with 2M parameters outperforms similarly sized Hyena- and Mamba-base based
Metri
Param, DNABERT (117M)
Enformer (252M)
(500M)
HyenaDNA
Manbo
XLSTM
XLSTM
Markers
HIS
MCC1 1
0.71g A4.44
0.785-8a111
0.784460047
0.70946522
0.8154284
0.70644022
0.524
H3KI4AC
MCC1 T
0.288
0.616-5--2
0.561
0.612
0.541
0.57y+h011
0.508
HIKMMES
MCC 1
0.34gpM0080
0.50 (-50)-8bdaa
0.62545000)
0.013500m
0:00040330
0.80
0.584tbaca
0.025
HEKAMEI
MCC 1
0.29159
0.511--9-2
0.5gg4rmm
0.512
0
0.48840-312
1,523==1 oos
0.49g2taa:2
0.526
MCC
0.211 pALDes
0:395-8uaa
0.31945.045
0.458
0.38g40332
0.087
0.489450224
0.500--9
H3KAME3
0.30g
0.362=4-472
0.410
0.44
0.544
0.5gutroa
0.537
H3K20MEX
24000
06615-haa
0.60gan
0:007
0:0024t0a:1
0.007 Ao
H1XS9C
0.542:5-12
0.5824rina)
0.58
0.00gani
0.822
0,02gtra:
1I4
MCC1 1
0.295 Bv927
0, 000-4002
0.
0.811
07704
0.813
HEAC
MCCT T
0.27799/1e
0.46g
0.52546-343
01.221
0.56xphia
0.5ag
Regulotory Assutattoo
Enhancer
MCC 1
0,451
0.51648800
0.50546u144
0.51750.11
0.10 0.101
0.54640
0:37544603
0.545-9
Enhancer Types
MCC 1
0.30g
0.425-8-051
0.424461122
0.386
0 0.4164
0.48g2tv04
0.400-mmm
Promoser. All
F1
0.971 --000s piavoss
4.99g4
0,0ggasuas
0:9074
0,070
0.90244xa22
0.967
NOGTATA
0.97256--0
0.30 on
0.96gttoa
0.377
4..05
0,94
0.9673
0.053-2006
0.059 s00
Spiice
Accuracy
0.
6.056
0.ggees nos
0.905
0.974 mo
Acceptoc
Ft
0.51 a
0.975 -8005
0.981 Accau
0.938
0.506 40.077
0.937
0.970
0.903
Donce
F1
1
0.50g49/19
0.5665
0.985
0.94g
0.874
0.90240001
0,301
Table A2: Downstream adaption of DNA language models on the Genomics Benchmarks. The Top-1 accuracy (1) for RC-equivariant PS and PH xL.TTM and Mamba-based Caduceus models, both with 500k parameters, are shown. Error bars represent the range of scores across five random seeds. XLSTM achieves comparable overall performance to Mamba and demonstrates superior accuracy when both models employ post-hoc conjoining. Scores for all models except xLSTM were obtained from Schiff et al. (2024).
Mamha-PH-500k
xLSSTM-PH-500k
Mamha-PS-500k
xL-STM-P--500K
Mouse Enhancers
0.754
0.780*00018
0.793
0.778=000007
Coding. vs. Intergenomic
0.905 00.003
003100.000
0.910+00000 0.03
0.934
Human vs. Worm
0.973 1:0.001
0.965 1.0001
0.968 L.0002
0.950 10.001
Human Enhancers Cohn
0.747 1.0004
0.742 1.0005
0.745 0.745-10.000 1-0.0.0000
0.734 1.000
Human Enhancers Ensemble
0.893 0.893-10.00s
0.920
0.900 0.0
0.902 10.004
Human Regulatory
0.872++0.01 1-0.011
0.872 +-0.0200002
0.873++0.00
0.869 0.86g$40.000
Human OCR Ensembl
0.828 +0.000
0.826 0o:---0000 .022
0.818 0.818-+0.000
0.800 + #0.002
Human Non TATA Promoters
40.007
0.951 20.004 004
0.945 10.010
0.949
(NT-v2) winning 8 of 18 tasks (see Table A1). On the Genomic benchmark, DNA-xLSTM is overall on par with Mamba-DNV DNAA and shows especially strong results with posthoc conjoining, winning 5 of 8 tasks compared to Mamba-DNA-PH. Results are reported in Table A2.
C.3 ARCHITECTURE AND HYPERPARAMETERS
The hyperparameters and composition of the DNA-xLSTM and DNA-Transformer++ models for pre-traing pre-training with context size 1k and 32k are reported in Tables A4 and A3. The hyperparameters were selected on separate validation set using manual hyperparameter selection due to limited computational resources.
C.4 REVERSE- COMPLEMENT INVARIANCE
We develop an xLSTM version that is invariant to the RC of an input sequence which is relevant for DNA applications following Schiff et al. (2024). In double-helix DNA structures, both strands
28

---

Published as a conference paper at ICLR 2025
Table A3: Pre-training pypeparraeeerss for DNA-Transformrr+ models with 2M and 4M parameters.
Comma-separated values represent hyperparameter sweeps, with the chosen values indicated in bold.
Hyperparameters
DNA- 2M
Embedding Dimension
256
256
Number Blocks
Number of Heads
8
Up-Projection Ratio
1.25
2.0
Norm Bias and Linear Bias
false
false
Context Length
1,024
32,768
Position Embeddings
RoPE
RoPE
Learning Rate
6c-3, 8c-3, le-2
6c-3, 8c-3, le-2
1.03
DNA-XLS TM
1.00
DNA-XLSTIM
Validation Loss
1.02
Validation Loss
0.99
0.98
1.01
0.97
1.00
0.96
0.99
0.98
10
2
10
Billion Tokens
Billion Tokens
Figure A4: Pre-training of RC-equivariant -LSTT--NNAPPS and Caduceus-PP models with 500k and 2M parameters trained on the human reference genome. Models were trained on Ik context windows using the MLM objective. Left: MLL losses (L) for models with 500k parameters. Right: MLM losses (4) for models in the 2M parameter range. ONA-xLSTM-PP outperforms Caduceus-PS in both settings, with the performance gap widening larger scales.
are semantically equivalent, as one strand the RC of the other. Given a strand, D. its RC, E. is oriented in the opposite direction with base conversion from A to T and Shrikumar et al. (2017) show that a data-driven approach to learning the equivalence between reverse- complement sequences can fail, which why Schiff et al. (2024) propose to enforce RC-equivariance by design, making use of two different inductive biases in the model architecture: PH (Zhou et al., 2022) and PS. For PH models, sequence-to seguencceto-seeuencce sequence models in our case realized by the xLSTM learn to handle both DNA sequences and their RC during pre-training by applying RC augmentation to the inputs. RC augmentation refers to the process randomly replacing input sequences by their RCs. For downstream tasks PH models are applied once to the original sequence and once to the RC and eventually outputs are summed:
(Schiff et al., 2024).
Y = XX) XLSTM(X).
(23)
For PS models - we assume models are realized by xLSTM architectures and therefore a block refers to single mLSTM sLSTM block - both the DNA sequence and its RC are provided simultaneously to each block in the architecture (for both pre-training and downstream task fine- tuning). Precisely, joint representation, originating from combining sequence representation and its RC representation, is split into X E R Dxt and X E Dxt and fed into the mLSTM or sLSTM block:
H. H [block(X]) X), RC(block(RC(())))
(24)
Notably, for each block the reverse-complement input built by the RC-function which flips both dimensions of X and [ indicates concatenation along the first dimension. Eventually, logits for the input sequence and its reverse complement are combined. For more details, we refer to Schiff al. (2024).
29

---

Published as a conference paper at ICLR 2025
Table A4: Pre-training hyperparameters of DNA-xLSTM Models from 500k 4M parameters.
Comma-separated values represent hyperparameter sweeps, with the chosen values indicated in bold.
Hyperparameters
DNA-xLSTM-500K
DNA-XLSTM-2M
DNA-XLSTM-IM
Embedding Dimension
Number of Blocks
Conv Kernel Size
Number of Heads
Up-Projection Ratio
1.25 alternating, blockwise truc, false
1.0
2.0
Bidirectionality
alternating, blockwise true, false
alternating. native, blockwise
Norm Bias and Lincar Bias
truc
QKV Projection Blocksize
m/sLSTM ratio
[0:1]. [1:0]
[0:1]. (1:0]
[U:1]. [1:0] 32,768
Context Length
1,024
1,024
Position Embeddings
None
None
RoPE
Optimizer
AdamW ( (0.9.
AdamW 8 (0.9,
Adam , (0.9.
Learning Rate
6c-3, 8e-3, 1c-2
6e-3, 8e-3, le-2
6c-3. 8e-3. 1c-2
Learning Rate Schedule
Cosine Decay
Cosine Decay 1,000
Cosine Decay 1,000
Learning Rate Warmup Steps
1,000
Weight Decay
0.1
0.1
0.1
Dropout
Batch Size
1,024
1,024
Update Steps
10,000
10,000
10,000
C.5 IMPLEMENTATION DETAILS
For both CLM and MIM pre-training we perform 10,000 update steps holding the number of tokens per step constant 220 CLM models are trained using autoregressive next-token prediction. For MLM pre-training, we follow the methodology presented by Devlin et al. (2019), where 15% of the input tokens are masked and the model tasked to predict the corrupted tokens. Concretely, 80% of the masked tokens replaced by special [MASK] token, 10% are replaced by random tokens sampled from the vocabulary and 10% remain unchanged. For MLM settings, we use weight-tie tied bidirectionality default (see Section B.3). For long-context bidirectional modeling, we unidirectional xLSTM cells and alternate the modeling direction at each block.
To fine-tune pre-trained models downstream tasks, follow the framework from Schiff al. (2024). Pre-trained models are augmented with task-specific classification head, which trained on average-pooled activations from model's final block. During fine-tuning, all model parameters are unfrozen. For the Genomic benchmark, we perform five randomly seeded train-validation splits, fine-tune models for 10 epochs, and early-stopping on validation performance. Final test results are reported as the mean performance max/min over the 5 seeds on held-out test set. For the Nucleotide Transformer tasks, we use 20 epochs and 10 seeds. For both the Genomic benchmarks and the Nucleotide Transformer tasks, we performed a hyperparameter search for both PH and PS models over batch sizes (64, 128, 256, 512), and learning rates (4e-4, 6e-4, 8e-4, le-3, 2e-3). The best results for each Nucleotide Transformer task can be found in Table A5.
30

---

Published as a conference paper at ICLR 2025
Table A5: Hyperparameter selection for DNA-xLSTM-PS and DNA-XLSTM-Ph on Nucleotide
Transformer tasks. Fine-tuning hyperparameters were chosen based on best scores averaged over train-validation splits.
DNA-XLSTM-Ph
DNA-XLSTM-PS
Learning Rate
Batch Size
Learning Rate
Batch Size
Histone Markers
H3
8e-4
128
4e-4
64
H3K14AC I4AC
6e-4
128
4e-4
64
Н3К36ME3
6e-4
64
4e-4
64
НЗK4ME1
8e-4
128
le-3
128
H3K4ME2
6e-4
64
2c-3
512
Н3K4ME3
8e-4
128
le-3
512
H3К79ME3
le-3
128
4e-4
НЗК9АC
4e-4
64
le-3
128
8e-4
6e-4
H4AC
4e-4
64
le-3
128
Regulatory Annotation
Enhancers
2e-3
512
2e-3
512
Enhancers Types
2c-3
512
2c-3
512
Promoter All
4e-4
64
1e-3
128
Promoter No TATA
le-3
128
le-3
128
Promoter TATA
3e-3
128
le-3
128
Splice Site Annotation
Splice Sites All
8e-4
64
2e-3
128
Splice Sites Acceptor
2e-3
128
2e-3
128
Splice Sites Donors
3e-3
128
2e-3
128
31

---

Published as a conference paper at ICLR 2025
D PROT-XLSTM: DETAILS AND ADDITIONAL RESULTS
This section provides further details regarding the architecture, training setup, and evaluation metrics for the Prot-XLSTM models. Additionally, we present supplementary results that complement the main findings discussed in Section 4.2. We adopted the experimental protocols outlined Sgarbossa al. (2024) train and evaluate our Prot-xLSTM models. We conducted three key experiments to assess the models' capabilities: a) protein language modeling (Section D.1), b) homology- conditioned protein design (Section D.2), and e) protein variant fitness prediction (Section D.3).
Aaanology-Awarr Pre- Training
Zero-Shot Fitness Prediction
Carnularga Leas
XL
F- ( logg(*?
Prot-xLSTM
Prot-XI.STM
BOD
BRAC
Up--San Oate
Homology-Awer Aware Generation
Texahee Proepp treasing
Prot-xLSTM
Prot-XLSTM
Figure A5: Prot-xLSTM Framework for Homology-Aware Protein Modeling. Left: Homology- Aware Pre-Traie Tre-Training Prot-XLSTM, a 100M parameter causal LSTM model, is pre-trained on homol- ogous proteins using fll-in-the-middll (FIM) augmentations. Homologous protein sequences from the OpenProteinSet are concatenated, creating input contexts ranging from 211 to 2 tokens. The model is trained with ext-tokee-ppredictin loss. Right Top: Zero-Shot Variant Fitness Prediction Prot-XLSTM predicts protein variant fitness by conditioning on homologous context. Fitness is calculated as the log-likelihood difference between mutant and wildtype sequences and evaluated using the ProteinGym benchmark, where predicted fitness scores are correlated with real-world assay results. Right Bottom: Homology-Aware Protein Generation Prot-xLSTM generates proteins by conditioning on sequences from specific families. Generated sequences are scored and compared to real proteins evaluate their alignment with family-specific properties.
D.1 HOMOLOGY-AWARY TRAINING
Data. The protein language model training data was derived from the filtered OpenProteinSet (Ahdritz et al., 2023), comprising 270k UniClust30 MSA clusters that included a total of 508M sequences and 110B residues. We used the ProtMamba pipeline to construct the training data, which illustrated in Figure 1 of Sgarbossa et al. (2024) and involved two key steps: (i) creating homology-aware but alignment-free training inputs by concatenating unaligned homologous sequences, and (ii) masking patches of tokens in each sequence and concatenating the unmasked patches at the end of each sequence to train the model autoregressively with the FIM strategy. We also use the train, validation (192 clusters), and test (500 clusters) split provided by ProtMamba.
Methods and hyperparameters. We trained two versions of the model: STM--X. and Prot-XLSTM-102M The larger model was designed to match the architecture and scale of the original ProtMamba model (ProtMamba-1077M in terms of layer count and embedding size. We optimized the Prot-xLSTM architecture, including block types and positional encodings, smaller Prot-XLSTTM-6NN model, and then applied these optimized architectural choices to the larger Prot-xLSTM- 102M model. For comparison, (ProtMamba-288) with an embedding dimension of 512) and implemented a LLaMA- model PPottTransformer+4-226 (Touvron al., 2023). Both models incorporate Absolute Positional
the
also trained smaller ProtMamba model
32

---

Published as a conference paper at ICLR 2025
Encodings (AbsPE) implemented in ProtMamba for xLSTM blocks, with RoPE applied specifically to the mLSTM blocks. As the sLSTM blocks (or Mamba) lack QK -ffffffomlaao,n RoPE cannot be directly applied them. The results of the hyperparameter search provided in The results of the hyperparameter search are reported in Table A7, and the composition of the Prot-xI.STM, and Trosfrransfomme++++ models reported in Table A6.
Table A6: Hyperparameter space considered for the Prot-xLSTMM and Prot-Transformer++ at different sizes, The selected values are marked in bold.
Hyperparameter
Prot-xl MMM-xLSTTM-26M
Prot-XL_STM-102M
Prot-Transformer+++26M
Embedding dimension
512
1024
512
Context length
2 2 17a
111118
2
Number of blocks
16
16
M/sLSTM ratio
[0:1], [1:0], [1:7]*
[1:0]
Conv 1D kernel size
4
QKV projection blocksize
4
4
Number of heads
4
Up projection dimension
1024
2048
2176
Norm bias and linear bias
False
False
False
Position embeddings
AbPE, RoPE
RoPE
RoPE
Context length was increased during training. sLSTM blocks at position 1 and 15.
Table A7: Prot-xLSTM hyperparameter search: Training loss comparison across different protein
language model architectures after training tokens.
Model type
#p (M)
Positional Encodings
Train loss
Mamba
27.7
АbPE
2.623
Transformer++
26.4
RoPE
2.568
25.8
2.694
sLSTM
26.3
AbPE
2.688
25.9
2.569
26.4
AbPE
2.545
mLSTM
25.9
RoPE
2.524
102
RoPE
2.482
25.9
2.554
XLSTM
26.4
AbPE
2.551
Training details. We trained our models using the ProtMamba pipeline with CLM with the FIM strategy. The pipeline efficiently handles long, concatenated sequences by extending the context length up = 288 supported by a context-length scheduling strategy. For the Prot-XLSTM-IOW model, we adhered to the ProtMamba protocol, gradually increasing the context length from 2 T 218 doubling 7 each stage when the loss plateaued. contrast, for the smaller models recommended in previous work (Devlin et al., 2019;
(Prot-xLSTM-261 and ProtMamba-28M), Press et al., 2021), we initially trained with T 2 for 20B tokens, then switched T 27 for an additional 10B tokens. Due to the quadratic scaling of Transformer architectures, Prot-Transsfrmerr++ 26M was only trained with 211 as it could not handle the computational demands of 217 Given the substantial computational resources required, did not fine-tune the training parameters. Instead, we used the default settings established by ProtMamba, which are reported in Table A8.
During training, we evaluated the next-token prediction capabilities of the models using negative log-like ligeliiil and token perplexity. The perplexity was calculated for four subset of tokens of the concatenated FIM sequence: the first protein sequence, the last protein sequence the FIM token and the entire concatenated sequence. Once the models were trained we evaluated their performance on the independent test set. We assessed the significance of the test results using pairwise Wilcoxon
Metrics.
33

---

Published as a conference paper at ICLR 2025
Table A8: Hyperparameters for training protein sequence models.
Effective batch sizeab
64-1
Optimizer
AdamW B (0.9, 0.95)
Learning rate Learning rate scheduler
6e-4
constant
Learning rate warmup steps Weight decay
500
0.1
Dropout
0
Decreaseed with context size maintain fixed total number of tokens per batch. For the larger model, the rule was relaxed for I' >= 2 enable multi-GPU training, with the batch size set to #GPUs. b Prot- Transformer++ was trained on 6 GPUs with an effective batch size of 96 and a learning rate of 9c-4, Due to unstable training of the larger model at 7 = 2" and 2 the learning rate was reduced to Ic-4.
signed-rank tests on the predictions of the different models. The resulting p-values from these tests are presented Figure A6.
Eokens
takens
ProtKiSTM
2-
36-70
26-73
04
00
26:26
10
aLs 201 Su
20
20112 3u
1o100
The Pambb 2014 Jo 07
Jel
2e110
22100
1c100
2652
20112 Xe 20107
1:100
20100
20100
NW INAASIN 5328
Aetimonna 137
POOVELST 30M
WeoMenma 201
PROMATE 201
Rose STM
Figure A6: Pairwise one- sndd Wilcoxon signed-ranh -aankd test valuee comparing test set perplexities of
different protein language models across various token subsets.
D.2 ICL: HOMOLOGY-CONII CONDITIONED PROTEIN GENERATION
Experimental setup. To evaluate the capacity of SSTM to autoregressively generate novel protein sequences given context of known homologs, we follow the protocol outlined in Section 3.4 of Sgarbossa et al. (2024). For a subset of 19 homology clusters from the test set, we generate sequences with contexts consisting of 10, 100, 500, 1000, and N (total number of sequences in the cluster) sequences. For each context length, we generate 100 sequences each with the following parameter combinations of generation temperature (r), top-k, which restricts the output selection to the & most probable tokens, and top-p. which limits the output to tokens reaching a cumulative probability p: (T, top-k, top-p) {00.8, 10, 0.9), (0.9, 10, 0.95), (1, 10, 0.95), (1, 10, 1), (1, 15, 1)) (Ferruz et al., 2022). This results in total of 2,500 sequences per cluster.
Methods compared. We compare both Prot-xLSTM models to ProtMamba models with a similar number of parameters. Additionally, we generated sequences using PoET (Truong Jr and Bepler, 2023), a Transformer-based homology-aware protein language model, following the same procedure. All generations were performed with a single A100 64GB GPU. While the xLSTM and Mamba- based models successfully generated sequences across all settings, PoET encountered out-of-memory issues for most clusters when using long contexts: it could handle up to 100 context sequences in 6 clusters, up to 500 in 11 clusters, and only clusters with the full N context sequences. Furthermore, we compared these models with baseline Hidden Markow Models (HMMs) with 100 states, trained on the respective cluster's MSA using the Baum- Welch algorithm (Baum et al., 1970). For each cluster, generated and evaluated 100 sequences using the HMMs.
Metrics. We evaluate the similarity of generated sequences by calculating the normalized Hamming distance to the closest natural sequence in the cluster using pairwise Smith- Waterman alignment (min dai). Additionally, we measure sequence similarity to natural homologs with the HMMER score from Hidden Markov Model (HMM) trained on the cluster's 's MSA (Eddy, 2020). To assess the
Smith-Waterman
34

---

Published as a conference paper at ICLR 2025
foldability, the generated sequences are also folded using ESMFold (Lin et al., 2023) and assessed by pTM (Evans et al., 2022) and average pLDDT confidence scores (Jumper et al., 2021). I). To compare these scores with natural sequences, computed the Satmmoorrv--mmmmnoe test statistic between the scores of 100 natural sequences and the 100 generated sequences with the lowest perplexity. Additionally, we analyzed the diversity of the generated sequences using adapted version of the #Circles metric to evaluate the number of diverse sequences among the top 100 (lowest perplexity) generated sequences. The #Circles metric, originally proposed by Xie al. (2023), exclusion-basee based method designed to quantify the diversity of small molecules in a set using Tanimoto similarity. We adapted this metric for protein sequences by substituting Tanimoto similarity with the Hamming distance and applied threshold of 0.3 to determine whether two sequences are considered diverse. Finally, we normalized the #Circles metric by the total number sequence and called this the Coverage.
sphere-
Results. We first examined the correlation between protein scores and sequence perplexities. Table A9 shows that the min daH, HMMER score, P'TM, and pLDDT all correlate with sequence perplexity for both Prot-XLSTM and ProtMamba. For the large models, the average Spearman correlation coefficients are 0.57 and 0.59, respectively. Given this correlation, we limit our subsequent analysis for each model and cluster to the 100 sequences with the lowest perplexity.
Table A9: Score distribution comparison of natural and generated proteins. Average Spearman correlation between model perplexity and sequence scores for sequences generated with Prot- xLSTM and ProtMamba models. Error bars indicate 95% confidence intervals across 19 test clusters.
Proi-XLSTM-26M
ProtMamba-28M
Prot-XLSTMM-10M
ProtMamba-
min dal
0.65 +0.08
0.46##0.11
0.68+40.10
0.61 #0.11
HMMER Score
0.61 #0.06
0.55#0.08
0.53#0.08
0.57#0.10
PLDDT
0.59
0.45=0.00
0.52#0.07
0.56#0.08
PTM
0.64 0.644...05 1:0.55
0.54
0.55
0..09
Figure A7 shows the distribution of scores for 100 randomly sampled natural sequences from each cluster as well as the 100 sequences with the lowest perplexity generated by STX--00M-- and ProtMamba- M models for 10 randomly selected clusters. The averages across all 19 evaluated test clusters are shown in Table 10 and in Table Table All we report Kolmogorov-Smimit
test statistics between metrics of the 100 natural sequences and the 100 generated sequences. First and foremost, both Prot-xLSTT and ProtMamba outperform the baseline HMM and Transformer models in generating higher-quality sequences. Both small and large Prot-xLSTM and ProtMamba models generate a diverse set of sequences that are, average, longer than their natural homologs. Notably, for the smaller models, Prot-XLSTM generates sequences with properties more closely resembling natural homologs compared to ProtMamba. For the larger models, the difference in quality between Prot-xLSTM and ProtMamba becomes less pronounced. However, Prot-xLSTM sequences remain slightly more similar natural homologs, indicated by lower Hamming distances and higher HMMER scores. This increased similarity in the larger Prot-xLSTM model comes at a slight cost to sequence diversity. Finally, all models, except ProtMamba-2881 produce sequences with foldability scores that are highly comparable to those of natural homologs.
PoET
Table A10: Score comparison of natural and generated proteins. Average sequence length, scores (min daH, HMMER, pLDDT, and pTM), and the coverage of diverse sequences across 19 test clusters for sequences generated with Pro and ProtMamba models. Error bars indicate 95% confidence intervals across clusters
Natural Sequences
HMM -12K
PoET -200M
Prot-xLSTM -26M
ProtManba -28M
Prot-XLSTM -102M
ProtMamba -107M
Seq. Len.
291
216
35
290
326+:3
28p238
276=40
min datt 1
0.51
0.72
0.66 0.66
0.55+t0.05
0.64
0.44
0.56
HMMER f
96=25
2.29900.8
80.15+17.35
1822556
122
165
163=45
PLDDT
0.81 20.03
0.49:40.02
0.65
0.79440.04
0.6740.07
0.80=m.00
0.80
PTM
0.7720000
0.2310001
0.3720.05
0.74
0.54
0.752000
0.74
Coverage
19
100=0
94
924.
92
87:110
99
35

---

Published as a conference paper at ICLR 2025
Modek
Natural
Pro-KLSTM--12M
ProtMamba-
Sequence Length
Min. Hamming
HMMER score
pTM
PLDOT
750
750
1.00
1.00
500
0.75
3.5
0.50
250
250
0.50
0.25
19'
9
0.25
750
750
1.00
1,00
A0X2410HH
500
500
0.75
0.75
250
250
0.56
0.25
0.25
750
500
500
0.75
0.75
250
250
0.50
0.25
0.0
0.25
750
1.00
1.00
ADAYCS
500
000
0.75
8
3.5
0.50
250
0.0
750
1.00
1.00
D85016
500
CED aonoe
0.75
0.5
0.50
250
0.50
800
0.25
0.25
750
750
1.00
1.00
STUZ45
500
500
0.75
0.75
0.5
250
0.50
0.25
0.25
750
750
G42478
500
1000
500
0.75
0.75
cod
250
250
0.50
00
0.25
0.0
0.25
1.00
1.00
F2C066
500
B
0.50
250
250
000
0.0
750
1.00
1.00
A041600322
300
500
0.75
2.5
0.50
250
0.50
0.25
0.25
750
75D
1.00
1.00
A04001TON7
500
500
0.75
0.75
0.5
250
0.50
0.25
0.25
Figure A7: Scores of natural and generated proteins. Boxplots of score distributions for 10 randomly selected clusters evaluated for 100 randomly chosen natural sequences and 100 generated sequences with the lowest perplexity values for large Prot-xLSTM and ProtMamba models.
D.3 PROTEIN VARIANT FITNESS PREDICTION
Experimental setup. We evaluate Prot-xLSTMM ability to predict mutational effects by leveraging its inpainting capabilities from the FIM training objective. This assessment follows the protocol described in Section 3.2 of Sgarbossa et al. (2024) for the ProteinGym DMS substitution benchmark (Notin et al., 2023), which consists of 217 datasets of single and multiple substitutions in protein
36

---

Published as a conference paper at ICLR 2025
Table All: Average Kolmogorov-Smirnon statistic (4) between scores of natural and gener- ated protein sequences with 95% confidence intervals across 19 homology clusters. For three of five metrics, score distributions of MeexxxLST---geeerr sequences are closest to natural sequences.
HMM -12K
Prot-XLSTM -26M
ProtMamba -28M
Prot-xLSTM -102M
ProtMamba -107M
PoET
-20M
Seq. Len.
0.37440.0
0.41 #0.09
0.52
0.4022008
0.36-20.0
0.46
min dall
0.9840.01
0.43#00.08
0.60#0.11
0.1740.009
0.4240.00 07
0.80
HMMER
0.97#0.02
0.57
0.54
0.44=00009
0.49
0.30
PLDDT
0.89+0.05
0.40
0.68
0.27=
0.30++.00
0.67#0.09
PTM
0.93+2004
0.38=0.06 0.
0.72
0.26
0.28:20.05
0. 0.65:
sequences, allowing comparison with state-of-the-art methods for protein variant fitness prediction. Briefly, for each wild-type sequence, three sets of 200 homologs were obtained by subsampling MSAs following the ColabFold protocol (Mirdita al., 2022) be used as context. The context sequences are ordered from the least similar to the most similar one. The wild-type sequence is then concatenated with the context, the mutated residue is masked, and this residue is predicted using the FIM method. Fitness is evaluated as the difference in likelihood between the concatenated sequence with the wild-type and the mutated amino acid and averaged over the triplicate. For multiple mutations, fitness is approximated the sum of the likelihoods of single mutations.
Results. ProteinGym's 's main metric the average Spearman correlation between the fitness predic- tions and the experimental DMS results. Table A12 summarizes reports ProteinGym's 's main metric, the average Spearman correlation between the fitness predictions and the experimental DMS results, for Prot-xLSTM and several other well-knoww protein models and the current top of the
Table A12: ProteinGym zero-shot DMS substitution benchmark. The average Spearman correlation (t) between predicted fitness scores and experimental measures over 217 DMS assays shown. While even small Prot-xI.STM models already yield high scores, models that use additional structure tokens, such as SaProt and ProSST, perform best.
Model Type
Model
Reference
#Params Spearman
Alignment- based
Indeeendnnt
Hopf (2017)
0.359
EVE
Frazer et al. (2021)
0.432
GEMME
Laine et al. (2019)
0.455
Protein language model (PLM)
Tranception L (w/o R)
Notin ct al. (2022a)
700M
0.374
VespaGi
Marquet et al. (2024)
3B
0.458
ProGen2 XL
Nijkamp al. (2023)
6B
0.391
ESM-2
(2023)
0.401
Alignment PLM
MSA Transformer
Rao et al. (2021)
100M
0.432
Tranception (w/ R)
Notin et al. (2022a)
700M
0.434
TranceptEVR I.
Notin et al. (2022b)
770M
0.456
Homology-awar PLM
Prot-xLSTM
Ours
26M
0.411
ProtMamba
Sgarbossa et al. (2024)
0.360
Prot-xLSTM
Ours
102M
0.416
ProtMamba
Sgarbossa et al. (2024)
107M
0.415
PoET
Truong Jr and Bepler (2023)
201M
0.470
Inverse folding
ESM-IF1
Hsu et al. (2022)
142M
0.422
Structure PLM
SaProt
Su al. (2024b)
35M
0.407
ProSST
Li et al. (2024)
110M
0.507
SaProt
Su al. (2024b)
650M
0.457
EVE parameters depend on the size of given MSA.
This work. All other values retrieved from ProteinGym 03/1 13/11/2024.
37

---

Published as a conference paper at ICLR 2025
E CHEM-XLSTM: DETAILS AND ADDITIONAL RESULTS
For chemical sequences, we perform two sets of experiments: a) unconditional molecule generation where we follow the experimental protocol of Özęelik et al. (2024). Additionally, we propose new and more challenging task: b) conditional generation with ICL, in which we generate new compounds conditional based on provided in-context compounds.
Domain-Conditioned Pre-Training
Molecular Demains
Language Modelling
Down
X
Sumple from
PROTACe
Domaln
Chem-XLSTI
Up
Kihase inhibiters
Context 4K Domain Specific SMILES Tokens
Domain-Conditionel Generation via ICL
Recurrent Decoding until
Parallel Prompt Encoding
Chem-XLSTM
Molecular Domain Conditioning
Figure A8: Chem-xLSTM Framework for Domain Loaain-Condiiionee Molecular Modeling. Top: Domain- Conditioned Pre-Training. An autoregressive Chem-XLSTM with 15M parameters is pre-trained on diverse set of molecular domains. Training contexts of 4,000 tokens are constructed by sampling molecules belonging specific chemical domains such as PROTACs natural products. Bottom: Domain-Condee Conditioned Generation. At test time, Chem-XLSTM generates novel molecules condi- tioned on molecular domains. Importantly, the model to rely its in- in-context learning capabilities to produce generation, the domain context can be encoded using the mLSTM's parallel mode and then switch to constant-memory recurrent mode for molecule generation.
evaluation is performed on unseen domains, requiring molecules. During
E.1 UNCONDITIONAL MOLECULE GENERATION
Unconditional molecule generation is the task of generating valid small molecules without imposing constraints on their characteristics or properties. Generative models aim learn general distribution by processing many examples of desirable results. To this end, models are trained on large training sets of arbitrary small molecules without particular conditions or constraints (Segler et al., 201 8; Gómez-Bon Bimazz-Bombarelli et al., 2018; Özęelik al., 2024). Following this approach, compared the ability of xLSTM and several other models to generate valid and diverse molecules.
Experimental setup. For comparability, we aligned our experiments with the setting and dataset of Özęelik et al. (2024). This means that all models are trained to generate molecules as SMILES strings (Weininger, 1988) using CLM paradigm. The dataset used in (Özçelik et al., 2024) is derived from ChEMBL with a random split in 1.9M training. 100k validation, and 23k test molecules, which have been encoded as SMILES. Before training, all SMILES strings were tokenized using a regular
38

---

Published as a conference paper at ICLR 2025
expression, containing all elements. This results in atoms being represented one token as well as additional SMILES symbols.
Methods and hyperparameters. We compared xLSTM with several other model classes. The first baseline is the default LSTM (Hochreiter and Schmidhuber, 1997) in Py which includes forget gate (Gers et al., 1999). This be considered the direct predecessor of the xLSTM architecture. We also included a variant GPT-2 (Radford et al., 2019) model based on the Transformer architecture (Vaswani al., 2017) with causal masking. Finally, we included two SSMs in our comparison. On one side, we considered an S4 model with the implementation from Gu et al. (2022), following (Özşelik al., 2024). On the other side, we incorporated Mamba model, using the official repository provided with (Gu and Dao, 2024). For our DMSmM, we used an xLSTM using only mL.TTM blocks (Beck et al., 2024). The 15M- parammeter model consists of layers with a hidden dimension of 512 and 8 heads. We trained the model for up to 100 epochs with a batch size of 1, ,,024, a context length of 100, a dropout rate of 0.25, and a learning rate of 0.005. All models were trained using the Adam optimizer (Kingma and Ba, 2015) using 3 (0.9, 0.999), and a learning-rate schedule with warm-up and cosine decay. We selected the best model based on the minimum validation loss observed at the end of each epoch. The hyperparameters tuned to match the model parameter count for a fair comparison. Detailed training information and learning eurves can be found in Appendix Section E.4.
le-8,
manually
Metries. We evaluated the models based on next-token perplexity and the Fréchet ChemNet Distance (FCD) (Preuer et al., 2018), metric adapted from the Fréchet Inception Distance, to assess the similarity between distributions of generated and test-set molecules within the ChemNet logits-space, model trained on the biological activity of molecules. A lower FCD indicates higher similarity and quality of generated molecules, making it particularly relevant for unconditional molecular design.
Additionally, we report auxiliary metries in Table A14, capturing aspects such as validity, novelty. diversity, and synthetic accessibility.
Validity reflects the ability to parse SMILES strings without chemical rule violations using RDKit (Landrum, 2013).
Uniqueness ensures distinct molecules by verifying unique InChl keys.
Novelty measures the proportion of generated molecules absent from the training set.
Synthetic Accessibility (SA) quantifies the ease of synthesizing molecule scale from (easiest) to 10 (most difficult) (Ertl and Schuffenhauer, 2009).
Diversity is defined the fraction of unique Murcko scaffolds among generated molecules.
Coverage is assessed using the #Circles metric (Xie et al., 2023), which counts unique solutions with Tanimoto similarity threshold of 0.7 (Renz et al., 2024).
These metrics collectively provide a comprehensive evaluation of the syntactic correctness, novelty, and realism of the generated molecules, as well as their alignment with the training distribution. While evaluating unconditional generative models remains challenging (Handa et al., 2023), our focus is on balancing validity, uniqueness, and novelty with computational efficiency.
Our proposed Chem-XLSTM model achieved the best results, with the lowest FCD (0.13) and a perplexity (1.68) that is competitive with that of GPT-based models. This indicates that Chem-XLSTM is able to generate realistic chemical structures that match the target distribution well.
Results.
All models in our comparison were able to produce valid, unique, and novel molecules. Even though these models have not been optimized for these properties. This is evidenced by the auxiliary metrics surpassing practical thresholds (see Table A14).
E.2 CONDITIONAL MOLECULE GENERATION WITH IN-CONTEXT LEARNING
Conditional molecule generation with in in-context learning (ICL) leverages contextual information to guide the design of novel molecules tailored for specific domains. By incorporating sequence of molecules as the input, models can conditionally generate new compounds of the same distribution, without the need for fine-tuning.
Experimental setup. Similar to the unconditional setup, the input consists of SMILES strings. In the conditional setup, additionally model sets of molecules from the same molecular domain
39

---

Published as a conference paper at ICLR 2025
sequence. Molecules from one molecular domain are serialized and concatenated, separated with the token. During training, the order of the molecules is permuted to improve generalization and robustness. We construct novel dataset derived from variety of molecular domains:
We consider natura -products domain and utilize the Coconut (Chandrasekhar et al., 2024) as source dataset.
Kinase inhibitors, wi withhwawn, malaría, tool compounás, pathogen, NIH mechanistic, lopac, natural product-based probes and drugs, zinc tool, axon medchem, adoog bioactive, novartis chemogenetic, drug matrix, PROTACs, covalentIn di,, DrugBank compounds, reframe, cayman dioactive all from the Probes Drugs portal (Skuta et al., 2017),
product molecules from the reaction dataset USPTO-50K (Lowe, 2012) split into 10
reaction classes.
The domains bio o, diversity, green, yellow, orange, and red, from ZINClick (Levré et al., 2018).
Active molecules from the domains BACE, BBBP, Clintox, HIV, SIDER, Tox21, Tox21-10k, and Toxcast from MoleculeNet (Wu et al., 2018).
Active molecules from 95 bioassays from FS-MOL (Stanley et al., 2021) considered each as separate domain.
Active molecules from 109 bioassays from PubChem (Kim al., 2023) considered each as separate domain.
A subset of active molecules from the BELKA challenge (Quigley et al., 2024) is modeled domain.
For the domains that are defined by the active molecules from a particular bioassay, selected assays with at least 300 active molecules and only use the active compounds. For the dataset each of the total 249 domains is limited to 100,000 compounds, where compounds selected at random. The final dataset is split at 8:1:1 into train-, validation- and test-domains, sorted by their character length descending order.
Methods and hyperparameters. We benchmark and orient our choices for the model classes as well as hyperparameters based the unconditional molecule generation results, We consider context length of 4,096 and adjust batch sizes as well as accumulation steps to accommodate GPU memory constraints. For the S4 model, we were only able fit a context length of 2,048.
Metrics. To evaluate conditional molecule generation we evaluate NTP loss. This metric quantifies how well the model predicts the next token in sequence, thus assessing whether model is able to generate molecules from an unseen, and potentially very special, molecular domain given only few molecules from that domain.
E.3 ARCHITECTURE AND HYPERPARAMETER SELECTION
Considered and selected hyperparameters for Chem-XLSTM are given in A13.
E.4 IMPLEMENTATION DETAILS
Unlike Özşelik al. (2024), we do not backpropagate the loss for [PAD] tokens, do we interpret them for decoding. We observed that not ignoring [EOS] and [PAD] token leads to more diversity but is not the standard way of decoding during decoding. They are primarily a pre-processing step to handle batches of data efficiently. In implementation, we end decoding the SMILES string with the [EOS] token. Further, do use SMILES augmentation, which could further improve the performance of all architectures.
e.g. NLP. Padding tokens are not typically generated
E.5 ADDITIONAL RESULTS
Practical thresholds are defined based on several key metrics. First, a high percentage of generated SMILES strings must correspond to chemically valid molecules, with threshold typically set above
40

---

Published as a conference paper at ICLR 2025
Table A13: Hyperparameter space considered for the Chem-xLSTM different sizes. The selected values are marked in bold.
Hyperparameter
Chem-xLSTTM-15M-d
Number of layers
Number heads
Embedding dimension
512
512
Hidden dimension
512
512
Batch size
16, 32, 64, 128
16, 32
Proj. factor
1.3
1.3
Learning rate
le-4, 2e-4, 3e-4, 5e-4
16, 1c-4, 2e-4, 3e-4, 5e-4
Optimizer
Adam, Adam W
Adam
Table A14: Diversity and correctness metrics for the 15M parameter models for small molecules (SMILES). The table reports the percentage of valid, unique, and novel molecules, the synthetic accessibilitty (SA), diversity (measured as the percentage of unique Mureko scaffolds), and coverage (#Cireles (Xie et al., 2023)) are reported as percentages relative to the total number of generated molecules.
Model
valid %
unique %
novel %
SA J
diverse %
coverage %
SMILES-LSTME
90.11=10.7
56.72 56.72
56.66 56.66+336
2.85+0.0
44.71
18.24
SMILES-GRI 6
99.05-0.6
62.09+12.1
61.81#12.
2.90
48.82#9.7
13.44
SMILES-S4 S4
97.48=0.0
61.4740.0
61.34
2.86#0.0
48. 48.49
13.01
Chem-Mambe
91.41 8.0
57.7543.2
57.6343.8
2.8420.0
45.65472
15.43
Chem-XLSTM (ours)
97.08=0.7
61.09
60.84
2.83#0.0
45.9745.5
14.53
"Segler al. (2018)
Adilov (2021)
Özgelik et al. (2024)
"adapted to SMILES in this work
90% ensure reliability, Additionally, practical threshold for uniqueness might require that over 80% of the generated molecules are unique, ensuring diversity in the explored chemical space. For novelty, least 50-70% of the generated molecules should be novel compared to known chemical databases, indicating the model's 's ability to explore new regions of chemical space. Finally, all models exhibit favorable synthetic accessibility (SA) scores, typically ranging between 2.5 and 5, ensuring that the generated molecules are feasible for synthesis. Further metrics and details are provided in the appendix.
F COMPUTE DEMAND AND RESOURCES
The experiments were conducted on multiple GPU servers with A000 GPUs. Model training was performed in both single-node and multi-node setups, utilizing 1-8 A100 GPUs per node. Prot- XLSTM- 102M training with a context length 28 was completed on a node with H200 GPUs. The largest models were trained across up to four nodes using distributed data parallelism. Some experiments leveraged compute resources provided by EuroHPC Joint Undertaking clusters, including Karolina I'4Innovations, Leonardo CINECA, and MeluXina Lux LuxProvide. The total amount of GPU hours required for the experiments in this paper is approximately 50k.
41

---

Published as a conference paper at ICLR 2025
G EXTENDED DISCUSSION
In this work, we demonstrated the potential of Bio-xL.TT variants as prime candidates to model biological and chemical sequences. We have provided clarity in key areas: a) how to tailor XLSTM for biological and chemical sequences, and b) comparing XLSTM-based models to other Spmain-specific LLMs, showcasing their robust performance in DNA, protein, and chemical sequence modeling tasks. Despite certain limitations, quence modeling, excelling in both masked and causal language tasks across different context sizes. In protein modeling. Prot-xI-STTM proved particularly effective at handling long-range dependencies, positioning as promising tool for generating homologous proteins. In small molecule modeling, Chem-xLSTI achieved the best FCD scores for unconditional generation and demonstrated strong ICL capabilities. Our findings underscore the potential xLSTM as a prime candidate for founda- tional models in molecular biology. The models we have introduced, trained, and made available can be used for example to generate rich learned representations for DNA sequences and homology- and chemical domain-conditioned generation of proteins and molecules without the need for fine-tuning.
showed strong performance in DNA se-
While Bio-xLSTTM has shown strong performance across DNA, protein, and chemical sequence modeling, it has several limitations. The manual hyperparameter selection process, which due to limited computational resources, may prevent optimal model configurations. We will explore the hyperparameter space further in the future, which might yield even better models. For DNA, the reliance on character-level tokenization might also restrict the performance and scaling to larger context sizes. Also for proteins, amino acid level tokenization without explicit structural information might limit it's performance. The DNA-xLSTM, currently constrained by the training dataset and their generalizability across organisms and chemical domains needs further exploration. Across all three domains, the training datasets contain biases whether is population biases in the genomic data, sequence distribution biases in protein datasets, chemical exploration biases in molecular datasets. These biases could influence the model's predictions and limit its generalizability in real-world applications. line with many works, we consider the perplexity metric, for example, next token perplexity, the related cross-entropy losses as a proxy for performance on downstream tasks. However, this metric might not capture the capacities of biological and chemical language models appropriately. Future work could address these limitations by expanding the training datasets and downstream evaluations of Bio-XLSTM, Finally, assessing Bio-xLSTM's performance parameter regimes beyond the billion scale remains an open question.
Prot-xLSTMM, and Chem-xLSTM, models are
42


Reviews:
[{"review_id": "hRKvX432Ki", "reviewer": "Reviewer_R8dF", "summary": "The paper introduces Bio-xLSTM, a set of models that is tailored towards modeling of biological sequences. Specifically, the authors apply xLSTMs to different tasks of DNA, protein and small molecular modeling, analyzing the capabilities of xLSTMs in these domains compared to transformer and state-space models. Bio-xLSTM shows remarkable performance across all three domains and achieves state-of-the-art performance at homology aware protein generation.", "questions": "- Do the authors have any explanations why Bio-xLSTM is strong on the histone tasks, but outperformed by NT-v2 500M on the regulatory annotation and splice site annotation tasks (Table A1)? Is there a general difference between the tasks that makes it harder to model it with xLSTMs?\n\n- Did the authors generally observe any patterns where the xLSTM approach is beneficial, and where it might be less effective?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The usage of xLSTMs in the biological domain is a reasonable approach, particularly in the field of DNA where long context appears to be of importance.\n- The paper is well-written, the evaluations and benchmarking is strong and includes reasonable baselines/competitors.\n- The overall performance of Bio-xLSTM is strong and Bio-xLSTM is a valuable model for future research in the field.", "weaknesses": "- I’m missing an evaluation of the diversity of the generated proteins and small molecules. \n- Regarding small molecule generation, I do not really see the benefit of large contexts. Specifically, the unconditional molecule generation uses a context size of 100 tokens. Maybe the authors could explain why the usage of xLSTMs should be beneficial in this domain (I see the point of ICL here, but for unconditional generation there doesn't seem to be any advantage, or?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hESO7MMNUK", "reviewer": "Reviewer_AJhs", "summary": "The authors adapt the xLSTM model architecture to the DNA, protein, and chemical informatics space. They compare with SoA models and benchmarks.", "questions": "“While Transformers have yielded impressive results, their quadratic runtime dependency” - In theory this is true, but it practice, there are more efficient implementations, such as Longformer (Beltagy 202), Linformer (Wang 2020), etc.\n\nFor the DNA-xLSTM tasks, to what degree do the PH or PS architecture additions cause a benefit vs just a xLSTM model? Same with the Mamba models (Table 1).\n\nWhat are the spikes in the validation loss in Figure 3?\n\nFor Table 3, I’m always curious about an HMM baseline. How would a HMM of that cluster perform? It’s most certainly going to have many many fewer parameters.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Generally I think the paper is well written, and the relevant baselines and comparisons across all domains are there.\n\nI think the DNA-xLSTM tasks and benchmarks presented in Table 1 are strong.\n\nGenerally, I think the models (parameter sizes, configurations, training data) are comparable across baselines.", "weaknesses": "Generally, I feel like this paper is a bit of a grab-bag of computational biology. While the xLSTM architecture is consistent, the additional model additions are quite varied and bespoke.\n\nI’m not sure the large blocks of sLSTM and mLSTM math contribute much to this paper. Do you ever refer back to these equations later in the work?\n\nTypo in header 3 - “BIO-XLSTM: LONGE-RANGE MODELING OF BIOLOGICAL AND CHEMICAL SEQUENCES”\n\n“Hamming distance, HMMER score, and structural scores correlate well with sequence perplexity, with an average absolute Pearson correlation of 0.57 across clusters for the large Prot-xLSTM model” I would not say those correlate well. The R**2 is approx 0.325, which is not a lot of variance explained. Moreover, are these distributions normal? Should you be using spearman?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kWhbsKqXUF", "reviewer": "Reviewer_zjFj", "summary": "The contribution describes application of an LSTM architecture to the modeling of sequential representation of molecules (biomolecules). The motivation for this choice of the architecture is to overcome quadratic scaling of transformers and offer linear scaling and constant memory requirements decoding. Three models are introduced as components of bio-xLSTM suite, specialized in modeling sequential representations of small molecules, proteins, and DNA. The reported results demonstrate performance improvement in certain tasks compared to the alternatives.", "questions": "The main motivation for the choice of LSTM is their favorable scaling. There is no scaling analysis in the paper, that clarifies the factual scaling and the observed prefactors. Even with fundamentally improved scaling, prefactors can be prohibitively unfavorable.\n\nThe authors provide performance measures in the form of averages and error bars over an ensemble of runs, which is great. At this point, however, it is more meaningful to test if the distributions of performance results are distinguishable instead of comparing averages. The authors do not have to do this exhaustively, but at least for the cases when their models are claimed to be outperforming the alternatives.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper explores alternatives to the modeling sequential molecular representations in chemistry that are expected to improve scaling and memory requirements. This is a great motivation, considering a) footprint of generative computing and b) its accessibility.\n\nThe authors discuss how to tailor the described architecture in relevant tasks.", "weaknesses": "The general weakness is that the paper contributes to an oversaturated field but does not offer any breakthrough. The case for LSTM is made by the appeal to their compute requirements, which are not discussed in terms of factual requirements of this study concerning memory bottlenecks, prefactors, scaling, etc. If all the reported results are produced on the same computing resource and performance improvement with LSTM  is marginal, why bother?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EaOcyaIAGp", "reviewer": "Reviewer_QpGg", "summary": "This paper presents the Bio-xLSTM framework, which encompasses three specialized models for distinct biological sequences: DNA-xLSTM, Prot-xLSTM, and Chem-xLSTM. DNA-xLSTM introduces reverse-complement equivariant blocks, essential for capturing the symmetry in DNA sequences. Prot-xLSTM is a homology-aware protein language model that employs in-context learning, addressing the variability in protein sequence lengths and context sizes. Chem-xLSTM is designed for SMILES representations of small molecules. Experiments demonstrate that the Bio-xLSTM framework is proficient in generative modeling and representation learning for DNA, proteins, and small molecules.", "questions": "1. Could you elaborate on the specific motivations behind choosing xLSTM over other SSM models? What unique advantages or improvements does xLSTM offer in the context of biological sequence design presented in this paper?\n2. The performance improvements demonstrated in the paper may not be solely attributable to the xLSTM architecture itself. To convincingly argue that the enhancements are a result of the architecture and not just parameter tuning, the authors could provide a more detailed ablation study.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper introduces the novel xLSTM architecture, demonstrating its versatility in biological sequence modeling.It incorporates homology awareness in Prot-xLSTM and reverse-complement equivariance in DNA-xLSTM, addressing key challenges in protein and DNA sequence analysis.", "weaknesses": "1. **Lack of Motivation**: The paper fails to provide a compelling justification for the use of xLSTM over existing SSM models like Mamba. The authors do not clearly articulate why xLSTM is a better choice for the tasks at hand, which leaves readers without a clear understanding of the advantages it offers over established models.\n\n2. **Lack of Novelty**: The concept of reverse-complement equivariance and the application of post-hoc conjoining (PH) and parameter sharing (PS) within the model architecture have been previously discussed in the literature, as seen in the Caduceus model (Schiff et al., 2024). This reduces the perceived novelty of the current work and may be perceived as a weak contribution to the field.\n\n3. **Results Lack Convincing Evidence**: The manuscript's results section lacks the depth required to fully convince the reader of the proposed xLSTM models' effectiveness. An ablation study would significantly strengthen the paper by demonstrating the impact of the xLSTM components on performance, which is currently not provided.\n\n4. **Redundant Writing**: The paper repetitively introduces the xLSTM architecture and training stategies, which has already been described in the original paper. This redundancy is unnecessary and detracts from the focus on the paper's new contributions and findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)