Tokenization to Transfer: Do Genomic Foundation Models Learn Good Representations?
Authors: Kirill Vishniakov
Karthik Viswanathan
Aleksandr Medvedev
Praveenkumar Kanithi
Marco AF Pimentel
Ronnie Rajan
Shadab Khan
Venue: ICLR 2026 Poster (2026)
Keywords: ai4science
foundation models
genomics
biology
pretraining
deep learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Pretrained Genomic Foundation Models offer little to no advantage over randomly initialized models on many genomic tasks.

Abstract: The success of Large Language Models has inspired the development of Genomic Foundation Models (GFMs) through similar pretraining techniques. However, the relationship between pretraining performance and effectiveness in downstream genomic tasks remains unclear. Additionally, the high computational cost of pretraining raises questions about its cost-efficiency. To assess the usefulness of pretraining in genomics, we evaluated seven different GFMs across 52 diverse genomic tasks, comparing them to their counterparts with randomly initialized weights. Across benchmarks, we find that randomly initialized models provide surprisingly strong baselines and tokenizer and architecture choices strongly shape both these baselines and the gains from pretraining. Specifically, character‑token models often match or exceed the performance of larger pretrained k‑mer or BPE models, whereas subword models appear to benefit from pretraining. We also find that the evaluated GFMs fail to capture clinically relevant genetic mutations, with embeddings and log‑likelihood ratios showing limited sensitivity to annotated variants. For the tasks we study, these results suggest that current NLP‑style pretraining strategies provide modest, tokenizer‑gated improvements over strong random baselines and motivate more biologically informed tokenization and variant‑aware objectives. Our code is available at https://github.com/m42-health/gfm-random-eval.

Published as a conference paper at ICLR 2026
TOKENIZATION TO TRANSFER: Do GENOMIC FOUN- DATION MODELS LEARN GOOD REPRESENTATIONS?
Kirill Vishniakoy!, Karthik Viswanathan', Aleksandr Medvedev?, Praveenkumar Kanithi², Marco AF Pimentel", Ronnie Rajan', Shadab Khan Ruya AI M42 ADII Lab
ABSTRACT
The success of Large Language Models has inspired the development of Genomic Foundation Models (GFMs) through similar pretraining techniques. However, the relationship between pretraining performance and effectiveness downstream ge- nomic tasks remains unclear. Additionally, the high computational cost of pre- training raises questions about its cost-efficiency. To assess the usefulness of pretraining in genomies, evaluated seven different GFMs across 52 diverse genomic tasks, comparing them to their counterparts with randomly initialized weights. Across benchmarks, we find that randomly initialized models provide surprisingly strong baselines and tokenizer and architecture choices strongly shape both these baselines and the gains from pretraining. Specifically, character-token models often match or exceed the performance of larger pretrained k-mer BPE models, whereas subword models appear to benefit from pretraining. We also find that the evaluated GFMs fail to capture clinically relevant genetic mu- tations, with embeddings and log-likelihood ratios showing limited sensitivity annotated variants. For the tasks study, these results suggest that cur- rent NLP-style pretraining strategies provide modest, tokenizer-gated improve- ments strong random baselines and motivate more biologically informed to- kenization and variant-aware objectives. Our code is available at comm/m2-. health/g wenggff--nnddnoe random-eval.
INTRODUCTION
Recent advances in language modeling have led to the application of similar unsupervised pre- training methods in genomics. This facilitated the emergence of Genomic Foundation Models (GFMs) (Consens et al., 2025) which learn representations from genomic sequences. This line of work has gained significant attention due the potential of GFMs revolutionize under- standing of genomics (Benegas et al., 2025b).
GFMs typically use two-step training approach akin Large Language Models: unsupervised pretraining on a large dataset, followed by a supervised finetuning on a downstream task. The pre- training phase usually involves either next token prediction (Brown al., 2020) or masked language modeling (Devlin et al., 2019). The promise of unsupervised pretraining is to extract knowledge from vast genomic datasets (Consortium et al., 2015) and compress it into the model's parameters, with the aim of producing generalist model applicable to a diverse set of tasks.
While some studies have explored scaling laws for GFMs (Nguyen et al., 2023: 2024), the rela- tionship between pretraining and downstream performance remains unclear, with no single GFM consistently proving to be the best (Marin al., 2024; Milovanović Orvieto, 2025). Combined with large model sizes (Dalla-Torre et al., 2024), long input sequences (Nguyen et al., 2023; 2024; Brixi et al., 2025) and massive datasets, the pretraining step demands substantial computational resources.
The natural question arises: how effective is unsupervised pretraining in the genomics domain? To answer this, we conduct extensive experiments with seven recent GFMs across finetuning, feature extraction and genomic variation analysis as summarized in Fig. 2.
"S.K. with M42 (initially) and ADIA Lab (subscquently) during this research work.

---

Published as a conference paper at ICLR 2026
Character
BPE
Kimer
NT Benchmark (Histone)
GUE Benchmark
Genomics Benchmark
0.75
0.70
DRABERT
WT-32OM
D.90
CHABEATA
Mutrel
Pretraines (MCC)
Pratrainad (MCC)
Caducads
DHABERT-2
WT.S0M
0.35
NTMON
0.-
Ceducaus
GENA-LM
0.50
Pretrained
0.56
HT 5OOM
0.45
MT:500M
0.5
0.8
0.3
0.5
0.5 Random Init (MCC)
080
0.88 Random Init (Accuracy)
Random Init (MCC)
Figure 1: Tokenizer choice determines random baseline quality, while pretraining gains are more complex. A comparison of averaged performance of identical models with random ini- tialization (I-axis) versus pretrained (y-axis) across (A) NT Benchmark (12 histone and enhancer tasks), (B) GUE (7 task categories), and (C) Genomic Benchmarks (8 tasks). Points above the di- agonal indicate a benefit from pretraining. Finding: The random baseline performance is strongly determined by the tokenizer, with character models (Orange) consistently outperforming subword models (Blue, Green). In contrast, the performance gains from pretraining are largest for subword models (NT-500M A + 0.242 MCC on GUE) but are highly variable for character models. Marker size is scaled with the number of model parameters.
(A) Finetuning
Feature Extraction
ATCGOC
PATTAC
Model
Functional Element
ATCGGC
CATTAC
Model
Biotype Prediction
GCARGC
Prodiction
GCARGC
(C) Genomic Variation
Mutation Database
Genomic Variation Dataset
Mutation Sensitivity
ATOG7A
Model
Analysis
HG38
Ancestry
Prediction
Figure 2: Overview of the experiments. (A) Finetuning: we finetune models on different func- tional element classification tasks. (B) Feature Extraction: For biotype classification, we extract embeddings from frozen models and train a simple classifier to predict gene types using these embeddings. (C) Genomic Variation: We evaluate models' ability to capture genetic variations through: (I) Mutation sensitivity analysis measures how well models distinguish between origi- nal and mutated sequences by computing embedding similarities, and (2) Ancestry prediction uses model embeddings with XGBoost to classify population groups based on genomic variants.
First, in standard finetuning tasks such as NT Benchmark (Dalla-Torre et al., 2024), GUE (Zhou et al., 2024), and Genomic Benchmarks (Grešová et al., 2023), we now compare each pretrained GFM directly to identically configured model with random initialization (Fig. 1). Random base- lines are surprisingly strong and depend strongly on tokenizer: character-level models (Caduceus, HyenaDNA, Mistral) achieve higher performance from scratch than models using k-mer or BPE kenization. At the same time, pretraining yields consistent gains for k-mer and BPE models, while gains for character-token models are smaller and more cetiitectur--eppnddeen a randomly initial- ized Caduceus model often matches or exceeds much larger pretrained GFMs across benchmarks.
This surprising trend continues to be observed in feature extraction task (Table 2), where embeddings from frozen models are used to train simple classifier. One might expect the benefits of pretraining to be most pronounced here, because randomly initialized models receive no tuning whatsoever and their weights remain fully random. Instead, we find that randomly initialized models can be competitive, and that simple architectural choices, particularly using a character tokenizer and a larger embedding dimension, substantially improve their features. On the biotype classification task, an untrained HyenaDNA variant yields the best performance among the GFMs we evaluate, and tokenizer ablation with matched HyenaDNA architectures (Table 3) shows that changing only
2

---

Published as a conference paper at ICLR 2026
Model
#Params
Architecture
Tokenizer
Vocab Size
Seq Len (tokens)
#Tokens
Data
HycnaDNA
450K
Decoder
Char
12
1024
2.6B
HRG
NT 500M
500M
Encoder
k-mer
4107
1000
300B
1000G
NTv2 50M
50M
Encoder
k-mer
4107
2048
300B
Multispecies
GENA-LM
110M
Encoder
BPB
32000
HRG+
DNABERTY2
117M
Encoder
BPE
4096
128
262B
Multispecics
Caduceus
8M
Encoder
Chur
12
131K
35B
HRG
Mistral
580M
Decoder
Char
12
4096
150B
1000G
Table I: Description of models evaluated in this study. The analyzed models differ in architecture, pretraining objective, tokenizer, model size, and pretraining dataset. We analyze the pretrained models and their randomly initialized counterparts. #Tokens refers to the number of tokens seen by the model during the pretraining. Data refers the pretraining dataset source.
the tokenizer from characters to 6-mers increases average downstream MCC by 0.19 despite slightly worse pretraining loss.
Finally, we assessed the models detecting subtle genomic variations (Section 3.5). This scenario requires models to be highly sen- sitive to single nucleotide changes within long sequences. In our evaluation, we find that most pretrained GFMs show limited sensitivity to these variants. For instance, even when up to half of the nucleotides in a DNA sequence are changed, some GFMs still produce embeddings with cosine similarity above 0.99, and Clin Var log-liielihood ratio tests yield AUROCs close to 0.5 (Table 4 and Table 5). These results suggest that, in their current form, the evaluated GFMs provide limited util- ity for applications that rely extensively on subtle mutation signals, including variant pathogenicity prediction, eQTL (Zhou Troyanskaya, 2015), sQTL (Garrido-Martte et al., 2021), and phenotype prediction (Lello et al., 2018).
of the most practically important applications for genomies:
Our results call for a more careful view of current unsupervised pretraining methods in genomics, suggesting that simply adapting NLP techniques is not yet sufficient to obtain broadly useful 'foun- dation' models on regulatory classification tasks. Rather than continuing to invest substantial com- putational resources in existing pretraining methods, advocate for critically rethinking the fun- damental building blocks of genomic foundation models. This includes developing biologically- informed tokenization strategies and establishing new robust benchmarks that comprehensively test for the understanding of genomic mechanisms.
2 MODELS
We selected six recently published GFMs for evaluation and also trained our own version of the Mistral (Jiang et al,, 2023) model on samples from the 1000 Genomes dataset (Consortium al., 2015). The models in analysis exhibit significant diversity in their architectures, pretraining objectives, tokenizers, model sizes, and pretraining datasets. Our model selection includes both en- coder and decoder architectures, transformer- based and state-space models, with model sizes rang- ing from 450K to 580M parameters. Interestingly, our Mistral outperforms all other previous GFMs on many tasks. We attribute the success of Mistral to advanced architecture recipe which includes RoPF, big embedding dimension and character tokenizer. Table lists model configurations; model descriptions in the Appendix.
We excluded the EVO model (Nguyen et al., 2024) from our analysis as it was trained on bacterial genomes and performed poorly in our preliminary tests the Nucleotide Transformer Benchmark.
EXPERIMENTS AND RESULTS
3.1 FINETUNING
To verify the usefulness of pretraining, we finetuned both pretrained and randomly initialized (rand. init.) versions of the models on Nucleotide Transformer Benchmark (Dalla-Torre et al., 2024), Genome Understanding Evaluation (GUE) (Zhou et al., 2024), and Genomic Benchmarks
3

---

Published as a conference paper at ICLR 2026
(Grešová et al., 2023) with exactly the same set of hyperparameters. snperparameeetcctee This set of benchmarks to- gether constitutes 52 genomic classification tasks. In total, we conducted nearly 10,000 finetuning experiments, this considers: seven models, both pretrained and random, evaluated across different tasks, folds, and learning rates.
To ensure robustness, performed broad hyperparameter search over learning rate, weight decay, batch size, warm-up steps, LoRA (Hu et al., 2022) vs full finetuning, and others. Because perfor- mance was most sensitive to the learning rate, the final run consisted of sweep over six learning-rate values, and we report the best result obtained. We also found that full finetuning consistently out- performed LoRA (see Table 8 in Appendix), suggesting that it provides the best opportunity for the model to reach the full score.
For each model, we now compare its pretrained checkpoint directly against an identically configured version with random initialization. In Fig. 1, the x-axis shows the average performance of the randomly initialized model and the y-axis shows the corresponding pretrained model on (a) NT histone and enhancer tasks, (b) GUE task categories, and (c) Genomic Benchmarks. Points on the diagonal correspond to identical performance, while the vertical distance from the diagonal measures the gain or loss from pretraining.
Reading Fig. along the x-axis reveals that the strength of the random baseline depends strongly on the tokenizer. For instance, sundomly-initializn character-token models (Caduceus, HyenaDNA) are often indifferent or superior than their pretrained counterparts. We also note a near-consistent trend of pretraining helping models trained using BPE or k-mer tokenizer, with the extent of gain ranging from 0.05-0.25 MCC on NT and GUE, with model architecture a strong determining factor of performance. Detailed per-task deltas comparing each pretrained model to the best random base- line are provided in Appendix in Fig. 7. Additionally we also provide separated apples-to-apples comparisons for each model between pretrained and rand. init. versions in Fig. 8 (NT Benchmark), Fig. 9 (GUE), Fig. 10 (Genomics Benchmark) in Appendix.
Even after accounting for pretraining gains, small randomly initialized models remain competi- tive in absolute terms. On NT Benchmark, for example, randomly initialized Caduceus (8M pa- rameters) achieves an average MCC of about 0.62 on the most challenging histone and enhancer tasks (Fig. IA), outperforming several larger pretrained models such as NT-500M, NTv2-50M, and GENA-LM (110M parameters), and often matching or exceeding its own pretrained version. DNABERTV2 (117M parameters) also provides a strong random baseline and, on several histone tasks, outperforms pretrained NT-500M by sizable margins (Table 18).
The results GUE in the middle part of Fig. 1 demonstrate that randomly initialized Caduceus shows remarkable performance outperforming its pretrained version by 1. MCC. For instance, on TF Prediction (Mouse) task in Fig. 9, randomly initialized Caduceus outperforms all 7 pretrained models. Similar trend is observed on Core Promoter Detection Group where rand. init. Mistral out- performs all pretrained models including its own pretrained version. In general on GUE benchmark the best randomly initialized model outperforms five to seven pretrained models (Fig. 7). In Ge- nomic Benchmarks similar trend of competitiveness of rand. init. models can be observed (Fig. 10).
The results across all three benchmarks demonstrate that while pretraining improves some GFMs relative to their own random initialization, we also identified several randomly initialized models like Caduceus, DNABERTV2, and HyenaDNA that can match or exceed pretrained performance across a wide range of tasks.
Unlike foundation models in other domains, such as computer vision (Radford et al., 2021) and NLP (Brown al., 2020), where pretraining typically leads to significant improvements in down- stream task performance, the current pretraining strategies in genomics are barely able to outper- form randomly initialized models. Moreover, even cases where pretrained models maintain an advantage, the gains from pretraining these modest gains may not justify the large amounts of compute required for pretraining in ge- nomics (Dalla-Torre et al., 2024), especially for these commonly used fine-tuning tasks.
surprisingly small typically within 2-3%. Together,
3.2 FINETUNING WITH LIMITED DATA
To examine whether pretraining is more beneficial when labels are scarce, we repeated finetuning for four representative models: 5T-50M (k-mer), DNABERT-2 (BPE) and GENA-LM (BPE), and
4

---

Published as a conference paper at ICLR 2026
Char tokenizer by default
Subword tokenizer by default
Tokenizer
Pretrain
Mistral
HyenaDNA
Caduceus
NTv2 50M
GENA-LM
DNABERTV2
NT 500M
defanlt
<
0.730
0.638
0.423
0.679
0.704
0.654
0.662
default
0.667
0.690
0.674
0.482
0.574
0.651
0.603
x
0 666
0.690
0.674
0.642
0.668
0.696
0.669
+larger embed dim
x
0.700
0.753
0.717
0.703
0.684
0.708
0.678
pretrained random
3.0%
-11.5%
-29.4%
-2.4%
2.0%
-5.4%
.1.6%
Table 2: Biotype classification F1 scores for pretrained vs. randomly initialized models. Optimizing the tokenizer and embedding dimension (rows 3-4) allows most random models to surpass their pretrained counterparts (last row).
M722-50M
Protraired
Fondim
H334mel
H3K4m33
cnhancors
0.50
0.23
10 50
10
10
10 Traningg dusa 261
Traniig
Tlaniig doaa
Figure 3: Pretraining benefit in low-resource regimes depends tokenization. Performance curves for H3K4mel and Enhancers at 1%, 5%, 10%, and 50% data fractions. Subword models (DNABERT-2, GENA-LM) show consistent gains from pretraining (solid / pretrained vs dashed / random lines), while the char-level model (HyenaDNA) often shows smaller benefits.
HyenaDNA (character), on four NT Benchmark tasks (H3K4me1, H3K4me3, H3K9ac, enhancers) using 1%, 5%, 10%, and 50% of the training labels. For each combination of model, dataset, and data fraction, we performed a full hyperparameter sweep over four learning rates (le-5, 5e-5, le-4, 5e-4) trained for 30 epochs, ensuring that both pretrained and random baselines were compared at their optimal settings. The results are displayed in Fig. 3. Pretraining yields consistent and some- times large gains for the BPE models across all label fractions (with the largest absolute improve- ments at 1-5%), whereas the character-level HyenaDNA model derives much smaller and sometimes negligible gains from pretraining in the same regime. This confirms that the tokenizer-deepndent pattern persists, and is more pronounced, in low-resource settings.
3.3 TOKENIZER EFFECT ABLATION
To causally isolate the effect of tokenizer on model performance, we performed a controlled ablation study. We pretrained two identical HyenaDNA models (same model size, same Human Reference Genome dataset, same # of training iterations) differing only in their tokenizer; one using character tokenizer and the other using 6-mer tokenizer. Both models were then finetuned on a representative subset of the NT Benchmark, specifically the enhancers, H3K4me3, and H3K9ac tasks, and report the performance averaged across these three tasks. Results are presented in Table 3.
The results in Table 3 offer critical insight. Although the character-level model achieved lower pretraining loss (1.180 vs 1.215). the k-mer model significantly outperformed it on downstream tasks (+0. 187 MCC advantage). This finding demonstrates that pretraining perplexity leading proxy for downstream task performance. The k-mer tokenization may be providing a useful inductive bias by forcing the model to learn representations of motif-like sequences from the outset, a process that is likely translating to more effective performance on these functional genomics tasks.
be mis-
3.4 FEATURE EXTRACTION
The biotype classification task assesses the quality of features extracted from both pretrained and rand. init. models. Unlike in finetuning, this task does not involve updating model weights. In other
5

---

Published as a conference paper at ICLR 2026
Table 3: Tokenizer Ablation. Compar- ison of two identical HyenaDNA models pretrained with different tokenizers. De- spite the character model achieving lower pretraining loss, the k-mer model vastly outperforms it on all downstream tasks.
Metric
Character
K-mer
Difference
Pretruining Loss І
1,180
1,215
Character better
H3K4me3 (MCC)
0.138
K-mer +0.185
H3K9ac (MCC)
0.141
0.349
K mer +0.208
Enhancers (MCC)
0,139
0.305
K mer +0.166 166
Avg. Downstream 1CC
0.139
0.326
K-mer +0..187
MyenaDNA Caduceus
Mistral
S0M
GENA-LM
DNABERTV2
NT 500M
51 Score
0.70
0.770
0.60
Rancem IeraDAa
128
512
1024
3400
2019
4005
1024
2048
4026
Eriudding Cimesin
Embedding Dimension
Embedding Dimension
Figure 5: Fl for pretrained/ ran- dom init HyenaDNA on the bio- type task; except d=64, random > pretrained over embed. dims.
Figure 4: Performance of rand. init models with char tokenizer improves as the embed dim increases. Squares mark default dimensions.
words, embeddings for randomly initialized models were extracted without any finetuning and were entirely bused on their initial random weights.
Using sequences and biotype labels from the Gencode (Harrow et al., 2012), we extracted features from models with frozen weights and applied max pooling along the token dimension. These pooled features were then used to train an XGBoost classifier to predict among nine biotype labels. See Appendix for dataset details.
We observed that the choice of tokenizer significantly impacts the performance of rand. init. encoder models. In particular, switching these models from their default k-mer or BPE tokenizers with large vocabularies (Table 1) to char tokenizer that has only four tokens substantially improved their performance (3rd row of Table 2, right part). For example, for NTv2 50M, the performance increased from 0.48 to 0.64. Char tokenizer is standard for decoder models, hence the identical results in the second and third rows of Table 2 for decoder models. This improvement likely stems from random models' difficulty handling large vocabulary sizes. Initially, the random HyenaDNA model achieved the highest F1 score (0.69) among random models despite its small 228-dimension embeddings, prompting us investigate how embedding size affects performance. We tested various embedding dimensions across all models, keeping other parameters constant and ensuring divisibility by the number of attention heads. Complete embedding configurations are detailed in the Appendix.
To decouple embedding dimension from initialization (random vs pretrained), we pretrained new HyenaDNA models from scratch across full range of embedding dimensions, and compared it to its randomly initialized counterpart under the identical encoder, tokenizer (character), pooling, and classifier setup. The paired curves in Fig, 5 show that pretraining provides advantage only at very small width (d=64); by d=128 the gap closes, and for all larger widths the randomly initialized model matches or exceeds the pretrained model. Thus, on biotype, the effect of pretraining is brittle with respect to model capacity.
In a related experiment, we find that (Fig. 4) as the embedding dimension of randomly initialized models increases, their performance improves, for nearly all of the models we tested. HyenaDNA shows consistent improvements, reaching an Fl score of nearly 0.75 at 4096 dimensions. NTv2 50M exhibited more dramatic improvement, with its Fl score rising from 0.53 to 0.71. Further, on GUE histone tasks (Appendix Table 13), random HyenaDNA (d=2048) is best on 9/10 tasks, again indicating that capacity interacts strongly with initialization. Also, shown in the fourth row of Table 2, increasing the embed. dim and using a char tokenizer allowed randomly init, models to outperform pretrained in 5 out of cases.

---

Published as a conference paper at ICLR 2026
3.6
1.8
Phatig 3pe
2.4
132
-2.
DHAUERT o2
NT 40M
GENNALM
Micrall
3.6
Fre
0,74
0.68
0.08
0.74
0.24
0.00
0.73
Warber Mayatees
Number MuRyoee
Strrber M.tiiees
KRn
0.68
0.00
0.1
0.3
(b)
Figure 6: (a) Mulation sensitivity: cosine similarity between reference and mutated sequences re- mains high under both pooling schemes, indicating limited sensitivity SNPs. (b) Ancestry clas-
sification (FI): random Caduceus (0.76) is highest; random HyenaDNA (0.69) slightly exceeds its pretrained counterpart; others are comparable.
3.5 GENOMIC VARIATION
This section transitions from functional element classification to genomic variation tasks, analyzing mutations like single nucleotide polymorphisms (SNPs), insertions, and deletions between individu- als. While functional elements remain largely consistent across populations, genomic variations individual-speciin and can significantly impact phenotype and disease risk. These tasks present unique challenge for GFMs, which must detect and interpret subtle sequence differences, often down to single nucleotide changes, to understand human genetic diversity and its health implications.
Ancestry prediction multilabel classification task that predicts an individual's 's ancestry using small portion of their genome. We constructed an ancestry dataset from 1000G data (Consortium al., 2015), using HG38 and applying mutations from each 1000G sample to obtain 32K-base consensus sequences. These sequences differ by 0.5% of positions, with an average of 33 variants (SNPs, insertions, and deletions). Embeddings generated from these sequences were used as features for XGBoost classification.
When generating the dataset, we selected eleven different regions of the genome, treating each as separate fold, and evaluated our models each region independently, reporting average (Fig. 6 (b)). See Appendix for dataset details.
Results in Fig. 6 (b) show that randomly initialized models generally match pretrained models' per- formance, Only Mistral and GENA-LM showed marginal improvement with pretraining (Fl differ- ence: 0.07 and 0.03). Caduceus achieved the highest F1 score (0.76) in both random and pretrained versions. The NT 500M model, despite being trained on 1000G variants, showed no advantage over its random initialization. This performance pattern could stem from two factors: the high masking probability (15%) ) in masked language modeling, which exceeds the natural mutation rate (0.5%), and the k-mer tokenization (6 nucleotides) that poorly captures single nucleotide variations.
Mutation Sensitivity Analysis. We further investigated models' limited ancestry prediction perfor- mance and assess their capability in capturing subtle genomic variations, at the SNP-level. These experiments verify the models' ability to detect differences between reference sequences and se- quences with inserted SNPs. We focused exclusively on SNPs to eliminate sequence length as confounding factor.
We measure the cosine similarity between the embeddings of the reference DNA sequence and the embeddings of the same sequence with SNPs introduced, using both global last / cls pooling and mutation-site pooling that aggregates representations only at tokens overlapping the SNPs. Lower similarity scores indicate better model sensitivity in detecting biologically significant changes, while high scores suggest the model fails to distinguish important genetic variations.
To ensure robustness, conducted this experiment by sampling twenty-five 1024-length sequences from chromosomes 7, 11, 12, 17, and 19 HG38, with five sequences per chromosome. The 1024- length chosen to avoid chunking effects, ensuring it fits within all models' context windows. For each sequence, we created variants by introducing mutations at increasing frequencies (1, 64, 128, 256, 512, 1024) at random positions. Embeddings for both the reference and mutated sequences

---

Published as a conference paper at ICLR 2026
Gene
NT 500M
NTv2
HyenaDNA
Mistral
Caduceus
GENA-LM
DNABERTY2
BRCA2
0.511
0.478
0.439
0.495
0.505
0.408
0.535
CFTR
0.442
0.365
0.454
0.345
0.442
0.421
0.536
Table 4: Near-random AUROC scores from log- likelihood ratio tests Clin Var variants, indicating poor model sensitivity to clinically relevant mutations.
Gene
Mutation Type
NT 500M
NTy2 50M
DNABERT<2
HyenaDNA
Mistral
GENA-LM
Caduccus
Benign
0,985
0.995
0.999
0.976
1,000
TP53
Pathogenic
0.983
0.993
0.996
0,999
0.988
1,000
0.990
Benign
0.999
0.984
0.964
0.996
0.907
0.996
0.996
BRCA2
Pathogenic
1.000
0,984
0,955
0,999
0.981
1,000
0,973
Benign
0.999
1,000
CETR
Pathogenic
1.000
0.999
0,998
1.000
0.996
1,000
0,999
Table 5: Gene-specific Variant Detection. Distinguishing benign vs. pathogenic variants in TP53, BRCA2 and CFTR genes. Lower values are better.
are then generated using different methods: last cls tokens (based on the decoder or encoder), and pooling the tokens only at mutation sites (denoted as "mutation" in Fig. 6). These approaches were tried to cover different ways of interpreting the embeddings generated by the model during downstream evaluations.
Fig. 6 illustrates the cosine similarity between reference and altered sequences across different pool- ing types. Despite the models using different tokenizers, the results generally poor. For most models, both last / cls and mutation pooling produced high cosine similarity values even for single mutation regime, typically 0.9 or higher. As the number of mutations increases, the cosine sim- ilarity also tends to increase, presumably due to averaging effect. Among the models tested, the Mistral-based DNA model showed the lowest cosine similarity for last pooling and relatively low cosine similarity for mutation pooling. In contrast, GENA-LM and NT 500M produced high co- sine similarity scores close to 0.999 for both pooling types. These results indicate that most models not significantly affected by mutations, thereby highlighting their limited ability to detect subtle sequence alterations, irrespective of their tokenizers.
Clin Experiments. To further investigate the sensitivity of GFMs to sequence alterations, conducted additional experiments using Clin Var data (Landrum et al., 2014), which includes genetic variations among individuals. These experiments aim to verify realistic setting, using real-world genetic variations from Clin Var. We analyze the TP53, BRCA2 and CFTR genes and obtained their gene sequences from NCBI (Sayers et al., 2022).
previous findings in more
First, we filtered the variants to include only exonic mutations. This ensures a focus on mutations that affect protein-coding regions, which are of greatest interest in clinical genetics. Next, cat- egorized the variants into two groups based on clinical significance: benign and pathogenic. The benign group included variants labeled 'Benign', 'Likely benign', "Benign/Likely benign', while the pathogenic group comprised variants classified as 'Pathogenic', 'Likely pathogenic", or "Pathogenic/Likes pathogenic'. This grouping enables us to compare the model's sensitivity to mutations with different clinical impacts.
After preprocessing the data, we take five chunks of 1024 base pairs for each gene independently that have both benign and pathogenic mutations. We created three sequences for each chunk: the original reference, one with only pathogenic mutations, and one with only benign mutations. The distribution of mutations is shown in the Appendix.
This variation in mutation density allows us to observe the model's sensitivity across different levels of sequence alteration. For each chunk, we applied max pooling the model outputs and computed the cosine similarity between the reference sequence and both the benign and pathogenic versions. Finally, we averaged cosine similarity over five selected chunks. The results presented in Table 5 showed consistently high similarity scores across all models and mutation types, indicating the con- sistent failure of models reflect genomic variance in their embeddings.
8

---

Published as a conference paper at ICLR 2026
Log-Likelihood Ratio Analysis. To further probe the sensitivity of the models to single nucleetieee changes, we evaluated them using log- oog-liielihood ratios, following the methodology of Benegas et al.
(2025a). This approach allows us to assess each model's ability to distinguish between alleles with- out any finetuning. Specifically, for a single-nucleotide variant defined by a reference base (RFF) and an alternative base (ALT) at given position, we compute the log- lelllkkelioodd ratio: log P(REF)' For encoder-only models, we follow (Benegas al., 2025a) and approximate these log probabilities using the softmax scores at the masked variant position, holding the surrounding context fixed. We applied this method each pathogenic variant in the BRCA2 and CFTR genes from our Clin dataset. Log probabilities for the mutated and reference nucleotides were computed directly from the pretrained models the specific mutation sites. We then used these log-likelihood ratio values to compute AUROC scores for distinguishing pathogenic from benign variants.
P(ALT)
The results, presented in Table show that model performance remained near random chance, with AUROC scores between 0.345 and 0.536. This finding further supports our earlier conclusions, providing site-specific evidence that existing pretrained GFMs have limited sensitivity to clinically significant mutations.
4 DISCUSSION
We evaluated seven Genomic Foundation Models (GFMs) across 52 tasks spanning three standard benchmarks under finetuning, frozen-feature, and variant sensitivity settings, and covering regu- latory, functional, and structural genomics tasks. Two consistent observations emerge. First, the random buseline is strongly influenced by tokenization, and the character-tokenized models often attain high performance without pretraining, matching or surpassing much larger pretrained subword models (Sec. 3.1). Second, the benefit of pretraining is conditional; we do observe it for subword tokenizers and model/recipe combinations (that pose a harder from-scratch representation learning problem), but find the pretraining gains are inconsistent for character tokenizers. The paired com- parison in Fig. 1 makes these patterns explicit by plotting each model's pretrained score against its own randomly initialized counterpart.
What drives the baseline, and when does pretraining help? Reading Fig. 1 left-to-right (ran- dom) and bottom-to-top (pretrained) shows that character tokenization provides strong starting point. Subword models (k-mer/BPE) frequently gain more from pretraining, whereas character models display smaller or variable gains. Architecture matters as well the character-tokenizeed Mistral benefits from pretraining, indicating that tokenizer, architecture, and scale jointly determine whether pretraining adds signal rather than noise. This tokenizer dependence persists in lahel-scarce regimes (Fig. 3). With only 1 1-5% of labels, subword models (e.g.. DNABERT-2, GENA-LM) show improvements from pretraining (e.g., DNABERT-2 gains ++0.25 MCC on H3K4mel at I 1%), while the character model (HyenaDNA) often shows negligible negative transfer. In short, pretrain- ing helps most when amortizes a difficult token-representation problem that supervised finetuning alone cannot solve within budget.
Mechanism - tokenizer inductive bias and a causal test. To ascertain the role of tokenizers in leading to the results we observed, we further fixed the architecture and training recipe and changed only the tokenizer for results reported in Table 3. Despite achieving a lower pretraining loss, the character-tokenizer model yielded substantially worse downstream MCC than its k-mer counter- part (+7.187 average advantage for k-mer). Two implications follow. (i) Pretraining loss can be a poor proxy for discriminative genomics: language-model objectives capture token predictability that does not necessarily align with downstream labels. (ii) Tokenizer inductive bias can dominate down- stream performance independently of loss. Intuitively, these can be explained as follows. A compact character vocabulary creates an easy input space for randomly init. models, producing strong base- line; whereas large subword vocabularies create a sparse, hard input space where pretraining confers more value by learning token representations that finetuning alone struggles to form.
Feature quality at zero training and evaluation bygiene. In the frozen-feature setting one might expect pretrained features to dominate, since the random arm receives no weight updates. Instead, modest architectural choices (character tokens and increased embedding dimension) make random features competitive and often superior (Table 2; per-task details in the Appendix). Further, un- der a matched HyenaDNA architecture and a full embedding dimension sweep. pretraining helps only d=64 and loses its advantage from 1=128 upward (Fig. 5), indicating that capacity rather

---

Published as a conference paper at ICLR 2026
than pretraining drives most of the gains on that task. Together with our LoRA vs. full-finetuning comparison (Table 8), we arrive at two main observations: (i) well-tuned random baselines (hy- perparameters tuned, with lightweight architectural choices) can achieve competitive representation quality; and (ii) cross-modal comparisons where random baselines are tuned with LoRA are likely under-reporting performance for random baselines.
Variant sensitivity is a central gap. Many practical applications hinge on single-nucleotide differ- ences. Across pooling schemes, including mutation-site pooling, embedding similarities between reference and mutated sequences remain high (Fig. 6), and ClinVar log-likelihood ratio tests yield AUROC values near chance (Table 4). Character-tokenizer models (e.g.. Mistral, Caduceus) some- times show lower similarity or stronger ancestry classification, but effect sizes are small. These findings indicate that, regardless of pretraining status, common objectives and tokenizers do not re- liably encode allele-level information needed for pathogenicity, eQTI/sQTI. or related tasks. This points to the need for mutation-ave aware objectives, tokenization that preserves single-base signals, and evaluation tasks that directly stress allcle-level sensitivity.
Limitations and Outlook. Our study targets discriminative sequence classification and frozen-feature quality. Several evaluated models also have short context windows (128-1024), which limits our ability to conduct long-rang: experiments often requiring 100k or longer context. We do not test generative sequence design or long-range modeling tasks such gene-expression regression or
promoter linkage, where specialized supervised models and new generative architec- tures (Evo2 (Brixi al., 2025), GenomeOcean (Wu et al., 2025), Enformer-style models) remain strong baselines.
L Limitations. (i) We analyze seven GFMs; broader coverage (e.g.. quantized graph-based models) is outside scope. (ii) Variant sensitivity is evaluated with cosine similarity and site-wise LLR on se- lected genes; further diagnostics (e.g.. ber-base in-silico mutagenesis with attribution maps; compar- isons to CADD/Enformer: eQTL/sQTL ground truths) are natural extensions. (iii) Our mechanism study isolates tokenizer effects; other factors (masking schedules, positional encodings, curriculum) could be grounds of evaluation for future work. (iv) We focus on maximizing performance of each model, and therefore limit our analysis to default precision. Extension to quantized models (e.g., GERM (Luo et al., 2025)) could be considered by the future studies.
Three practical paths follow from these results. (i) Short-range classification - com- pact, character-tokenizer models are strong and compute-efficiet random baselines; reporting well- tuned random baselines should become standard practice. (ii) Subword models and label-scarce regimes expect gains from pretraining, but align the objective with use (c.g., mutation-aware masking, contrastive signals anchored at variant loci, and multi-scale local-global supervision). (iii) Variant-centric applications clear evidence that the community should prioritize tokenizer and objective function redesign before further scaling. In parallel, community benchmarks that couple allele-level probes with long-range regulatory tasks, and require transparent architectural controls, will better track genuine representation gains rather than proxy loss reductions.
Outlook.
5 CONCLUSION
Our comprehensive evaluation of GFMs highlights significant limitations in current pretraining approaches. Across standard classification benchmarks, pretrained GFMs offer most modest, tokenizer- and anchitecture-deeeenentt advantages over tuned randomly initialized counterparts. We also found that existing GFMs exhibit insufficient sensitivity to variants, which limits their utility in tasks requiring variant interpretation. We identify areas for methodological refinement, including optimizing masking approach, employing chavacter-level tokenization, and designing specialized architectures better attuned to biological sequence complexity. Additionally, our extensive hyperpa- rameter optimization helped ensure the robustness of these conclusions. While genomic pretraining still holds promise, particularly in specialized generative contexts, realizing the vision of broadly applicable, clinically relevant GFMs will require a fundamental reassessment of current practices. We hope our findings encourage the genomic modeling community to develop more biologically informed approaches, rigorous benchmarks, and targeted strategies, ultimately bridging the gap be- tween computational advancements and tangible biomedical impacts.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Our code is available at gituu..cm/nn22hhetttgg-----ooo-e In our study we used such pub- licly available datasets as:
1000 Genomes Project VCF files can be accessed at International Genome Sample Resource (IGSR) (Fairley al., 2020).
GRCh38 reference genome assembly can be downloaded from (National Center for Biotechnology Information (NCB1), 2024).
Gencode (Frankish et al., 2019) gene annotation used for biotype labeling.
NT Benchmark datasets are introduced (Dalla-Torre et al., 2024).
Genome Understanding Evaluation (GUE) multi-species benchmark is introduced in (Zhou et al., 2024).
Genomic Benchmarks is introduced (Grešová et al., 2023).
Clin Var variant records for TP53, BRCA2 and CFTR and corresponding gene sequences are down- loaded from NCBI (National Center for Biotechnology Information (NCBI), 2024).
Details on training hyperparameters for evaluations are provided in the Appendix. Finally, the Ap- pendix includes description of the computational environment used for experiments.
REFERENCES
Gonzalo Benegas, Carlos Albors, Alan J Aw, Chengzhong Ye, and Yun S Song. A dna language model based on multispecies alignment predicts the effects of genome-wide variants. Nature Biotechnology, 2025a.
Gonzalo Benegas, Chengzhong Ye, Carlos Albors, Jianan Canal Li, and Yun S Song. Genomic language models: Opportunities and challenges. Trends Genetics, 2025b.
Garyk Brixi, Matthew G Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A Gonzalez, Samuel H King, David B Li, Aditi Merchant, et al. Genome modeling and design across all domains of life with evo 2. bioRxiv preprint 0...2ri:2222.02.18.33.3.3
2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
Micaela E Consens, Cameron Dufault, Michael Wainberg, Duncan Forster, Mehran Karimzadeh, Hani Goodarzi, Fabian Theis, Alan Moses, and Bo Wang. Transformers and genome language models. Nature Machine Intelligence, 2025.
1000 Genomes Project Consortium et al. A global reference for human genetic variation. Nature, 2015.
Hugo Dalla-Torre Liam Gonzalez, Javier Blendzza-Reville Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida. Hassan Sirelkhatim, al. The nucleotide transformer: Building and evaluating robust foundation models for human genomics. Nature Methods, 2024.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Susan Fairley Ernesto Lowy-Gallego, Emily Perry, and Paul Flicek. The international genome sample resource (igsr) collection of open human genomic variation resources. Nucleic Acids Research, 2020.
Haonan Feng, Lang Wu, Bingxin Zhao, Chad Huff, Jianjun Zhang, Jia Wu, Lifeng Lin, Peng Wei, and Chong Wu. Benchmarking dna foundation models for genomic and genetic tasks. Nature communications, 2025.

---

Published as a conference paper at ICLR 2026
Veniamin Fishman, Yuri Kuratov, Maxim Petrov, Aleksei Shmelev, Denis Shepelin, Nikolay Chekanov, Olga Kardymon, and Mikhail Burtsev. Gena-lm: A family of open-source founda- tional models for long dna sequences. Nucleic Acids Research, 2025.
Adam Frankish, Mark Diekhans, Anne-Maud Ferreira, Rory Johnson, Irwin Jungreis, Jane Love- land, Jonathan Mudge, Cristina Sisu, James Wright, Joel Armstrong, et al. Gencode reference annotation for the human and mouse genomes. Nucleic Acids Research, 2019.
Diego Garrido-Martín, Beatrice Borsari, Miquel Calvo, Ferran Reverter, and Roderic Guigó. Iden- tification and analysis of splicing quantitative trait loci across multiple tissues in the human genome. Nature Communications, 2021.
Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimeček, and Panagiotis Alexiou. Ge- nomic benchmarks: a collection datasets for genomic sequence classification. BMC Genomic Data, 2023.
Jennifer Harrow, Adam Frankish, Jose M Gonzalez, Electra Tapanari, Mark Diekhans, Felix Kokocinski, Bronwen Aken, Daniel Barrell, Amonida Zadissa, Stephen Searle, et al. Gencode: the reference human genome annotation for the encode project. Genome Research, 2012.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformat- ics, 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint urXiv:2310.06825 2023.
Melissa J Landrum, Jennifer M Lee, George R Riley, Wonhee Jang, Wendy S Rubinstein, Deanna M Church, and Donna Maglott. Clinvar: public archive of relationships among sequence variation and human phenotype. Nucleic Acids Research, 2014.
Louis Lello, Steven G Avery, Laurent Tellier, Ana I Vazquez, Gustavo de Los Campos, and Stephen DH Hsu. Accurate genomic prediction of human height. Genetics, 2018.
Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, Cheng Tan, Jiangbin Zheng, Yufei Huang, and Stan Z. Li. Vqdna: Unleashing the power vector quantization for multi-species genomic se- quence modeling. In ICML, 2024.
LeAnn M Lindsey, Nicole L Pershing, Anisa Habib, Keith Thomult-Toompson, W Zac Stephens, Anne J Blaschke, Xiaofang Jiang. and Hari Sundar. The impact of tokenizer selection in genomic language models. Bioinformatics, 2025.
Haozheng Luo, Chenghao Qiu, Maojiang Su, Zhihan Zhou, Zoe Mehta, Guo Ye, Jerry Yao-Chieh Hu, and Han Liu. Fast and low-cost genomic foundation models via outlier removal. In ICML., 2025.
Frederikke Isa Marin, Felix Teufel, Mare Horrender, Dennis Madsen, Dennis Pultz, Ole Winther, and Wouter Boomsma. Bend: Benchmarking dna language models on biologically meaningful tasks. In ICLR, 2024.
Vera Milovanović and Antonio Orvieto. No clear winner at small scale: Comparing modern se- quence architectures and training strategies for genomic language models. In ICML Generative Al and Biology (GenBio) Workshop, 2025.
National Center for Biotechnology Information (NCBI). NCBI [Internet]. https: /twp:///wwww../ ncbi.
nlm. nih. gov/ /, 2024.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin W Thomas, Michael Wornow, Callum Birch- Sycch,Sykes, Stefano Massaroli, Aman Patel, Clayton M. Rabideau, Yoshua Bengio, Stefano Ermon, Christo- pher Re, and Stephen Baccus. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. In NewrIPS, 2023.
12

---

Published as a conference paper at ICLR 2026
Eric Nguyen, Michael Poli, Matthew G. Durrant, Brian Kang, Dhruva Katrekar, David B. Li, Liam J. Bartie, Armin W. Thomas, Samuel H. King, Garyk Brixi, Jeremy Sullivan, Madelena Ng, Ash- ley Lewis, Aaron Lou, Stefano Ermon, Stephen A. Baccus, Tina Hermandez-Boussard, Christo- pher Ré, Patrick D. Hsu, and Brian I.. Hie. Sequence modeling and design from molecular to genome scale with evo. Science, 2024.
Lifeng Qiao, Peng Ye, Yuchen Ren, Weiqiang Bai, Chaoqi Liang, Xinzhu Ma, Nanqing Dong, and Wanli Ouyang. Model decides how to tokenize: Adaptive dna sequence tokenization with mxdna. In NeurlPS, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Melissa Sanabria, Jonas Hirsch, Pierre M Joubert, and Anna R Poetsch. Dna language model grover learns sequence context in the human genome. Nature Machine Intelligence, 2024.
Eric W Sayers, Evan E Bolton, J Rodney Brister, Kathi Canese, Jessica Chan, Donald C Comeau, Ryan Connor, Kathryn Funk, Chris Kelly, Sunghwan Kim, et Database resources of the na- tional center for biotechnology information. Nucleic Acids Research, 2022.
Yair Schiff, Chia- Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arxiv:24030032. 2024.
Ziqi Tang, Nirali Somia, Yiyang Yu, and Peter K Koo. Evaluating the representational power of pre-trained dna language models for regulatory genomics. bioRxiv preprint bioRsiv:2024.00.29.55222 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In EMNLP, 2020.
Weimin Wu, Zhihan Zhou, Robert Riley, Mohammad Abdulqader, Xuefeng Song, Satria Kautsar, Rob Egan, Steven Hofmeyr, Gefei Liu, Shira Woldhaber-Gordon, Mutian Yu, Harrison Ho. Yaping Liu, Andrei Stecca Steindorff, Fengchen Liu, Feng Chen, Rachael Morgan-Kiss, Lizhen Shi, Han Liu, and Zhong Wang. Uncovering the genomic manifold via scalable learning from the global microbiome. bioRxiv preprint
2025.
Xiang Zhang, Mingjie Yang, Xunhang Yin, Yining Qian, and Fei Sun. Deepgene: An effi- cient foundation model for genomics based on pan-genome graph transformer. bioRxiv preprint bioRxiv:2024.044. 0.....0...4.. 24. 24.990879, 2024.
Yao-zhong Zhang, Zeheng Bai, and Seiya Imoto. Investigation of the bert model on nucleotide sequences with non-standard pre-training and evaluation of different k-mer embeddings. Bioin- formatics, 2023.
Jian Zhou and Olga G Troyanskaya. Predicting effects of noncoding variants with deep learning- based sequence model. Nature Methods, 2015.
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V Davuluri, and Han Liu. DNABERT-2- Efficient foundation model and benchmark for multi-species genomes. In ICLR, 2024.
A APPENDIX
A.1 RELATED WORKS
Genomic Foundation Models. Encoder-only approaches have proven effective in sequence pre- diction tasks, using k-mer tokenization (Ji et al., 2021; Dalla-Torre et al., 2024), Byte Pair Encod- ing (Zhou et al., 2024; Sanabria et al., 2024), and learned tokenization strategies (Qiao et al., 2024; Li al., 2024) to enhance efficiency and manage longer sequences. Certain encoder architectures
13

---

Published as a conference paper at ICLR 2026
have been enhanced with recurrent memory mechanisms (Fishman et al., 2025) capture long- range dependencies more effectively, while others utilize whole -eoomee aligmments (Benegas et al., 2025a) to incorporate evolutionary context. More recent work has explored pan--ennnnnn genome graph rep- resentations (Zhang et al., 2024) to better capture genetic variation diversity.
Meanwhile, decoder-only architectures have shown potential by integrating structured state-space models (Nguyen et al., 2023; Schiff et al., 2024), achieving competitive performance with minimal parameters and supporting long context lengths. Hybrid architectures (Nguyen al., 2024; Brixi et al., 2025), incorporating both attention and state-space blocks, have emerged, demonstrating great generative capabilities spanning from molecular to genome scales. Our work introduces GFM based on Mistral architecture (Jiang et al., 2023) and performs performance analysis of the most recent GFMs.
Genomic Foundation Models Analysis. It was shown that k-mer embeddings pretrained on ran- dom DNA sequences can reach similar performance those trained on the world biological data (Zhang et al., 2023). Another study found that character tokenization outperforms other meth- ods in state-space models (Lindsey et al., 2025). Evaluation of GFMs across the BEND benchmark reveals that they capture limited information on long-range features (Marin et al., 2024). It was also shown that mean pooling improves performance of GFMs for genomic sequence classifications and closes the performance gap between them (Feng et al., 2025). Pretrained DNA models were bench- marked (Tang et al., 2024) showing they do not offer great advantage over conventional machine learning methods. In contrast to this study, analysis includes finetuning and variant- -aaset tasks, more models and also shows that randomly initialized models can be better as feature extractors.
A.2 MODELS
We use the following GFMs in our analysis:
HyenaDNA (Nguyen et al., 2023): Decoder-only state-space model with 450K parameters. Uses character tokenizer and was pretrained on the Human Reference Genome with 1024 base pair sequence length.
Caduceus (Schiff et al., 2024): Decoder-only model with 8M parameters. Trained on sequences of 131k base pairs on HRG. Combines a bidirectionally equivariant decoder with character tokenizer.
Mistral (our version): Decoder-only transformer model with 580M parameters. Uses character tokenization and was trained on the 1000 Genomes dataset (Consortium et al., 2015).
Nucleotide Transformer (Dalla-Torre et al., 2024): Encoder-only model presented in two ver- sions: 500M parameter model trained on the 1000 Genomes Project data and its v2 with 50M parameter model trained on multispecies data. Both use k-mer tokenization.
GENA-LM (Fishman et al., 2025): Encoder-only model with 110M parameters. Employs BPE tokenizer and was pretrained on the HRG with 1000G augmentations.
DNABERTy2 (Zhou et al., 2024): Encoder-only model with 117M parameters. Uses BPE tok- enization and was trained on multispecies data.
A.3 RANDOM WEIGHT INITIALIZATION
We initialized the model weights following procedure using standard Hugging Face Transformers library (Wolf et al., 2020) initialization methods:
For Linear Layers: Weights were initialized from normal distribution (0, 0.02), biases were
initialized to zero.
For LayerNorm: The scaling factor (gamma) was initialized to 1. The bias term (beta) was initialized to O.
For Embedding Layers: Embeddings were initialized from the same normal distribution (O, 0.02).
For Caduceus and HyenaDNA we performed prenorm residual rescaling, which is the default weight initialization procedure for these models. Biases for linear layers were initialized zeros.
14

---

Published as a conference paper at ICLR 2026
A.4 MISTRAL PRETRAINING
We pretrain a Mistral model on 50 random individual samples from the 1000 Genomes Project. Table 6 provides the Mistral configuration details and Table provides the Mistral training configu- ration. Specifically, reverse complement of sequences formed with Genome VCFs is used with probability of 0.5. All the chromosomes (chrl chrX) are used for sequence formation from 50 individuals. Individuals are sampled in stratified way, 10 from each superpopulation. We filtered out sequences where number of unknown nucleotides was more than half of sequence length. Total number of tokens is 150B. We used Py PyTorhh FSDP framework. Training was done on GPU cluster using 6 nodes with Nvidia H100 GPUs per node.
config
value
tokenizer
character
config
value
sequence_eet
4096
16
num_epochs
1
num attention heads
16
initial Ir
7.2c-4
hidden_size
1408
final.Ir
4.2e-5
vocab_sizz
12
optimizer momentum
31, B2 = 0.9, 0.95
intermediate.st _size
7168
Ir schedule
cosine with warmup
batch_size
64
Table 6: Mistral model architecture.
num-genomes
50
Table 7: Mistral training configuration.
A.5 COMPUTING INFRASTRUCTURE
We used the following computing infrastructure the experiments:
Hardware Specifications:
GPU Cluster: 6 nodes with 8 Nvidia H100 GPUs per node (48 H100 GPUs total)
GPU Memory: 80GB
Software Specifications:
Framework: PyTorch with FSDP (Fully Sharded Data Parallel)
CUDA Version: 12.1
A.6 FINETUNING EXPERIMENTS
We use the following datasets for finetuning, more details about them can be found in the corre- sponding original papers:
NT Benchmark (Dalla-Tone) et al., 2024) consists of the following group of tasks histones, en-
hancers, promoters and splice sites.
Genomic Benchmarks (Grešová et al., 2023) contains several datasets focused on regulatory element classification tasks across three organisms: human, mouse, and roundworm.
Genome Understanding Evaluation (GUE) (Zhou et al., 2024) is a comprehensive multi-species benchmark containing 28 datasets across 7 genomic analysis tasks including promoter detection, transcription factor prediction, splice site detection, etc. with sequence lengths ranging from 70 to 1000 base pairs.
We finetune random and pretrained initializations of the chosen model using the configuration pro- vided in Table 9. In our preliminary experiments, found that max pooling performed better than cls last pooling for randomly initialized models while maintaining performance for pretrained, so used max pooling consistently across all experiments.
Additionally, we perform an ablation for full finetuning vs LoRA (Hu et al., 2022) finetuning and present the result in Table 8. In all the cases, full finetuning outperforms LoRA, suggesting that
15

---

Published as a conference paper at ICLR 2026
NT Benchmark
10.1
MMCC
types
MMCC
4.1
Meteral
Nhv soe
Deseean
Soote
Caloenu
Motonal
NT12 Sole
hyreaatle
Ar Scote Conta oou
Miserol
N712 soee Hy preasbii
czus CENALL Lt DISceEnd
sonte
Cadocruu
Mistral
Nhv Sole
Hyrenebuu azne-Le
Castomu
Mttorel
Nt hyyoubble
catae de DNGGEEAT
Ny soole Caloouu
Misenll wiv2 Soe
Hy hyycaabis Gasth
sioee
Cadornu
GUE Benchmark
10.2
Bex oodeet CHoNeess
Epigenetic Marts
Promsier Delection BEX rommm Maro
If Predicion (humen)
Bost rendon Piti!
Lirw Homoler Dsteciion
DISTORMM Ealsons
Prediction (Nowe)
Bas naNdOrT CHRNDOS
Spice
M.CC
4.3
4.4
Mesrol
inemablle
Ar seos canarous
sauunr NTiz some Hyyeneti
Stote
Mastral
1912 Stod
pperuauu
art scnte
Castereuo
Mtz Some
sooe
cabseaa
1rt2 Soaee
cententr
Mitccee Mtt Some
Ar Scou
Genomic Benchmarks
rislboory
Accurscy
0.1
Marral
Nr sopee
Mantar aT12 Spa ny NymonNA GENA Dauaenn WT sacceuus
Mastcal
NT sonee Cadureos
Motial w7-2 Soze
HycnsDN
GERA-4 mekeranc
HT Soove Sutoomus
Mastall Suit
tyvers IyvenOON
cone- DAKBEATT
NT souet
Cedweea
Mottal NT-2 soe
Hyumsoe
WT Soute
wnz2
Figure 7: Pretraining provides limited diverse genomic tasks. We plot the performance difference (A MCC or Accuracy) between seven pretrained GFMs and the best-performing randomly initialized model for each task. Red bars indicate tasks where the randomly initialized model outperformed, whereas green bars indicate tasks where the pretrained model performed better. The predominance of red bars provides strong evidence that current pretraining paradigms, adapted from NLP, offer minimal benefit for genomics tasks. Notably, smaller randomly initialized models (c.g., Caduceus) frequently achieve the best performance, challenging the assumption that larger pretrained models are inherently superior.
advantage over randomly initialized models across
full finetuning method gives the best chance for the models (both pretrained and random) to achieve their best scores.
16

---

Published as a conference paper at ICLR 2026
config
value
optimizer
AdamW
Model
Method
eohancors
H3K4mel
promoter.all
splice all
Ic-5, 30-5,
GENA-LM
LoRA
0.449
0.260
0.908
0.501
learning rate
8c-5, 1c-4, 3e-4
Tull
0,560
0.466
0.966
0.935
weight decay
0
Mistral
LoRA
0.470
0.267
0.906
0.525
0.9,
Fall
0,550
0.557
0.965
0,980
optimizer momentum
0.999
50M
LoRA
0.405
0.267
0.852
0.458
batch size Ir schedule
32
Full
0.551
0.524
0.957
0.979
cosine
epochs
20/ / 100
Table 8: LoRA full finetuning four representative NT tasks. The metric is MCC for enhancers and histone, and F1 for promoters and splice sites. Across all cases full finetuning performs better than LoRA.
Table 9: Hyperparameters for fine- tuning experiments. For GUE we finetune for 20 epochs for NT Bench- mark and Genomic Benchmarks we use 100 epochs.
A.7 DETAILED FINETUNING PERFORMANCE BY TASK SUBGROUP
To showcase the individual performance of randomly initialized models, we present their results on NT Benchmark alongside pretrained models in Fig. 8. For instance, the "Enhancers" subgroup includes all enhancer-related tasks, while the "Histone" subgroup covers all histone tasks, and so In addition, we also show this for GUE and Genomic Benchmarks in Fig. and Fig. 10. We also provide results for all models on NT Benchmark in Table 18.
The results presented in Fig. highlight that randomly initialized models can perform remarkably well across all subgroups of the NT Benchmark. In the "Enhancers" subgroup, all randomly initial- ized models perform comparably their pretrained counterparts. In histone tasks, the best random models, DNABERTv2 and Caduceus, reach average MCC scores of 0.62 and 0.63, outperform- ing pretrained NTv2 50M, HyenaDNA. GENA-LM, and NT 500M. In case of randomly initialized Caduceus it also outperforms its own pretrained version.
Histones
Splice Sites
Promoters
1.0
Aug MCC
0.6.
27 0.4
0.5
Mistral
NTV2 SOM
HyenaDNA
GENA.AA LM
DMABERT22
NT 500M
Caducaus
Motraa
NTV2 SOM
HyenaDNA
GENA-LM LM
DNABERTO2
NT NT 500M
Ceducous
0.0
Mismal
mTv saw
HyeraDNA
GENA-LA LM
DMABERTI22
NT 500M
Casurruus
Mistral
NYv2 SOM
HyenaDNA
GENA- LM
DMABERTV2
500M
Cadurouu
Figure 8: NT Benchmark performance per subgroup. Pretrained models (clear bars) vs randomly initialized (dashed bars). Random models competitive across all subgroups, with random Ca- duceus outperforming five pretrained models on challenging histone tasks. Red dashed line shows
best random model score.
17

---

Published as a conference paper at ICLR 2026
Epigenetic Marks
Promoter Detection
Prediction (Human)
MCC
0.77
0.00
0.5
Moma
4712 Sode
GG AM
NT SonM
Cudkenss
mestral
NY12 SOM
Mymnspuu
GEMe LM
NT SOUM
Corlccoss
Morral
NY22 SoM
HymaDDi
GEA64444
Ar SOOM
Cudbcasa
Care Promoter Detection
TF Prediction (Mouse]
Splice
MCC
0.0 NA
GoT
Mornal
N122 Soe
GENALA
DIAGERIIVZ
Show
Cutverus
Mastral
N712 SOM
GBIA-L
DRABERIVZ
Galsonus
Mammal
GE
DIARDRIZZ
NT
Figure 9: GUE performance for each model. Solid bars indicate pretrained model, dashed bars indicate random. Horizontal red dashed line indicates the performance of the best random model. The best rand. init. model is consistently competitive with pretrained models.
Human Nontata Promaters
Human E Ensembl Regulatory
Human Enhancers Ensembl
1.0-
Accuracy
0.8
0.98 10
0.33
NAXA
0.95
097
0.96
047
50
0.01
0.
0.22
D550
Histral
NT&2 S0M
HyenaDNA
CENA Lt
DNABERTV22
NT SOOM
Caluccuue
Mistral
NTV2 504
HyenaDNA
GENA 1t
DNASERTV2
NT sool
Carbucus
Misraa
KYV2 50M
HyenaDNA
GENA-L4
DHABERTV2
N7500H
Caubrruus
Human E Enhancers Cohn
Drosaphila Enhancers Stark
Demo Human Or Worm
1.0
Accuracy
0.8
0.97
0.99
0:57
0.02
020
0.23
0023
0.75
0.22
60
100
0.79
6878
60
Histral
NTV2 S0M
HYsnaDNA
CENA.U
DRABERTVZ
NT 5O0M
Caluccus
Mistral
urv2 50M
HyenaDMA
DNABERTV2
NT so0M
Carbcaus
Misral
N0M SOM
HyenaDNA
GENA-LM
DMABERTV2
SO0M
Cadbraus
GENA
Figure 10: Genomic Benchmarks performance for each model. Solid bars indicate pretrained
model, dashed bars indicate randomly initialized. Horizontal red dashed line indicates the perfor- mance of the best random model.
A.8 BIOTYPE CLASSIFICATION
Biotype task a sequence classification task into nine different labels. Our dataset consists total of 19605 sequences. The detailed statistics for sequences belonging each gene type is provided in Table 10. For the supervised training step, we perform a train-test split of 80% 20% using stratification by class label. We use XGBoost with the hyperparameters provided in Table 12. All metrics are reported on the test set.
In addition, also perform similar feature extraction experiments on the subset of GUE benchmark displayed in Table 13. Randomly initialized HyenaDNA with large embedding size outperforms all pretrained models.
18

---

Published as a conference paper at ICLR 2026
Gene Type
Count
Avg Length
Max Length
Min Length
TEC
1056
1613.26
18662
87
IncRNA
3000
32359.59
957949
87
miRNA
1879
81.89
180
41
misc RNA
2212
206.49
464
57
processed pseudogene
3000
798.02
12016
28
protein coding
3000
69971.51
2059620
159
saRNA
1901
110.46
328
50
snoRNA
943
118.86
791
55
unprocessed pseudogene
2614
5025.27
233909
28
Table 10: Statistics biotype genes.
Model
Embedding Dimensions
config
value
HyenaDNA
64, 128, 256, 512, 1024, 2048, 4096
objective
multi:softmax
Caduceus
64, 128, 256, 512, 1024, 2048
num classes
NTv2 50M
64, 128, 256, 512, 1024, 2048, 4096
max_depth
DNABERTv2
48, 96, 192, 384, 768, 1152, 1536, 2304, 3072
learning rate
0.1
GENA-LM
48, 96, 192, 384, 768, 1152, 1536, 2304, 3072
n_estimators
1000
500M
100, 320, 640, 1280, 2560, 3840
eval metric
mlogloss
Mistral
64, 128, 256, 512, 1408, 2048, 4096
doe_method
hist
Table 11: Embedding dimensions for biotype experiments.
Table 12: Biotype XGBoost configuration.
Pretrained
Task
HyenaDNA Random EDD 2048
Mistral
HyenaDNA
NTv2-50M
GENA-LM
DNABERT<2
500M
H3
0,650
0.626
0.510
0.502
0.346
0,566
0,557
H3K14ac
0.275
0.227
0.190
0.272
0.208
0.338
0,220
H3336me3
0.408
0.267
0.252
0.330
0.321
0.397
0.308
13344ml1
0.320
0.224
0.211
0.275
0.244
0,295
0.267
113K4me2
0.265
0,243
0.186
0.176
0,218
0.185
0.245
H3K4mm33
0.207
0.126
0.105
0.147
0.113
0.189
0.121
H3K79me3
0.522
0.428
0.367
H3K9ac
0.429
0.373
0.288
0.343
H14
0.671
0.649
0.491
0.612
0.282
0.227
0.202
Average
0.403
0.339
0.280
0.324
0.375
0.330
Table 13: Feature Extraction on Histone Tasks from GUE, Embeddings extracted from pre- trained and randomly initialized models were used to train an XGBoost classifier. Randomly initial- ized HyenaDNA with embed_ 2048 outperforms every pretrained model on every task except H3K 14ac. MCC on test set reported.
A.9 ANCESTRY BENCHMARK
Each task is the sequence classification task with five labels, South Asian, European, African, Amer- ican, East Asian. Each label is a superpopulation from 1000 Genomes dataset. We selected eleven different regions on chromosome with the length of 32K nucleotides, where each region corre- sponds different variant. The start indices with respect to the human reference genome used for sequence construction provided in Table 14. Each task has 3202 samples.
Training involves two stages: embedding generation from the model interest and supervised train- ing on the embeddings with XGBoost. During the embedding generation step, sequence embeddings are constructed similarly to the biotype classification task. For the supervised training step, we split the dataset into train, validation and test set with sizes 72%, 8%, and 20% respectively. We use XGBoost with hyperparameters mentioned in Table 15. All metrics are reported on the test set and averaged eleven tasks across each chromosome.
19

---

Published as a conference paper at ICLR 2026
chromosome
start position
chrl
119478211
config
value
chr3
2015011
objective
molti:sffmmax
chr5
85769129
num num_class
chr?
74672986
max-depth
chr9
75197358
learning
0.1
chrl
62543311 I
n.cstimators
1000
chrl3
52182164
colsample bytree
0.5
chrl5
45995594
eval_metric
mlogloss
chrl7
36628720
tree.methov
hist
chr19
24308808
stopping rounds
100
chr21
18354991
Table 15: Ancestry XGBoost configuration.
Table 14: Sample chromosome positions.
A.10 CLINVAR EXPERIMENTS
Each chunk used for Clin Var experiments consists of benign and pathogenic mutations. Three types of sequences are formed: reference sequence, sequence with benign mutations, and sequence with pathogenic mutations. The distribution of mutations in these chunks for all three genes is presented in Table 16.
TP53
BRCA2
CFTR
Chunk Index
Benign
Pathogenic
Benign
Pathogenic
Benign
Pathogenic
122
27
138
46
32
18
2
60
61
268
74
19
11
3
51
50
187
57
32
30
76
13
38
37
6
11
Table 16: Mutation Data Distribution by Gene and Chunk. Distribution of benign and pathogenic mutations across different chunks for TP53, BRCA2, and CFTR genes.
A.11 MODEL CHECKPOINTS
Checkpoints for all the pretrained models were obtained from Hugging Face. Table 17 provides detailed checkpoint IDs which can be loaded using the transformers library.
Model
Checkpoint
NTv2 50M
NT 500M
Caduceus
1
HyenaDNA
LongSafriilyoaaannaaiyyy--ssaaa
DNABERTy2
2hiban1196/DNAABBER22221111
GENA-LM
LIRl-Institute/genaa bew.-eeeen-------e------------
Table 17: Checkpoints used for pretrained models.
20

---

Published as a conference paper at ICLR 2026
Caduoors
01,606 Lon0
0.795 2.0000
0.615 0100
0.555 0.555 1.00c
0.493 1.000
0.65 1o0
0.795 Lone
0.451 0.45: 2001
0.967 0.9672000 1.000
0.970 1.000
0.955 2
0.968 5:000
0.685
0.534
0.525
0.962
0.960
NY 500N
0.266 tuno
0.102 100s
0.20 1.000
0.35 1I01
0397 LIDS
0454 1000
0951 0951 1000
0934 1,000
0841 1002
1000
0.620
0254
0.500
QU00
0508
DNABERTY2
0.00
0.395 0AR0
0.605 sone
0,511 2:00
0611 2006
0.742 1000
0.088 2000
0.835 2.05
0,900 2000
0.932 0.932 5001
0.669
0,499
0,529
0,850
09999
Randem
GENA-LM
0.43 1.00
0.223 S-112
0,316 0.00
0.16 LOGG
0/556 LO00
0,474 2000
0.726 000
0,787
1.000
0.856 000
0 2:000
0.619
0,905
0.33
0.5.5
0,767
0.856
0.899
Table 18: Pretrained and randomly initialized models performance on NT Benchmark.
IleenaNA
0,552 0002
0.354 1.000
0,302 2.06
0.403 LOME
0,770 LOMe
2002
0.95
2001
0.958 1000
0.787
0.576
0.421
0.556
0.75
0.976
0,960
0.988
NTV2 50M
0.503 2000
0.245 LO0
0.168 S-C0NN
0.31 LOON
0.73 LONS
0448 0
2005
00082000 2005
0574 2OND
0836 2009
0.066 LINE
0.007 2000
0.45
0.260
0.52
0.08
0.470 0o00
0.444 001
0.573 DOM
0.679 0OG
0 0OM
0,977 0000
0,922 0000
0,958 = 000
0.955 000
Mistral
0,760
0,770
0.422
0.561
0,91
0.957
Caduerus
0.688 IL0S
0.56 HUSS
0-433 LOTT
0613 LIS
O.EG LIMES
0-47 1ORS
0.999 1003
1003
0.665 1.00
0.778
0,42
0L52
0.88
0.95
01977
NY S0MNN
0.475 0I00
0,259 2-40n
0.289 2002
D-469 LOIE
0.741 LAWS
0433 2005
0,977 2000
0,955 2OMS
0,947 20000
0,953 0.953s: 5:000
0,702
0,307
0,557
0,954
DNABERUYA
0,009 0000
0.644 DOM
0.507 MARE
0.564 0096
0.66 DON
0.826 Bes
0-469 0O00
0,000 & 000
0.973 A 000
0.965 DOW
0.973 000
0.585
0.336
0.955
Pretrained
GENA-LM
DOWN
0.551 0O0E
0.487 DO0X
0-445 DO0S
D.555 0001
0.776 0OWS
0477 2002
0041 0002
0,953 1000
00S
5 0001
0,773
0.63
0,408
0,587
0,500
0.955
0,970
0,969
HyymaDNA
1004
0.427 100C
0012
LOOK
0,778 WOST
0.444 000S
0.987 0006
0000
0002
0001
0.783
0.541
0.367
0.441
0.588
01552
0.988
0.950
0961
0.600
N1Y2 S0M
(100
0,792 2.000
0,500 2-007
2000
0.56 2008
09900 20000
2000
5000
0.5-0
0,595
0,303
0,518
0:51
0,769
0,57
0.93
0,956
0998
0957
001
0,682 0008
0007
0,09 0On
0,800 0.00
0000
0.71 0002
Misral
0,802
0.664
0.53
0.744
0,00
0.73
0.24
0,78
0557
0981
0962
0070
Medric
MCC
MCC
MCC
MCC
MCC
MCC
MCC
MCC
MCC
FI Scons
FI Soone
FI Score
Fi Score
spice. donors
Dataset
H5X70mm
H5S44mel
H5K36003
enhancers
splion sytioraiteaa siteell
promoter ints
poonnte
pommter..al
114
HS
21


Reviews:
[{"review_id": "FwRoAIwdWC", "reviewer": "Reviewer_zXbv", "summary": "This submission evaluates seven Genomic Foundation Models (GFMs) on 52 tasks spanning NT Benchmark, GUE, and Genomic Benchmarks, comparing pre-trained checkpoints against randomly initialized counterparts under fine-tuning, frozen‑feature (“feature extraction”), and genomic variation settings. The headline result is that models trained from scratch often match or surpass their pre-trained versions; moreover, embeddings from multiple GFMs are notably insensitive to clinically relevant variants, with cosine similarity remaining ~0.9–0.999 even after many SNPs are introduced. The paper argues that (i) existing NLP‑style pre-training is a poor investment for regulatory genomics classification and (ii) tokenization/architectural choices (e.g., character tokenization, larger embedding dimensions) dominate performance.", "questions": "- **(Q1)** Can the authors provide apple-to-apple ablations where architecture, tokenizer, embedding size, positional encoding, and training schedule are identical, and the only difference is with vs. without pre-training? This would directly test the causal value of pre-training beyond the confounds noted in Table 2 and Figure 3.\n\n- **(Q2)** How do the findings change in label-scarce regimes (e.g., 1%, 5%, and 10% of labels)? Please include learning curves and area under the data curve statistics to test whether pre-training is more helpful at low data.\n\n- **(Q3)** For mutation sensitivity, could the authors further provide in-silico mutagenesis with base-resolution attribution (e.g., Grad-CAM) and compare to CADD/Enformer scores and to eQTL/sQTL ground truths (AUPRC, per-gene AUROC)?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **(S1)** Breadth and scale of evaluation with reproducibility intent. The study spans 7 models and 52 tasks across three popular benchmarks, with nearly 10k fine-tuning runs and LR sweeps, and reports model‑wise subgroup results. Datasets and checkpoints are standard, and code is linked (anonymized) in the paper, which, taken together, improves reproducibility. Meanwhile, the Figure 1 summary is easy to parse and consistently shows a limited advantage for pre-training when judged against a strong random baseline. The authors also empirically prefer full fine-tuning over LoRA, reducing a common confound in cross‑model comparisons.\n\n- **(S2)** Variant-sensitivity diagnostics reveal a critical gap. The mutation-sensitivity experiments and ClinVar log-likelihood–ratio AUROCs near 0.5 jointly show that many GFMs’ representations are remarkably insensitive to clinically relevant point mutations, echoing long-standing concerns that subword/k‑mer tokenization blurs SNV granularity.\n\n- **(S3)** Positioning within ongoing benchmarking discussions. The paper’s thesis aligns with emerging benchmark evidence that SFT matters and that supervised long-range models can still dominate on gene expression, underscoring that what we benchmark matters as much as how.", "weaknesses": "- **(W1)** “Pre-training vs. random” comparisons are confounded by architecture/tokenizer changes. The feature-extraction claim that random models can beat pre-trained hinges on changing the tokenizer (char rather than original k‑mer/BPE) and increasing the embedding size for the random arm (Table 2 and Figure 3), whereas pre-trained arms keep their native tokenizers/widths. This violates ceteris paribus, conflating the benefit of model design with the absence of pre-training. A fair test needs identical architectures/tokenizers/widths with/without pre-training. As is, the headline conclusion is directionally plausible but not causally isolated.\n\n- **(W2)** Variant sensitivity method is too blunt to support strong claims. The mutation‑sensitivity analysis relies on global pooling and cosine similarity of full‑sequence embeddings, while high similarities that sometimes increase with more mutations likely reflect pooling/normalization effects rather than true biological blindness. Moreover, the LLR analysis lacks detail for encoder-only masked LMs (pseudo-likelihood vs. left-to-right), complicating an apples-to-apples comparison with decoders. Token-level distances, attribution at mutated loci, and clearly specified pseudo-LLR for encoders are necessary.\n\n- **(W3)** Task coverage underweights where long-range biology is known to matter. The selected tasks span mainstream benchmarks, but the suite lacks gene-expression regression (e.g., bulk RNA/CAGE) and enhancer–gene linkage tasks for which long-range inductive biases and explicit supervised training (e.g., Enformer) remain strong baselines and stress tests.\n\n- **(W4)** Positioning more concurrent architecture findings could be sharper. The paper’s narrative at times attributes the observed wins to the absence of pretraining, when the architecture/inductive bias could be dominant. Independent recent results indicate that simple, well-tuned CNNs can outperform SSM/Transformer DNA models on many tasks without pretraining (e.g., ConvNova), suggesting that architectural priors and receptive-field design can rival or exceed pretraining gains. This alternative explanation deserves explicit treatment in the Discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5DoVrXeAFY", "reviewer": "Reviewer_4ABt", "summary": "This paper challenges the prevailing assumption that large-scale unsupervised pretraining is inherently beneficial for Genomic Foundation Models (GFMs). The authors conduct an extensive empirical study comparing seven different GFMs against their randomly initialized counterparts across 52 diverse genomic tasks, spanning finetuning and feature extraction. The core finding is that randomly initialized models, when properly tuned, often match or even exceed the performance of their billion-scale pretrained counterparts.\n\nFurthermore, the paper introduces a critical set of analyses on genomic variation, demonstrating that current GFMs are largely insensitive to subtle, clinically relevant mutations like SNPs. The models perform at near-random chance on variant classification tasks and produce nearly identical embeddings for reference and mutated sequences. The authors conclude that current pretraining strategies, largely adapted from NLP, are insufficient for genomics and that the field must rethink its approach to tokenization, pretraining objectives, and evaluation.", "questions": "1. The authors convincingly argue that k-mer/BPE tokenizers are a major issue, especially for the genomic variation tasks. Their own Mistral model, which uses a character tokenizer and performs well, seems to support this. Could the paper's central finding be more narrowly (and perhaps more accurately) stated as \"Current k-mer-based pretraining strategies are ineffective\" rather than a blanket statement about all pretraining?\n\n2. Following on the previous point, the feature extraction results (Table 2, Fig. 3) suggest that a randomly-initialized model with a character tokenizer and an optimized embedding dimension is a top performer. Is this also true for the finetuning tasks? A comparison of \"best random (char-tokenizer)\" vs. \"best pretrained\" would be very illuminating.\n\n3. The authors attribute their different findings from prior work (e.g., Dalla-Torre et al., 2024) to their more rigorous hyperparameter search. As a sanity check, were they able to reproduce the original paper's results (i.e., showing a benefit for pretraining) by using the original fixed hyperparameters? This would definitively confirm that the HPO sweep is the key methodological difference.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Large-Scale, Rigorous Evaluation: The sheer scale of the study (7 models, 52 tasks) provides very strong evidence. The authors' commitment to a rigorous hyperparameter search for all models (random and pretrained) makes their comparison fair and robust.\n\n2. Novel and Critical Analysis: The genomic variation analysis (Sec 3.3) is a major strength. By showing that GFMs are insensitive to SNPs and fail on ClinVar data, the authors expose a critical blind spot in current models and evaluation methods. This finding alone is a significant contribution.\n\n3. Actionable Insights: The paper provides clear hypotheses for these failures (k-mer tokenization obscuring SNPs, high masking rates) and points toward concrete areas for improvement (e.g., character-level tokenizers, as used in their own Mistral model).", "weaknesses": "1. Limited Scope (Generative Tasks): The paper's claims are based entirely on classification and feature extraction tasks. The authors briefly concede in the discussion and conclusion that pretraining might still be valuable for generative tasks. This is an important limitation, and the bold title (\"Pretraining Does Not Promise Performance\") might be a slight overstatement. The weakness is minor, as the paper's scope is already large, but it should be stated more prominently.\n\n2. \"What\" over \"Why\": The paper excels at showing that pretraining fails but is less definitive on why. The discussion of tokenization and masking rates is insightful but largely correlational. The paper would be strengthened by even a single targeted ablation, e.g., pretraining two identical small models, one with k-mer and one with character-level tokenization, to prove that the tokenizer is the key factor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pCm71Kjrxz", "reviewer": "Reviewer_rYMn", "summary": "This paper evaluates the effectiveness of pretraining in Genomic Foundation Models (GFMs). The authors benchmark seven GFMs across 52 genomic tasks, comparing them with randomly initialized counterparts. Results show that randomly initialized models often match or even outperform pretrained GFMs in both fine-tuning and feature extraction settings. The study concludes that current research lacks a clear understanding of how pretraining contributes to model performance and offers new insights into how to achieve more efficient genomic modeling.", "questions": "See the weaknesses section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Meaningful Perspective: The paper focuses on evaluating the effectiveness of pretraining in GFMs and raises new research questions and perspectives for genomic modeling, providing valuable insights for the community.\n\n- Comprehensive Evaluation: The study benchmarks 7 representative GFMs across 52 diverse genomic tasks, covering structural, functional, and regulatory dimensions.\n\n- Extensive Experimental Setup: The authors conduct detailed comparisons of fine-tuning techniques, including full fine-tuning and LoRA. Also perform extensive hyperparameter searches over learning rate, batch size, and other factors with nearly 10,000 fine-tuning experiments.", "weaknesses": "- Potentially Unfair Comparison Setup: Randomly initialized models received embedding dimension optimization, but similar architectural optimization was not performed for pretrained models. What would happen if the pre-trained model also used a character tokenizer and a larger embedding dimension?\n- Analysis of Pretraining: The observation that a character-level tokenizer benefits randomly initialized models may indicate design issues in the pretrained models rather than reflecting the true effect of pretraining on genomic models.\n- Task Distribution: The paper evaluates 52 tasks, but it is unclear what criteria were used to select them. Are there specific types of tasks like long-range dependency tasks where pretrained models might perform better?\n- Generative Tasks: All seven models evaluated in the paper are classification models, and the 52 tasks focus primarily on classification, with no assessment of generative tasks. Although the authors explain their reasons for not using Evo, recent large-scale genomic models such as Evo2[1] and GenomeOcean[2] are designed for generative purposes. Do the conclusions drawn in this study also hold for generative models, or are they only valid when generative models are applied to classification tasks?\n- Quantized Model: In this paper, all models are tested under full precision. However, quantized genomic foundation models (GFMs) like GERM [3] also exist. Does the same conclusion hold for these models?\n\n[1] Genome modeling and design across all domains of life with Evo 2.\n\n[2] GenomeOcean: An Efficient Genome Foundation Model Trained on Large-Scale Metagenomic Assemblies.\n\n[3] Fast and Low-Cost Genomic Foundation Models via Outlier Removal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)