Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Authors: Haozheng Luo
Chenghao Qiu
Maojiang Su
Zhihan Zhou
Zoe Mehta
Guo Ye
Jerry Yao-Chieh Hu
Han Liu
Venue: ICML 2025 poster (2025)
Keywords: DNA
Outlier-free
Genomic Foundation Models
Outlier Removal
Primary area: applications->everything_else
TL;DR: GERM is a genomic foundation model optimized for low-resource settings by removing outliers, enhancing low-rank adaptation and quantization, achieving up to 64.34% efficiency gains and 37.98% better fine-tuning performance over baseline models.

Abstract: To address the challenge of scarce computational resources in genomic modeling, we introduce GERM, a genomic foundation model optimized for accessibility and adaptability. GERM improves upon models like DNABERT-2 by eliminating outliers that hinder low-rank adaptation and post-training quantization, enhancing both efficiency and robustness. We replace the vanilla attention layer with an outlier-free mechanism inspired by associative memory models. By removing outliers during both pre-training and fine-tuning, this approach accelerates adaptation, reduces computational costs, and enhances quantization robustness within acceptable loss margins. Additionally, we propose GERM-T, a strategy that employs small-step continual learning within the outlier-free framework, leveraging original checkpoints to avoid retraining from scratch. Empirically, GERM improves fine-tuning performance by 37.98% and quantization by 64.34% over the baseline model. It also reduces average kurtosis by 92.14% and maximum infinity norm by 82.77%. Compared to leading methods, GERM consistently delivers superior performance, offering a practical solution for genomic modeling in resource-constrained settings.

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Haozheng Luo"
Chenghao Qiu
Maojiang Su
Zhihan Zhou'd
Zoe Mehta
Guo Ye' Jerry Yao-Chieh Hu
Han Liu
Abstract
1 Introduction
We introduce novel model named GERM by utilizing outlier-free Hopfield layer (Hu et al., 2024a) to replace tra- ditional attention layer (Vaswani et al., 2017). GERM offers quantization- and rapidly adaptable DNA genomic foundation model (GFM), making ideal for deployment and fine-tuning sssource-constraingg devices.
To address the challenge scarce computational resources in genomic modeling, we introduce GERM, genomic foundation model with strong compression performance and fast adaptability. GERM improves upon models like DNABERT-2 by climinating outliers that hinder low www-rank adap- tation and post-training quantization, enhancing both efficiency and robustness. We replace the vanilla attention layer with outlier-free mecha- nism inspired by associative memory models. By removing outliers during both pre-training and fine-tuning, this approach accelerates adaptation, reduces computational costs, and enhances quan- tization robustness within acceptable loss mar- gins. Additionally, we propose GERM-T, a strat- egy that employs small-step continual learning within the outlier-free framework, leveraging orig- inal checkpoints to avoid retraining from scratch. Empirically, GERM improves fine-tuning perfor- mance by 37. 37.98% and quantization by 64.34% over the baseline model. It also reduces average kurtosis by 92.14%% and maximum infinity norm by 82.77%. Compared to leading methods, GERM consistently delivers superior performance, offer- ing practical solution for genomic modeling in resource-consttrinee settings. Code is available at https://ggaubb.coo//MlccccsAAAA LAB/GGRM..
Existing GFMs, such as DNABERT-2 (Zhou et al., 2024) and GenomeOcean (Zhou et al., 2025b), achieve state-of-the- art performance on various genomics tasks. However, many GFM users include not only professional computational re- searchers but also researchers from traditional biomedical labs, who often operate on resource-constrained platforms such mobile phones, edge devices, and IoT systems. The large size and high computational cost of these models make them challenging use in such devices. Also, if researchers require the model to adapt to new tasks, it should be able to fine-tuned on those tasks without demanding substantial computational resources. Efficient methods such as low- rank adaptation fine-tuning (e.g., LoRA (Hu et al., 2022)) and post-training quantization (e.g., SmoothQuant (Xiao et al., 2023)) help reduce training and inference costs for GFMs. However, directly applying these techniques to orig- inal models without modification leads to huge performance drops. This results from outlier values in the model's atten- tion mechanisms, inherited from pretrained models (Clark et al., 2019; Kovaleva et al., 2019). Prior studies (Hu et al., 2024a; Bondarenko al., 2024; Clark al., 2019) show that transformer-based models often direct attention toward less useful tokens, referred to as outliers. These outliers cause inefficiencies that reduce the overall model performance. Additional studies (Wu et al., 2024c; Huang et al., 2024; Hu et al., 2025) reveal that low-rank adaptation worsens the outlier issue. Outliers from both pretrained models and low-rank adaptation fine-tuning processes distort outputs and lower accuracy.
Equal contribution 'Northwestern University Tiaajin High
University 'Vernon
Hills
School.
Correspon-
dence to: Haozheng
Luo
<hluo wetttno@uunottwssttmm.mmm
Chenghao Qiu <q1320460765 0000:/000060t...ccdu.......
Mao-
jiang Su <maojiangsu22330@e htw.://0eu..otttwwseeee...d
Zhi-
To address inefficiencies caused by outliers in transformer- based genomic foundation models, GERM draws inspiration from associative memory models (Hu et al., 2024a;b; 2023; Xu et al., 2024; Wu et al., 2024a;b; Ramsauer et al., 2021). We replace the standard transformer attention mechanisms with an outlier-free attention layer proposed by Hu al. (2024a), which detects and removes outliers occurring dur-
han
Zhou
rz2hihanzho2222@@unoooo www.toowwtteeeeeeeeeeo seuu,
Zoe Mehta <zoe.mehta@ @ vhhscougars. orrg>,
Guo
Ye
<guoye2018@e w.w.ttwweee.nttteee..mmmmmm edui, Jerry Yao-Chich
Hu
<jhu@u. thuu@u.ootthwessem..che Han Liu <hanliu northwestem.cdd eduu..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
B-DEEBVNG (0)
AACGCACTATNTA
Q
Softmax
SentencePiece
ALiBi
Layer
FFN
{A,T, G, C,TA,AC
Ditribation w: outteer
AACGCACTATATA
GERD (q)
Outlier-feee Layer
SentencePiece
ALiBi
Layer
FFN
(A,T, G, C,TAAC
Distrilution wib outlier
Figure 1: Structural Comparison of DNABERT-2 and GERM Models, This diagram illustrates the differences in processing pipelines between DNABERT-2 and GERM. Both DNABERT-2 and GERM the SciencePiece tokenizer with BPE for tokenization. Following that, both models employ ALiBi positional encoding embedding layer. However, as shown in (a), DNABERT-2's transformer architecture outputs the outliers. We propose replacing the vanilla Softmaxwith an outlier free layer. In (b), the output of the attention mechanism removes outliers from the original output.
ing pretraining and low-rank adaptation (LoRA).
enhance the model's ability to handle and mitigate outliers during pretraining and fine-tuning, Additionally, compro-
This outlier mitigation in GERM results in "triple win" for genomic foundation models: faster low-rank adapta- tion, reduced computational demands, and more reliable post-training quantization. On resource--onssraiiee de- vices, incorporating fine-tuning techniques such as QLoRA (Dettmers et al., 2024a) and quantization methods like Om- niQuant (Shao et al., 2024) enables efficient fine-tuning and inference with minimal performance degradation. This sig- nificantly enhances its accessibility and usability, promoting broader deployment without specialized hardware.
we introduce continual learning strategy mise version to avoid retraining the model from scratch. This strategy ensures suboptimal performance in terms of model quantization robustness and low-rank adaptation.
Experimentally. We evaluate the performance and effi- ciency of our method using the existing DNABERT-2 model (Zhou et al., 2024) structure. Additionally, we benchmark it against the state-of-the-ar low-rank adapta- tion methods and post-training quantization techniques. Compared to the standard framework, the proposed frame- work achieves average performance improvements of 37.98% in finetuning and 64.34% % in quantization, re- spectively. Additionally, GERM shows reduction of 92.14% in the average kurtosis and 82.77% % in the maxi- mum infinity norm on average.
Additionally, addressing the limitations of (Hu et al., 2024a), particularly its resource-heat heavy training from scratch, we in- troduce GERM-T. GERM-T adds an outlier-free layer to existing GFM and uses small-step continual training to effi- ciently achieve near-optimal performance.
Contributions. We propose GERM, outlier-free GFM with enhanced quantization robustness and rapid low-rank adaptation. Our contributions are as follows:
Related Work
Quantization. Considering the quantized object, exiting foundation models (FMs) quantization can be classified into two fields: weight-only quantization and weight-activation quantization. For weight-only quantization, prior stud- ies focus on converting weights to low-bit values. For instance, GPTQ (Frantar al., 2023) uses block-wise re- construction for 3/4-bit quantization. SpQR (Dettmers et al., 2024b), OWQ (Lee et al., 2024), and AWQ (Lin et al., 2024) emphasize the significance of weights tied to higher-magnitee activations. Therefore, SpQR and OWQ employ mixed-precision quantization to safeguard vital weights, while AWQ opts for channel-wise scaling avoid
We propose outlier-free model structure address and mitigate outliers introduced by pretrained models and low-rank adaptation. This approach enables rapid low-rank adaptation and robust post-training quantization, significantly enhancing the overall performance of the quantized model and model finetuning. Notably, our model fine-tunes DNABERT in just 5 minutes on a single NVIDIA GeForce RTX 2080 Ti GPU.
Methodologically, we replace the standard transformer at- tention mechanism in the GFM with an outlier-free layer
2

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
mixed-precision's 's hardware inefficiency. QLoRA (Dettmers et al., 2024a), LoftQ (Li al., 2023) and QUIP (Chee et al., 2023) restore the capabilities of the quantized model through parameter-efficient fine-tuning. For weight- activation quantization, prior studies compress both weights and activations. SmoothQuant (Xiao et al., 2023), LLM.int8() (Dettmers et al., 2022), and Outlier Suppres- sion (Wei et al., 2022) achieve W8A8 quantization by man- aging activation outliers. LLM.in880 uses mixed-precisioon decomposition, while the other two employ channel-wise scaling. Furthermore, Outlier Suppression+ (Wei et al., 2023) adds channel-wise shifting to drive W6A6 quanti- zation. In comparison to other quantization approaches, including prior works (Wei et al., 2023; Xiao et al., 2023) that address the outier issue during quantization, the outlier- free layer in GERM is more effective managing outliers within the model's s attention mechanism. It provides GERM with unique advantage in terms of quantization robustness.
dependencies. Furthermore, GenomeOcean (Zhou al., 2025b) provides an efficient 4-billion-paramete genome foundation model for diverse, context-aware DNA sequence generation. However, these models demand significant computational resources and lack robustness to quantiza- tion, rendering them unsuitable for deployment on resource- constrained devices. Specifically, GenomeOcean utilizes 64 NVIDIA A100 80G GPUs over a span of 14 days for train- ing. This limits accessibility for research labs with limited computational capacity. More recently, Evo (Nguyen et al., 2024a), a generative genomic model, integrating Trans- former and Hyena operator to efficiently capture long-range dependencies in genomic sequences, achieving a context window of 131k nucleotides. Furthermore, Evo uniquely bridges bridges the DNA-RNA-protein central dogma via cross-modal inference without task-specific supervision.
2 GERM
Outlier Values in Quantization. Numerous studies (Hu et al., 2024a; Ma et al., 2024; Heo et al., 2024; Puccetti et al., 2022; Kovaleva et al., 2021; Bondarenko et al., 2021; Luo et al., 2021) observe outlier values in the transformer-basee language models such as BERT (Devlin et al., 2019) and early GPT (Radford et al., 2019) models. Since the advent of FMs (Zhou et al., 2024; 2025a; Zhang et al., 2022; Brown et al., 2020) root in the GPT and BERT, recent studies by Xiao et al. (2023); Ahmadian et al. (2023); Dettmers al. (2022) tackle the existence of outlier values in FMs. Ac- cording to them, these outliers exhibit a large magnitude of values at the shared dimensions of hidden states across tokens. More recently, Bondarenko et al. (2024); Sun et al. (2024); Hu et al. (2024a) explain that the outliers attribute to the vertical pattern in the attention mechanism (Xiao et al., 2024; Kovaleva et al., 2019), influencing the performance of FMs. In partieular, Sun et al. (2024) claim a different type of outlier existing in the hidden states of specific tokens. However, most of these studies concentrate on language and vision models, leaving the impact of outliers on genomic foundation models largely unexplored. Additionally, meth- ods like Hu et al. (2024a) require training from scratch to eliminate outliers, which is computationally expensive.
This section introduces the proposed method, which com- prises the outlier-free architecture, small-step continual learning, and the DNA genomic foundation model (GFM). The outlier-free architecture is designed mitigate chal- lenges posed by outliers during the model fine-tuning pro- cess. Meanwhile, the small-step continual learning tech- nique extends the training process using smaller learning steps after the initial training, aiming to address and mitigate the outliers present in the original model checkpoints.
In our study, we develop the GFM framework to train DNA sequence-based genomic foundation models that employ Transformer-based architectures such as DNABERT (Ji et al., 2021) and Nucleotide Transformer (Dalla-Torre et al., 2024). These models use DNA tokenization and Trans- former attention mechanisms, making them well-suited for integrating techniques like LoRA and the outlier-free mech- anisms proposed in our approach. Alternatively, models like HyenaDNA (Nguyen et al., 2024b) and Caduceus (Schiff et al., 2024) utilize different architectures, such convolu- tional layers or the Mamba architecture. While these models introduce novel features, they are not currently the most widely adopted in genomic modeling and require further research. Therefore, we adopt DNABERT-2 as the baseline in this paper, as it best represents the Transformer-baset DNA GFMs central to our study. The proposed outlier-free architecture of GERM is illustrated in Figure 1.
Genomic Foundation Model. The majority of genomic foundation models (GFMs) use transformers to model sequence dependencies, similar to BERT (Devlin et al., 2019) and GPT (Brown et al., 2020) in NLP. Specifically, DNABERT (Ji al., 2021) and DNABERT-2 (Zhou al., 2024) leverage transformers for DNA sequence analysis by employing masked language modeling and fine-tuning for biological tasks. In addition, Nucleotide Transformer (Dalla- Torre et al., 2024) excels at molecular phenotype predic- tion and variant prioritization, while HyenaDNA (Nguyen et al., 2024b) is optimized for modeling long-range genomic
Outliers Challenge in Transformer Architecture. Pre- vious studies (Clark al., 2019; Kovaleva et al., 2019) demonstrate that structural elements, including delimiters and sentence boundaries, attract unexpectedly high atten- tion weights in BERT's attention mechanism. Conse- quently, these tokens dominate the attention mechanism, overshadowing more informative tokens. Further anal-
3

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
ysis by Kobayashi et al. (2020) show that tokens with smaller value vector magnitudes paradoxically tend to ob- tain greater attention weights. These phenomena indicate that transformer-ben based models may focus on less relevant information, leading to inefficient processing. Studies by Hu et al. (2024a); Bondarenko et al. (2024) highlight the underlying cause of the outlier challenge in transformer- based models, proposing that transformers do not require updates when the attention inputs are sufficiently informa- tive. However, the normalization nature of the Softmax function forces non-zero attention weights even for irrele- vant tokens, creating numerical instability. Such outliers distort gradient updates and hinder model performance. Ad- ditionally, this issue increases computational and memory demands during training and results in significant perfor- mance degradation after model quantization. Consequently, implementing a strategy address outliers during both the pretraining and fine-tuning stages is crucial. Numerous studies address the outlier problem across different model stages, including pre-training (Hu et al., 2024a), fine-tuning (Hu et al., 2025), and inference (Bondarenko et al., 2024; Xiao et al., 2023). In our study, we extend the work of Hu al. (2024a) by utilizing the memory-associate retrieval dynamics function Softmax,. This function is defined as
DNA Genomic Foundation Model. We implement a sim- ple yet effective design for the DNA genomic foundation model (GFM) following DNABERT-2 (Zhou et al., 2024), Initially, we employ Sentence (Kudo & Richardson, 2018) with Byte Pair Encoding (BPE) (Sennrich et al., 2016), a subword tokenization method, to process DNA sequences. SentencePiece is particularly effective for handling the large number of unique tokens present in DNA without assuming any pre- pretookenization, such as k-mer segmentation (Chor et al., 2009). SentencePiece with BPE is used in natural language processing for word segmentation and learns a fixed-sized vocabulary of variablc-length tokens based on character co- co-occurrence frequencies. Due to the significant difference between natural language and DNA sequences, the vocabulary sizes used in the NLP domain (Zhang et al., 2022; Radford al., 2019; Kenton & Toutanova, 2019) are not suitable for DNA sequences. In our study, set vocab- ulary size to 4096, as it best balances model performance with computational efficiency among candidates.
We then adopt the BERT architecture (Kenton & Toutanova, 2019) to train our GFM on DNA sequences, with several modifications to better accommodate the unique characteris- tics of the DNA data. Standard positional encoding methods, such as Rotary Positional Encoding (Su et al., 2024) and Sinusoidal Positional Encoding (Vaswani et al., 2017), face limitations when applied to sequences longer than those en- countered during training due to their inherent input length restrictions. To address these limitations, we employ the Attention with Linear Biases (ALiBi) method (Press et al., 2022), it is more robust to variations in sequence length and can handle longer sequences compared to traditional po- sitional encoding methods. Instead of adding positional em- beddings to the input, ALiBi introduces linear biases into the attention mechanism, allowing the model to learn positional information inherently from the input sequence. Specifically, let q; E IR' represent the query vector for the i-th token in a sequence of length L, and K E RR Lxd denote the key matrix for all tokens. The attention score for query 9: is computed as: Soft (a:lmmxxyy: K + m x [-(i
exp(S)
Softmax; (5) == 1+ Σ exp(S.)
where S is the input to the activation function. This ap- proach addresses outlier problems in GFMs. Additionally, we provide a theoretical analysis of the expressive guarantee of low-rank adaptation for sereformer-bssee GFMs with Softmaxi in Appendix A.
Small-step Continual Learning. The outlier removal technique introduced in OutEffHop (Hu et al., 2024a) is highly effective in reducing the impact of outliers during model pretraining. However, a major limitation of this ap- proach is the need to retrain the model from scratch, which is a significant challenge for large-scale models like GFMs due to the extensive time and computational resources required. To address this issue, suggest small-step continual learning approach as compromise to the existing GERM structure, called GERM-T. T. It involves resuming training with an outlier-free model structure after the initial training phase to address and mitigate outliers in the original model checkpoints. This approach aims to lower the computational cost and time needed for retraining while still optimizing performance. Although this small-step continual learning technique may not be effective full retraining, it of- fers a more efficient and cost-effective solution for outlier removal in GFMs. For users with limited computational resources who cannot train a model from scratch and rely on 8-bit or 6-bit quantization during inference, GERM-T offers viable compromise strategy.
1),...,. -1, -1, -,,
(LL- - 1 - 1)))), where m is a fixed scalar. ALiBi uses a geometric sequence of different m values for each attention head, allowing model to learn positional information from the input sequence itself. By replacing learned positional embeddings with ALiBi, GERM can process arbitrarily long sequences during fine-tuning and inference, despite being pre-trained on relatively shorter sequences.
3 Experimental Studies
In this section, we perform a series of experiments to demon- strate the effectiveness of our proposed method. In par- ticular, we compare the performance of our method with DNABERT-2 detailed in (Zhou et al., 2024).
4

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 1: Comparing GERM and GERM-T with DNABERT-2 in a Post-Training Quantisation (PTQ) setting. We perform experiments GERM with baseline models using four quantization methods (Traditional W8A8, SmoothQuant, Outlier Suppression. OmniQuant) three quantization configurations Bright-4ii--Atiiattion
(W8A8), Weight-bhii-AAiiiatton---h (W6A6), and (W4A4)). The evaluation metrics include Matthews Correlation Coefficient (MCC), the difference MCC (Delta MCC) compared to the official DNABERT-2 checkpoint, the average kurtosis, and the maximum infinity norm |x for outler values at FP16. The best results are highlighted in bold, while the second-best results are underlined. In most configurations, GERM demonstrates superior fine-tuning performance compared to DNABERT-2.
Briigh--bit--Ativattoonn
Quantization Method
Delta MCC (
Avg Performance Drop (1)
Max inf.
Model
#Bits
MCC (1)
Avg. Kurtosis (1)
norm (4)
Official
16W/16A
66.
39.68
53.61
16W/16A
59.11
7.00
8W/8A
33.60=10.41
32.51
43.81%
8W/8A
36.51=00.02
45.37
38.63%
DNABERI-2
6W/6A
SmoothQuam
2..74=00004
45.37
66.18%
4W/4A
-1.03=10006
67.06
101.24%
270.90
61.64
8W/SA
25.26:10.02
40.85
57.60%
Outlier
6W/6A
27.84=0.228
38.27
52.71%
8W/8A
49.92
16.19
15.76%
6W/6A
OmniQuant
48.47:10.14
17.64
18.61%
4W/4A
2.94
63.17
94.78%
16W/16A
59.73
6.38
8W/8A
5..30
8.81
3.77% %
8W/8A
56.65+0.11
9.46
4.82%
6W/6A
SmoothQuant
56.48
9.63
5.45%
GERM
4W/4A
20.05
46.06
69.44%
21.29
10.62
8W/8A
45.87=0.08
20.24
25.23% %
Outlier
6W/6A
00.57
25.54
36.27%
8W/8A
55.99-10.09
10.12
5.95%
6W/6A
OmniQuan
55.70=10.03
10.41
6.41 %
4W/4A
49.42+0000
16.69
17.17% %
16W/16A
59.30
6.81
8W/8A
38.3
27.73
35.27%
8W/8A
57.52-10.00
8.59
3.01 %
GERM-T
6W/6A
SmoothQuan Quant
0.00
35.77
48.83%
4W/4A
0.22+10.00
65.89
99.63%
251.40
28.49
8W/8A
42.57:10.05
23.54
28.31%
Outlier
6W/6A
46.02+00.06
20.06
22.34 %
8W/8A
56.80=10.12
9.31
4.21
6W/6A
OmniQuant
55.41+00000
10.71
6.57%
4W/4A
3.86+10.00
62.25
93.49%
Following Zhou et al. (2024), we validate our strategy with DNABERT-2 model. we adopt the DNABERT- 2 model of size 117 million parameters' We pretrain this model with the masked language modeling (MLM) tech- nique, following the original DNABERT-2 (Zhou et al., 2024). Each model trained from scratch undergoes total of 200K training steps. For models utilizing small-step tinual learning, we initially train the model from scratch
Models.
using the DNABERT-2 architecture, followed by continual learning with the outlier-free structure for the remaining steps. In our experiment, we use the 40K continual learning steps model as the representative example of GERM-T to compare against DNABERT-2 and GERM.
Datasets. We utilize 27 datasets spanning 7 tasks and 4 species, as outlined in (Zhou et al., 2024). As shown in Appendix B.3, most downstream tasks in GFMs are clas- sification tasks. Consequently, the datasets are designed
hhtps//huuginnfaaccc////hht1199
5

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 2: Comparing GERM and GERM-T T with D in Low-Rank Adaptation Setting, We perform experiments on GERM with baseline models across three Low-Ran Adaptation methods (LoRA, QLoRA, LoftQ). The evaluation metrics include the Matthews Correlation Coefficient (MCC), the Delta MCC performance difference relative to the official DNABERT-2 checkpoint, the average kurtosis, and the maximum infinity norm xx for outlier values. Additionally, we measure the average performance drop after low-rank adaptation to evaluate the efficiency of GERM in this setting. The best results are highlighted in bold, while the second-hest results are underlined. In most configurations, GERM demonstrates superior tine-tuning performance compared to DNABERT-2. The GERM demonstrates average performance improvement of 37.98% % compared to the DNABERT-2 model.
Low-Rank Adaptation Method
Delta MCC
Avg Performance Drop (L)
Max inf. norm(()
Models
MCC (t)
Avg. kurtosis(
different (J)
Full
59.11
7.00
270.90
61.41
DNA BERT-2
LoRA
10.91
15.2
13.87%
219.20
QLoRA
50.65+0011
15,46
14.31%
292.85
53.91
LoftQ
50.76:10.06
15.31
14.05%
299.18
54.18
Full
59.73
6.38
21.29
10.62
GERM
LoRA
57.27+00.70
8.84
4.12%
19.41
QLoRA
53.16110.21
12.95
10.99% %
34.29
27.27
LoftQ
53.11+0.088
13.00
11.08%
33.02
27.41
Full
59.30
6.81
251.40
28.49
GERM-T
LoRA
55.60++0228
10.51
6.23%
140.86
QLoRA
51.05:0.007
15.06
13.90%
287.95
53.92
LoftQ
51.20:+0.13
14.91
13.65%
286.16
53.35
for genome sequence classification problems, with input lengths ranging from 70 to 1000.
conducted three times with different random seeds, and we report the average and standard deviation for each metric.
To evaluate the performance of our method against the official DNABERT-2 model, we also full fine- tune the official pretrained DNABERT-2 model as base- line downstream classification task to demonstrate the absolute performance of those three models. We fur- ther compare the performance of these three models on the same downstream classification task using FP16 precision and four PTQ methods: Traditional W8A8 (Weights- Activations-8bij as outlined in Bondarenko et al. (2024), SmoothQuant (Xiao et al., 2023), Outlier Suppression (Wei et al., 2022), and OmniQuant (Shao et al., 2024). With the exception of the W8A8 method, evaluate and com- pare the quantization performance of SmoothQuant, Out- lier Suppression, and OmniQuant at W8A8 (Weights-Sbit, Activations and W6A6 precision levels. Addition- ally, we present the quantization performance for W4A4 using OmniQuant and SmoothQuant. We use the same hy- perparameters specified in their respective studies. This approach guarantees that our evaluations are conducted un- der standardized conditions, enabling precise comparisons and assessments of each quantization method.
Baselines.
Evaluation Metrics. To evaluate the performance of out- liers in our strategy, report the maximum infinity norm ||x]]x of the activation tensors across all transformer layers as metric for detecting outliers. Additionally, we present the average kurtosis of x, calculated only from the output tensors from the Seed-Forward Network (FFN) layer and Layer Normalization. These two components are known to contain outliers, as confirmed by our experiments and prior studies (Hu et al., 2024a; Bondarenko et al., 2024; 2021). Both metrics have demonstrated strong correla- tion with model quantizability (i.e., robustness to outliers) (Bondarenko et al., 2021; Chmiel et al., 2020). For pre- quantization performance, we also report the FP16 (16- bit (oatin)-point) Matthews correlation coefficient (MCC) score assess model's downstream classification ability.
3.1 Post-Traine fost-Training Quantization (PTQ)
To assess the efficiency of our method for Post- Dost-Training Quantization (PTQ), we replace the standard attention layer in DNABERT-2 (Zhou et al., 2024) with the Softmax, ac- tivation function. We utilize the pre- pre-trained checkpoints of these three models and fine-tune them at full rank fol- lowing the procedure described in (Zhou et al., 2024). In this experiment, we evaluate the models on the test datasets using FP16 precision and apply PTQ to measure perfor- mance degradation due to quantization, Each evaluation is
Results. Referring to Table 1, it is clear that the GERM surpasses the DNABERT-2 in scenarios involving W4A4, W6A6 and W8A8 post-training quantization using of the-art PTQ methods. Specifically, when both weights and activations are quantized to bits (W8A8), GERM exhibits
6

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
minimal average performance decline of only 4.82% on SmoothQuant. Additionally, the proposed strategy remains effective as the quantization bit size decreases. For exam- ple, when models are quantized to W4A4, GERM exhibits a minimal average performance decline of only 17.17% on OmniQuant, compared to an 94.78% drop in the DNABERT- 2. This demonstrates that GERM outperforms the vanilla structure by enhancing the robustness of model quantiza- tion and improving performance across outlier-free methods like SmoothQuant and Outlier Suppression. Additionally, GERM-T exhibits strong quantization performance at both 8-bit and 6-bit levels, with a minimal performance drop, even smaller than that of GERM. The only exception is with W4A4 quantization, where GERM-T experiences a notable performance drop due larger outliers in GERM-T compared to the GERM. The outlier metries indicate the improvement in aiverage kurtosis is minimal, though sub- stantial reduction in the maximum infinity norm compared to the DNABERT-2. Outlier metrics show that GERM re- duces the average kurtosis by --92.14% and the maximum infinity norm by ~82777% across 27 datasets. Additionally, GERM-T achieves a reduction of approximately 7.20% in average kurtosis and 53.78% in the maximum infinity norm across the same datasets.
methods as described in (Dettmers al., 2024a).
Results. In Table 2, our results highlight the effective- ness of GERM in low-rank adaptation. In most configura- tions, GERM significantly enhances fine-tuning performance. Specifically, GERM achieves an average performance im- provement of 37.98% in low-rank adaptation compared to DNABERT-2 model. Similarly, GERM-T shows an aver- age performance improvement of 20.01 % over the same baseline. These results demonstrate that both GERM and GERM-T can greatly enhance low-rank adaptation for the model. When considering outlier metrics, we observe that LoRA exhibits significantly larger outlier values compared to full fine-tuning. Also, in most configurations, the out- lier values in LoRA are much higher than those in QLoRA and LoftQ. One potential reason is that QLoRA and LoftQ utilize quantization stabilize parameter updates and com- press model representations, which helps minimize the plification of extreme outlier values, Furthermore, GERM demonstrates substantial reduction in outlier values com- pared to DNABERT-2, and GERM-T also experiences a significant decrease in the maximum infinity norm, though the reduction in averuge kurtosis remains limited.
Table 3: Quantization Robustness Performance Comparison
with Different Continual Learning Steps. We evaluate the quan- tization robustness of different models, using SmoothQuant 16-bit, 8-bit, and 6-bit quantization levels. The evaluation metric the Matthews Correlation Coefficient (MCC) with average per- formance drop following quantization also noted. The best results are highlighted in bold, and the second- bentdbestttne results are underlined.
3.2 Low-Rank Adaptation
Fine-tuning models for downstream tasks is often compu- tationally expensive. To enhance the efficiency of fine- tniing with fewer parameters, various parameter-effiiien fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), are commonly used. To assess the efficiency of our method for fine-tuning tasks with LoRA, we evaluate our framework on LoRA methods. We use the pretrained checkpoints of the three models and fine-tune them using multiple LoRA approaches following similar procedure as described in Section 3.1. In this experiment, we evaluate the models on the test datasets using full-rank fine-tuning to compare the performance drop across various LoRA ap- proaches. Each evaluation is conducted three times with different random seeds, and we report the average and stan- dard deviation for each metric.
Avg Performance Drop (L)
Method
#Bits
MCC (t)
DNABERT-2
16W/16A
59.11
GERM
16W/16A
59.73
Out20k
16W/16A
59.21
GERM-T
16W/16A
59.30
Out100k
16W/16A
60.56
DNABERT-2
8W/SA
36.51
38.23%
GERM
8W/8A
56.78
4.93%
Out20k
8W/8A
54.75
7.53%
GERM-T
8W/8A
57.52
3.00%
Out100k
8W/8A
58.77
2.96%
LoRA Methods. We compare our method with the vanilla version across three different LoRA methods: LoRA (Hu et al., 2022), QLoRA (Dettmers et al., 2024a), and LoftQ (Li al., 2023). For the Full Fine-Tuning method, we fine-tune the model at full rank using mixed-precision FP16 training. For the LoRA method, following (Hu et al., 2022), we fine- tune the model with low-rank adaptations using a rank of 128 and an alpha value of 256. For the QLoRA and LoftQ methods, we fine-tune the model with quantized low-rank adaptations, maintaining the same rank and alpha values as in LoRA. Both A and LoftQ utilize 4-bit quantization
DNABERT-2
6W/6A
20.74
64.91% 1%
GERM
6W/6A
56.48
5.44% %
Out20k
6W/6A
27.61
53.36%
GERM-T
6W/6A
28.32
52.24%
Outl00k
6W/6A
30.44
49.74%
3.3 Additional Experiments
In this section, we conduct additional experiments to eval- uate the effectiveness of our method in various scenarios.
7

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
316
500
7001
5
Inference Time (S)
4:13 4.52..22
Train Time (S)
Time (S)
5
3.1
323
325
4
2.83
300
P.24 3.25
2:30
4
3
200
204
377
2
100
16W/16A
8W/8A Bits
4W/4A
Full
LoRA
QLoRA LoftQ
Full
LoRA QLORA LoftQ
Fine-Tuning Method
Fine-Tuning Method
DNABERT-2
GERM
GERM-T
Figure 2: Comparison of Performance in Resource-Constie Constrained Computing Environments. Comparison of three models on the quantization and fine-tuning task. All models were trained the same computing infrastructure (Nvidia GeForce RTX 2080 Ti 11GB) for comparison. The training time represents the average time per epoch, with OmniQuant used as quantization example this figure.
We conduct ablation studies investigate the impact of different continual learning steps, the influence of adaptor rank. Also, we conduct a case study on model deployment and fine-tuning using single 2080 Ti to evaluate model latency and fine-tuning speed per epoch.
larger performance drop observed in the model using 100K steps of continual learning, compared to GERM-T, It is at- tributed to the superior performance achieved through full fine-tuning. These results indicate that employing 40K con- tinual learning steps GERM-T optimal for enhancing model performance in our approach.
Impact of Different Continual Learning Steps. To eval- uate proposed method's performance across various continual learning steps, we conduct experiments on three different step sizes: 20K. 40K, and 100K. We train the models for a total of 200K steps, employing a combina- tion of vanilla and outlier-free pretraining as outlined in (Zhou et al., 2024) and (Hu al., 2024a). To assess perfor- mance degradation after quantization using SmoothQuant, perform full-rank fine-tuning of the models using the training datasets. Additionally, we assess the performance decline across models using three iterations of LoRA- LoRA-based fine-tuning technology. We use the "Outxxk" prefix to in- dicate the number of continual learning steps applied to the outlier-free structure. For instance, 100 represents model trained with 100K continual learning steps using the outlier-free structure. This notation helps illustrate how different levels of continual learning impact the model's performance.In In most configurations, GERM outperforms DNABERT-2 in both quantization robustness and low-rank adaptation. The only exception is in the 8-bit quantization scenario, where the Out and GERM-T models exhibit better performance and smaller average performance drop than GERM. Overall, GERM-T achieves better near-optimal performance across all continual learning model settings. The results, as shown in Tables 3 and 4, demonstrate that our method outperforms the vanilla approach across all test sets. Also, we observe that GERM-T exhibits the most op- timal performance drop during quantization and low-rank adaptation compared to other continual learning steps. The
Performance of GERM on Alternative Trans- former-based Models. To assess the performance of our proposed method on alternative transformer-based GFMs, conducted experiments on Nucleotide Trans- former (NT) (Dalla-Torre et al., 2024), significant genomic foundation model in this domain. The results, as shown in Tables 19 and 20, demonstrate that our method outperforms the vanilla approach across all test sets. GERM achieves an average performance improvement of 52.01 in low-rank adaptation and 67. 77.6%% PTQ experiments. We also conduct experiments on the NT-2.5B model demonstrate the scalability of GERM on larger-scale GFMs, as shown in Appendix D.7.
Performance of GERM with Alternative Outlier Removal Methods. To evaluate the performance of proposed method against existing outlier removal techniques, we com- pare GERM with approaches such clipped softmax and gated attention (Bondarenko et al., 2024). As shown in Ap- pendix D.8, GERM achieves a performance improvement of 2.59% in 4-bit and 1.99% in 8-bit quantization.
Performance of GERM with Differrnt Adaptor Rank. In our experiments, we evaluate the performance of our proposed method under the influence of adapter rank. All results are presented in Appendix D.I.
8

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 4: Low-Rank Adaptation Performance Comparison with Different Continual Learning Steps. We evaluate the low- rank adaptation performance (LoRA, QLoRA, LoftQ) for various models with different continual learning steps. The evaluation metric is the Matthews Correlation Coefficient (MCC), and the average performance drop after adaptation is also shown. The best results highlighted in bold, and the second-best results underlined.
users during both the inference and fine-tuning processes.
Case Study 2: Performance in CPU-Only Computing En- vironments. To demonstrate GERM'S capability in CPU- only computing environments, we perform performance tests on CPU-only devices. We compare GERM'S per-epoch training and inference times for the LoRA and QLoRA fine- tuning methods. The results, presented in Appendix D.4, indicate that both GERM and GERM-T achieve shorter fine- tuning times per epoch compared DNABERT-2, with the only exception being QLoRA when deployed, where the time is slightly longer.
Fine- Muninn Method
Avg Performance
Method
MCC (f)
Drop ())
DNABERT-2
Full
59.11
GERM
Full
59.73
Out20k
Full
59.21
GERM-T
Full
59.30
4 Discussion and Conclusion
10t
Full
60.56
DNABERT-2
LoRA
50.91
13.87%
We introduce GERM, a versatile and accessible GFM de- signed function on limited computational resources. By replacing the vanilla attention layer with an outlier-free layer, eliminate outliers during both model pretraining and fine-tuning. This approach ensures robust quantization and enables effective low-rank adaptation. In addition to presenting a novel architecture that enhances DNABERT by mitigating outliers, GERM incorporates a compromise strategy with continual learning, eliminating the need for ex- tensive retraining, Empirically, GERM achieves an average reduction of average kurtosis by~ ~9..114% and the maximum infinity norm by ~82.77% across 27 datasets. Additionally, GERM enhances model quantization robustness by decreas- ing the average quantization performance drop by 64.34% and the average low-rank adaptation performance drop by 77.8%. For the compromise model GERM-T TT quantization robustness is improved by reducing the average quantiza- tion performance drop by 31.42% and the average low-rank adaptation performance drop by 20.01%.
GERM
LoRA
56.78
4.94%
Out20k
LoRA
54.75
7.53%
GERM-T
LoRA
55.60
6.24%
Out100k
LoRA
56.61
6.52%
DNABERT-2
QLoRA
50.65
14.31 %
GERM
QLoRA
53.16
11.00 %
Out20k
QLoAA
50.61
14.52%
GERM-T
QLoAA
51.05
13.91% %
Out100k
QLoRA
51.24
15.39%
DNABERT-2
LoftQ
50.76
14.13%
GERM
LoftQ
53.11
11.08 %
Out20k
LoftQ
50.94
13.97%
GERM-T
LofiQ
51.20
13.66%
Out100k
LoftQ
50.77
16.17%
3.4 Case Study: Performance in
Computing Environments.
Case Study 1: Performance in Single 2080-Ti GPU Com- puting Environments. To demonstrate GERM'S capabil- ity in formance tests on a single NVIDIA GeForce RTX 2080 11GB GPU, where GERM requires only minutes fine- tune, demonstrating its practicality and efficiency. We com- pare GERM'S performance with the DNABERT-2 model on the same downstream classification task using OmniQuant. Consistent hyperparameters from their respective studies are applied across all models. Additionally, we provide the per-epoch training time and inference time for the LoRA, QLoRA, and LoftQ fine-tuning methods. The results, as shown in Figure show that both GERM and GERM-T achieve shorter full-rank fine-tuning times per epoch com- pared to DNABERT-2. Additionally, the model quantization latency for both GERM and GERM-T is lower than that of DNABERT-2, while delivering superior quantization perfor- mance. These observations indicate that GERM offers faster adaptation and requires fewer computational resources for
Limitations and Future Work. While successful in many settings, our proposed GERM-T still faces challenges in eliminating outliers in GFM, leading to significant perfor- mance drops during 4-bit quantization and low-rank adap- tation methods such as QLoRA and LoftQ. In future work, we aim to develop strategies that efficiently remove outliers without necessitating retraining from scratch. We provide more detailed discussion of the limitations of GERM-T in Appendix E.
environments, we conduct per-
Impact Statement
We believe this methodology presents an opportunity to strengthen the core of foundation models, including large language models, by improving robustness through quanti- zation and enabling faster low-rank adaptation. However, this approach may also amplify biases in the training data, potentially leading to unfair or discriminatory outcomes for underrepresentedd groups

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Acknowledgments
Clark, K., Khandelwal, U., Levy, O., and Manning, C. D. What does bert look at? analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. pP. 276. Association for Computational Linguistics, 2019.
The authors would like to thank Yegna Jambunath for en- lightening discussions on related topics, and Jiayi Wang for facilitating experimental deployments. The authors would like thank the anonymous reviewers and program chairs for constructive comments.
Molla-Torre, H., Gonzalez, L., Mendoza-Reville J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop. E., de Almeida, B. P., Sirelkhatim, H., et al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, pp. 1-11, 2024.
Han Liu is partially supported by NIH 10ILM137220) NSF AST-2421845, Simons Foundation MPS-AI 100000115,,, Abb Vie and Dolby. Haozheng Luo is partially supported by the OpenAl Researcher Access Program. This research was supported in part through the computational resources and staff contributions provided for the Quest high perfor- mance computing facility at Northwestern University which is jointly supported by the Office of the Provost, the Or- fice for Research, and Northwestern University Information Technology. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. GPT3. 8-bit matrix multiplication for transformers at scale. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized 1lms. Advances in Neural Information Processing Systems, 36, 2024a.
References
Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantizer representa- tion for near-lossless IIm weight compression. In ICLR, 2024b.
Ahmadian, A., Dash, S., Chen, H., Venkitesh, B., Gou, S., Blunsom, P., Üstün, A., and Hooker, S. Intriguing properties of quantization at scale. Advances in Neural Information Processing Systems, 01334288-42994 2023.
Bondarenko, Y., Nagel, M., and Blankevoort, T. Under- standing and overcoming the challenges of efficient trans- former quantization. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Pro- cessing, pp. 7947-7969, 2021.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio. T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Min- neapolis, Minnesota, June 2019. Association for Compu- tational Linguistics.
Bondarenko, Y., Nagel, M., and Blankevoort, T. Quantiz- able transformers: Removing outliers by helping atten- tion heads nothing. Advances Neurul Information Processing Systems, 36, 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neclakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Eckart, C. and Young. G. The approximation of one matrix by another of lower rank. Psychometrika. 1(3):211-218 1936.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023.
Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. QuIP: 2-bit quantization of large language models with guarantees. In shirth- Conference on Neural Information Pro- cessing Systems, 2023.
Heo, J. H., Kim, J., Kwon, B., Kim, B., Kwon, S. J., and Lee, D. Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models. In The Twelfth International Conference on Learning Representations, 2024.
Chmiel, B., Banner, R., Shomron, G., Nahshan, Y., Bron- stein, A., Weiser, U., et al. Robust quantization: One model to rule them all. Advances in neural information processing systems, 33:5308-5317, 5317, 2020.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Chor, B., Horn, D., Goldman, N., Levy, Y., and Massingham, T. Genomic dna k-mer spectra: models and modalities. Genome biology, 10:1-10, 2009.
10

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Hu, J. Y.-C., Yang, D., Wu, D., Xu, C., Chen, B.-Y., and Liu, H. On sparse modern hopfield model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
inference of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13355-13364, 2024.
Li, Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W., and Zhao, T. Loftq: Lora-fine ferrafiie-tttnngg aware quantiza- tion for large language models, 2023.
Hu, J. Y.-C., Chang, P.-H., Luo, H., Chen, H.-Y., Li, W., Wang, W.-P., and Liu, H. Outlier-effccien hopfield lay- ers for large transformer-basee models. In Forty-first International Conference on Machine Learning, 2024a.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-awes aware weight quantization for Ilm compres- sion and acceleration. In MISys, 2024.
Hu, J. Y.-C., Lin, T., Song, Z., and Liu, H. On computa- tional limits of modern hopfield models: A fine-grained complexity analysis. 2024b.
Liu, Z., Li, J., Xin, L.., Li, S., Yu, C., Zang, 7., Tan, C., Huang, Y., yajingbai, Xia, J., and Li, S. Z. Genebench: Systematic evaluation of genomic foundation models and beyond, 2025.
Hu, Y-C.,, Su, M., jui kuo, E., Song, Z., and Liu, H. Computational limits of low-rank adaptation (IoRA) fine- tuning for transformer models. In The Thirteenth Interna- tional Conference on Learning Representations, 2025.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019.
Huang, X., Liu, Z., Liu, S.-Y., and Cheng, K.-T. Rolora: Fine-tuning rotated outlier-free ilms for effective weight- activation quantization. In Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 7563- 7576, 2024.
Luo, Z., Kulmizev, A., and Mao, X. Positional artefacts propagate through masked language model embeddings. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- une 1: Long Papers), pp. 5312-5327, Online, August 2021. Association for Computational Linguistics.
Ji, Y., Zhou, Z., Liu, H., and Davuluri, R. V. DNABERT: pre-trai Bidirectional Encoder Representations from Transformers model for DNA-language in genome. 2021.
Kenton, J. D. M.-W. C. and Toutanova, L. K. Bert: Pre- training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume I, pp. 2. Minneapolis, Minnesota, 2019.
Ma. Y., Li, H., Zheng, X., Ling, F,, Xiao, X., Wang, R., Wen, S., Chao, F., and Ji, R. Outlier-awars slicing for post- training quantization in vision transformer. In Forty-first International Conference on Machine Learning, 2024.
Kohayashi, G., Kuribayashi, T, Yokoi, S., and Inui, K. Attention is not only a weight: Analyzing transformers with vector norms. pp. 7057-7075, 2020.
Mirsky, L. Symmetric gauge functions and unitarily invari- ant norms. The quarterly journal of mathematics, 11(1): 50-59, 1960.
Kovaleva, O., Romanov, A., Rogers, A., and Rumshisky, A. Revealing the dark secrets of bert. 2019.
Kovaleva, O., Kulshreshtha, S., Rogers, A., and Rumshisky, A. BERT busters: Outlier dimensions that disrupt trans- formers. In Zong, C., Xia, F,, Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3392-3405, Online, August 2021. Association for Computational Linguistics.
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J.. Thomas, A. W., King, H., Brixi, G., al. Sequence modeling and design from molecular to genome scale with evo. Science, 386(6723): eado9336, 2024a.
Nguyen, E., Poli, M., Faizi, M., Thomas, A., Wornow, M., Birch-Sykes, C., Massaroli, S., Patel, A., Rabideau, C., Bengio, Y., et al. Hyenadna: Long-range genomic se- quence modeling single nucleotide resolution. Ad- vances in neural information processing systems, 36, 2024b.
Kudo, T. and Richardson, J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Blanco, E. and Lu, W. (eds.), Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, Novem- ber 2018. Association for Computational Linguisties.
Press, O., Smith, N., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapo- lation. In International Conference on Learning Repre- sentations, 2022.
Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. Owq: Outlier- aware weight quantization for efficient fine-tuning and

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Puccetti, G., Rogers, A., Drozd, A., and Dell' Orletta, F. Outlier dimensions that disrupt transformers are driven by frequency. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2884-134,, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties.
Wu, D., Hu, J. Y.-C., Hsiao, T.-Y., and Liu, H. Uniform memory retrieval with larger capacity for modern hop- field models. In Forty-first International Conference on Machine Learning, 2024a.
Wu, D., Hu, J. Y.-.,, Li, W., Chen, B.-Y., and Liu, H. STanhop: Sparse tandem hopfield model for memory- enhanced time series prediction. In The Twelfth Interna- tional Conference on Learning Representations, 2024b.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. 2019.
Wu, S., Lu, Y.J.,, Luo, H., Hu, J. Y-C.,, Wang, J., Dehak, N., Villalba, J., and Liu, H. Fast adaptation and robust quanti- zation of multi- foundation models from associative memory: A case study in speechlm. In Workshop on Ef- ficient Systems for Foundation Models II@ ICML2024, 2024c.
Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Adler, T., Kreil, D., Kopp, M. K., Klambauer, G., Brandstetter, J., and Hochreiter, S. Hopfield networks is all you need. In International Conference on Learning Representations, 2021.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Schiff, Y., Kao, C.-H., Gokaslan, A., Dao, T., Gu, A., and Kuleshov, V. Caduceus: Bi-directional equivariant long- range dna sequence modeling. In ICML, 2024.
Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Proceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 1715-1725, 2016.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. 2024.
Xu, C., Huang, Y.C..,
Li, W., Gilani, A., Goan, H.-S., and Liu, H. Bishop: Bi-directional cellular learning for tabular data with generalized sparse modern hopfield model. In Forty-firt International Conference on Ma- chine Learning, 2024.
Shao, W., Chen, M., Zhang, 7., Xu, P., Zhao, L., Li, Z Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Ommidirectionally calibrated quantization for large lan- guage models. In ICLR, 2024.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Zeng, Y. and Lee, K. The expressive power of low-rank adaptation. 2024.
Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activations in large language models. 2024.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M.. Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.. Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R. V., and Liu, H. DNABERT-2: Efficient foundation model and benchmark for multi-species genomes. In The Twelfth International Conference on Learning Representations, 2024.
Wei, X., Zhang, Y., Zhang, X., Gong, R., Zhang, S., Zhang, Q., Yu, F., and Liu, X. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402- 17414, 2022.
Zhou, Z., Wu, W., Ho, H., Wang, J., Shi, L., Davuluri, R. V., Wang, Z., and Liu, H. DNABERT-s: Pioneering species differentiation with awecies-aware DNA embed- dings. 2025a.
Wei, X.. Zhang, Y., Li, Y., Zhang, X., Gong, R., Guo, J., and Liu, X. Outlier suppression+: Accurate quantiza- tion of large language models by equivalent and effective shifting and scaling. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, pp- 1648-1665, Singapore, December 2023, Association for Computational Linguisties.
Zhou, Z., Wu, W., Wu, J., Shi, L., Wang, Z., and Liu, H. Genomeocean: Efficient foundation model for genome generation, 2025b.
12

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Supplementary Material
Theoretical Analysis
13
Experimental Setup
16
B.1
Computational Resource
16
B.2 Hyperparameters
16
B.3 Downstream Tasks Across Different Models
16
C Formula of Average Kurtosis
17
D Additional Numerical Experiments
17
D.I Influence of Adaptor Rank
17
D.2 All Results in Low-rank Adaptation
18
D.3 All Results in Post- pest-Training Quantization
23
D.4 All Results of Performance Comparison in Cesource-Constraan Computing Environments
27
D.5 Evaluation of Common Genomic Foundation Models
28
D.6 Performance of GERM on Alternative
soaeffommer-bbsee Models
29
D.7 Performance of GERM on Large-scale GFMs
31
D.8 Performance of GERM with Different Outlier Removal Techniques
33
D.9 PTQ Performance of GERM Following LoRA Fine-Tuning
33
D.10 Attention Distribution Visualization in GERM
34
Additional Discussion and Limitation
34
F Definition of Outlier in Our Paper
34
A Theoretical Analysis
In this section, we provide the expressive guarantee of Low-Rank Adaption for transformer model with Softmax,. Moreover, we identify the conditions for the existence of low-rank adapters. We summarize our main findings in the following informal theorem.
Theorem A.1 (Informal). Let fo be the frozen model and f be the target model. Under certain non non-singularity assumption and L-RA-rank conditions, their exist low-rank adapters such that the adopted model f exactly equal f.
The formal version is stated in Theorem A.2. Then we provide detailed illustration. We start with the definition of the target model and the adopted model f-
Definition A.1 (Definition of target model F and adopted model f). For any input X E R where D denotes the dimension of token embedding and N denotes the number tokens. We consider a H-heads transformers TFp. consist of
Krs L- L-Transformer blocks and an output layer with parameter 0 ((W",, W Wh. Wh W Wn, W2,, duG b2) W. Specifically, we formulate it as
Hidden layer: Attn Zi-1) -)= Σ W or
W Z Softmax; (Z,_ W ki W (Z,,, Zi-1),
h=1
Zi W2 ReLU(W1; Aunn(Z-1) +,, b,,1)) b1 + bz 1
OutputLayer: TF (X) Softmax, (W.2L)
where we define % == X. Here, Wh. W W Q E DDD are weight matrices in I-th attention layer. Further, Wu, Wa I Dx are weight matrices and bıt but are the bias vectors in the /-th feedforward layer. Then we define the
13

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
target model f and the adopted model f are
= = TF Or = (wo'r W vi, W W WoU W 11, 22,, bic, b2) W
f: TF.. A = ((W)) + AW w + AW W - AW W + AW@U)
Wii + AWI, W2r + AW2, bir, bx) ) W. + AW.)
Moreover, we define the best low-rank approximation for matrix W.
Definition A.2 (Best Low-rank Approximation for W). For any matrix W E DDD the singular value decomposition (SVD) of W is expressed as W = UDVT, Above U, V E R DxD are orthonormal matrices and D IR DxDx is diagonal matrix. Let the singular value of W are denoted as 01 (W) .... .p.2) 2 0. When d > D,let aa(W) ) = O. For any
rank r > 0, we define
LRA, (W) = Σ o:WWWuu,,
where u, of are the i-th column of V, respectively.
According to Eckart & Young (1936) and Mirsky (1960), LRA,(W) are the best rank-r approximation the Frobenius norm the 2-norm of W. To present our results, we now introduce non-singularity assumption based on Definition A.2.
Assumption A.1 (Non-Singularity). For a fixed [D]. the weight matrices of both the target model, the frozen model and the following matrices are non-singular, for all E [R]. Specifically,
W W LRA, AT W W W&.), for all [H] and W 2,1 W Wh W 2,-1 Wki Wa)), for all h E [H] e [LM)11),
W W LRA.
W LR,,
w wh)), for all h e [M] and ( 1, 1,
W W LRA, W w.W W W. W2 w&)) for all h e [H] and I e [L]M[},
WoW2L + LRA, W W 2L W.W2t),
are non-singular, where LRA denotes the rank-r approximation follows Definition A.2.
Under non-singularity assumption (Assumption A.I), apply another helper lemma from Zeng & Lee (2024) to construct the weight matrices in Theorem A.2.
Lemma A.1 (Exactly represent target model, Lemma 7 of Zeng & Lee (2024)). Define error matrix E := - = Wi and denote its rank by RØ = rank E). For given LoRA-rank R E [D]. assume that all the weight matrices of the frozen model (W and II Wi + (RA,(E) are non-singular for all < R(L - 1). Then, the approximation error
I
L
min AWrrank|AWsss
II (Wi AW)
W
RRL+1
W
1=1
Error matrix
and the optimal solution to the matrix approximation problem satisfies I (W1 + AW) = -I',, W LRARLAR Therefore, when R > Rg we have 11 (Wi + AWi) W. implying J.
With Assumption A. and Lemma A. we show that for any input X E DDD, their exist adapted model f capable of approximating target model f exactly, i.e, f(X) = f(x).
Theorem A.2 (Express capability of transformers). Suppose LoRA-rank R E [D]. Let Assumption A.1 hold. Define the
14

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
rank-based functionality gap G, to i-th Outlier-ffficient Hopfield block (i [L]) output layer (i L + 1) as
max; (rank( W Ki W
W W V (ra w. oo Wiw.&wW..)) i= 1.
qi
maax W 2, W
Ki W Q W 2,i-1
- W 22111)
G.
V Vmmx,, (rank ( W Oz W vi W 2,
2 s L. == L L++
rank(W. .. 2r. W.W,r.),
R 2
11 1 1H AW AW oL
then there exists low-rank adapters with rank lower than AW21, AW, with other low-rank adapters set to O, and updated bias
((WW)) Rt: AW Q vectors (0,1 ba)t=n such that for any input X E RDx,, the adapted model f exactly approximates target model f, i.e.
f(X)= = x(X).
Proof. We build our proof on Zeng & Lee (2024).
First, ensure that, for each Outlicr-fficch Hopfield block, the output from the first feedforward layer in the target model matches that in the adapted model. Then, we select an appropriate output layer weight matrix to complete the proof.
We define H, E DxN and Zi E DxN as the intermediate and final outputs of the I-th transformer block in the target model J. respectively. For any 1 [T.), they are formulated as
A
H = ReLU(W M
Σ
Wo Zi- Softmax, Softmax,(Z1_11 Wki W KI Q Z(1)) + b,,
A=1
Z = W W2 + b
Correspondingly, we introduce H, and Zi to denote the intermediate output of the first feedforward layer and the final output of the 1-th Outlier-Efficient Hopfield block for the adapted model f.
i1 ReLUUWIi( (wå + AWO) + 2i-1
(A.1)
A=1
Softmax| (2T ( + AWk) T(W + AW) + bx
2. 12 + 12
(A.2)
for any 1 E [4]. Note that Zo 2o = X. Next, we inductively construct the adapter weight matrices ((Aw), on AWA, AW AW bu: b2) such that H. = H, for all E [L]. We then select the low-rank adapters for Wa and the W, to approximate the output of the target model. For unmentioned low-rank adapters, we them O.
When / 1.
To achieve Å. H, for all X, the following conditions must be satisfied:
Bias Vector: by = =b11-
Query and Key: (WA + AWA T (W AW Whi AT KI
W Qt.
Value and First Feedforward Layer: (Wor + AWW, + AW
WT: W. W. of W vI-
It is simple to check that we only need to set byz = b to, and select rank-At or lower matrices AWA, AWA AW AWh, KI Qu
as suggested by Lemma A.I. This ensures H. H, for =
When / 1. For the cases 2, ........ L, we assume the induction hypothesis holds for 1 which is Å H
We let 62,111 = W2,-1W 2,1- b2,1-1, then it holds.
2 W2 W
21-11
(A.3)
Substituting (A.3) into (A.1) and (A.2), the necessary conditions become:
15

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Bias Vector: by but.
Query and Key: (Wk) AWA KI) ( AW 27 2,1-1 Ki
W 2,1- 2,11 2,1-1
Value and Output Projection: (Wor + A W + AWD) WTw W.. OF Vi W 2.-1
By setting bi bu and adjusting AW Rt AW Qr: AW AW for all h E [H] based on Lemma A.I, we satisfy all three conditions above, thereby obtaining f1 H1 for I E [4]][]]
Output Layer Analysis. By applying the induction method, we have established A, = H, for all E [r]. We only need to select appropriate weight matrices to ensure that f(X) fX) for all X E X. The final output of the target model f with input X can be written as
f(X) = Softmax; (W zz) = Sooftmax (W (W. 22L2HHH + bor
Similarly, the final output of the adapted model f with input X can be written as
f(X) = Softmax ((W, + AW,)
= Softmax, (WW, + AW.)(WWa Hr + b2)
To achieve F(X) f(X), we select 2W21. and AW, based on Lemma A.1, and let b2g. (W, + AW,, -W .br. where W. + AW, is invertible as shown the proof of Lemma A.1. Combining above, we complete the proof.
B Experimental Setup
B.1 Computational Resource
We perform all experiments using 2 NVIDIA A100 GPU with 80GB of memory and 24-core Intel(R) Xeon(R) Gold 6338 CPU operating 2.00GHz. Our code is developed in PyTorch and utilizes the Hugging Face Transformer Library for experimental execution.
B.2 Hyperparameters
We present the hyperparameters used in the fine-tuning stage for each model. We use AdamW (Loshchilov & Hutter, 2019) as the optimizer. Most of the other hyperparameters remain the same across all models and datasets, including batch size of 32, warmup step of 50, and weight decay of 0.01. A learning rate of 3e-5 is used for all models during fine-tuning, For low-rank adaptation, we use learning rate of 1 with a LoRA rank of 8 and LoRA alpha set to 16. For each task, we use different training steps as shown Table 5. During pre-training, the model is trained for 200,000 steps with batch size of 1024 and maximum sequence length of 512, using the AdamW optimizer with 31 0.9, 82 0.98, and 1e-6 The pre-training stage takes approximately 4 days using 2 NVIDIA A100 80G GPUs.
Table The number of training steps. We present the number of training steps we use in our experiments. In the task of Transcription Factor Prediction on the Mouse genome, we train the model for 1000 steps on each dataset.
EMP
TF-M
CVC
TF-H
PD-tata
PD-o
CPD-tata
CPD-o
SSP
Epochs
lk
8
3
10
4
10
B.3 Downstream Tasks Across Different Models
We analyze the downstream tasks of various genomic foundation models (GFMs), specifically comparing DNABERT-2 (Zhou et al., 2024). HyenaDNA (Nguyen et al., 2024b), and Nucleotide Transformer IDalla-Tore et al., 2024). As shown in Table 6, all of these GFMs utilize classification tasks as their primary downstream applications. Additionally, we analyze related GenBench datasets (Liu et al., 2025) and find that, uniquely, GenBench includes some regression downstream tasks, providing broader evaluation spectrum.
16

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 6: Comparison of Models (Benchmarks) and Their Tasks.
Model
Tasks
Classffication-On]
DNABERT-2
GUE (28 Classification tasks)
Yes
Nucleotide Transformer
Nucleotide Transformer Benchmark (18 Classification tasks)
Yes
HyenaDNA
GenBench (Classiiiction-Onlty) Nucleotide Transformer Benchmark
Yes
GenBench
Classification Regression (e.g., Drosophila Enhancer Activity Prediction)
No
C Formula of Average Kurtosis
As shown in Bondarenko et al. (2024); Hu al. (2024a), average kurtosis is significant metric for measuring outliers. Kurtosis is statistical measure that quantifies the "tailedness" of distribution relative to normal distribution. The formula for kurtosis is:
K E (c: - x²1 (I: - 2)
where represents the data points, T is the mean, and is the number of data points. As a result, when a distribution has higher kurtosis, it indicates that the data distribution has heavier tails. In other words, there are more outlier values present in the distribution. The GERM reduces kurtosis by evenly distributing attention across informative tokens, rather than focusing excessively on specific outlier tokens (e.g., delimiters or cos markers).
D Additional Numerical Experiments
D.1 Influence of Adaptor Rank
We perform an in-depth analysis to assess the performance of our proposed method across different ranks when implementing Low-rank Adaptation (LoRA), comparing to the standard vanilla approach. We compare the model performance with different rank value of LoRA and keep the alpha value double of the rank value. The results, as shown in Table 7, demonstrate that our method outperforms the vanilla approach across all tested ranks. We observe that a rank of provides optimal performance compared to the vanilla method. The lack of further performance improvements with higher ranks because a higher rank in LoRA usually introduces more trainable parameters into the model. This leads to significant performance improvement over the vanilla structure, thereby narrowing the performance gap with GERM.
17

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 7: Comparison of Different Ranks Using LoRA. We conduct experiments to evaluate how different ranks affect the performance of LoRA. The evaluation metric used is Matthews Correlation Coefficient (MCC), We measure the average performance decline following low-rank adaptation, with best results highlighted in bold.
Fine- Muni-Tnnng
Method
Rank
MCC
Method
DNABERT-2
Full
N/A
59.11
GERM
Full
N/A
59.73
GERM-T
Full
N/A
59.30
DNABERT-2
LoRA
16
55.71
GERM
LoRA
16
58.91
GERM-T
LoRA
16
57.41
DNABERT-2
LoRA
52.87
GERM
LoRA
57.27
GERM-T
LoRA
55.60
DNABERT-2
LoRA
51.02
GERM
LoRA
55.64
GERM-T
LoRA
52.07
D.2 All Results in Low-rank Adaptation
In this section, we present a comprehensive evaluation of various models under different low-rank adaptation (LoRA) strategies. The experiments compare DNABERT-2, GERM-T, and GERM models across multiple biological prediction tasks, including epigenetic marks prediction, promoter detection, and transcription factor prediction in both human and mouse datasets. The adaptation methods assessed include Full Fine-tuning, LoRA, QLoRA, and LoftQ.
18

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 8: Performance Comparison of Full Fine-tuning with DNABERT-2. This table shows the performance of all models on the full fine-tuning task.
Epigenetic Marks Prediction
Model
H3
H3K 14ac
H3K36me3
H3K4mel
H3K4me2
H3K4me3
DNABERT-2
75.03
33.07
48.63
33.67
31.63
24.31
GERM-T
75.02
47.31
51.32
34.53
27.72
23.19
GERM
75.58
50.36
53.13
33,36
36.02
23.97
Epigenetic Marks Prediction
Promoter Detection
Model
H3K79me3
H3K9ac
H4
H4ac
all
notata
DNABERT-2
60.88
49.87
77.67
26.87
82.45
90.29
58.20
GERM-T
59.61
52.63
80.20
45.37
80.82
90.24
59.27
GERM
59.80
55.08
81.25
48.36
79.40
90.51
59.75
Transcription Factor Prediction (Human)
Core Promoter Delection
Model
2
notata
tata
DNABERT-2
66.51
70.10
57.43
39.67
71.35
66.61
65.34
71.98
GERM-T
68.17
68.45
62.85
50.62
72.25
48.96
65.09
75.90
GERM
67.29
70.88
58.21
51.75
72.21
51.91
63.07
70.40
Transcription Factor Prediction (Mouse)
Virus
Splice
Model
0
4
Covid
Reconstruct
DNABERT-2
45.61
80.20
78.05
72.70
41.94
66.17
68.85
GERM-T
40.90
57.18
71.25
70.25
36.88
66.69
77.60
GERM
41.98
59.19
65.53
72.58
38.68
66.02
76.59
19

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 9: Performance Comparison of LoRA with DNABERT-2. This table shows the performance of all models the low-rank adaptation (LoRA) task.
Epigenetic Marks Prediction
Model
H3
H3K 14ac
H3K36me3
H3K4mel
H3K4me2
H3K4me3
DNABERT-2
65.93
35.97
44.19
38.93
28.30
21.86
GERM-T
69.98
42.59
48.34
41.15
29.11
24.45
GERM
70.61
37.12
48.57
35,46
30.20
25.62
Epigenetic Marks Prediction
Promoter Detection
Model
H3K79me3
H3K9ac
H4
H4ac
all
notata
DNABERT-2
55.69
45.65
74.15
33.52
81.84
89.72
53.64
GERM-T
58.25
49.96
78.08
36.18
84.19
91.71
60.18
GERM
57.99
47.54
78.37
35.55
82.97
90.35
58.51
Transcription Factor Prediction (Human)
Core Promoter Delection
Model
2
all
notata
tata
DNABERT-2
61.68
69.70
38.80
42.17
58.94
66.61
65.34
65.34
GERM-T
65.21
69.47
57.42
46.86
66.18
48.96
65.09
75.90
GERM
66.77
71.03
55.95
48.65
69.63
51.91
63.07
70.40
Transcription Factor Prediction (Mouse)
Virus
Splice
Model
0
4
Covid
Reconstruct
DNABERT-2
45.61
80.20
78.05
72.70
41.94
13.15
56.16
GERM-T
40.90
57.18
71.25
70.25
36.88
23.36
60.85
GERM
41.98
59.19
65.53
72.58
38.68
41.37
64.62
20

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 10: Performance Comparison of QLoRA with DNABERT-2. This table shows the performance of all models on the QLoRa task.
Epigenetic Marks Prediction
Model
H3
H3K
H3K36mc3
H3K4me1
H3K4mc2
H3K4mc3
DNABERT-2
57.67
32.22
38.84
29.17
28.74
20.47
GERM-T
59.52
33.03
35,84
28.85
28.02
21.53
GERM
60.55
37.30
39.52
33.22
26.61
21.29
Epigenetic Marks Prediction
Promoter Detection
Model
H3K79me3
H3K9ac
H4
H4ac
all
notata
tata
DNABERT-2
53.11
41.69
67.02
28.92
81.27
88.66
54.98
GERM-T
54.22
44.76
72.82
29.68
81.15
87.58
55.07
GERM
56.80
47.41
74.76
30.85
82.13
90.02
56.97
Transcription Factor Prediction (Human)
Core Promoter Detection
Model
1
2
all
notata
tata
DNABERT-2
62.62
66.97
49.46
41.86
63.86
60.41
64.26
63.24
GERM-T
62.00
67.31
50.84
41.57
65.87
60.48
65.14
55.80
GERM
62.10
69.00
53.28
44.80
65.71
61.15
64.73
57.27
Transcription Factor Prediction (Mouse)
Virus
Splice
Model
0
4
Covid
Reconstruct
DNABERT-2
36.56
75.32
70.55
50.47
35.41
6.44
49.99
GERM-T
35.63
72.05
70.82
48.77
34.46
14.81
51.87
GERM
37.01
76.59
72.95
52.34
33.48
16.72
59.12
Across all adaptation methods, GERM consistently outperforms DNABERT-2 and GERM-T in various prediction tasks, particularly excelling in epigenetic marks prediction and promoter detection. This consistent superiority suggests that GERM possesses higher degree of flexibility and effectiveness in low-rank adaptation scenarios. While DNABERT-2 and GERM-T show competitive performance in certain tasks and adaptation methods, GERM's robust performance across diverse biological datasets and adaptation strategies underscores its potential as more reliable tool for complex genomic predictions.
21

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 11: Performance Comparison of LoftQ with DNABERT-2. This table shows the performance of all models the
LoftQ task.
Epigenetic Marks Prediction
Model
H3
H3K 14ac
H3K36me3
H3K4mel
H3K4me2
H3K4me3
DNABERT-2
58.92
32.36
36.59
28.84
27.19
18.46
GERM-T
60.03
32.03
37.23
28.78
27.94
20.16
GERM
62.17
36.79
39.45
33.12
27.37
21.84
Epigenetic Marks Prediction
Promoter Detection
Model
H3K79me3
H3K9ac
H4
H4ac
all
notata
DNABERT-2
60.88
46.21
67.86
29.63
80.44
88.71
56.79
GERM-T
59.61
52.63
80.20
30.57
80.10
88.78
58.52
GERM
59.80
55.08
81.25
31.23
82.00
89.81
59.86
Transcription Factor Prediction (Human)
Core Promoter Delection
Model
2
all
notata
tata
DNABERT-2
63.30
67.63
50.39
43.41
64.09
59.80
64.24
56.93
GERM-T
63.35
66.70
51.82
40.13
64.13
60.74
64.50
57.46
GERM
62.08
68.84
54.25
44.70
64.57
61.43
65.12
59.10
Transcription Factor Prediction (Mouse)
Virus
Reconstruct
Model
0
4
Covid
Reconstruct
DNABERT-2
35.61
74.25
71.16
51.63
34.37
8.31
52.17
GERM-T
39.46
75.27
71.24
57.41
34.28
5.01
55.89
GERM
42.13
76.55
71.96
51.46
36.04
9.14
60.12
22

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
D.3 All Results in Post-Training Quantization
In this section, we present the results of all experiments conducted for the post-training quantization (PTQ).
Table 12: Performance Comparison of Outlier Suppression with DNABERT-2, This table shows the performance of all models on the Outlier Suppression.
Epigenetic Marks Prediction
Model
Bits
H3
H3K14ac
H3K36me3
H3K4mel
H3K4me2
H3K4me3
8W/SA
31.81
0.11
16.17
0.74
14.33
1.69
DNABERT-2
6W/6A
26.56
1.44
13.76
2.12
13.50
3.04
8W/SA
65.55
33.07
34.81
6.82
23.29
15.60
GERM-T
6W/6A
62.50
11.13
38.51
27.66
21.31
15.68
8W/8A
66.29
27.78
28.54
32.06
13.47
14.77
GERM
6W/6A
60.19
13.89
21.67
28.26
3.72
5.49
Epigenetic Marks Prediction
Promoter Detection
Model
Bits
H3K79me3
H3K9ac
H4
H4ac
all
notata
tata
8W/8A
17.90
13.30
2.30
11.46
29.46
21.55
26.88
DNABERT-2
6W/6A
12.66
14.30
2.30
10.68
25.19
18.33
23.10
8W/8A
46.91
32.02
75.01
32.99
53.15
46.88
29.94
GERM-T
6W/6A
44.79
15.91
76.85
11.45
71.60
82.13
44.51
SW/SA
44.44
29.83
77.00
19.42
75.36
83.04
31.92
GERM
6W/6A
22.82
11.73
21.67
3.77
69.04
82.75
23.43
Transcription Factor Prediction (Human)
Core Promoter Detection
Model
Bits
tf0
tfl
tf2
tf3
tf4
all
notata
tata
8W/SA
41.72
41.84
40.79
6.84
42.45
54.50
28.58
54.53
DNABERT-2
6W/6A
41.43
42.05
39.64
9.96
43.17
55.54
26.75
56.20
SW/SA
35.24
71.82
64.96
37.81
23.54
58.13
60.69
57.54
GERM-T
6W/6A
54.18
76.65
67.42
27.09
23.54
57.69
56.73
73.47
SW/AA
54.37
52.53
41.99 1.99
45.58
59.72
47.09
55.61
66.39
GERM
6W/6A
54.62
51.94
41.34
44.74
60.88
46.63
54.85
66.47
Transcription Factor Prediction (Mouse)
Virus
Splice
Model
Bits
0
2
3
Covid
Reconstruct
8W/8A
35.39
23.62
52.89
41.51
23.73
23.74
6.69
DNABERT-2
6W/6A
34.55
23.20
55.55
39.24
25.13
25.55
6.87
8W/8A
35.24
71.82
64.96
37.81
23.54
19.42
30.25
GERM-T
6W/6A
54.18
76.65
67.42
27.09
29.01
53.21
28.47
8W/8A
50.97
48.73
41.80
50.21
30.48
61.04
36.76
GERM
6W/6A
49.96
48.50
39.87
47.53
28.95
59.94
36.79
23

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 13: Performance Comparison of SmoothQuant with DNABERT-2. This table shows the performance all models on the SmoothQuant.
Epigenetic Marks Prediction
Model
Bits
H3
H3K14ac
H3K36me3
H3K4mel
H3K4me2
H3K4mc3
8W/8A
44.40
-1.47
42.17
13.12
23.30
8.39
DNABERT-2
6W/6A
28.07
-0.08
22.77
2.96
15.73
0.10
4W/4A
-1.85
0.83
-1.09
-0.94
2.31
-0.90
8W/8A
72.44
46.34
50.15
28.18
26.59
23.25
GERM-T
6W/6A
27.19
8.68
28.30
2.15
14.21
4.82
4W/4
0.00
-2.43
0.94
0.80
-0.07
8W/A
72.07
49.89
52.95
33.11
32.28
24.87
GERM
6W/6A
72.23
48.60
53.67
31.94
32.09
24.76
4W4A
18.17
6.83
20.84
4.03
0.00
1.43
Epigenetic Marks Prediction
Promoter Detection
Model
Bits
H3K79me3
H3K9ac
H4
H4ac
all
8W/8A
55.54
14.16
2.30
10.19
44.80
58.87
38.19
DNABERT-2
6W/6A
25.14
20.25
3.99
0.21
78.44
57.23
0.54
4W/4A
0.45
-0.90
-5.07
1.20
0.00
-1.78
-5.42
8W/8A
59.78
48.39
76.85
43.62
73.81
81.40
58.84
GERM-T
6W/6A
49.94
41.77
40.94
20.73
72.97
70.85
12.03
4W/4A
-2.53
2.26
0.00
3.10
-1.31
-1.16
0.00
8W/8A
61.36
50.31
79.61
48.66
80.60
92.51
58.08
GERM
6W/6A
62.15
49.05
78.99
48.37
79.95
92.18
58.73
4W/4A
21.16
0.00
56.10
0.17
35.32
76.43
8.04
Transcription Factor Prediction (Human)
Core Promoter Detection
Model
Bits
3
4
all
SW/BA
45.06
48.20
47.29
4,70
52.27
60.82
41.31
68. 15
DNABERT-2
6W/6A
28.41
29.85
34.60
6.51
20.96
47.72
36.38
31.69
4W/4A
5.55
-2.74
-1.51
-0.72
4.52
1.53
-3.00
8W/SA
57.74
52.51
53.17
46.36
65.22
64.15
61.01
75.31
GERM-T
6W/6A
46.18
43.36
38.43
36.12
43.69
52.24
60.68
4W/4A
2.33
-3.19
-2.42
1.79
4.53
0.78
3.33
-5.73
8W/8A
58.54
52.44
48.14
45.42
66.42
49.26
64.46
70.31
GERM
6W/6A
57.55
51.81
48.29
45.19
65.87
47.55
64.35
69.67
4W4A
43.41
35.76
25.56
9.77
22.75
21.99
40.51
12.46
Transcription Factor Prediction (Mouse)
Virus
Splice
Model
Bits
Covid
Reconstruct
8W/8A
45.74
47.95
67.98
57.32
29.55
27.30
24.70
DNABERT-2
6W/6A
13.28
25.38
30.48
7.67
12.96
-0.44
0.00
4W/4A
0.06
-1.95
-15.92
-2.76
2.59
0.13
0.95
8W/8A
46.21
82.36
75.48
67.31
33.53
66.74
73.63
GERM-T
6W/6A
0.00
27.14
42.68
21.42
6.52
7.95
0.00
4W/4A
-1.22
1.22
0.00
0.00
1.49
-0.22
0.00
8W/8A
46.87
53.88
54.71
65.01
38.48
64.51
75.14
GERM
6W/6A
44.96
60.08
53.97
64.14
37.28
64.69
73.20
4W/4A
8.07
26.40
10.00
27.02
9.80
19.48
0.00
24

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 14: Performance Comparison of OmniQuant with DNABERT-2. This table shows the performance of all models on the OmniQuant.
Epigenetic Marks Prediction
Model
Bits
H3
43K14ac
H3K4mel H3K36me3
H3K4me2
H3K4me3
8W/8A
67.33
20.74
41.80
32.98
29.08
19.58
DNABERT-2
6W/6A
66.05
14.45
38.10
32.74
27.87
19.58
4W/4A
2.54
-0.73
0.93
2.14
6.01
0.98
8W/AA
72.14
46.57
50.37
35.38
24.90
23.52
GERM-T
6W/6A
71.47
44.40
49.26
35.16
23.22
22.87
4W/AA
0.00
0.00
0.00
-1.65
1.83
-1.30
SW/8A
72.20
50.02
$3.20
33.43
32.80
24.68
GERM
6W/6A
71.52
49.17
53.10
32.14
32.92
25.61
4W/A
70.33
44.65
49.23
28.33
23.55
22.43
Epigenetic Marks Prediction
Promoter Detection
Model
Bits
H3K79me3
H3K9ac
114
H4ac
all
notata
8W/8A
60.11
38.01
$7.40
18.80
75.24
87.94
37.51
DNABERT-2
6W/6A
56.96
38.26
59.30
16.63
72.26
89.34
38.56
4W/4A
-0.48
2.11
0.00
0.56
0.00
5.89
0.00
8W/8A
61.12
49.38
78.25
44.21
82.19
91.75
59.80
GERM-T
6W/6A
62.14
48.76
78.25
41.23
72.25
91.75
56.87
4W/4A
0.00
2.08
3.31
8.71
1.09
0.31
8W/SA
61.64
49.87
79.19
48.48
79.24
92.48
57.42
GERM
6W/6A
62.01
50.08
79.33
48.36
79.33
92.13
57.42
4W/4A
59.70
46.49
71.66
42.71
79.79
89.34
49.81
Transcription Factor Prediction (Human)
Promoter Detection
Model
Bits
3
4
all
notata
tata
8W/8A
52.40
52.31
48.78
17.02
$5.88
63.98
61.25
69.12
DNABERT-2
6W/6A
52.53
49.67
45.04
9.14
52.14
62.94
55.92
69.12
4W/4A
5.68
11.69
11.10
2.62
0.00
4.91
7.66
16.69
8W/SA
60.47
82.30
73.72
68.07
44.32
63.48
66.55
72.93
GERM-T
6W/6A
44.96
51.87
72.70
63.35
41.84
63.72
66.55
72.93
4W/4A
9.50
13.09
0.00
6.22
0.00
12.19
35.90
14.08
SW/SA
45.92
54.67
56.42
67.66
39.22
48.40
64.28
68.68
GERM
6W/6A
45.18
53.28
55.45
65,81
38.44
48.37
63.60
68.68
4W/4
54.40
49.31
43.44
44.76
59.61
38.51
60.73
56.20
Transcription Factor Prediction (Mouse)
Virus
Splice
Model
Bits
2
3
Covid
Reconstruct
8W/8A
48.88
64.61
62.21
65.79
41.44
46.29
63.82
DNABERT-2
6W/6A
51.31
67.05
64.04
59,92
37.17
44.42
62.13
4W/4A
-1.40
2.44
6.08
0.82
-0.74
44.60
0.00
8W/8A
60.47
82.30
73.72
68.07
44.32
39.59
75.30
GERM-T
6W/6A
44.96
51.87
72.70
63.35
41.84
39.59
68.19
4W/4A
0.00
0.00
9.05
0.00
3.53
-0.18
0.00
8W/8A
45.92
54.67
56.42
67.66
39.22
46.90
70.33
GERM
6W/6A
45.18
53.28
55.45
65.81
38.44
47.12
70.33
4W/4A
42.15
52.01
32.18
62.06
32.30
44.60
33.59
25

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 15: Performance Comparison of Traditional W8A8 PTQ with DNABERT-2. This table shows the performance of all models on the Traditional W8A8 post-training quantization (PTQ).
Epigenetic Marks Prediction
Model
H3
H3K 14ac
H3K36mc3
H3K4me1
H3K4me2
H3K4mc3
DNABERT-2
50.39
24.73
26.09
21.80
26.80
5.05
GERM-T
63.62
29.51
39.58
26.13
19.86
17.98
GERM
70.63
50.93
53.15
33.07
35.75
24.79
Epigenetic Marks Prediction
Promoter Detection
Model
H3K79me3
H3K9ac
H4
H4ac
all
notata
tata
DNABERT-2
48.50
42.11
70.95
3.67
71.94
60.04
34.55
GERM-T
56.87
35,84
75.44
24.97
67.34
80.33
25,41
GERM
61.46
50.33
78.53
47.56
80.97
92.28
60.05
Transcription Factor Prediction (Human)
Core Promoter Detection
Model
1
2
all
notata
tata
DNABERT-2
6.98
26.21
57.43
22.41
42.90
49.62
38.69
34.68
GERM-T
50.73
16.09
44.11
21.43
46.94
53.68
61.83
68.45
GERM
57.11
53.20
51.67
45.65
67.63
50.85
64.42
69.35
Transcription Factor Prediction (Mouse)
Virus
Splice
Model
0
4
Covid
Reconstruct
DNABERT-2
21.06
65.35
65.32
29.29
11.28
2.33
14.58
GERM-T
19.18
57.24
16.69
14.64
27.57
6.25
6.88
GERM
45.50
59.92
53.15
62.44
38.58
66.99
75.55
GERM demonstrates exceptional adaptability and performance in post-training quantization tasks, outperforming both DNABERT-2 and GERM-T across various quantization methods. GERM-T also shows commendable performance, especially in 8-bit quantization, making it viable alternative when GERM may not be applicable. These models collectively represent significant advancements in deploying efficient and accurate genomic prediction tools in environments with limited computational resources.
26

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
D.4 All Results of Performance Comparison in Resource-Constraine Computing Environments
In this section, we present the results of Performance Comparison in Sessource-Consraiigg Computing Environments. All models were trained on the same computing infrastructure (Nvidia GeForce RTX 2080 Ti 1 I1GB) to ensure a fair comparison. The training time represents the average time per epoch, with OmniQuant used as quantization example.
GERM demonstrates superior adaptability and performance DNABERT-2 and GERM-T. Its consistent high MCC scores and reduced training and inference times across various quantization levels and fine-tuning methods establish GERM the most robust and efficient model, with GERM-T following as commendable second-best option. These attributes make GERM promising candidate for further research and application in settings demanding both high performance and computational efficiency.
pnoource-constrainee computing environments compared
Table 16: Comparison of Performance Resource--Constrained Computing Environments. Comparison of three
models on the quantization and fine-tuning task.
Method
#Bits
MCC (1)
Time (sec.)
DNABERT-2
16W/16A
59.11
7.66
GERM
16W/16A
59.73
6.70
GERM-T
16W/16A
59.30
7.01
DNABERT-2
8W/8A
49.92
5.47
GERM
8W/8A
55.99
4.79
GERM-T
8W/8A
56.80
5.01
DNABERT-2
4W/4A
-1.03
3.81
GERM
4W/4A
20.05
3.33
GERM-T
4W/4A
0.22
3.49
Fine-Tuning Method
Method
MCC (t)
Time (sec.)
Train
Inference
DNABERT-2
Full
59.11
516.49
3.85
GERM
Full
59.73
323.10
3.24
GERM-T
Full
59.30
326.91
3.25
DNABERT-2
LoRA
50.91
197.13
4.12
GERM
LoRA
57.27
154.67
3.30
GERM-T
LoRA
55.60
167.76
3.32
DNABERT-2
QLoRA
50.65
206.15
5.28
GERM
QLoRA
53.16
164.10
4.13
GERM-T
QLoRA
51.50
177.95
4.17
DNABERT-2
LofiQ
50.76
251.37
5.77
GERM
LofiQ
53.11
199.58
4.52
GERM-T
LoftQ
51.20
220.37
4.52
27

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Case Study 2: Performance in CPU-only Computing Environments. To demonstrate GERM's capability in CPU-only computing environments, perform performance tests 64-core Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 50GB RAM. We compare GERM's per-epoch training and inference times for the LoRA and QLoRA fine-tuning methods. The results, presented in Table 17, indicate that both GERM and GERM-T achieve shorter fine-tuning times per epoch compared to DNABERT-2, with the only exception being QL when deployed, where the time is slightly longer. QLoRA can be slower than LoRA during inference and fine-tuning due to hardware limitations when bfl6 (bfloat16) support is unavailable. QLoRA relies on ultra-low-prestt -owe--peiiiiinn quantization (e.g., 4-bit weights) to reduce memory usage and increase efficiency, which works best on systems that support 6 similar mixed- fexxed-precision operations. However, without 6 support, these low-precision operations must be emulated by converting back higher precision, introducing computational overhead. This diminishes the intended speed advantage of QLoRA, potentially making slower than LoRA on incompatible hardware.
Table 17: Comparison of Performance in CPU-only Computing Environments. Comparison three models on the tinettuning task.
Fine-Tuning Tuning Method
Method
MCC (t)
Time (sec.)
Train
Inference
DNABERT-2
LoRA
50.91
808.23
29.66
GERM
LoRA
57.27
618.68
23.10
GERM-T
LoRA
55.60
674.40
23.57
DNABERT-2
QLoRA
50.65
516.04
63.17
GERM
QLoRA
53.16
358.34
45.28
GERM-T
QLoRA
51.50
418.13
46.91
D.5 Evaluation of Common Genomic Foundation Models
In this section, we conduct the experiments to show the performance of common GFMs. As there is no official fine-tuning code for the Nuclcotide Transformer (Dalla-Tore Torre et al., 2024), we utilize its open-sourced checkpoints available on HuggingFace Model Hub and fine-tune using our code base with LoRA. We implement HyenaDNA (Nguyen et al., 2024b) using its official implementation available on HuggingFace and the HuggingFace Trainer. Since HyenaDNA does not natively support LoRA for fine-tuning, do not evaluate LoRA's performance on HyenaDNA. As shown in Table 18,
Table Performance Comparison of Common Genomic Foundation Models.
Fine- Tuning Method
Model
MCC
Average Performance Drop
Full
59.11
DNABERT-2
LoRA
50.91
13.87%
Full
51.31
HyenaDNA
LoRA
Full
56.05
NT-500M-hum -hmman
LoRA
52.66
6.44%
medium--45kk-sellen--
28

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
D.6 Performance of GERM Alternative Transformer-based Models
In this section, we conduct our experiment to validate the effectiveness of the outlier removal approach using alternative transformer-based models, evaluating performance through Matthews Correlation Coefficient (MCC) and average perfor- mance drop. We use the NT-500M- -uuman" as the target model for our evaluation. Table 19 compares these metrics across NT-500M-human, GERM, and GERM-T models using different low-rank adaptation methods. Table 20 examines the impact of various quantization techniques on the same models. The results demonstrate the effectiveness of outlier removal across diverse adaptation and quantization strategies, highlighting the balance between performance and resource efficiency.
Table 19: Low-Rank Adaptation Methods Comparison. This comparison evaluates the performance of different low-rank adaptation methods, including Full, LoRA, QLoRA, and LoftQ, on Nucleotide Transformer 500M models. The best results highlighted in bold, while the second-best results underlined.
Tuninnuning Method
Model
MCC
Delta MCC
Average Performance Drop
Full
56.05
LoRA
52.66
3.39
6.44%
NT-500M-humas humman
QLoRA
51.46
4.59
8.19%
LoftQ
51.89
4.16
7.42%
Full
55.52
0.53
LoRA
54.32
1.73
2.16%
GERM (NT-500M --OM--hmmn)
QLoRA
53.78
2.27
3.13%
LoftQ
54.24
1.81
2.30%
Full
56.53
-0.48
LoRA
54.89
1.16
2.90%
GERM-T (NT-500M-human)
QLoRA
52.78
3.27
6.63%
LoftQ
53.45
2.60
5.45%
http://liieenneeocoooooooeoeeeee human-ref
29

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 20: Quantization Methods Comparison. This comparison analyzes the performance of various quantization methods, including FP16, W8A8, Outlier, SmoothQuant, and OmniQuant, on Nucleotide Transformer 500M models. The best results are highlighted in bold, while the second-best results are underlined.
Model
#Bits
Quantization Method
MCC
Delta MCC
Average Performance Drop
16W/16A
56.05
8W/8A
34.66
21.39
38.17%
8W/8A
32.95
23.10
41.21
Outlier
6W/6A
26.65
29.40
52.45%
8W/8A
38.23
17.82
31.79%
NT-500M-human
6W/6A
SmoothQuant
28.67
27.38
48.84%
4W/4A
3.54
52.51
93.68%
8W/8A
47.35
8.70
15.52%
6W/6A
OmniQuant
43.63
12.42
22.16%
4W/4A
5.34
50.71
90.47%
16W/16A
55.53
0.52
8W/8A
53.67
2.38
3.35%
8W/8A
45.71
10.34
17.68 %
Outlier
8W/8A
41.38
14.67
25.48%
8W/8A
53.18
2.87
4.23%
GERM (NT-500M- (NT-500M-human)
6W/6A
SmoothQuant
52,43
3.62
5.58%
4W/4A
24.96
31.09
55.05%
8W/8A
52.45
3.60
5.55
6W/6A
OmniQuant
51.56
4.49
7.15%
4W/4A
46.45
9.60
16.35 %
16W/16A
56.53
-0.48
8W/8A
40.71
15.34
27.99%
8W/8A
45.98
10.07
18.66%
Outlier
6W/6A
43.38
12.67
23.26 %
8W/8A
54.19
1.86
4.14%
GERM-T (NT-500M-humanl
6W/6A
SmoothQuant
38.67
17.38
31.59%
4W/4A
10.57
45.48
81.29%
8W/8A
52.46
3.59
7.20%
6W/6A
OmniQuant
51.34
4.71
9.18%
4W/4A
23.57
32.48
58.31% 1%
30

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
D.7 Performance of GERM on Large-scale GFMs
In this section, we present experiments to validate the effectiveness of GERM on large-scale GFMs. We use the NT-2.5B- multi² as the target model for our evaluation. Table 21 compares these metrics across NT-2.5B-multi, GERM, and GERM-T models using different low-rank adaptation methods. Table 22 extends this analysis to evaluate the impact various quantization techniques on the same models. In the larger- parar-p model, we adopt stricter quantization bits. This choice aims to save computation and improve efficiency, as finer compression is crucial when model parameters scale up. Additionally, experiments conducted with removal consistently enhances performance and resource efficiency across diverse adaptation and quantization strategies.
model further validate these findings, demonstrating that outlier
Table 21: Comparison of Low-Rank Adaptation Methods in Large-Scale Models. This comparison evaluates the performance of different low-rank adaptation methods, including Full, LoRA, QLoRA, and LoftQ, on Nucleotide Transformer 2.5B models. The best results highlighted in bold, while the second-best results are underlined.
Fine- Fininn
Model
MCC
Delta MCC
Average Performance Drop
Method
Full
56,98
LoRA
53.50
3.48
6. 6.11%
NT-2.5B-multi
QLoRA
52.29
4.69
8.19%
LoftQ
52.89
4.09
7.17%
Full
57.16
-0.18
LoRA
55.98
1.18
2.06%
GERM (NT-2.5B-multi)
QLoRA
55.52
1.64
2.87
LoftQ
55.80
1.36
2.38 %
Full
56.82
0.16
LoRA
55.24
1.58
2.78%
GERM-T (NT-2.5B-mult)
QLoAA
53.32
3.50
6.16%
LoftQ
53.74
3.08
5.42%
http:///iinnnniaa....ooo//eoeeee 31

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Table 22: Comparison of Quantization Methods in Large-Scale Models. This comparison analyzes the performance of various quantization methods, including FP16, W6A6, W4A4, Outlier, SmoothQuant, and OmniQuant, on Nucleotide Transformer 2.5B models. The best results are highlighted bold, while the second-best results are underlined.
Model
#Bits
Quantization Method
MCC
Delta MCC
Average Performance Drop
16W/16A
56.98
6W/6A
18.52
38.46
67.50%
4W/4A
1.39
55.59
97.56%
6W/6A
50.23
6.75
11.85%
Outlier
NT-2.5B-mull
4W/4A
40.74
16.24
28.50%
6W/6A
47.23
9.75
17.11%
SmoothQuant
4W/4A
35.16
21.82
38.29%
6W/6A
49.55
7.43
13.04%
OmniQuant
4W/4A
43.63
13.35
23.43%
16W/16A
57.16
-0.18
6W/6A
45.96
11.2
19.59%
4W/4A
42.48
14.68
25.68%
6W/6A
52.24
4.92
8.61%
Outlier
GERM ANT-2.5B-muld
4W/4A
49.00
8.16
14.28 %
6W/6A
51.95
5.21
9.11 %
SmoothQuant
4W/4A
48.15
31.09
15.76%
6W/6A
52.55
4.61
8.07%
OmniQuant
4W/4A
49.26
7.90
13.82
16W/16A
56.82
0.16
6W/6A
32.58
24.24
42.66%
4W/4A
10.49
46.33
81.54%
6W/6A
52.14
4.68
8.24%
Outlier
GERM-T (NT-2.5B-mult)
4W/4A
46.24
10.58
18.62%
6W/А
51.61
5.21
9.17%
SmoothQuant
4W/4A
48.12
8.70
15.31 %
6W/6A
52,43
4.39
7.73%
OmniQuant
4W/4A
47.28
9.54
16.79%
32

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
D.8 Performance of GERM with Different Outlier Removal Techniques
In this section, we present experiments to validate the effectiveness of GERM compared to other common outlier removal techniques, including clipped softmax and gated attention (Bondarenko et al., 2024). We utilize SmoothQuant as the quantization method our experiments. As shown in Table 23, the results demonstrate the superior effectiveness of outlier removal by GERM compared clipped softmax and gated attention. GERM achieves performance improvement of 2.59% in 4-bit quantization and 1.99% in 8-bit quantization.
Table 23: Performance of GERM with Clipped Softmax and Gated Attention
Method
#Bits
MCC (1)
Average Performance Drop
DNABERT-2
16W/16A
59.11
GERM
16W/16A
59.73
Clipped Softmax
16W/16A
59.17
Gated Attention
16W/16A
59.49
DNABERT-2
8W/8A
49.92
15.55%
GERM
8W/8A
55.99
6.26 %
Clipped Softmax
8W/8A
53.26
9.99%
Gated Attention
SW/8A
54.58
8.25%
DNABERT-2
4W/4A
-1.03
101.74%
GERM
4W/4A
20.05
66.43 %
Clipped Softmax
4W/4A
13.49
77. 77.20%
Gated Attention
4W/4A
18.66
69.02%
D.9 PTQ Performance of GERM Following LoRA Fine-Tuning
In this section, we present experiments to validate the effectiveness of GERM post-training quantization (PTQ) perfor- mance following LoRA adaptation. We employ SmoothQuant as the quantization method in our evaluation. As shown in Table 24, the results highlight the superior outlier mitigation capability of GERM, leading to improved PTQ perfor- mance. Specifically, using W8A8 quantization GERM achieves 84.72% improvement over the baseline, demonstrating its effectiveness in low-rank adaptation and quantization scenarios.
Table 24: Comparison of Post-Training Quantization (PTQ) Performance Across Low-Rank Adaptation Methods in Large-Scale Models This comparison assesses the performance of SmoothQuant (SQ) following low-rank adaptation fine-tuning using LoRA DNABERT-2 117M models. The top-performing results are shown in bold, and the second-best results are underlined for clarity.
Model
Method
MCC
Delta MCC
Average Performance Drop
Full
59.11
7.00
DNABERT-2
LoRA LoRA + SQ
50.91+1.677
15.2
13.87% 42.09%
3..2
31.88
Full
59.73
6.38
GERM
LoRA
57.27+1.770 70
8.84
4.12 %
LoRA + -Q
55.89+10.93
10.22
6.43 %
Full
59.30
6.81
GERM-T
LoRA
55.60+10.28
10.51
6.23%
LoRA + SQ
54.26+10.65
11.85
8.50%
33

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
D.10 Attention Distribution Visualization in GERM
In this section, to better understand the impact of outlier removal in GERM, we visualize the attention score distributions across transformer layers, shown in Figure We visualize the attention distributions for three different DNA sequences from the mouse 0 dataset, showing both attention probabilities and attention scores for the vanilla and GERM versions of DNABERT-2. The attention probability captures the probability between each token and all other tokens, while the attention score is computed by multiplying the attention probability with the attention value, yielding a tensor of shape (number of tokens x hidden dimension). For visualization purposes, we use the last hidden layer of the model and display the first 32 dimensions of the attention score. These visualizations reveal that GERM produces smoother and more stable attention patterns compared to baseline models such as DNABERT-2, which exhibit sharp spikes and irregularities due to the influence of outliers. The reduced variance and kurtosis in GERM'S ' attention maps confirm its ability to suppress low-information tokens, resulting in more efficient and interpretable attention behavior.
E Additional Discussion and Limitation
In this section, we provide additional discussion on the limitations of GERM-T. As a trade-off approach, GERM-T is designed practical extension of GERM for continual low-resource adaptation, minimizing recomputation by reusing pre-traie pre-trained checkpoints and applying small-step updates. While it achieves balance between performance and computational cost under 8-bit quantization, one key limitation is that GERM-T performs worse in certain quantization and low-rank adaptation settings, particularly under low-bit quantization scenarios. This performance degradation results primarily from the restricted optimization scope imposed by small-step fine-tuning and the accumulation of approximation errors. Comparing with GERM, these errors prevent GERM-T T from fully mitigating inherent outliers, leading to increased average kurtosis and maximum infinity norm. In future work, explore the underlying causes of this degradation in greater depth and develop more robust quantization-awar training (QAT) approach to better manage these trade-offs.
F Definition of Outlier in Our Paper
In this section, provide more detailed deification of outlier in our paper. In our work, we define outliers as tokens or activations that disproportionately influence the atlention mechanism, despite containing little or meaningful information. These outliers emerge when the softmax function amplifies the attention probabilities of tokens that ideally should receive minimal or zero focus. We use the following attention mechanism to analyze this behavior:
Output = Residual
Softmax
QK
V + x
va
As shown in Hu et al. (2024a), if the attention input X already contains sufficient information, the attention mechanism within the residual connection should ideally behave like an identity transform, producing near-zero attention outputs:
QK
Softmax
V=
Vd
In such cases, tokens with high values in which may represent biologically significant features- still receive near-zero attention probabilities.
Why Classic Softmax Fails. The problem arises from how the softmax function normalizes probabilities. Softmax enforces that all probabilities sum to one, which inherently magnifies the attention probabilities assigned low-value tokens. This unwanted amplification broadens the attention score distribution and introduces outliers tooens that exert disproportionate influence despite their low informational value.
These outliers are particularly problematic in genomic models like DNABERT-2, where certain regions of genomic sequences-Ste suceeesssssshhae as repetitive patterns, low-complexity sequences, or non-coding regions regions-resemble no-op tokens. While these regions carry minimal biological relevance, the classic softmax mechanism may assign them higher-than chan-expeeted attention scores, diverting focus away from meaningful genomic features.
34

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Intuition Behind Outliers in Genomic Models. Outliers in genomic models typically arise from sequence patterns that produce anomalous query-key interactions in the attention mechanism. While genomic data lacks traditional "words" as in language models, certain biological patterns exhibit similar behavior. Key examples include:
Low-Complyxity Regions (e.g,, Poly-A or Poly-T Sequences): Genomic sequences often include regions with runs of identical bases (e.g., AAAAA. ... TTTTT. These sequences contain minimal unique information yet can produce large, uniform dot-product values in the attention mechanism. This causes softmax to assign exaggerated probabilities to these low-information tokens, effectively making them outliers.
Repetitive Motifs and Tandem Repeats: Certain genomic regions, such as microsatellites and tandem repeats, involve recurring nucleotide patterns that behave similarly to low-value tokens in the attention mechanism. These patterns exhibit strong internal correlations, often resulting in softmax overemphasizing them as if they were biologically significant.
Boundary and Spacer Elements (e-g., Alignment Padding or Non-coding Spacer Sequences): In genomic datasets, artificial padding sequences, non-coding segments, spacer sequences sometimes introduced to ensure proper sequence alignment. These tokens are intended to have no biological relevance, softmax's behavior inad inadvertently amplifies their attention scores, creating noise that distorts meaningful patterns.
35

---

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Attention Probability
Attention Score
(al)
(b1)
(a2)
(b2)
(a3)
Figure Attention Distribution Visualization in GERM. Comparison attention probabilities and outputs for genomic sample between DNABERT-2 and NT-550M--human Heatmaps from the final hidden layers are scaled from (blue) (red). In the figure, all rows labeled (a) correspond to the vanilla DNABERT-2, while all rows labeled (b) represent the GERM version. The vanilla model exhibits broad attention spread, which dilutes focus across tokens, whereas GERM concentrates attention on key tokens, enhancing both efficiency and interpretability.
36


Reviews:
[{"review_id": "C6RorkBn7F", "reviewer": "Reviewer_sdac", "summary": "This paper addresses the limitations of current GFMs, particularly DNABERT-2, when applying low-bit quantization and parameter-efficient fine-tuning methods like LoRA. The authors attribute performance degradation to outliers in attention mechanisms and propose GERM, a variant using an outlier-free attention mechanism (softmax₁). They also present GERM-T, a continual learning-based model for adapting GFMs under resource constraints. Experiments show that GERM and GERM-T perform better than baselines under low-precision and constrained environments.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The updated manuscript can be accessed anonymously at [link](https://www.dropbox.com/scl/fi/itpm5n21pfu3at01bofab/germ_icml2025.pdf?rlkey=zl8noukikpz1s4493b752s9uj&e=1&st=qma9ihc9&dl=0).\n\n\n> **Reviewer's Comment**: However, the motivation ….\n\n\n**Response**: We thank the reviewer for suggesting a direct attention score analysis to validate the outlier hypothesis.\nTo address this point, we add detailed visualizations of the attention score distributions in **Figure 3 and Appendix E.10**, where we compare DNABERT-2 and GERM. The results demonstrate that GERM suppresses attention outliers, leading to more focused and efficient attention patterns. This analysis provides direct evidence supporting our hypothesis.\nWe clarify this connection in the revised manuscript and invite the reviewer to examine Figure 3 and Appendix E.10 for a detailed analysis of these findings.\n\n\n> **Reviewer's Comment**: While statistics like ..\n\n\n**Response**: We sincerely appreciate the reviewer’s insightful suggestion about the need to more rigorously establish the connection between attention outliers and downstream performance degradation. In response, we have added detailed attention score distribution visualizations in Appendix E.10 and Figure 3, including per-layer heatmaps comparing DNABERT-2 and GERM.\nFor a formal definition of “outliers”, we respectfully refer the reviewer to our response to Reviewer `AD8a`. We also discuss in our response to Reviewer `8UsM` how existing literature has demonstrated the negative impact of attention outliers on model performance.\nWe believe these additions strengthen the theoretical and empirical motivation of our work and clarify the causal relationship between attention outliers and performance.\n\n\n> **Reviewer's Comment**: Limited novelty…\n\n\n**Response**: We appreciate the reviewer’s feedback on the importance of novelty. While our approach builds upon established methodologies, we would like to highlight our key innovation: we are the first to integrate outlier removal to simultaneously enable **(1) robust quantization and (2) accelerated low-rank adaptation** for genomic foundation models. This contribution is important because genomic data presents unique challenges—such as extreme sparsity, high variability, and frequent outliers in attention mechanisms—which differ significantly from those in traditional NLP.\n\n\n1. **First to achieve accelerated LoRA for Genomic Models via Systematic Outlier Removal**: Our work pioneers the use of LoRA in genomic foundation models like DNABERT-2. Applying LoRA directly to DNABERT-2 results in performance degradation due to genomic data-specific outliers. This novel integration of the outlier-free Hopfield mechanism enables effective low-rank adaptation alongside robust quantization, achieving a 37.98% improvement in fine-tuning performance compared to DNABERT-2.\n2. **Adapting Techniques for Genomic Challenges**: The genomic domain demands unique adaptations due to sparse and highly variable tokenization methods like k-mer and BPE. The outlier-free Hopfield layer required significant adjustments to mitigate domain-specific outliers, reducing kurtosis and infinity norm values by 92.14% and 82.77%, respectively, across 27 genomic datasets. This ensures both robust quantization and efficient fine-tuning in resource-constrained settings.\n3. **GERM-T for Continual Learning**: Beyond LoRA, GERM-T introduces a novel continual learning strategy that avoids training from scratch while effectively leveraging outlier-free layers. This approach focuses on resource-constrained genomic research, making adaptable fine-tuning without compromising performance possible.\n4. **Empirical Validation**: Our work provides the first empirical evaluation of integrating outlier mitigation into LoRA fine-tuning and quantization for genomic models, achieving a 64.34% improvement in quantization robustness compared to DNABERT-2. These results validate the effectiveness of our proposed modifications and their impact on genomic tasks.\n\n\nBy adapting and extending these methods, we address domain-specific challenges while advancing genomic modeling. We hope this response clarifies the novelty and significance of our contributions, and we are happy to provide further details or analyses if needed.\n\n\n> **Reviewer's Comment**: What is the operational definition …\n\n\n**Response**: We appreciate the opportunity to provide more details about the significance of outliers in transformer-based models. A more detailed explanation is provided in the response to the reviewer `AD8a`.\n\n\n> **Reviewer's Comment**:.Further reference ….\n\n\n**Response**: We appreciate the reviewer's suggestion to expand our literature review by including additional GFMs. In response, we incorporate Evo in the related work section of our revised version.", "claims_and_evidence": "Some central claims, such as the harmful effects of outliers in attention distributions on quantization and LoRA performance, are not fully supported with direct quantitative evidence. While statistics like kurtosis and max norm are provided, detailed distributional plots or causal analysis connecting these outliers to downstream performance degradation are lacking. The claim of improved efficiency through outlier removal is empirically supported but not rigorously motivated.", "methods_and_evaluation_criteria": "The evaluation framework is reasonable, employing multiple benchmarks (e.g., variant effect prediction, promoter identification) and standard metrics like MCC. The methods, such as GERM and GERM-T, are clearly described and evaluated under realistic constraints (e.g., low-precision hardware). However, the lack of detailed analysis on where and how outliers hurt model behavior weakens the methodological clarity.", "theoretical_claims": "Supplementary material (Section A) includes theoretical analysis on the softmax₁ function, particularly proving that it produces attention distributions with a bounded second moment, which helps prevent outlier values. While the result is cited from prior work (Hu et al., 2024), the paper correctly restates the theoretical guarantees that motivate the use of softmax₁. No new proofs are introduced, but the prior claims are accurately presented.", "experimental_designs_or_analyses": "Experiments are well-structured and provide comparisons across multiple quantization settings. However, the motivation—attention outliers—is not directly evaluated through visualization or per-layer analysis of attention score distributions. This gap undermines the connection between hypothesis and results.", "relation_to_broader_scientific_literature": "The paper builds on DNABERT-2 and outlier mitigation in transformers (e.g., softmax₁ from Hu et al. 2024). While its application to genomics is relevant and timely, the primary technique has already appeared in prior literature, reducing the novelty of the proposed approach.", "essential_references_not_discussed": "The authors appropriately cite related work on outlier-free transformers and GFMs including DNABERT, DNABERT-2, HyenaDNA and Nucleotide Transformer. Further reference that could enhance the current reference would be Evo [1].\n\n[1] Sequence modeling and design from molecular to genome scale with evo", "other_strengths_and_weaknesses": "- **Strengths**: Practical focus on resource-limited settings; strong empirical results under quantized and efficient setups.\n\n- **Weaknesses**: Limited novelty (reuses existing attention method), weak quantitative motivation, and missing connection between proposed changes and biological interpretability.", "other_comments_or_suggestions": "Some useful comments: \n1. Clarify how “outliers” are defined and provide attention distribution visualizations. \n2. Consider evaluating biological interpretability or downstream relevance more thoroughly.", "questions_for_authors": "1. Can you provide attention score distributions (e.g., histogram or density plots) to support the outlier hypothesis?  \n2. What is the operational definition of an “outlier” in your analysis (e.g., percentile-based, norm threshold) and how it relates and impacts the biological findings?   \n3. Given that softmax₁ is not novel, what do you consider the core technical contribution of this work?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pYlA6YRRZY", "reviewer": "Reviewer_AD8a", "summary": "This paper describes an adaptation of the DNABERT genomic foundation model to reduce the impact of outliers in attention mechanism. The outlier phenomenom was first observed in large language models, where it was shown attention mechanisms can learn to pay large attention to irrelevant tokens, like the [SEP] token. This is believed to stem from cases where an attention head needs something equivalent to a no-op, where it does not attend anywhere. These outliers manifest themselves as certain embeddings in each layer having large magnitude and cause problems for quantization, due to the resulting large range of values to be quantized. Quantization is a key component of making genomic foundation models practical, in the sense that they could be deployed in APIs for non-computational scientists to use in exploratory research. \n\nThe authors incorporate the outlier-free Hopfield layer of (Hu et al., 2024a) into the DNABERT model and demonstrate that this reduces the impact of outliers and retains better performance after quantization than the original BERT model. They also demonstrate that this model has better performance after low-rank adaptation and provide an efficient continual learning approach that enables the incorporation of this layer in a pretrained model without retraining from scratcn, and that the performance of this version falls between the GERM model (outlier-free Hopfield layer retrained from scratch) and the original DNABERT model on almost all evaluations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The updated manuscript can be accessed anonymously at [link](https://www.dropbox.com/scl/fi/itpm5n21pfu3at01bofab/germ_icml2025.pdf?rlkey=zl8noukikpz1s4493b752s9uj&e=1&st=qma9ihc9&dl=0).\n> **Reviewer's Comment**: what would lead to an outlier in a GFM...\n\n\n**Response**: Thank you for your insightful question about the nature of outliers in genomic foundation models. In our work, we define **outliers** as **tokens or activations that disproportionately influence the attention mechanism**, despite containing little or no meaningful information. These outliers emerge when softmax amplifies the attention probabilities of tokens that ideally should receive minimal or zero focus. We use the following attention mechanism to analyze this behavior:\n$$\n\\text{Output} = \\text{Residual} \\left( \\text{Softmax} \\left( \\frac{Q(K)^\\top}{\\sqrt{d}} \\right) V + X \\right)\n$$\nAs shown in (Hu et al., 2024), if the attention input \\(X\\) already contains sufficient information, the attention mechanism within the residual connection should ideally behave like an **identity transform**, producing near-zero attention outputs:\n$$\n\\text{Softmax} \\left( \\frac{QK^\\top}{\\sqrt{d}} \\right) V \\approx 0\n$$\nIn such cases, tokens with **high values in \\(V\\)** — which may represent biologically significant features — should still receive **near-zero attention probabilities**.\n\n\n## Why Classic Softmax Fails\n\n\nThe problem arises from how the softmax function normalizes probabilities. Softmax enforces that probabilities sum to 1, which inherently magnifies the attention probabilities assigned to **low-value tokens**. This unwanted amplification broadens the attention score distribution and introduces **outliers** — tokens that exert a disproportionate influence despite their low information value.\nGenomic models such as DNABERT-2 face a critical challenge from outliers - including repetitive elements, low-complexity regions, and non-coding segments - similar to no-op tokens. Despite their limited biological significance, these regions receive disproportionately high attention weights through standard softmax operations, consequently suppressing the model's focus on biologically relevant genomic features.\n\n\n## Intuition Behind Outliers in Genomic Models\n\n\nOutliers in genomic models typically arise from sequence patterns that produce anomalous query-key interactions in the attention mechanism. Though genomic data lacks traditional \"words\" like language models, certain biological patterns produce similar effects. Key examples include:\n### 1. **Low-Complexity Regions (e.g., Poly-A or Poly-T Sequences)**\nGenomic sequences frequently contain repetitive base patterns (e.g., `AAAAA...`, `TTTTT...`). These sequences carry minimal unique information yet can produce large, uniform dot-product values in the attention mechanism. This causes softmax to assign exaggerated probabilities to these low-information tokens, effectively making them outliers.\n### 2. **Repetitive Motifs and Tandem Repeats**\nGenomic repetitive elements such as microsatellites and tandem repeats, which contain patterns (e.g., `(CA)n`, `(GAA)n` repeats) that generate artificially inflated attention scores due to their inherent self-similarity. However, such regions lack corresponding biological information, often resulting in softmax overemphasizing them as if they were biologically significant.\n### 3. **Boundary and Spacer Elements (e.g., Alignment Padding or Non-coding Spacer Sequences)**\nIn genomic datasets, artificial padding sequences, non-coding segments, or spacer sequences are sometimes introduced to ensure proper sequence alignment. These tokens are intended to have no biological relevance, yet softmax’s behavior inadvertently amplifies their attention scores, creating noise that distorts meaningful patterns.\n## Impact on Genomic Analysis\nIn genomic foundation models like DNABERT-2, these outliers negatively impact performance by:\n- **Increasing Error Rates:** Outliers divert attention away from biologically meaningful regions, reducing prediction accuracy for tasks like mutation site identification.\n- **Destabilizing Fine-Tuning:** During fine-tuning, excessive focus on low-information tokens increases noise in gradient updates, limiting convergence stability.\n- **Masking Important Features:** Outliers may overshadow rare but critical genomic patterns, reducing the model’s capacity to detect subtle but meaningful biological signals.\n\n> **Reviewer's Comment**:.One possible criticism is….\n\n**Response**: We appreciate the reviewer’s feedback on the importance of innovation. We also appreciate the reviewer's comments, acknowledging that our paper makes a solid contribution as an empirical study that others can build upon. This paper presents experimental evidence demonstrating that outlier-free methods offer an efficient solution for improving low-rank adaptation and quantization. We restate our contribution in the response to the reviewer `sdac`.", "claims_and_evidence": "The authors claims, as described above, are demonstrated with a comprehensive set of experiments.", "methods_and_evaluation_criteria": "Ther authors use the same set of benchmarks as previous genomic foundation models.", "theoretical_claims": "I did not check the theoretical claims (in the appendix) in detail.", "experimental_designs_or_analyses": "The experiments look sound to me.", "relation_to_broader_scientific_literature": "This work is highly relevant to the broader scientific literature, given the increased interest in genomic foundation models and the likely requirement to use techniques like quantization and low-rank adaptation to make their use practical.", "essential_references_not_discussed": "To the best of my knowledge, the authors discuss all the relevant research.", "other_strengths_and_weaknesses": "The paper is well-written, with clear motivation and clearly described experiments. As mentioned above, the contributions are useful and carefeully evaluated.\n\nOne possible criticism is that the paper combines ideas that have previously been demonstrated for large language models, rather than providing a technical innovation in itself. However, I think the paper makes a solid contribution as a piece of empirical work others can bulld on.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "While we have a good explanation of the source of outliers in language models, it wasn't obvious to me what kind of input feature or token would lead to an outlier in a genomic foundation model. Do the authors have an intuition for this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tgbmPcCUTI", "reviewer": "Reviewer_PKFp", "summary": "This article introduces the outlier-free Hopfield layer into the Genomic foundation model to achieve a better trade-off between performance and efficiency. They also propose a continued training approach to avoid the additional cost of training from scratch. Comprehensive experimental results demonstrate that the outlier-free design significantly reduces performance degradation during quantization or fine-tuning.\n\n## update after rebuttal\nI have read the authors' rebuttal and my concerns have been addressed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The updated manuscript can be accessed anonymously at [link](https://www.dropbox.com/scl/fi/itpm5n21pfu3at01bofab/germ_icml2025.pdf?rlkey=zl8noukikpz1s4493b752s9uj&e=1&st=qma9ihc9&dl=0).\n> **Reviewer's Comment**: does not clearly explain what an outlier is or how to quantify it...\n\n**Response**: We thank the reviewer for the helpful comments and the opportunity to clarify our definition of outliers and provide theoretical analysis for their mitigation.\n\n## **Definition of Outliers in Our Work**\n\nThe definition of **outliers** in our work differs from the classical statistical definition. In traditional statistics and machine learning, outliers are typically defined as data points that fall outside the modeled distribution. In contrast, in our context, we define **outliers** as **tokens or activations that disproportionately influence the attention mechanism**, despite containing little or no meaningful information. A more detailed explanation is provided in the response to the reviewer `AD8a`.\n\n## **Q2: How Does softmax_1 Mitigate Outliers?**\n\n  \n\nIn our paper, we introduce **Softmax1** equation, a modified softmax function designed to mitigate the effects of outliers:\n\n  \n\n$$\n\\text{Softmax1}(S)_i = \\frac{\\exp(S_i)}{1 + \\sum_j \\exp(S_j)}\n$$\n\n  \n\nWhere $\\( S = QK^\\top / \\sqrt{d} \\)$ represents the scaled dot product in the attention mechanism.\n\n  \n\n**Key Improvements in Softmax1:**\n\n  \n  \n\n1. **Suppression of Low-Value Tokens:**\n\nUnlike standard softmax, which assigns **non-zero probabilities** to all tokens — even those with highly negative scores — Softmax1 allows low-information tokens to receive **near-zero probabilities**. This behavior is crucial in genomic models where repetitive sequences or spacer elements resemble no-op tokens.\n\n  \n\n2. **Controlled Attention Distribution:**\n\nSoftmax1 suppresses the broadening of the attention distribution, ensuring that the model remains focused on biologically relevant regions rather than noisy patterns.\n\n  \n\n---\n\n  \n\n## **Theoretical Support for Softmax1’s Outlier Resistance**\n\n- **Standard Softmax Behavior:**\n\nStandard softmax assigns non-zero probabilities to all tokens, even those that ideally should receive no attention. In extreme cases:\n\n  \n\n$$\n\\lim_{x_1 \\to -\\infty} \\ldots \\lim_{x_k \\to -\\infty} \\text{Softmax}(x)_i = \\frac{1}{k} > 0\n$$\n\n  \n\n- **Softmax1 Behavior:**\n\nIn contrast, Softmax1 ensures that tokens with highly negative scores are assigned probabilities that collapse to zero:\n\n  \n\n$$\n\\lim_{x_1 \\to -\\infty} \\ldots \\lim_{x_k \\to -\\infty} \\text{Softmax1}(x)_i = 0\n$$\n\n  \n\nThis limiting behavior ensures that low-information tokens, such as repetitive motifs or spacer tokens, are effectively ignored, stabilizing the attention mechanism.\n\n  \n\n---\n\n  \n\n## **Empirical Evidence for Outlier Mitigation**\n\nTo validate that Softmax1 effectively mitigates outliers, we provide both **quantitative** and **qualitative** evidence:\n\n  \n\n- **Quantitative Evidence:**\n\nAs shown in Table 1, GERM achieves a **92.14% reduction in kurtosis** and an **82.77% reduction in the maximum infinity norm** compared to DNABERT-2. These reductions demonstrate that GERM effectively suppresses extreme values associated with outliers.\n\n  \n\n- **Qualitative Evidence (Attention Distribution Plots):**\n\nVisualizations in **Appendix E.10** in revision illustrate that DNABERT-2 exhibits sharp, irregular attention spikes corresponding to outlier tokens, while GERM maintains a smoother, more stable attention distribution.\n\n---\n\n## **Conclusion**\n\nWe appreciate the opportunity to clarify our definition of outliers and their impact on genomic foundation models. Our theoretical analysis highlights how the attention mechanism illustrates the formation of outliers, while the Softmax1 equation mitigates their influence by reducing the amplification of low-information tokens. The combination of improved theoretical design and strong empirical evidence reinforces GERM’s ability to suppress outliers, enhancing its stability and performance in genomic modeling tasks.", "claims_and_evidence": "The article does not clearly explain what an outlier is or how to quantify it. I suggest that the authors provide quantitative or qualitative evidence to demonstrate that introducing the outlier-free structure can indeed mitigate the outlier phenomenon.", "methods_and_evaluation_criteria": "Yes, they have conducted comprehensive evaluations on various baselines, including downstream fine-tuning and quantization, to support their conclusions.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes, the experimental design here includes a solid and thorough ablation study.", "relation_to_broader_scientific_literature": "Their research contributes to improving the accessibility of genomic foundation models by providing the community with a more lightweight yet efficient model, which helps accelerate scientific research within the community.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tZOf715orP", "reviewer": "Reviewer_8UsM", "summary": "The paper \"Making Genomic Foundation Models more Foundational Requires Outlier Removal: A Case Study on DNABERT-2\" introduces GERM, an outlier-free genomic foundation model (GFM) designed to improve quantization robustness and low-rank adaptation efficiency. The authors argue that eliminating outliers in attention mechanisms significantly enhances computational efficiency and model performance under resource constraints. Empirical results demonstrate substantial improvements in quantization and fine-tuning metrics compared to DNABERT-2. While the work addresses critical challenges in deploying GFMs, several methodological and experimental aspects require clarification and validation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The updated manuscript can be accessed anonymously at [link](https://www.dropbox.com/scl/fi/itpm5n21pfu3at01bofab/germ_icml2025.pdf?rlkey=zl8noukikpz1s4493b752s9uj&e=1&st=qma9ihc9&dl=0).\n> **Reviewer's Comment**: a detailed comparison to ...\n\n**Response**: We thank the reviewer for the feedback and the opportunity to clarify our evaluation of outlier suppression techniques. We would like to highlight that we have already included a detailed comparison of GERM with alternative outlier suppression methods in **Table 23** and **Appendix E.8** of the supplementary material. We are happy to provide further clarification if needed.\n> **Reviewer's Comment**: Definition of the outlier…\n\n**Response**: We appreciate the opportunity to clarify the definition of **outliers** in our work and how it differs from the classical statistical definition. A more detailed explanation is provided in the response to the reviewer `AD8a`.\n> **Reviewer's Comment**: The theoretical analysis in Appendix B assumes…\n\n**Response**: Thank you for the helpful feedback. We acknowledge that our assumptions may not hold universally. However, we believe they remain practical in real-world GFMs for three reasons:\n1. Weight matrices in large foundation models rarely become singular. Overparameterization and regularization typically prevent this.\n\n\n2. The low-rank assumption may not hold in every scenario. Yet it helps us show the strong expressiveness of LoRA tuning transformers with outlier-free layers. Many theoretical analyses use similar assumptions, so they form a reasonable setup.\n\n\n3. Under these conditions, our results show that LoRA-tuned outlier-free transformers match or exceed the expressiveness of softmax-based architectures. These assumptions may not apply everywhere, but they do not undermine our theoretical insights.\n\n> **Reviewer's Comment**: LoRA Experiments...\n\n**Response**: We appreciate the reviewer’s suggestion to disentangle the effects of LoRA and quantization for a more granular analysis. In our revised manuscript, we add additional experiments and analysis for LoRA+Quantization in  **Table 24 and Appendix E.9**.\n\n> **Reviewer's Comment**: larger models available?...\n\n**Response**: Thank you for your advice to include larger models for a more comprehensive evaluation. In our study, we select **NT2.5B** as the largest model for the classification task because it represents the most suitable model architecture designed for genomic sequence classification.\nWhile larger models such as **Evo** and **GeneOcean** exist, they are fundamentally designed as generation models rather than classifiers. These models prioritize sequence generation capabilities rather than directly optimizing for classification accuracy. As a result, they differ significantly in architecture, objective function, and training strategy. So, making a direct comparison with GERM in a classification context is less appropriate.\n> **Reviewer's Comment**: Unclear practical impact of outlier metrics...\n\n**Response**: Thank you for the comment. We clarify that the outlier metrics, such as kurtosis and max infinite norm, are empirically shown to correlate with model quantizability—that is, robustness to performance degradation under quantization [1,3]. Prior studies [2,4] demonstrate that outliers significantly reduce quantized model performance, and thus reducing these metrics has direct implications for improved performance of quantization.\n\n[1] Bondarenko, et al. \"Quantizable transformers\"\n\n[2] Wei, Xiuying, et al. \"Outlier suppression\"\n\n[3] Chmiel, Brian, et al. \"Robust quantization\"\n\n[4] Dettmers, Tim, et al. \"Gpt3. int8 ()\"\n\n> **Reviewer's Comment**: other GFMs should include in literature review...\n\n**Response**: We appreciate the reviewer's suggestion to expand our literature review by including additional GFMs. In response, we incorporate Evo in the related work section of our revised version.\n\n> **Reviewer's Comment**: Incomplete discussion of limitations?...\n\n**Response**: We thank the reviewer’s suggestions regarding the need for a more complete discussion of GERM-T’s limitations. We update the discussion of GERM-T's limitations in our revised manuscript, specifically in **Appendix F**.", "claims_and_evidence": "1. The claim that GERM reduces computational costs and improves quantization robustness is supported by experiments showing a 92.14% reduction in average kurtosis and 82.77% in maximum infinity norm. Also, the authors validate the connection between outlier metrics and practical deployment, such as inference time in resource-constrained environments.\n2. The authors demonstrate that GERM outperforms DNABERT-2 by 37.98% in fine-tuning and 64.34% in quantization, validating the effectiveness of the proposed \"outlier-free\" method.\n3. The small-step continual learning strategy (GERM-T) avoids retraining from scratch, which is pragmatic. However, its performance degradation in 4-bit quantization (Table 1) suggests limitations in outlier mitigation for extreme compression.", "methods_and_evaluation_criteria": "**Outlier-Free Attention**: Replacing Softmax with a modified Hopfield layer is innovative, but it lacks a detailed comparison to other outlier suppression techniques mentioned in supplementary material, such as clipped attention and gated mechanisms.", "theoretical_claims": "1. Definition of the outlier in the context of the paper. Especially considering that classical definition of outlier in statistics and ML is that observation is not from the distribution that is being modeled.\n\n2. The theoretical analysis in Appendix B assumes non-singular weight matrices and ideal low-rank adaptation conditions. While mathematically sound, these assumptions may not hold for real-world GFMs with complex parameter interactions.", "experimental_designs_or_analyses": "The experimental design does not entirely make sense to me. I have several concerns:\n\n**LoRA Experiments:** I would also suggest to separate the experiments to two fold, if LoRA+Quantization is one of the focus on this work. In a first experiment, we can investigate the impact of LoRA (with FP 32/16)\n\n**Model Size**: The largest GFM used in this paper is NT-2.5B. If there are larger models available, I recommend that the authors include them in the experiments to provide a more comprehensive evaluation.", "relation_to_broader_scientific_literature": "This work aligns with efforts to optimize transformer-based models for resource-constrained settings, such as QLoRA and SmoothQuant. The authors also discuss state-of-the-art models like HyenaDNA and NT-2.5B.", "essential_references_not_discussed": "There are some other GFMs should include in literature review like Evo which Integrates Hyena operators for efficient sequence modeling.", "other_strengths_and_weaknesses": "**Strengths**:  \n\n- Novel integration of outlier-free attention with ALiBi for variable-length sequences.  \n- Comprehensive evaluation across multiple quantization methods and adaptation strategies.  \n\n**Weaknesses**:  \n\n- Unclear practical impact of outlier metrics, questions like how kurtosis reduction translates to real-world deployment gains need further discussion.  \n- Incomplete discussion of limitations, particularly GERM-T’s performance trade-offs.", "other_comments_or_suggestions": "I do not have additional comments or suggestions", "questions_for_authors": "Questions are already listed above. I have no further questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission is proposed to address the limitations of the current GFMs, particularly the DNABERT genomic foundation model. It mainly focuses on the outliers and reduces their impact on the attention calculation. The extensive results show the promise of the proposed methods. More in-depth analysis about the influence of outliers could be very helpful. In general, it is a good paper and we recommend to be accepted.
Decision: Accept (poster)