L2G: Repurposing Language Models for Genomics Tasks
Authors: Wenduo Cheng
Junhong Shen
Mikhail Khodak
Jian Ma
Ameet Talwalkar
Venue: Accepted by TMLR ()

Abstract: Pre-trained language models have transformed the field of natural language processing (NLP), and their success has inspired efforts in genomics to develop domain-specific foundation models (FMs). However, creating high-quality genomic FMs from scratch is resource-intensive, requiring significant computational power and high-quality pre-training data. The success of large language models (LLMs) in NLP has largely been driven by industrial-scale efforts leveraging vast, diverse corpora and massive computing infrastructure. In this work, we aim to bypass the data and computational bottlenecks of creating genomic FMs from scratch and instead propose repurposing existing LLMs for genomics tasks. Inspired by the recently observed 'cross-modal transfer' phenomenon -- where transformers pre-trained on natural language can generalize to other modalities -- we introduce L2G, which adapts a pre-trained LLM architecture for genomics using neural architecture search and a novel three-stage training procedure. Remarkably, without requiring extensive pre-training on DNA sequence data, L2G achieves superior performance to fine-tuned genomic FMs and task-specific models on more than half of tasks across multiple genomics benchmarks. In an enhancer activity prediction task, L2G further demonstrates its capacity to identify significant transcription factor motifs. Our work not only highlights the generalizability and efficacy of language models in out-of-domain tasks such as genomics, but also opens new avenues for more efficient and less resource-intensive methodologies in genomic research.

Published in Transactions on Machine Learning Research (08/2025)
L2G: Repurposing Language Models for Genomics Tasks
Wenduo Cheng
wendhucco@@ndrre..cm
Ray and Stephanie Lane Computational Biology Department Carnegie Mellon University
Junhong Shen
junkoongs@endreeu.ccme cww..
Machine Learning Department
Carnegie Mellon University
Mikhail Khodak
mbhooaak@hprincoton..dk
Princeton Language & Intelligence, Princeton Lab
Princcton Language
Jian Ma
fannaaass.
wwt.raaaassccccu..co. cmau. edu
Ray and Slephanie Lane Compulational Biology Department Carnegie Mellon University
Ameet Talwalkar
tabwalkar @emma edu
Machine Learning Department
Carnegie Mellon University
Reviewed on OpenReview: https: /ttps://openreeee openevvee. net/ pet/forum? id 5NM4guc990
Abstract
Pre-trained language models have transformed the field of natural language processing (NLP), and their success has inspired efforts in genomics develop domain-specific founda- tion models (FMs). However, creating high-quality genomic FMs from scratch is resource- intensive, requiring significant computational power and high-quality pre-training data. The success of large language models (LLMs) in NLP has largely been driven by industrial-scale efforts leveraging vast, diverse corpora and massive computing infrastructure. In this work, we aim to bypass the data and computational bottlenecks of creating genomic FMs from scratch and instead propose repurposing existing LLMs for genomics tasks. Inspired by the recently observed 'cross-modal transfer' phenomenon where transformers pre-trained on natural language can generalize other modalities we introduce L2G, which adapts pro-trained LLM architecture for genomics using neural architecture scarch and novel three-slage training procedure. Remarkably, without requiring exlensive pre-training DNA sequence data, T.2G achieves superior performance to fine-tuned genomic FMs and task-specific models on more than half of tasks across multiple genomics benchmarks. In an enhancer activity prediction task, L2G further demonstrates its capacity to identify sig- nificant transcription factor motifs. Our work not only highlights the generalizability and efficacy of language models in out-of-domain tasks such as genomics, but also opens new avenues for more efficient and less resource-intensive methodologies in genomic research.
1 Introduction
In recent years, large-scale pre-trained models, often referred to as foundation models (FMs) (Bommasani et al.] 2021). have revolutionized the field of natural language processing (NLP). Models such as BERT (De vlin al. 2019), LLaMA (Touvron et al.] 2023), and GPT (Brown al.) (0200 leverage self-supervised pre-training on vast amounts of unlabeled text data to develop rich and nuanced understanding of lan- guage. Similarly, DNA sequences be viewed strings of nucleotides, with recurring patterns analogous

---

Published in Transactions on Machine Learning Research (08/2025)
to reusable elements in natural language. This parallel has inspired the development of genomic FMs, such as DNABERT (Ji et al. 2021), Nucleotide Transformer (Dalla-Torre et al. 2023), and HyenaDNA (Nguyen (et al. 2024), which are pre-trained on large-scale genomic sequence data with subsequent fine-tuning These models have shown potential in predicting complex genomic features, such as regulatory elements, chromatin accessibility, and gene expression.
However, constructing these domain-speciffe FMs from scratch is costly and resource-intensive For instance, training the Nucleotide Transformer required approximately 174 billion tokens and 28 days of continuous training 128 NVIDIA A100 GPUs. Even smaller models like DNABERT-2 required two weeks of training on 8 GTX 2080 Ti GPUs (Zhou et al. 2023b). More recent architectures with fewer parameters than transformer-basee models, such as HyenaDNA (Nguyen et al. 2024 and Mamba (Gu & Dao 2023), still require substantial compute budget and massive training corpora (see Fig. 1 and Table for details).
To address these challenges, we explore an alternative strategy to bypass genomic pre-training altogether. Our framework, Language-to-Genonee (L2G),, adapts pre-trained language models to genomic prediction tasks. This work is motivated by recent advances in the cross-modol transfer paradigm, which leverages the general reasoning capacity of pre-trained LLMs in domains such as protein property prediction (Vinod ct al. 2023 and solving partial differential equations (Shen et al., 2024a). These studies demonstrate that translerring existing models from well-studied text and vision modalities to scientific applications holds the promise of not only drastically reducing the required computational and data resources associated with pre- training, but also improving downstream model performance. L2G builds on general-purpose cross- mrosll transfer approach (Shen et al. 2023) but incorporates neural architecture search (NAS) and novel three- stage training procedure (Fig. 2 to adapt to the unique features of genomics data, significantly enhancing
empirical effectiveness. Fig.1 contrasts L2G with the traditional approach of building genomic FMs.
Sources
Downstream Tasks
L2G
75
i-modal
Enhancer Activity
Caduceus
DNA
Mean Test Score (%)
TECTCTCCTCG
ATGRATTTAIC
DNA
FM
70
HyenaDNA
NT
DNABERT-2
Pre-training
Fine tuning
Histone Marks
65
60
DNABERT
Text discovered principlles Inheritance 1865'
Language FM
Crose-modal with L2G
Gene Splicing
Enformer
55
10a
1000
10
Num of genomics data (log scale) used for pretraining and fine-tuning
Figure 1: L2G is a data-efficient cross-modal fine-tuning method for genomics. A. Schematic overview contrasting genomic foundation models (FMs) with L2G during pre-training and fine-tuning. Genomic FMs pre-train with massive DNA sequencing data, while L2G bypasses genomic-specific pre-training altogether by leveraging existing pre-trained language models. Both approaches perform fine-tuning for specific down- stream genomic task, but L2G's three-stage cross-modal fine-tuning workflow on par with vanilla fine-tuning in terms of compute and data requirements. B. L2G (red dot) achieves higher mean test score than leading genomic FMs on the Nucleotide Transformer benchmark, with higher values indicating better performance. By skipping pre-training, L2G requires significantly less genomic data and computational resources.
The advantages of L2G are three-fold. First, by sidestepping pre-training, L2G is much more data and compute efficient. For a given downstream task, our three-stage workflow is comparable to vanilla fine-tuning in terms of required compute and data resources. All our experiments can be performed on a single A6000 GPU in matter of hours by leveraging existing open-source language models, compared days of training needed to develop genomic FMs from scratch. Second, L2G demonstrates better average performance than
2

---

Published in Transactions on Machine Learning Research (08/2025)
fine-tuned genomic FMs various genomies benchmarks (Fig. 13. including GenomicBenchmarks (Grešová et al. 2023 and NucleotideTTanssfomeee Benhhemaeeeeeeereeeee (Dalla-Torre et al. 2023). Third, we show that T.2G can tackle challenging regulatory activity prediction tasks, such as predicting developmental and housekeeping enhancer activity, where it consistently outperforms expert-designed modcls. Additionally, L2G learns relevant transcription lactor motifs.
Overall, L2G leverages pre-trained LLMs for genomic prediction, achieving performance competitive with in-modal transfer various genomic tasks while bypassing the massive costs associated with collecting and processing large amounts of unsupervised genomic sequencing data.
2 Methods
2.1 Cross-modal fine-tuning
Fine-tuning has been a highly effective tochnique for adapting pre-trained language models to various down- stream tasks. However, most existing research focuses on in-modal adaptation, where the fine-tuning data originates from the same modality as the pre-training data but is tailored for a more specialized focus, such as sentiment analysis or text classification. In such cases, the model operates on the same type of input it was originally trained on. In contrast, cross-modal finc-tuning adapts pre-trained model to work with data from unseen modalities, such using lext-pre-trained LLMs to address biological quest Fig. D illustrates the distinction between in-modal and cross-modal fine-tuning strategies for genomic tasks.
Cross-modal fine-tuning is more challenging than the in-modal fine-tuning due to the modality gap between the pre-training and target. task data (Tuu et al. 2022). Bridging this gap often requires additional data align- ment. For instance, to repurpose a pre-trained BERT model for predicting physicochemical and biomedical properties of protein sequences, Vinod al. (Vinod al.) 2023) introduced R2DL (Representation Repro- gramming via Dictionary Learning), token-level alignment method that learns sparse linear mapping between English vocabulary embeddings and amino acid embeddings.
Recently, Shen et al. (0223) proposed a more general distributional alignment technique, ORCA, that adapts various pre-trained transformer models to diverse non-text, non-vision inputs. ORCA employs a convolu- tional neural network to transform input data into sequence features, minimizing the distribution distance between target data embeddings and standard English token embeddings prior to fine-tuning. ORCA achieves state-of-the-art results three benchmarks rks containing over 60 datasets from 12 modalities, outperforming wide range of general-purpose, automated machine learning (AutoML) and task-specific methods. However, genomics is one domain where ORCA does not show superior results (Shen et al. 2023). Specifically, on DeepSEA- - a well-known dataset for predicting functional effects of genomic sequence ORCA falls behind non-pre-trained AutoML baselines. This motivates us to study how cross-modal alignment can be improved specifically for genomic prediction tasks.
Beyond ORCA, several other studies have also proposed different cross-modal fine-tuning strategies (Cai al. 2024 Ma et al., 2024 Shen et al. 2024b Zhou et al., 2023a Chang et al.] 2024 Roberts et al. However, they are not tailored to biological domains. To address this gap, we aim to develop the first cross-modal fine-tuning framework specifically designed for genomic applications.
2023
2.2
Motivation for the framework
Since ORCA is the only previous method that has attempted genomics (asks, we thoroughly examine its workflow to identify limitations. Specifically, ORCA first creates custom embedder and predictor networks to support various tasks. The embedder is trained to minimize the optimal transport dataset distance (OTDD) between a target and proxy dataset, aiming to map the target dataset into the embedding space of the pre-trained model. Finally, the entire model - comprising the embedder, transformer, and predictor - is fully fine-tuned on the target task data to adapt the pre-trained model to the target modality.
However, ORCA has two major limitations when applied to genomics both model-wise and training wise. Model-wise, ORCA employs a universal CNN structure the input embedder for all tasks. This embedder,
3

---

Published in Transactions on Machine Learning Research (08/2025)
consisting of a single-layer CNN with small kernel sizes and strides designed for computer vision, may not be well-suited for genomics tasks. It cannot effectively model the long-range dependencies of genomic sequences and may fail to capture important features from genomic datasets. To address this, we propose redesigned embedder architecture tailored for genomics data.
Training-wise, by reproducing ORCA experiments on DeepSEA, we revealed that a lower alignment loss at the end of embedder training does not necessarily lead to better downstream performance on genomics tasks. This suggests more complex dynamic between embedder training and fine-tuning than the ORCA paper indicated. For instance, in many cases, training the embedder for longer epochs can hurt the final performance on the target task. We hypothesize that this occurs becausc a single alignment loss is insufficient for effective embedder training; closer mapping to text embeddings can result in the loss of important class information in the target genomic dataset. To address this, we propose a new embedder training objective that jointly optimizes for both distribution alignment and downstream task performance. Another possible factor is overfitting, as ORCA relied on a relatively small and imbalanced source dataset during the alignment step. To mitigate this, we sampled additional data points from the source dataset, ensuring they were evenly distributed across different categories.
By tailoring the embedder architecture and objective design to genomics data, we significantly improve empirical performance and develop a new cross-modal fine-tuning workflow, named L,2G, to effectively adapt language models for genomic applications.
2.3 Model design
2.3.1
Problem Setup
A modality M consists of a feature space X, a label space y, and a joint probability distribution P(X, ))). We focus on the cross-modal setting in this paper. That is, the target genomics modality M, and source language modality M2 have different feature spaces, label spaces, and joint probability distributions, i.c., X # X², }d + ye, and P(X*,)') + ((X"))") Our goal is to adapt a model pre-trained in M, to the tasks in Mt.
Following previous work (Cai et al. 2024 Shen et al.; 2023), the model architecture of L2G is composed of three parts: a CNN embedder, a transformer encoder, and a linear predictor (Fig. 2 The embedder maps input genomics data to an embedding space, the encoder extracts fcatures from these embeddings, and the predictor maps the encoder output the label space.
2.3.2 Embedder
Denote f as the source embedder language model, which transforms the source raw data X' into source language embeddings heext xD where N denotes the embedding length and denotes the embedding dimension. Following ORCA, we use the CoNLL-2003 dataset (Sang & De Meulder 2003) as the reference dataset for text. This dataset contains nine classes for a named entity recognition (NER) task, from which we sampled 350 data points per class, resulting in total of 3,150 data points. The source language embedder is taken from a pre-trained language transformer and remains frozen during the entire training process. Reference data is passed through the embedder to obtain reference embeddings for alignment.
Denote I as the custom target embedder, which transforms the target genomics sequence data in X into target emboddings I.DNA RND. The key to cross-modal transfer to learn f so that it maps h DNA into the shared representation space with text, As mentioned above, existing work typically uses generic small-kernel convolutional layer for ft, which is unsuitable for modeling long-sequence genomics data. To address this, we propose using larger, more capable dilated CNN as the backbone architecture for ft.
We chose CNN as the backbone for DNA embeddings because CNNs are well-suited for DNA embeddings as evident in previous work. Landmark models like DeepBind (Alipanahi et al. 2015) and DeepSEA (Zhou & Troyanskaya 2015) used shallow CNNs predict protein binding and chromatin features, and they remain standard baselines in computational genomics. More recent architectures [Kelley et al.] 20188 Fudenberg
et al. 2020
Avsec et. al. 2021b Yang & Ma 2022) build on this foundation by incorporating dilated
4

---

Published in Transactions on Machine Learning Research (08/2025)
convolutions residual connections to capture longer-range dependencies, but the remains central. CNNs excel at learning local patterns, and in genomic data, they can effectively identify short regulatory patterns, such as transcription factor binding motifs, splice signals, and repeat elements, regardless of their position within a long genomic sequence. Unlike prior work, which fixes the convolution hyperparameters (e.g., kernel size and dilation rate) for the model architecture before seeing the task and data, we employ a data-driven approach that automatically learns the architecture configuration from the end task data (see later section). This new approach effectively improves downstream task performance.
of convolutions
2.3.3 Transformer Encoder
A Architecture
The transformer encoder, denoted as g. takes ADNA as input and outputs in- termediate representations (last hidden states) inttermediate RND While L2G is compatible with various lan- guage models, chose RoBERTa-base in our experiments in this work because its model size smaller than or compa- rable to most transformer- based genomic FMs, such as DNABERT, D DNABERT-2, Enformer, and Nucleotide Transformer- 500M. This choice ensures fair compari- son of different methods. The embedding dimension D for RoBERTa-base is 768.
AOW DNA
DNA Embedder
Transformer Encader
Linear Predictor
0.6 0.4
Predictions
Language Embedder
Trainable Frozen
Text
B
Complete Training Workflow
Stage 1
Stage DNA Embedder Pre-trsining
Stage 3 Supervised Fine-tuning
Neural Architecture
Tasc-fpecific DNA Embedder
Closing the Modality Gap
Final Prodictions
C Details on Stage 2 Embedder Pretraining
xD4
DNA Embedder
0
Predictor
DNA
Predictions
Labels
2.3.4 Linear Predictor
Language Embedder
h
Trainable
The predictor, denoted as p', takes Intermediate input and returns task- specific output tensor. The goal of p' is to map the learned representations to the desired output dimension. Following ORCA (Shon et al.) 2023), we use aver- age pooling along the sequence length di- mension. A single linear layer then trans- forms the pooled outputs of the language models to produce the final prediction.
Frozen
Text
min
Dow
min
(Predictions, Labels)
The overall workflow. A. The architecture of L2G. The architecture of the L2G model composed of CNN em- bedder, transformer layers from pre-trained language model, and linear predictor. B. The model is trained in three stages. In stage 1, L2G performs a Neural Architecture Search to op- timize the embedder architecture for a given task. In stage 2, the CNN embedder is pre-trained to minimize the modality gap between DNA embeddings and language embeddings. In stage 3, the entire model fine-tuned task-specific data in su- pervised manner by minimizing the task-specific loss between the final predictions and the true labels. C. In stage 2, L2G closes the modality gap by pertaining the embedder with joint objec- tive. The CNN embedder is attached to linear predictor and trained with a joint objective, which simultaneously minimizes a) the distribution distance between language embeddings (huen) and DNA embeddings (hDNA), and b) the task-specific loss be- tween predictions from the lincar predictor and the true labels. After the embedder pretraining, we retain only the encoder, at- tach it to the transformer backbone and new linear predictor, and then perform final fine-tuning.
Figure 2:
2.4 Model Training
L2G trained three stages: neural ar- chitecture search, embedder pre-training, and fine-tuning (see Algorithm and Fig. 2)
2.4.1 Neural Architecture Search
Neural Architoctural Scarch is a machine learning technique that automates the design of deep neural network architec- tures. Instead of relying substantial manual efforts by human experts, NAS identifies architectures that perform well on given task through algorithmic solutions (Liu et al. 2018 Given its success in certain genomics applications (Zhang et al. 2021), we utilize NAS to tailor embodder architectures different tasks.
5

---

Published in Transactions on Machine Learning Research (08/2025)
A Algorithm Pseudocode for the L2G workflow.
To achieve optimal performance across various downstream tasks, we use two-step process for selecting the embedder network in L2G. First, we select an optimal backbone CNN ar- chitecture from a pre-defined search space. In this study, we consider ResNet (He et al. 2016) and UNet (Ronneberger et al., 2015 both of which have been effectively applied in ge- nomics. ResNet, with its deep residual connec- tions, is well-suited for classification tasks that require capturing hierarchical features from se- quential data. In contrast, UNet's U-shaped encoder-decoder structure with skip connec- lions makes it particularly effective for dense prediction tasks. Each architecture is trained for one epoch, and the one achieving the high- est validation score is selected as the backbone.
Input: Genomic Datasct G, Set of Embedder Backbone Architec- tures B, Language Model L, Alignment Loss Weight a, Task- Specific Loss Weight 8 for each architecture E B do
Initialize
eal scoree + Train for one epoch on G
best baargg a _at__serre ;
// Select the embedder backbone with the best validation score
d) + DASH( [_]] // Optimize the kernels and dilations h text 4- Inference L on the source text dataset : // Generate
text embeddinga Initialize best with (k for epoch E embedder epochs do
pred 1 DNA <- best b(G)
loss 1 LMMD(h text, AN
loss 2 Lrask (pred 1, l,labets)
embedder <- min(( loos__111 + 3 lloss_ss
model + embedder + transformer blocks from + linear predictor
pred 4 Train model on return pred_
Next, I.2G applies NAS to optimize layer op- erations for the specific task. Specifically, we learn the optimal kernel size and dilation rate for each convo- lutional layer in the CNN using the DASH (Diverse-task Architecture Search) algorithm (Shen et al., 2022). which has demonstrated state-of-the-art, performance among AutoMT. methods on the DeepSEAA dataset. After this two-step process, both the backbone architecture and the convolutional layers of the embedder f³ are tailored for the target lask, effectively capturing meaningful larget embeddings from genomics datasets.
2.4.2 Embedder Pre-training
The embedder pre-training stage critical for minimizing the modality gap between DNA and the pre- trained language models, enabling cross-modal adaptation. We propose a joint objective to address the training limitations discussed carlier. The first objective minimizes the distribution distance between DNA and text data, performing modality aligument. Unlike ORCA (hen et al. 2023), which uses OTDD loss, we utilize Maximum Mean Discrepancy (MMD) as the distance metric in L.G due to its better empirical performance in our ablation studies (Table 9).
Additionally, we introduce a second objective - a task-specific loss (Ltask) during embedder pre-training, This enables the embedder incorporate class information while performing distribution alignment. This task-specific loss either cross-entropy loss for classication tasks or Mean Squared Error (MSE) for regression tasks. By including Lrask, the embedder learns to model the class information effectively when mapping genomics data to the language model's embedding space. Training with only alignment loss but not task- specific loss can result in worse downstream performance, as shown in the ablation studies in the Results section.
In summary, the embedder pre-training objective is defined as:
Liotal = aCLMD (Manugggeeagg
BLesek (Y, 8),
where a and B are weights for the MMD loss and task-specific loss, respectively. To optimize performance, set up scheduler for these weights, minimizing the task-specific loss first before minimizing the joint objective. We did this because empirically, introducing the alignment loss in later epochs of pre-training embedders achieves the best performance. Specifically, we set the aligument loss weight a = 0 during the first two thirds of the total epochs of pre-training, effectively optimizing only the task-specific loss Crask in the carly stage. We then set a to a positive value and optimize the full joint objective. This staged approach empirically improves performance, as delaying the alignment loss allows the embedder first learn meaningful task-relevant representations before enforcing modality alignment via LMMD-
6

---

Published in Transactions on Machine Learning Research (08/2025)
2.4.3 Fine-tuning
After pre-training the embodder, the entire model including embedder, transformer encoder, and linear predictor fine-tuned using task-specific loss on the larget data. To optimize the hyperparameter config- uration (e.g., learning rate, dropout rate, weight decay) for fine-tuning, we use ASHA (Li et al.) 2020).
3 Results
3.1
Overview
Directly applying transformer models trained on natural language data to out-of-domain tasks like genomics degrade the quality of the pre-trained weights, resulting in inefficiencies and inaccuracies due to the fundamental mismatch between the two modalities. To address this, developed L2G, an effective and efficient workflow designed to repurpose pre-trained language models for genomics tasks through cross rross-modal transfer learning. Unlike traditional in-modal transfer learning, where transformer models are first pre- trained on large-scale DNA sequencing data before fine-tuning, our approach is not only more competitive in prediction quality but also significantly more efficient. L2G eliminates the need for large-scale self-supervised pre-training, reducing both data and computational requirements while still generalizing effectively across a variety of genomics tasks through funing.
We demonstrate the empirical effectiveness and efficiency of L2G through extensive experiments on two genomics benchmarks and challenging regression lask for enhancer activity prediction. Beyond presenting results on predictive accuracy, we assess L2G's ability to learn relevant TF motifs and evaluate the efficacy of cross-modal finc-tuning through embedding analyses and ablation studics.
3.2 L2G matches or outperforms fine-tuned genomic FMs
Predicting the regulatory function of non-coding DNA based on its sequence is cru- cial for prioritizing functional non-coding variants and remains a major challenge in ge- nomics Hiill et al., 2023 Zhou & Troyanskaya 2015). We evaluated L2G on two ex- isting benchmarks, Genomic Benchmarks [Gresová et al. 2023 and the Nucleotide Trans- former Benchmarks
Dalla- Holla-Torre al. 2023), demonstrate generalizability and efficacy.
We first evaluated L2G on the Nucleotide Trans- former Benchmarks (Dalla- Torre et al.) 2023), one of the most widely used benchmarks for genomic FMs. This benchmark suite includes eighteen tasks for predicting regulatory elements from four categories: enhancers, promoters, epi- genetic marks, and splice sites from DNA quences with lengths ranging from 300 to 600 bp (Table 3). We compared I.2G against sev- eral representative genomic FMs, including En-
DNABERT-1 DNARFRT->
Enformer
HyynaaDNA
Caduceus
Average Test
a1
83.2
97.0
61.2
90.6
66.4
92.0
80.4 21.5
26.2
96.7
61.0
63.5
67.0
former (Avsec et al. 2021a), DNABERT-1 (Ji et al.
55.1
47.4
49.3
46.7
2021), DNABERT-2 (Zhou al. 2023b), H HyenaDNA (1kb) (gguyen et al. 2024). Nu-
36.3
cleotide Transformer Multispecies (2.5B) and Caduceus-ph (Schiff et al.) 2024), which outper- forms Caduceus-ps in most of the tasks (Dalla- Torre et al. 2023). All these models have been pre-trained and then fine-tuned.
Splico
Figure 3: Average performance across task types (his- tone marks, enhancers, promoters, and splice sites) on the Nucleotide Transformer benchmarks. Fach bar repre- sents the average test score of a model. L2G outperforms all other models on histone marks and enhancers tasks and ranks second on promoters and splice sites prediction tasks. 7
The complete benchmarking results are dis- played in Table 4 and Fig. 3 L2G achieves the best results ten tasks and ranks second

---

Published in Transactions on Machine Learning Research (08/2025)
six others. It demonstrates a clear advantage in predicting histone marks and enhancers from DNA sequences, outperforming all other genomic FMs. For promoter and splice site prediction tasks, the Nucleotide Transformer is the top-performing model, followed by L2G. It is worth noting that the Nucleotide Transformer is the largest model evaluated on this bench- mark, with 2.5 billion parameters and the most extensive pre-training data, while L.2G uses significantly fewer parameters and no pre-training data. Overall, L.2G achieves the highest average test score on the Nucleotide Transformer benchmark with the least training (pre-training + fine-tuning) data, underscoring its ability to generalize effectively across tasks even with limited data.
The Genomic Benchmarks dataset (Grešová et al. 2023) includes eight classification tasks: seven binary and one three-way classification task. These tasks focus on predicting regulatory elements such as promoters, enhancers, and open chromatin regions from several species, inchuding humans and Drosophila (Grešová
et al. 2023 Marin et al., 2023 Nguyen et al.] 2024). The inputs sequences have median lengths ranging from 200 to 2,381 bp. This benchmark includes three baselines: a supervised CNN model, a supervisod transformer model, and fine-tuned genomic FM, HyenaDNA (Nguyen et al., 2024]. The results are shown in Table 5 L2G outperforms all other models in five out of eight tasks and is the second best in the remaining three, slightly behind HyenaDNA. As demonstrated by the aggregated results using performance profiles Fig. 6C, L2G achieves the best overall performance on the Genomic Benchmarks. Additionally, we evaluated L.2G on a challenging cell-type-specffic element classification task from the very recent DART- Eval Pateel et al. 2024) benchmark, with the results shown in Table 6 L2G outperformed all fine-tuned genomic FM basclines and the ab initio basclines.
Overall, 1.2 matches or outperforms fine-tuned genomic FMs across variety of regulatory element predic- tion tasks. This is particularly significant L2G does not rely extensive pre-training on unsupervised DNA data, a standard practice for most genomic FMs. These results highlight the efficacy of the cross- modal transfer learning approach employed by L2G, which effectively leverages the pre-trained knowledge embedded in language models to address genomics tasks.
3.3 L2G reveals transcription factor motifs
While we have demonstrated the strong benchmark performance of L2G, we also sought to showcase its utility in downstream applications, such as discovering functional regulatory syntax. Here, we focused on regression task to predict the activities of developmental and housekeeping enhancers (de Almeida et al., 2022 from DNA sequences. Using the DeepSTARR dataset (de Almeida et al. 2022), which predicts enhancer activity for two distinct promoters in Drosophila S2 cells, we compared L2G to baseline methods.
The comparison included the DeepSTARR model, an adaptation of the Basset convolutional ncural net- work (Kelley et al. 2016), as well as several genomic FMs. Full results are presented in Table 7 Fig. A shows that predictions by T.2G align well with measured values for both developmental (PCC=0.66) and housekeeping (PCC-0.76) enhancers. Compared to other models, L2G outperforms all others in the house- keeping enhancer prediction task and ranks second in the developmental enhancer prediction task, slightly behind DeepSTARR (PCC=0.68).
To determine whether L2G learned regulatory syntax, we quantified how each nucleotide contributes to predicted enhancer activities using DeepLiftShap (Scott et al.) 2017) and identified predictive sequence patterns with TF-Modisco-lite (Shrikumar et al. 2018). Full motif results are shown in Fig. S and Fig. g Interestingly, although both L2G and DeepSTARR achieve high PCC values in predicting developmental and housekeeping enhancer activities, they identified different sets of TF motifs (Fig. 4B).
For developmental TF motifs, both models identified AP-1, GATA and SREBP, but L2G uniquely revealed the da motif. The daughterless (da) gene, part of the basic helix-loop-helix (bHLH) family, is essential for several developmental pathways, including sex determination and neurogenesis (Caudy el al., 1988). For housekeeping TF motifs, L2G uniquely identified BEAF-32 and CRP. The Drosophila Boundary Element- Associated Factor (BEAF) of 32kDa primarily binds near the promoters of numerous housckceping genes, contributing to chromatin domain boundary activity and promoter function (Jiang et al. 2009 Bushey al. 2009). Similarly, the CRP motif may be associated with housekeeping promoters (Zhimulev et al.] 2024).
8

---

Published in Transactions on Machine Learning Research (08/2025)
We acknowledge that differences in identified motifs may arise from several factors. First, used different implementations of DeepI.FFTShap and TF-MoDISco algorithms, DeepSTARR is based on Keras and TensorFlow, while our implementation uses PyTorrc. These framework differences can contribute to varia- tions in motif interpretation and sensitivity. Additionally, DeepSTARR's simpler CNN-based architecture is inherently easier to interpret than L.2G's CNN-Lransformer hybrid model, and hyperparameter variations, such as the similarity thresholds for merging patterns, could also affect motif detection.
Nevertheless, demonstrate that L2G predict enhancer activities and reveal relevant TF motifs as- sociated with developmental and housekeeping enhancers, suggesting it is effective in identifying important sequence patterns for prediction tasks.
A
Dev TF Motifs L2G
Hk Motifs
PCC: 0.76
DeapSTARR
Predicted (lg2))
Predicted (ogzz
STAT
GATA
BEAF-32
Ohleri
SREBP
twist
crp
Ohler7
Ohlerr
AP-1
ETS
Measured (log2)
(log2)
C
Enhancer
SREBP
da
da
chr3R: 887 8877679
0.2
0.0
Hk Enhancer
crp
BEAF-32
chr2R: 7870334-7870582
80
-0.5
20
180
220
240
Figure 4: L2G predicts enhancer activities and reveals TF motifs. Performance of L2G on predicting developmental and housekeeping enhancer activity from DNA sequences in Drosophila S2 cells, measured using Pearson Correlation Coefficient (PCC). Scatter plots show predicted observed enhancer activity for developmental (left) and housckeeping (right) enhancers. B. Venn diagrams indicating the common and unique TF motils identified by L2G (blue) and DeepSTARR (red) for developmental (left) and housekeeping enhancers (right). Motif identified by DeepSTARR were retrieved from (de Almeida et al. 2022), with un- known and redundant motifs excluded. While both methods achiove high PCC values in predicting enhancer activities, they identify different sets of motifs. C. Nucleotide contribution scores for strong developmental (top) and housekeeping (bottom) enhancer sequences, respectively.
3.4
L2G closes the modality gap
L2G bridges the modality gap through joint loss optimization during the embedder pre-training step, si- multancously aligning distributions between text and DNA while optimizing downstream task performance. This approach addresses the challenge where directly fine-tuning language models on genomic tasks often results in weight shifts and poor performance. To evaluate the effectiveness this strategy, we examined the lcarned representations of the target modality data generated by different fine-tuning approaches. Specifi- cally, we selected three binary classification tasks from the Nucleotide Transformer benchmark and visualized the learned embeddings for the two classes. We also calculated the Silhouette Score, which quantifies cluster

---

Published in Transactions on Machine Learning Research (08/2025)
separation. Scores range from -1 to +1, with +1 indicating well-separated clusters, 0 suggesting overlap- ping clusters, and -1 indicating incorrect class assignments (Rousseeuw 1987) Across all three tasks, T.2G achieved higher positive Silhouette Scores (Fig. 7 demonstrating improved class separation compared to vanilla fine-tuning. This clear distinction in embedding space resulted in better performance.
To better understand the factors contributing to the success of cross-modal fine-tuning I.2G, we conducted a series of ablation studies on selected tasks from the Nucleotide Transformer benchmark, H3, Enhancers, and Promoters TATA.
First, to validate the value of a pre-trained LLM backbone and the effectiveness of cross-modal adaptation in L2G, we compared two configurations: using a pre-trained RoBERTa- base transformer versus a randomly initialized backbone trained from scratch. As shown in Table g and Fig. 5A, L2G with the pre-trained transformer consistently outperforms training from scratch. This confirms that L2G benefits from cross- modal adaptation and that its gains are not simply due to the large size and capacity of the transformer.
We then examined the impact of different distribution alignment metrics, specifically Maximum Mean Dis- crepancy (MMD) and Optimal Transport Distance for Distributions (OTDD). In principle, OTDD's joint matching of features and labels should capture richer cross-modal structure. However, on our genomics tasks, MMD alignment consistently outperforms OTDD (see Table 9 and Fig. 5B). We believe this gap stems mainly from OTDD's optimization difficulty: the MMD loss decreases smoothly and converges quickly, whereas OTDD is much harder to learn and yields noisier gradients empirically. Thus, although OTDD offers theoretical appeal for joint- disrriistiinttinn matching, its practical use in genomics-language embedding spaces is hampered by training instability. In contrast, MMD's simpler optimization landscape makes it a more reliable alignment loss in our setting. Addressing OTDD's optimization challenges would be intcresting direction for future work.
Next, we examined the impact of loss functions during embedder pre-training by comparing joint optimization of task-specific and MMD losses against using only the task-specific loss or only the MMD loss. Table I0 and Fig. 5D confirm that joint optimization consistently produces the best performance, demonstrating the benefits of optimizing both alignment and task objectives together during embedder pre-training.
Finally, we evaluated the optimal choice of embedder architecture by comparing the neural architecture search method DASH, used in L2G, with four alternatives: unsearched UNel, unsearched ResNet, the domain-specific CNN model DeepSEA and the embedder from the original ORCA model (Shen et al. 2023). Table I and Fig. 5C indicate that using unsearched UNet or ResNet as embedders already yields strong results, but DASH provides more robust performance and additional gains, although the magnitude of this benefit varies by dataset.
The results of these ablation studies showed that the pre-trained transformer, using MMD as alignment loss, joint loss optimization, and the DASH-based embedder consistently outperformed their respective alterna- tives (Fig. 5) These findings highlight the critical role of pre-training, optimized loss functions, and robust embedder architccttre in the success of L2G.
4 Discussion
In this work, we investigate the efficacy of modal transfer in genomics. By analyzing a general-purpose cross-modal fine-tuning method, we identified key limitations in both architectural design and objective function. To address these challenges, we introduced L,2G, new method that incorporates a carcfully designed architecture and improved alignment between different modalities. This enables L2G to harness the capabilities of pre-trained language models for genomics tasks. Our evaluations across multiple genomics tasks demonstrate superior average performance compared to fine-tuned genomic FMs and domain-specific expert models, notably without requiring large-scale pre-training
The success of our cross-modal methods raises important questions: Is the current pre-training approach in genomics the most effective? Do we truly need vast amounts of unsupervised genomics data for pre- training? By leveraging pre-trained language models, L2G bypasses the need for extensive unsupervised pre-training, reducing computational and data demands while still achieving competitive performance with
10

---

Published in Transactions on Machine Learning Research (08/2025)
100 A
8
100
80
00
80
80
MCC/F1 (1)
NCC F2 (%)
60
Loss (N)
60
60
60
MCC
DASH
20
Joint losses
Uhot
RosNet
DeepSEA ORCA
Scratch
OTOOD Loss
MMD Loss Alone
0
H3
Enharcera
Promoters TATA
H3
Enhancers
Promaters TATA
H3
Entancere
Promsher TATA
H3
Enhancers
Promatert TATA
Figure 5: Ablation results for three selected lasks in the Nucleotide Transformer benchmark. A: Performance comparison between a pre-trained RoBERTa- base transformer and a randomly initialized backbone trained from scratch. B: Evaluation aligument losses, comparing MMD and OTDD. C: Impact of different loss functions during embedder pre-training, comparing joint loss optimization, task-specific loss alone, and MMD loss alone. D: Comparison of embedder architectures, including DASH, UNet, ResNet, DeepSEA, and the original ORCA embedder.
in-modal transfer. This challenges the conventional approach of building domain-spocific FMs from scratch, suggesting that language models originally developed for NLP be repurposed for seemingly unrelated domains like genomics. This opens avenues for cross-disciplinary applications of LLMs, highlighting their versatility and questioning the necessity of developing entirely new models for every domain.
A few contemporaneous studies have also raised concerns about the effectiveness of current genomic FMs. For example, one study observed that genomic FMs offer little to no advantage over traditional models based on one-hot encoded sequences (Tang et al.) 2024). Another work found that a supervised-only pipeline named DASHA surpassed the latest genomic FMs on the Nucleotide Transformer benchmark (Xu et al., (224)). Notably, L2G outperforms DASHA 11/18 tasks in the benchmark and reaches better average score. At the time of this writing, L2G is the only fine-tuning based approach that outperforms such strong supervised baselines on the Nucleotide Transformer benchmark, despite not being pre-trained on genomics data. Our work along with these contemporaneous studies collectively challenge the prevailing pre-training seehiinngthee---e then-fine- tuning paradigm for genomic FMs and highlight the need to rethink their development and applications.
Our study has several limitations. First, our evaluations did not cover broad range of genomics tasks, including long-range prediction tasks that involve more complex interactions and regulatory mechanisms, Future work could extend L2G's application to more complex genomic tasks, such sequenco-based gene expression prediction (Kelley al. 2018 Avsec al. 2021a). Second, have not explored whether the scaling laws common in NLP apply to cross-modal transfer learning for genomics. Tt remains to be seen whether using increasingly larger language models (trained on natural language) would yield proportionally better performance. Third, interpretability remains a challenge (Chen et al.) 2024). While we have used ablation studies and embedding analyses to explain T.2G's 's effectiveness, the underlying mechanisms and in- terprctability of L2G require further investigation. Lastly, our current cross-mode transfer approach relies fine-tuning. Future work could explore combining cross-modal transfer with continued pre-training (Gu- rurangan al. 2020), leveraging both unsupervised text and genomic data to potentially further enhance the performance of domain-specifie FMs, albeit at the cost of increased data and computational requirements.
In summary, L2G demonstrates the patential of cross-modal transfer learning to address genomics tasks effectively and efficiently, providing a compelling case for leveraging existing pre-trained models from natural language rather than building domain-specific ones from scratch. This work lays the foundation for further advancements in cross-disciplinary applications of pre-trained language models, extending their utility to a diverse range of biological problems.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Acknowledgments
This work was supported in part by the National Institutes of Health Common Fund 4D Nucleome Program grant UM1HG011593 (J.M.), National Institutes of Health Common Fund Cellular Senescence Network Pro- gram grant UH3CA268202 (J.M.), National Institutes of Health grants ROLHG007352 (J.M.), RO1HG012303 (J.M.), R21DA061481 (J.M.), and U24HG012070 (J.M.), as well as National Science Foundation grants IS1170511 (A.T.), IIS1838017 (A.T.), IIS2046613 (A.T.), and I1S2112471 (A.T.). J.M. was additionally supported by the Ray and Stephanie Lane Professorship, Guggenheim Fellowship from the John Simon Guggenheim Memorial Foundation, a Google Research Award, and a Single-Cell Biology Data Insights award from the Chan Zuckerberg Initiative. A.T. received additional funding from Meta, Morgan Stanley, Amazon, Google, and Scribe. Any opinions, findings, conclusions, recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of any of these funding agencies.
References
Babak Alipanahi, Andrew Delong, Matthew T Weirauch, and Brendan J Frey. Predicting the sequence specificities of dna-and rna-binding proteins by deep learning. Nature biotechnology, 33(8):831-833) 838, 2015.
Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R. Ledsam, Agnieszka Grabska SaanskaaBarwiskaa, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R. Kelley. Effective gene expression prodiction from sequence by integrating long-range interactions. Nature Methods, 18(10):1166-1208, 2021a.
Ziga Avsce Mclanie Weilert, Avanti Shrikumar, Sabrina Krueger, Amr Alexandari, Khyati Dalal, Robin Fropf, Charles McAnany, Julien Gagneur, Anshul Kundaje, et al. Base-resolution models of transcription- factor binding reveal soft motif syntax. Nature Genetics, 53(3):354-366) 2021b.
Timothy I. Bailey, James Johnson, Charles E Grant, and William S Noble. The MEME suite. Nucleic Acids Research, 43(W1):W39 W49, 2015.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. Xiv preprint TrXii:2108.07955 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Newral Information Processing Systems, 33:1877- 1901, 2020.
Jason Buenrostro, Beijing Wu, Howard Y Chang, and William Greenleaf. Atac-seq: a method for assaying chromatin accessibility genome-wide. Current protocols in molecular biology, 109(1):21-22)) -2, 2015.
Ashley M Bushey, Edward Ramos, and Victor G Corces. Three subclasses of Drosophila insulator show distinct and cell type-specific genomic distributions. Genes & Development, 23(11):1338 1350, 2009.
Lincan Cai, Shuang Li, Wenxuan Ma, Jingxuan Kang, Binhui Xie, Zixun Sun, and Chengwei Zhu. Enhaneing cross-modal fine-tuning with gradually intermediate modality generation. ar preprinl arXiv:2406.0900 09003, 2024.
Michael Caudy, Harald Vässin, Michael Brand, Rabiya Tuma, Lily Yeh Jah, and Yuh Nung Jan. daughterless, Drosophila gene essential for both neurogenesis and sex determination, has sequence similarities to myc and the achaete-scute complex. Cell, 55(6):106111065) 1988.
Ching Chang, Wei-Ye Yao Wang, Wen-Chih Peng, and Tien-Fu Chen. LLM4TS: Aligning pre-trained LLMs as data-officient timo-serios forocastors. arXiv preprint arXim:2308. 08469, 2024.
Valerie Chen, Muyu Yang, Wenbo Cui, Joon Sik Kim, Ameet Talwalkar, and Jian Ma. Applying inter- pretable machine learning computational dology-pitfally, recommendations and opportunities for new developments. Nature Methods 21 (8): 1461, 2024.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Nigel Collier, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Jin-Dong Kim. Introduction to the bio- entity recognition task at julpba. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications
pp. 73 78, 2004.
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Reville Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oleri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkha- tim, al. The Nucleotide Transformer: Building and evaluating robust foundation models for human genomics. bioRrie, 2023.
Bernardo P de Almoida, Franziska Reiter, Michaela Pagani, and Alexander Stark. DeepSTARR predicts enhancer activity from DNA sequence and enables the de novo design of synthetic enhancers. Nature Genelics, 54133 624, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapler of the associalion for compulational linguislies: human language lechnologies, volume 1 (long and short papers), pp. 4171-4188 2019.
Elizabeth D Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical Programming, 01:201-213, 2002.
Veniamin Fishman, Yuri Kuratov, Aleksei Shmelev, Maxim Petrov, Dmitry Penzar, Denis Shepelin, Nikolay Chekanov, Olga Kardymon, and Mikhail Burtsev. Gena-lm: family of open-source foundational dna language models for long sequences. Nucleic Acids Research, (2ga)ag 2025.
Geoff Fudenberg, David R Kelley, and Katherine S Pollard. Predicting 3D genome folding from DNA sequence with Akita. Nature Methods, 17(11):111-1111, 2020.
Katarína Grešová, Vlastimil Martinek, David Čochák, Petr Šimcōck, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Dala, 24(1):25, 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.001 007522 2023.
Shobhit Gupta, John A Stamatoyanmopoulos, Timothy L Bailey, and William Stafford Noble. Quantifying similarity between motifs. Genome Biology, 8:1 9, 2007.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint vrXiv:2004.10900 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Christopher Hill, Sanjarbek Hudaiberdiev, and Ivan Oveharenko. ChromDL: A next-generation regulatory dna classifier. bioRxiv, pp. 2023 01, 2023.
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana Davuluri. DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics, 37(15):2112-2133 2021.
Nan Jiang, Eldon Emberly, Olivier Cuvier, and Craig M Hart. Genome-wide mapping of boundary element- associated factor (BEAF) binding sites in drosophila melanogaster links BEAF to transcription. Molecular and Cellular Biology, 2009.
David R Kelley, Jasper Snock, and John L Rinn. Basset: Learning the regulatory code of the accessible genome with deep convolutional neural networks. Genome Research, 26(7):990 -99000999-000 2016.
13

---

Published in Transactions on Machine Learning Research (08/2025)
David R. Kelley, Yakir A Reshef, Maxwell Bileschi, David Belanger, Cory Y McLean, and Jasper Snock. Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome Research, 28(5):739 750, 2018.
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-Tzur, Moritz Hardt, Ben- jamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning. Proceedingss of Machine Learning and Systems, 2:230 246, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.0900 2018.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers as universal computation engines. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 7628-7636, 2022.
Wenxuan Ma, Shuang Li, Lincan Cai, and Jingxuan Kang. Learning modality knowledge alignment for cross-modality transfer. arXic preprint arXiv:2406. 18864, 2024.
Frederikke Isa Marin, Felix Teufel, Mare Horlacher, Dennis Madsen, Dennis Pultz, Ole Winther, and Wouter Boomsma. BEND: Benchmarking DNA language models on biologically meaningful tasks. In The Twelfth International Conference on Learning Representations, 2023.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. HyenaDNA: Long-range genomic se- quence modeling at single nucleotide resolution. Advances in Neural Information Processing Systems, 36, 2024.
Anusri Pampari, Anna Shcherbina, Evgeny Z Kvon, Michael Kosicki, Surag Nair, Soumya Kundu, Arwa S Kathiria, Viviana I Risca, Kristiina Kuningas, Kaur Alasoo, et al. Chrombpnet: bias factorized, base- resolution deep learning models of chromatin accessibility reveal cis-regulatory sequence synlax, transcrip- tion factor footprints and regulatory variants. bioRrin, pp. 2024-12, 2024.
Aman Patel, Arpita Singhal, Austin Wang, Anusri Pampari, Maya Kasowski, and Anshul Kundaje. Dart- eval: A comprehensive dna language model evaluation benchmark regulatory dna. arXie preprint 142.:2441.0049000 2024.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp. 28043-28078. PMLR, 2023.
Nicholas Roberts, Samuel Guo, Cong Xu, Ameet Talwalkar, David Lander, Lvfang Tao, Linhang Cai, Shuaicheng Niu, Jianyu Heng, Hongyang Qin, et al. Automl decathlon: Diverse tasks, modern meth- ods, and efficiency at scale. In NeurIPS 2022 Competition Track, pp- 151-170. PMLR, 2023.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI 2015, pp. 234-241. Springer, 2015.
Peter Roussceuw. Silhouettes: graphical aid to the interpretation and validation of cluster analysis. Journal of Compulational and Applied Mathemalics, 20:53 65, 1987.
Erik F Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Langnage-independent named entity recognition. orXiv preprint cs/0306050, 2003.
Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caducous: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint vrXiv:2403.0c3 2024.
M Scott, Lee Su-In, et al. A unified approach interpreting model predictions. Advances in Neural Information. Processing Systems, 90:4765-4774 2017.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Shula Shazman, Hunjoong Lee, Yakov Socol, Richard S Mann, and Barry Honig. OnTheFly: database of drosophila melanogaster transcription factors and their binding sites. Nucleic Acids Research, 42(D1)) D167 -171,, 2014.
Junhong Shen, Mikhail Khodak, and Ameet. Talwalkar. Efficient architecture search for diverse tasks. In Advances in Neural Information Processing Systems, 2022.
Junhong Shen, Liam Li, Lucio M Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Tal- walkar. cross-modal fine-tuning: Align then refine. In International Conference on Machine Learning, pp. 11030-31056, PMLR, 2023.
Junhong Shen, Tanya Marwah, and Ameet Talwalkar. UPS: Towards foundation models for PDE solving via cross-modal adaptation. arXiv preprint ar arXiv:2403.07188, 2024a.
Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicolo Fusi. Tag-LLM: Repur- posing general-purpose LLMs for specialized domains. arXiv preprinl arXim:2402 00nm24422-51440 05140, 2024b.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Leurning, pp. 3145 3153. PMIR, 2017.
Avanti Shrikumar, Katherine Tian, Ziga Avsec, Anna Shcherbina, Abhimanyu Banerjee, Mahfuza Sharmin, Surag Nair, and Anshul Kundaje. Technical note transcription factor motif discovery from importance scores (TF-MoDISco) version 0.3. 6.5. arXiv preprint orXiv:1811.04116 2018
Ziqi Tang, Nirali Somia, Yiyang Yu, and Peter K Koo. Evaluating the reprosentational power of pre-trained dna language models for regulatory genomies. bioRriv, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprint ar Xix:22300 13971, 2023.
Ria Vinod, Pun-Yu Chen, and Payel Das. Reprogramming pretrained language models for protein sequence representation learning. arXiv preprint arXiv:2301.02120 2023.
Zongzhe Xu, Ritvik Gupta, Wenduo Cheng, Alexander Shen, Junhong Shen, Ameet Talwalkar, and Mikhail Khodak. Specialized foundation models struggle to beat supervised basclines. orXiv preprint 1220096, 2024.
Muyu Yang and Jian Ma. Machine learning methods for exploring sequence determinants of 3D genome organization. Journal of Molecular Biology, 434(15):167666, 2022.
Zijun Zhang, Christopher Y Park, Chandra I, Theesfeld, and Olga G Troyanskaya. An automated framework for efficiently designing deep convolutional neural networks in genomics. Nature Machine Intelligence, 3 (5):392 400, 2021.
Igor Zhimulev, Tatyana Vatolina, Victor Levitsky, and Anton Tsukanov. Developmental and housekeeping genes: Two types of genetic organization in the Drosophila genome. International Journal of Molecular Sciences, 25(7):4068, 2024.
Jian Zhou and Olga G. Troyanskaya. Predicting effects of noncoding variants with deep lcarning- sequence model. Nature Melhods, 12:931 934, 2015.
Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in Neural Information Processing Systems, 36:43322 43355, 2023a.
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Lin. DNABERT-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint ar Xix::23306 15006, 2023b.
Lihua Julie Zhu, Ryan G Christensen, Majid Kazemian, Christopher J Hull, Metewo Selase Enuameh, Matthew D Basciotta, Jessie A Brasefield, Cong Zhu, Yuna Asriyan, David S Lapointe, et al. FlyFactor- Survey: a database of drosophila transcription factor binding specificitics determined using the bacterial one-hybrid system. Nucleic Acids Research, 30(supp__1):DL1 2011.
15

---

Published in Transactions on Machine Learning Research (08/2025)
A Appendix
A.1 Code Availability
The souree code of L2G can be accessed at: https: github.cttm.mmmm. htwpu/bbcco//w/owcccconoogggg L2G
A.2 Data Availability
In this work, we utilized several public datasets.
The Genomic Benchmark available at:
benchmarks)
The Nucleotide Transformer benchmarks can be downloaded from HuggingFace at:
http:///wwuuuulecccoo//////////s _transfomerr downstream tasks
The DART-Eval benchmars is available at:
http://gtttubbcoom/uudddaaaaae DARRaaIeelll//DR1--va-aaa
The DeepSTARR dataset is available Zenodo at:
A.3 Pre-training resources of various genomic FMs
Table provides provides a comparative summary the computational resources, model parameters, and pro-training data with various genomic FMs.
Model
Params
GPUs
Wall clock
Pre-training Data
DNABERT (Ji
2021
110M
8-GTX 2080ti-L1GB
days
3.2B
DNABERT-22 Zhou et al. 2023b
8-GTX 2080ti-11GB
days
Enformer Avsec et al. 2021a
252M
64-TPU v3 cores-32TB
days
14.1B
Nucleotide Transtormer Dalla Torre al. 2023
128-A100-80GB
days
174B
HyenaDNA (Nguyen et al 2024
1-A100-40GB
80 mins
3.2B
Caduceus Schiff et al. 2024
1.9M
?
35B
Table 1: Pre-training resources and data of various DNA foundation models. The pre-training data is reported in nucleotides. The computing resources required for pre-training Caduceus are unknown. Note that Enformer is a supervised model trained for the gene expression prediction task and is not pre-trained on unsupervised genomic sequencing data. However, we included it here as was used as DNA FM baseline the Nucleotide Transformer benchmark.
A.4 Description of the downstream tasks
Genomic Benchmarks dataset consists of eight classification tasks: seven binary and one three-way, focusing on regulatory elements such as promoters, enhancers, and open chromatin regions from several species, including humans, mouse (Mus musculus), and roundworm (C. elegans) (Grešová et al. 2023 A three-
layer CNN serves the baseline model this benchmark, and the HyneaDNA study (Nguyen al. 2024)
included supervisod trained transformer bascline. Inputs are DNA sequences with lengths between 200
500 bp, except for the Mouse Enhancer Ensembl dataset, which has the longest inputs (median 2,381 bp: maximum 4.707 bp). Metadata for the tasks included in the Genomic Benchmarks (Grešová et al. provided in Table
2023 is
Nucleotide Transformer Benchmarks dataset is another widely used benchmark for evaluating genomic FMs.
This benchmark suite, introduced alongside the Nucleotide Transformer (Dalla- Torre et al. 2023 eval-
uates genomic FMs on 18 classification tasks such as predicting regulatory elements for enhancers (hu- man), promoters (human/mont) epigenetic marks (yeast), and splice sites DNA sequences 300-600 bp long. Performance metrics for several models including Enformer (Avsec
from
16

---

Published in Transactions on Machine Learning Research (08/2025)
Dataset
Samples
Classes
Max Length
Metric
dummy_ mouse_ enhancers ensembl
www.mmuueeehhnnrrssss.t.eo umhancersseensmbll
1,210
4,707
Accuracy
100,000
200
Accuracy
demo_human__e _r__a_rr____wmm
100,000
2
200
Accuracy
gumma__chhancer__chm
27,791
2
500
Accuracy
154,842
2
573
Accuracy
human_
aaaaan__nneebllreguattoro Tuman_ocrremsembl
289,061
3
802
Accuracy
36,131
2
251
Accuracy
174,756
2
593
Accuracy
Table 2: Description of datasets in Genomic Benchmarks. Each dataset is described by the name, the total number of samples, the number of target classes, the maximum sequence length in nucleotides, and the evaluation metric used.
ct a..
2021a
DNABERT-1 (Ji et al. 2021 DNABERT-2 Zhou et al. 2023b). HyenaDNA (Poli et al.
2023 and Nucleotide Transformer (Dalla- Torre et al. 2023 are included, along side results the recent
Caduceus- (Schiff et al. 2024 Metadata for the tasks included in the Nucleotide Transformer Bench-
marks (Dalla- Torre et al. 2023 is presented Table 3
Dataset
Samples
Classes
Max Length
Metric
H3
13,468
500
MCC
H3K4mel
28,509
500
MCC
H3K4me2
27,614
500
MCC
H3K4me3
33,119
500
MCC
H3K9ac
25,003
500
MCC
H3K14ac
29.743
500
MCC
H3K36me3
31,392
500
MCC
H3K79mc3
25,953
500
MCC
13,140
500
MCC
H4ac
30,685
500
MCC
Enhancers
14,968
200
MCC
Enhancer types
14,968
200
MCC
Promoter all
53,276
300
F1
Promoter TATA
5.517
300
F1
Promoter non TATA
47,759
300
F1
Splice sites all
27,000
400
Accuracy
Splice sites acceptors
19.961
600
F1
Splice sites donors
19.775
600
F1
Table Description of datasets in Nucleotide Transformer Benchmarks. Each dataset is described by the name, the total number of samples, the number of target classes, the maximum sequence length in nucleotides, and the evaluation metric used. Metrics include MCC (Matthews Correlation Coefficient). F1 score, and accuracy, used
in the Nucleotide Transformer study (Dalla- Torre et al. 2023
DART-Eval is recent benchmark that curates biologically significant tasks focused on regulatory DNA Paa-
tel et al.) 2024). Among these, learning functional regulatory syntax is one of the most biologically relevant downstream applications. DART-Evall includes a task aimed at learning cell-type-specific regulatory syntax by distinguishing uniquely active elements identified from ATAC-seq (Buenrostro et al. 2015) experiments across five cell lincs. We evaluated L2G against both fino-tuned and ab initio basclines provided by DART- Eval. The fine-tuned baselines include DNABERT-2 (Zhou et. al. (023b),, GENA-LM (Fishman et al. 2025
HyenaDNA (Poli et al. 2023), and Nucleotide Transformer (Dalla- Torre al. 2023 The ab initio baseline, ChromBPNet (Pampari et al. 2024), is strong non-pretrained supervised model that outperformed all other genomic FMs in this task.
Developmental and Housekeeping Enhancer Activity Predictions is a two-class regression task that predicts enhancer activitics for housckeeping and developmental enhancers in Drosophila $2 cells using 249 bp se- quences. The dataset, soureed from the DeepSTARR project (de Almeida et al. 2022). includes a CNN model baseline. The evaluation metric is the Pearson Correlation Coefficient (PCC).
17

---

Published in Transactions on Machine Learning Research (08/2025)
Histone Datasets
Models
DNABERT-2
Text Scate %
Score
Eriformer
HycnaDNA
L2G
10
HOK3mel
H3k4maz
HoKime3
HOKRac
HSK1 decc
HDK36ined
HGK79m
Enhaneers
Enhancer typer
Promoter ron TATA
Prompter
Spice sioes all
Splice sites accsotor
Splice sitee donare
Promoter
Fractonn of sihoptinal tasto
Models
Fosaton of a mmmm taska
DNABERT-1
DNABERT 2
CNN
NT
Transtorer
HyenaDNA Caduceus
HyeraDNA L2G
50
Figure 6: L2G matches or outperforms recent DNA foundation models on the Nucleotide Transformer (NT) Benchmarks and Genomic Benchmarks. A. Test, scores on the Nucleotide Transformer Benchmarks for histone mark prediction tasks (left) and enhancer, promoter, and splice site prediction tasks (right). The bar for DNA-BERT-I missing because it could not predict splice sites. B. Aggregating results the Nucleotide Transformer Benchmarks (Table using performance profiles (Dolan & Moré 2002). Larger values (fractions of tasks on which a method is within a r-factor of the best) indicate better performance. L2G's curve in the top-left corner demonstrates it is often the best or second best method. C. Aggregating results on the Genomic Benchmarks, which include supervised CNN and transformer baselines.
A.5 Complete results
The complete results on the Nucleotide Transformer Benchmarks are shown in Table and Fig. A We
used batch size of 64 and cross-entropy loss across all datasets. Test scores for other genomic FMs are from Supplementary Table 6 of Dalla-Torre et al. (DallDaaTorrre et al. 2023). Results for Caduceus-Pi Schill et al. 2024), which outperforms the Caduceus-Ps on 17 out of 18 tasks, are included. To provide a holistic comparison of methods across all datasets, we utilized performance profiles (Dolan & Moré 2002). Each
curve shows the proportion of problems it solves within varying thresholds of a performance factor т. As
shown in Fig. 6B, I.2G achieves the best or second-best performance across all tasks.
The complete results on the Genomie Benchmarks are shown in Table 5 cross-entropy loss across all datasets. We trained the CNN and HyenaDNA (32k) baselines, while results for the transformer baseline were obtained from the HyenaDNA paper (guuyn et al. (024)., as the code is not open-sourced. We also computed the performance profiles for the results on the Genomic Benchmarks. As shown in Fig. 6C, L2G achieves top performance across all tasks.
We used batch size of 64 and
The complete results for the cell-type-specific element classification task on the DART-Eval benchmark are shown in Table 6 L2G outperforms both fine-tuned and ab initio baselines in overall accuracy. Notably, I.2G is the only fine-tuned-basee method to surpass the non-pre-trained supervised model ChromBPNett.
18

---

Published in Transactions on Machine Learning Research (08/2025)
The complete results for the Developmental and Housekeeping Enhancer Activity Prediction Task are pre- sented in Table 7 For this task, we used batch size of 128 and mean squared error (MSE) loss. To benchmark the performance of L2G, we included the expert model, DeepSTARR, which was a CNN specif- ically designed for this task. Additionally, we compared with two genomic FMs, HyenaDNA (32k) and Nucleotide Transformer (v2, 500M). PCC was used as the evaluation metric.
Dataset Histone Markers
Enformer
DNABERT-1
DNABERT-2
HyenaDNA
Caduceus- Ph
L2G
H3
79.3
72.4
76.3
78.5
78.1
81.5
82.5
H3K4me1
54.1
29.1
39.6
51.2
51.2
52.3
58.6
H3K4me2
32.4
20.7
28.2
33.3
45.5
48.7
56.2
H3K4me3
40.8
15.6
25.8
35.3
55.0
54.4
66.3
H3К9ac
54.7
41.5
50.5
54.5
58.6
62.2
65.1
H3K14ac
53.8
28.4
40.3
51.5
60.8
63.1
69.4
H3K36mc3
61.8
34.5
47.4
59.1
61.4
60.1
68.8
H3K79me3
62.3
49.8
57.8
61.5
66.9
69.7
70.7
H4
80.8
73.5
78.4
79.7
76.3
81.1
78.8
H4ac
27.5
35.9
46.5
56.4
62.1
65.1
Average
61.0
63.5
67.0
Enhancer
Enhancers
54.5
45.4
49.5
52.5
52.0
54.6
55.8
Enhancer types
44.4
31.2
36.7
42.3
40.3
43.9
62.6
Average
49.5
38.3
43.1
47.4
46.2
49.3
59.2
Promoter
Promoter TATA
95.9
91.8
91.0
90.9
87.9
95.3
96.0
Promoter non-TAI TATA
97.7
90.9
92.4
94.3
91.9
96.9
97.2
Promoter all
97.5
90.9
92.2
94.3
91.9
97.0
96.2
Average
97.0
91.2
91.9
93.2
90.6
96.4
96.5
Splice
Splice sites
77.2
96.2
90.9
93.4
94.0
97.9
Splice sites acceptors
82.9
94.9
91.6
93.7
96.4
Splice sites donors
98.7
81.2
92.5
89.4
94.8
95.5
Average
98.5
80.4
92.8
91.5
94.2
96.7
Table 4: The performance of each model on the Nucleotide Transformer Benchmarks. Metrics used by task: MCC for histone markers, F1-score for enhancers and splice site acceptors/ acceptors/donors, /onors, and accuracy for splice site all. Bold indicates the best performance, and underline indicates the second-best. NT stands for nucleotide transformer (multispecies, 2.5B version). The results for other baselines are retrieved from the Nucleotide Transformer paper (Dalla- Torre et al. 2023). DNABERT-1 could be trained on two splice site prediction tasks because the input sequence length exceeded the maximum context length allowed by DNABERT-1.
Dataset
CNN
Transformer
HyenaDNA
L2G
Mouse Enhancers
72.8
80.1
82.6
79.3
Coding vs Intergenomic seqs
88.2
88.8
89.6
Human vs Worm
92.8
95.6
Human Enhancers Cohn
71.6
70.5
73.0
Human Enhancers Ensembl
80.2
83.5
86.9
88.4
Human Ensembl Regulatory
93.9
91.5
92.0
93.9
Human Nontata Promoters
85.8
87.7
94.3
92.9
Human OCR Ensembl
67.8
73.0
79.1
78.9
Average
81.7
83.8
86.8
86.9
Table 5: The performance of each model the Genomic Benchmark dataset. The evaluation metric is accuracy (the higher, the better). Bold indicates the best performance, and underline indicates the second-best.
19

---

Published in Transactions on Machine Learning Research (08/2025)
Model
Overall
GM12878
H1ESC
HEPG2
IMR90
K562
ChromBPNet-like
0.667
0.887
0.886
0.874
0.874
0.865
DNABERT-2
0.650
0.878
0.886
0.875
0.863
0.836
GENA-LMM
0.636
0.869
0.881
0.874
0.858
0.836
HyenaDNA
0.610
0.866
0.868
0.862
0.853
0.838
Nucleotide Transformer
0.632
0.868
0.881
0.871
0.859
0.836
L2G
0.690
0.879
0.876
0.878
0.877
0.871
Table 6: sell-type-specific element classification task from the DART-Eval benchmark. We report the overall accuracy across all classes and the accuracy specific each cell line. Bold text indicates the best performance, and underlined text indicates the second-best. L2G is the only fine-tuned method that outperformed the non- pretrained supervised model ChromBPNet.
Dataset
HyenaDNA
Nucleotide Transformer
DeepSTARR
L2G
dev
0.57
0.64
0.68
0.66
hk 0.65
0.75
0.74
0.76
Mean 0.61
0.70
0.71
0.71
Table The performance of each model on the Drosophila enhancers prediction regarding the developmental (dev) and housekeeping activity (hk).
20

---

Published in Transactions on Machine Learning Research (08/2025)
H3
enhancers
promoter tata
S5: 0.30
SS: 0.08
SS: 0.68
L2G
Class
Neg
SS: 0.23
SS: 0.07
SS: 0.32
Pos
Vanilla
Figure 7: Visualization of the learned embedding of target modality data for models trained with L2G (top), and vanilla fine-tuning (bottom). The target modality data is from three representative downstream tasks from the NT Benchmark: histone modification (H3, left), enhancer regions (enhancers, middle), and promoter regions (promoter tata, right). Pink dots represent positive class, while yellow dots indicate negative class. Each plot includes the Silhouette Score, widely used metric for evaluating cluster separation. The score ranges from -1 to +1, where: +1 indicates well-separated clusters, 0 suggests indifferent clusters, and -1 indicates potential misclassification of points between clusters (Rousseeuw 1987.
A.6 Embedding analysis
To evaluate the quality of the learned representations of the target modality data, we analyzed the embed- dings generated by L2G and compared them to those obtained through vanilla fine-tuning, which refers to directly training the model with task-specific loss without the NAS embedded pertaining steps. This analysis was conducted on three binary classification tasks from the Nucleotide Transformer benchmarks: H3, enhancers, and promoter tata. We visualized the embeddings of the two classes in each task using t-SNE, as shown in Fig. 7 This visualization provided a qualitative assessment of how well the embeddings separate the classes in a deluuced-dimensionee space. To quantitatively measure cluster separation, we cal- culated the Silhouette Score for cach set of embeddings. The Silhouette Score ranges from -1 to +1, where +1 indicates well-separated clusters, 0 signifies overlapping clusters, and -1 suggests incorrect class assign- ments Rousseeuw 1987). Across all three datasets, L2G consistently achieved higher Silhouette Scores compared to vanilla fine-tuning, demonstrating its superior ability to produce distinct class separations in the embedding space.
21

---

Published in Transactions on Machine Learning Research (08/2025)
A.7 Ablation studies
We conducted a series of ablation studies lo evaluate the contributions of different components in our method. Specifically, we assessed the impact of pre-trained transformers, the losses during embedder pretraining, and the embodder architecture. The results on three tasks from the Nucleotide Transformer benchmarks are shown. Detailed results are provided Table g for ablation study for the pre-training Body, Table | for ablation study for the distribution alignment metrics, Table 10 for ablation study for the losses during embedder pre-training, Table [1 for the embedder architecture, Table 12 for different training strategies, Table 13 for different transformer model backbones, and Table 14 for different source language data.
Dataset
Pre-trained
Scratch
82.5
67.1
Enhancers
55.8
54.0
Promater TATA
96.0
86.7
Table 8: Performance comparison of pre-trained RoBERTa with L2G versus randomly initialized RoBERTa trained from scratch on three datasets from the Nucleotide Transformer benchmark.
Dataset
MMD
OTDD
H3
82.5
67.1
Enhancers
55.8
54.7
Promoter TATA
96.0
95.2
Table Performance comparison of Maximum mean Discrepancy (MMD) and Optimal Transport Dataset Dis- tance (OTDD) as alignment metric used during the embedder pre-training stage on three selected datasets on the Nucleotide Transformer benchmark.
Dataset
Joint losses
Task-specific loss only (alpha=0)
MMD loss only (beta=0)
H3
82.5
79.5
69.0
Enhancers
55.8
54.5
52.4
Promoter TATA
96.0
86.2
89.8
Table 10: Performance comparison of different losses used during the embedder pre-training stage three selected datasets the Nucleotide Transformer benchmark.
A.8 Motif analysis
We calculated the muclcotide contribution scores using a DecpLiftShap mothod from Catum GitHub Reposi- tory for developmental and housekeeping enhancer aclivities. DeepLiftShap combines the DeepLIFT (Shriku- mar al. (2117) algorithm with SHAP (SHapley Additive s<lanaaions) values to attribute model predictions to input fcatures by calculating the contribution relative to reference bascline (Scott al.) 2017 Each feature is nucleotide at specific position.
Following the DeeRSTARR methodology (de Almeida et al. 2022) we used 100 dimucleotide-shuufled ver- sions of each input sequence as baseline sequences. Hypothetical importance scores for each sequence were multiplied by its one-hot encoded matrix to derive the final nucleotide contribution scores.
Motifs were identified using TF-Modisco-litoe, a more efficient implementation of TF-Modisco (Shrikumar et al.] 2018), on the nucleotide contribution scores for each enhancer type separately (de Almeida et al. 2022 For motif annotation, downloaded two reference databases for Drosophila: OnTheFly (Shazman ct al. 2014) and Flytooorryrrvey (Zhu et al. 2011), from the MEME suite (Bailey et al. 2015), and compared motifs using TOMTOM (Gupta et al. 2007]. Motifs with fewer than 30 seqlets were discarded. The resulting motifs for developmental and housekeeping enhancers are visualized in Fig. and Fig. respectively,
22

---

Published in Transactions on Machine Learning Research (08/2025)
Dataset
DASH
UNet
ResNet
DeepSEA
ORCA
H3
82.5
76.4
80.5
54.9
Enhancers
55.8
55.4
55.5
52.4
44.3
Promoters TATA
96.0
96.1
94.0
94.9
84.0
Table 11: Performance comparison of different embedder architectures three selected datasets from the Nucleotide Transformer benchmark. The architectures compared include DASH, unsearched UNet and ResNet, DeepSEA (a three-layer CNN), and the embedder from the original ORCA model (a single-layer convolutional model). DASH, through neural architecture search, optimizes the configuration of UNet and ResNet backbones, while the unsearched versions of UNet and ResNet use a fixed kernel size and fixed dilation rate of
Dataset
Train
Frozen
No
82.5
74.9
76.0
Promoters TATA
96.0
95.1
92.8
Enhancers
55.8
54.9
48.0
Performance comparison of three strategies three datasets from the Nucleotide Transformer bench- mark. To isolate the contribution of the pre-trained transformer trunk, we evaluate: (1) full fine-tuning, where all transformer parameters are trainable; (2) frozen transformer backbone, where all transformer weights are frozen; and (3) no-transformer, where only the encoder and predictor are trained.
Table 12:
Dataset
RoBERTa BaBERTa-Base
RoBERTa-Lare
82.5
85.5
Promoters TATA
96.0
96.2
Enhancers
55.8
57.1
Table 13: Performance comparison between using pre-trained RoBERTa Base (149M parameters) and RoBERTa- Large (387M parameters) as transformer backbones on three selected datasets from the Nucleotide Transformer benchmark.
Dataset
CONLL2003
JNLPBA
82.2
Promoters TATA
96.0
96.2
Enhancers
55.8
56.5
Table 14: Performance comparison of different language source data on three selected datasets from the Nucleotide Transformer benchmark. We evaluated the impact of language source data by comparing the CoNLL-2003 (Sang & De Meulder 2003) and JNLPBA (Collier et al., 2004) datasets. We chose JNLPBA because specifically designed for biomedical named entity recognition, making more domain-aligned with our genomic sequence analysis tasks than CoNLL-2003.
23

---

Published in Transactions on Machine Learning Research (08/2025)
Forward
Reverse
# Seqlets
Motif
Q
value
Match
CC
EEERTFF LRUISERS SEEEER FOSSS02
IGACTCA
GIOTGA
1059
AP-1
0.006236
ATATC
GATAA
GATAAG
616
GATA
0.028951
A
Aa CASETG
CA .IG x
325
da
0.005764
ATCACCCcAE
TCACGGEAS
187
SREBP
0.001854
Figure 8: Motifs discovered for developmental enhancers by L2G. Details include forward and reverse se- quences, seqlet count, motif name, Q- Value, and closest database match. The Q- Value is statistical measure that represents the false discovery rate (FDR) for the motif. Lower Q-values indicate more significant results.
Forward
Reverse
# Seqlets
Motif
Q value
Match
11131111 FEUEESE LESTERD SEESEES E5774793
GGECACACI
GEGTGACC
848
Ohler1
0,000003
ClTT
ATCG.T TA
TATCGATA
704
DRE (Dref)
0,043078
CLCOTG
CAGCIG.I GLE
CAGETGA
263
crp
0.013335
IAG G TG
CA CTA
CAACACTA
247
Ohler7
0.051141
EIA ATCCAT CLAGGGG GAcHA
CIATAL TA &o
TA TATIG I
63
BEAF-32
0.021631
Cla cla cla
Co BATeo ArA
30
DRE+DRE
0.019005
Figure 9: Motifs discovered for housekeeping enhancers by L2G. Details include forward and reverse se- quences, seqlet count, motif name, Q- Q-Value, and closest database match.
24

---

Published in Transactions on Machine Learning Research (08/2025)
A.9
Implementation details
Table 15 provides the hyperparameter settings used for training L2G.
Hyperparameter
Value
Distribution Alignment Metric
MMD
Transformer Backbone
RoBERTa-base
Target Sequence Length
512
Training Epochs
25
Embedder Pre-training Epochs
80-100
Warm-up Epochs
Decay Epochs
(Weight for Alignment Loss)
(Weight for Task Loss)
Dropout
0.05
Gradient Clipping
[-1, 1]
Batch Size
64-128
Embedder Pre-training Optimizer
SGD
Embedder Pre-training Learning Rate
Searched by DASH
Fine-tuning Optimizer
Adam
Fine-tuning Optimizer Betas
[0.9, 0.98]
Fine-tuning Learning Rate
1e-5
Weight Decay
1e-5
Scheduler
Step Decay
Table 15: Hyperparameter settings for training L2G.
For all tested datascts, we applied data during training by randomly shifting input sequence by up to bp and revplee-co sequences. During testing, predictions from the forward and reverse complement sequences were averaged. This approach is commonly used in genomics to improve the prediction accuracy of deep learning models (Zhou & Troyanskaya 2015
Avsec et al. 2021a).
25


Reviews:
[{"review_id": "Usuk8PPvuA", "reviewer": "Reviewer_kD7P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper demonstrates that it is possible to efficiently learn genomic models from large language models using the proposed L2G transfer learning technique. On the included benchmarks, the L2G model performs better than similarly-sized models trained on considerably more genomic data. The benchmarking is thorough, and exploration of model properties are included. It is a clearly written paper with figures that generally presents their setup and findings well.\n\nThe proposed adaptations of the ORCA setup are sensible and well-justified. This includes introducing a task-specific loss to enable cross-modal transfer learning, and replacing the OTDD loss from the ORCA setup to MMD, which seemingly allows for better imitation of the reasoning of the base language model. \n\nWeaknesses:\n\nThe paper presents some ablations on the setup, but more comparisons of design choices would be desirable. Some general examples include 1. How does the choice of the RoBERTa-base model impact performance compared to similar (or larger) language models? 2. Usage of DASH: On the shown tasks in Fig. 5, UNet embedder architecture consistently performs better than ResNet. Further, on 2 of 3 shown tasks, UNet also performed better than using DASH. It would be interesting to see a comparison of which CNN backbones (ResNet vs UNet) is selected for different tasks by DASH, and some additional results that indicate how using DASH improves over the base UNet. \n\nThe paper would also benefit from some general clarifications on some technical differences between embedder pre-training, which also seems to include the task loss and hence the predictor, and the final fine-tuning (such as the data used for pretraining vs fine-tuning, and whether the task loss for pretraining is tied to the task the model is fine-tuned on). Figure 2 part C does not sufficiently clarify this for me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes L2G, a technique for cross-modal transfer learning from language to genomic data. The technique is based on ORCA, but has suitable adaptations that better enables cross-modal learning from language to genomic data.\nThe paper provides a comparison to pre-trained (and fine-tuned) genomic models on three benchmarks consisting of a varied set of sub-tasks, demonstrating good performance. Further, some ablations and demonstrations of model capabilities are provided.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper would primarily benefit from expanding the ablations to increasingly demonstrate how the proposed cross-modal learning improves upon regular fine-tuning. Examples include ablations on the LM backbone, whether there are (qualitative) differences between embeddings of L2G vs. pure genomic models (expanding on Fig 7) etc. \n \nIt would also be interesting and useful if the authors could describe intuitions of why language pre-training + cross-modal transfer learning seems to lead to better results than pre-training on genomic data + fine-tuning on genomic  data (e.g. how the proposed L2G setup leverages the LM logic in a manner beneficial for the downstream genomic tasks), although this request is admittedly more vague.\n\nSome minor typos: \"an unseen modalities\" (page 3), \"Bridge this gap often\" (page 3), \"[...] to learn f into map h into [...]\" (page 5). In Table 5, the second HyenaDNA accuracy is bolded when it shouldn’t be (please also consider using bold for equal top accuracies).", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sru6LlIiVy", "reviewer": "Reviewer_eAGp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper identifies limitations of previous works (e.g. architecture design, difficulties in cross-modality predictions, computational expense) to motivate the proposed framework. The authors address each limitation using existing techniques in neural architecture search and domain adaptation to build a 3-stage training pipeline.\n- The empirical results demonstrate efficiency and effectiveness of the L2G framework. The ablation studies tease apart the impact of difference design decisions. \n- They also show a meaningful biological application in discovering functional regulatory syntax, highlighting the relevance of the method in real tasks.\n\nWeaknesses\n\n- The motivation behind the choice for a CNN architecture for DNA embeddings was not clear, given that the data is sequential in nature, and might be more amenable to sequential modeling.\n- Discussion about important hyperparameters (e.g. weights on different loss terms) or the choice of pretraining data is lacking. \n- Other limitations, like limited complexity of the benchmarked tasks and lack of interpretability, are already discussed by the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work introduces the L2G -- Language-to-Genome -- framework that  adapts pre-trained language models to genomic prediction tasks, leveraging a carefully designed pipeline of neural architecture search. a domain-adaptation-style pretraining of a DNA embedder to close the modality gap between DNA and language modalities, and finally, task-specific supervised fine-tuning. The authors empirically show the advantages of L2G in data and computational efficiency, along with performance that is competitive with more expensive genomic foundation models. They further study the impact of different choices on performance through ablation studies. This line of work is especially useful to allow practitioners to adapt and use high performing foundation/language models under resource constraints.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As discussed above, some discussion and/or empirical evidence about the following questions would be appreciated:\n\n- Why is a CNN a natural choice for DNA embeddings, especially as opposed to other architectures designed for sequential modeling?\n- How sensitive is the method to choices of hyperparameters and pretraining data?", "broader_impact_concerns": "The authors sufficiently address broader impact and limitations in the Discussion section of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d40Lu94Iw6", "reviewer": "Reviewer_n2NV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The authors do well to frame the question of whether very large models pre-trained with self-supervised learning are worth the expense.\n- The authors do a good job of contextualizing the ORCA framework, as well as laying out the case for why the model family for learning the embedder needed to be enlarged so that genomic data could work for cross-modal transfer in this way: it's ORCA, but with a broader class of models for the encoder and an additional loss for subsequent finetuning after the alignment.\n- The experiment in section 3.3 (summarized in Figure 4) is nice, and shows that some predictions made by L2G align with existing methods tailored to genomic DNA for enhancer identification.  The use of DeepLIFT -> TF-modisco is a nice touch.\n\n### Weaknesses\n\nUnfortunately, the authors left out a few key experiments that I would have liked to see.  The most necessary is the ablation for training only the embedder network along with the predictor network, leaving out the RoBERTa transformer weights entirely, to validate how much of this performance can be explained by feature repurposing versus simply training an intensively selected encoder network.   \n- The end of section 3 (and Figure 5) contained many ablation studies, but they seem incomplete and raise further questions: Why does a randomly initialized RoBERTa trunk perform so well?  It seems to be due to the encoder training, but this option (which seems so clearly needed for me) is missing.\n- While the reasons for expanding the size of the model family for choosing the encoder network are well motivated, the choice for abandoning OTDD over MMD (section 2.4.2) is not, beyond a confusing result presented in Figure 5.  It also seems like the auxiliary loss chosen to incorporate class information is under-motivated by interpreting the results.  The original Alvarez Melis and Fusi paper argued convincingly why OTDD was preferred, so it's curious why the authors did not observe this here and why.  Practitioners will want to know why.\n- Again in section 2.4.2, the task specific loss is motivated a bit weakly by saying it includes class information.  OTDD is able to use class information for aligning distributions, I'm again curious why MMD is preferred, and cross-entropy (or MSE of all things) is preferred here?  It feels like this part of the story is not complete\n- Section 3.2 relies a lot on the NT benchmarks as being suitable for differentiating among models.  I'm not sure the benchmark has aged well, and would prefer to see more thoughtfully constructed benchmarks like BEND or DART-eval used here.\n- I think it's true that L2G's surprising performance raises the question of whether self supervised learning on large transformers is an effective way to build DNA lms, but I honestly do not think the rest of the paper pursues this question.  What it interrogates is how to correct the failure of ORCA to perform well on adapting LLMs to tasks involving genomic DNA.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present L2G, an extension of a repurposing model for LLMs to out-of-domain classification or regression tasks.  They extend work by Shen et a. who develop an auto-ML driven method for repurposing the trunk of LLMs for cross-domain purposes by aligning newly trained encoder and prediction modules for various domains extending beyond text or images, called ORCA.  While ORCA did not show success in tasks involving genomic DNA inputs, the authors of the present work identify the causes of this failure and show how to remedy it.  They perform some necessary ablations to show the effects of an improved auto-ML search over encoders, as well as a multi-loss fine-tuning procedure.  They conclude that LLMs can indeed work for cross modal learning for problems involving genomic DNA, and suggest that resource-intensive self-supervised training of transformers ab-initio may not be worth the expense.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "These changes would greatly improve my likelihood of recommending acceptance:\n- It would be helpful if the authors could explain why it's the case that OTDD is not able to incorporate class label information when the original paper of Alvarez-Melis and Fusi argue convincingly that OTDD is superior.  Is it a case of a failure of approximation?  Or something else?\n- The task specific loss is motivated a bit weakly by saying it includes class information.  OTDD is able to use class information for aligning distributions, I'm again curious why MMD is preferred, and cross-entropy (or MSE of all things) is preferred here?  It feels like this part of the story is not complete.\n- Panel B and C of Figure 4 are puzzling.  What information does this tell us about L2G?  Should we infer that L2G is finding relevant motifs for `da` and `crp` and `BEAF-32`, or is it missing those identified by DeepSTARR?  And by what measure it it failing to find those TF motifs it missed?  It's not clear what to take away from these panels.\n- The backbone ablation in section 3.4 is missing.  This experiment isn't really the one I wanted to see; it's sure to show that it will fail using a randomly initialized backbone. What I want to see as a measure of how much of the performance can be attributed to the knowledge encoded in the RoBERTa-base weights is how it performs just against the trained encoder and predictor, with no transformer at all.\n- In panel B of Figure 5, is it a mistake that OTDD was unable to perform at all in the enhancer classification task?  And what does it mean that the random weights RoBERTa model performs comparably to the pre-trained weights model ??  There really should be more discussion of all four panels of Figure 5.\n\nThe following are not critical for my recommendation for acceptance, but would improve the work in my view.\n\n- In 2.4.3, the authors reveal they are using ASHA for hyper-parameter optimization. After using NAS for learning the embedder, it would be good to see some data that shows this additional compute still conforms to the claim in the introduction that it vastly saves on compute compared to the much larger in-modal comparators, perhaps an estimate of flops would help here?\n- The embedder pre-training loss objective is desribed a bit vaguely.  What do the authors mean by saying they 'schedule' the weighted losses?  Could this concluding part of 2.4.2 be clarified?\n- Some of the reference do not seem to render properly.  For example, `Devlin et al` 2019 (BERT) is rendered as `Kenton & Toutanova`.\n- The concluding sentence for the first paragraph of the introduction uses quite vague language. \"Genomic element\" versus \"chromatin state\" can be overlapping, or different.  Some citations to clarify these tasks would help here.   Also, \"genome function\" isn't really a meaningful term, maybe the authors mean gene function prediction?\n- At the beginning of section 3.1, the authors say that \n\n> Directly applying transformer models trained on natural language data ... can lead to the corruption of pre-trained weights\n\n'corruption' of the pre-trained weights is a strange phrase here.  I think the authors mean degraded performance on the source domain, with no guarantee of good performance in the target domain?\n- The second sentence of 3.1 restates part of the introduction, and I think can be skipped.  Maybe move more directly to the following sentence \"We demonstrate the empirical ...\"", "broader_impact_concerns": "None required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)