Multi-modal Transfer Learning between Biological Foundation Models
Authors: Juan Jose Garau-Luis
Patrick Philippe Bordes
Liam Gonzalez
Maša Roller
Bernardo P de Almeida
Christopher F. Blum
Lorenz Hexemer
Stefan Laurent
Maren Lang
Thomas PIERROT
Guillaume Richard
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-Modal
Transformers
BioAI
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Biological sequences encode fundamental instructions for the building blocks of life, in the form of DNA, RNA, and proteins. Modeling these sequences is key to understand disease mechanisms and is an active research area in computational biology. Recently, Large Language Models have shown great promise in solving certain biological tasks but current approaches are limited to a single sequence modality (DNA, RNA, or protein). Key problems in genomics intrinsically involve multiple modalities, but it remains unclear how to adapt general-purpose sequence models to those cases. In this work we propose a multi-modal model that connects DNA, RNA, and proteins by leveraging information from different pre-trained modality-specific encoders. We demonstrate its capabilities by applying it to the largely unsolved problem of predicting how multiple \rna transcript isoforms originate from the same gene (i.e. same DNA sequence) and map to different transcription expression levels across various human tissues. We show that our model, dubbed IsoFormer, is able to accurately predict differential transcript expression, outperforming existing methods and leveraging the use of multiple modalities. Our framework also achieves efficient transfer knowledge from the encoders pre-training as well as in between modalities. We open-source our model, paving the way for new multi-modal gene expression approaches.

Multi-modal Transfer Learning between Biological Foundation Models
Juan Jose Garau-Luis*
Patrick Bordes
Liam Gonzalez" InstaDeep
InstaDeep
InstaDeep
Masa Roller
Bernardo P. de Almeida
Lorenz Hexemer
InstaDeep
InstaDeep
BioNTech
Christopher Blum
Stefan Laurent
Jan Grzegorzewski BioNTech
Maren Lang BioNTech
BioNTech
BioNTech
Thomas Pierrot
Guillaume Richard
InstaDeep
InstaDeep
Abstract
Biological sequences encode fundamental instructions for the building blocks of life, in the form of DNA, RNA, and proteins. Modeling these sequences is key understand disease mechanisms and is an active research area in computational biology. Recently, Large Language Models have shown great promise in solving certain biological tasks but current approaches are limited to single sequence modality (DNA, RNA, or protein). Key problems in genomics intrinsically involve multiple modalities, but it remains unclear how to adapt general-purpose sequence models to those cases. In this work we propose a multi-modal model that con- nects DNA, RNA, and proteins by leveraging information from different pre-trained modality-specifet encoders. We demonstrate its capabilities by applying it to the largely unsolved problem of predicting how multiple RNA transcript isoforms originate from the same gene (i.e. same DNA sequence) and map to different tran- scription expression levels across various human tissues. We show that model, dubbed IsoFormer, able to accurately predict differential transcript expression, outperforming existing methods and leveraging the of multiple modalities. Our framework also achieves efficient transfer knowledge from the encoders pre- training well as in between modalities. We open-source model, paving the way for new multi-modal gene expression approaches.
Introduction
Foundation models have ignited revolution in numerous scientific fields, starting NLP and computer vision, and more recently in several domains within the life sciences. Within the biological sciences, these models have enabled predicting protein structures from sequences [1, 2]. deciphering the genome functions [3, 4, 5, 6] and interactions of biomolecules [7], and crafting
molecules
not found nature [8]. Specifically, significant progress has been made with foundation models tailored to DNA [3, 4, 5, 9], RNA [10, 11, 12, 13], and protein [2] sequences. These foundation models have typically been developed and trained separately, using either self-supervised learning techniques
"Equal Contribution
1 Equul Supervision
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

such as Masked Language Modeling (MLM), as seen in models like ESM [2] and the Nucleotide Transformer (NT) [3], or supervised learning approaches on large datasets, as in AlphaFold [1] and Enformer [14]. These models have been instrumental in advancing our understanding of biology by accurately predicting the structures and functions of biological sequences.
While existing methods provide relevant insights, they are still limited by the fact that they only consider single sequence modality. In biology, the central dogma describes the flow of genetic information from DNA to RNA to proteins [15]. This fundamental concept underscores the intercon- nectedness of these three types of biological sequences and highlights the potential for a unified modeling approach. An architecture that integrates DNA, RNA, and protein modalities should provide a comprehensive model for biology that mirrors the natural processes within cells. Furthermore. by enabling transfer learning across modalities, the model can capitalize on the vast amounts of pre-training already performed on individual DNA, RNA, and protein datasets.
Developing deep learning models using multiple biological sequence modalities has been mainly limited by the lack of matched available data; existing databases usually isolate a specific modality and thus relationships between modalities are not easily obtainable. As more multi-modal datasets are made available [16, 17], it is becoming possible to develop models that extract and combine the information from DNA, RNA, and protein sequences to better model the different cellular processes. Such multi-modal models have already been successful in other domains, such as mixing language and visual imputs [18, 19, 20, 21, 22. 23, 24, 25], but until now there are no models that can handle multiple biological sequence modalities.
In our work, we propose the first multi-modal architecture to connect DNA, RNA, and proteins (Fig. 1). Our approach is based three main components: (i) pre-trained modality-speciffe encoders that produce one embedding per modality, (ii) aggregation layers that combine information from the encoders and create multi-modal representation, and (iii) task-specific head that predicts the desired output. We show that our multi-modal approach transfers and aggregates knowledge of pre-trained mono-modal encoders and outperforms previous single-modality baselines (Enformer [14], NT [3], and ESM [2]). We also demonstrate the flexibility of our approach by comparing different encoders for a specific modality and different aggregation techniques. While some previous approaches have modeled specific interactions between modalities, such as protein-to-DN. interaction using structure information and modules [26], our approach is involving one or more biological sequence types.
and can be adapted to any task
Multi-modal
Multi-modal RNA embeddings
Multi-modal Protein embeddingg
DNA ombeddings
h
peox
Choss atlention
Cross attention
Cross attennioon
Crosp-attention
Crose-attention
Crosi-atteniion
hmm
prot
Uni-modal
Uni-modal
Uni-modal Protein embeddinge
DNA embedtingg
RNA embeddinga
Figure 1: Our aggregation module compiles information from the different biological sequence modalities of DNA, RNA, and proteins by using successive cross-attention layers and residual connections.
Significant problems in genomies intrinsically involve multiple sequence modalities [27, 26), and it is still unclear how to adapt general-purpose sequence models to those cases. In order to validate our multi-modal approach, we focused on the study of crucial task in genomics that has been challenging to tackle using single sequence modality- namely the prediction of RNA transcript isoform expression across different tissues [28]. When gene DNA sequences are transcribed into mRNA molecules to produce proteins, they generally do not express it in just single way producing single protein isoform. After DNA sequences are transcribed, pre-mRNA transcripts undergo a process called RNA alternative splicing where they are cut and re-assembled to form different variants of mature mRNA molecules that can be translated into proteins. This process allows for the generation of multiple RNA and protein isoforms from single gene DNA sequence, that can differ in structure, function, localization, or other properties (Fig. 2a). Therefore, predicting which isoforms are expressed in a given cell or under specific conditions is key to understand gene regulation and discase mechanisms.
2

---

This task is multi-modal in nature, since only looking at the DNA sequence present in the cell does not provide complete picture of the different RNA isoform landscape. Focusing on this single important task in opposition to tackling more but less challenging tasks should provide a stronger evidence for the effectiveness of our approach to advance biological knowledge.
In summary, our contributions are the following: (i) We built the first multi-modal model for the integration of DNA, RNA, and protein sequences. (ii) We show that our model achieves efficient transfer learning between the three modalities, not only leveraging intra-modalities pre-training but also inter- modalliilsss transfer. (iii) We use our architecture to tackle a new central task in biology that requires multi-modat approach, namely RNA transcript isoform expression prediction, and obtain oftte--o--he-att results, overcoming limitations of existing gene expression models such as Enformer. (iv) Finally, we performed ablation studies to validate our different architectural choices and release our trained isoform expression prediction model IsoForm providing new framework and baseline to the community and opening the door to future research on multi-mode sequence modeling and multi-modal biological problems.
2 Related Work
Biological sequence modeling Researchers have explored different ways to process DNA, RNA, and protein sequences for multiple applications. Initial approaches included dynamic programming [29], hidden Markov models [30], and genomic hash tables [31]. Recently, deep learning, via supervised [1] and unsupervised frameworks [3, 4, 13, 2]. has gained thrust in the community. These methods, influenced by advancements in Large Language Models [32, 33], have focused on DNA [3, 4, 5, 9], RNA [10, 11, 12, 13], and protein sequences [2]. They capture complex biological patterns and perform tasks like protein structure prediction [2] and variant effect prediction [34]. These efforts have become more targeted given the different challenges of each sequence modality. For instance, models like HyenaDNA [5, 9] or Caduceus [35] tackled the long-range dependencies in DNA. Few models consider multiple modalities, mainly focusing on structural information (e.g. predicting DNA interacting residues in proteins [26]). Our work. IsoFormer, is the first general- purposse model integrating three biological sequence modalities.
Multi-Modal Integration Efforts to address mudti-modality in deep learning models have been prominent in NLP [36, 37, 38, 39, 40] and computer vision [41, 42, 43, 44, 45). In computer vision, integrating image and audio for video classification [46] audio-visual segmentation [47] is common, often using cross-attention [48]. Unsupervised approaches using contrastive learning also integrate image, audio, and other modalities [49]. Large Language Models have driven multi-modal efforts for text and image with methods like Flamingo [19] and CLIP [18], leading to various integration architectures, including the Perceiver Resampler [50, 19] and C-Abstractor [51, 52). IsoFormer integrates DNA, RNA, and protein sequences, leveraging past integration architectures.
Gene Expression Prediction Transcript isoform expression prediction is more refined and challenging task within gene expression prediction. Traditionally, gene expression has been addressed through tailored approaches and experimental annotations [53, 54]. Recently, deep learning models have demonstrated improved results by predicting gene expression directly from DNA sequence [55, 56). Enformer [14] leveraged dilated convolutions and attention layers to extend the context window to 190 kilo base pairs (kbp), achieving new state-of-the-ar results on gene expression. However, these models are limited to gene-level expression levels and cannot predict isoform-specific expression as DNA sequence information is not sufficient to solve this task. We introduce IsoFormer, the first multi-modal model that is able predict transcript isoform expression.
3 Background
Central dogma of biology In this work, we consider three biological sequence types: DNA, RNA, and proteins. These sequences (composed of nucleotides or amino-acids, are fundamental to biological processes in living organisms. They are also intricately intertwined: DNA dictates RNA synthesis during transcription, and RNA guides protein synthesis through Iranslation (see Fig. 2: [15)). Thus, changes in DNA can alter RNA and protein sequences, impacting an organism's 's phenotype (i.e. function). DNA and RNA sequences consist of four nucleotides (ACGT and ACGU, respectively). while proteins are made of 20 amino. Thus, these sequences are commonly modelled as linear strings.
3

---

Obtaining data in this format is becoming more available thanks to recent advances in high- tigh-throughput sequencing technologies.
Gene expression and isoforms Gene expression is the process by which a gene's 's DNA sequence is transcribed into various RNA molecules that code for specific proteins. This process is complex, as a single gene can produce different RNA and protein isoforms with varying abundances across tissues (Fig. 2a). RNA isoforms are mRNA molecules of different exon compositions derived from the same gene, produced via processes like alternative splicing. The expression level of each isoform commonly measured by counting the amount of its RNA molecules in cells. Accurate isoform expression prediction across tissues can aid in understanding genetic variants' effects on cellular processes and phenotypes. However, this task cannot be tackled solely using DNA sequences as the same DNA sequence produces different isoforms in different cellular contexts and types. In addition, both DNA and RNA sequence features are crucial as DNA sequences contain regulatory elements that control transcription levels, while RNA isoform sequences have features affecting their stability and degradation.
4 Method
4.1 Multi-modal framework
We consider respectively DNA, RNA, and protein sequences Xdna Adna. Xrna E Ama and Xprot E Apoot where Adm is the DNA base alphabet ACGT, Ama is the RNA base alphabet ACGU, and Aprot is the set of 20 acins. We then consider three associated modality- encoders f with respective weights ama mma prox) that encode sequences x into corresponding embeddings h:
h
f (hdna: hyma: hprot), h'm.
JRRE (hmaa hana- hpros), h/mot = -/0 (hprots Idnas hma)
(I)
We assume that the weights (fana: O Ppeot) have been obtained through independent pre-training
processes that can involve supervised or oul--supereiee training techniques over large corpus of biological data. Note that in practice, commonly used in recent works [57], models pre-trained over DNA sequences can be re-used to produce embeddings for RNA sequences, replacing artificially the uracil base (U) by thymine (T) in the input. In this work, aim connect these encoders and train them jointly to learn a multi-modal embedding hmulti- We start by learning a multi-mode embedding per modality defined as
h/m = f// (hdna hma: hprot). h'm
f
(hma: hdnas hprot). D
f
(hprots Idna. hma)
(2)
where 1 is an aggregation function with weights 6. Then, we define the multi-modal embedding as the concatenation of the per modality multi-modas embeddings:
hmulti (mm h'ma b
Note that this definition is general as it allows the use of any aggregation function over the modality embeddings h. The multi- modality embeddings per modality h' are introduced to solve tasks that are mmodallty-centeredd For instance, h' could be used to solve a task that involves nucleotide-level annotation over a DNA sequence while requiring other modalities as input. We rely otherwise on the concatenated multi-modal embedding hmuiti to solve any other task.
4.2 Genes and isoforms expression
We now introduce our notations specific the task of RNA isoform expression prediction (Fig. 2a). We consider a DNA sequence Xdna of length L to contain gene g. In practice, given the input size limitation of the existing foundation models, the sequence length L. might be shorter than the full length of the gene. In this case, we choose the DNA sequence X to be centered on the start of the gene, i.e., where transcription begins, which also surrounds the promoter regions known to be important for transcription. This way, we also capture all the enhancer regulatory elements upstream of the transcription site.
We denote by X X the n existing transcripts for gene across all tissues. Coding transcripts (s) translated into proteins and denote by xpeat the amino-acid sequence of the protein associated
(1)

---

to the transcript Xma. We define the expression of genes and transcripts across tissues 7 as:
76
VI, (Xima> T)
Σ
T
R.
While deep learning models have been trained predict the overall expression genes across tissues with great accuracy [55, 56, 14, 58], to our knowledge no model can predict the expression of the different RNA transcripts across tissues directly from the sequence. In this work, we leverage our multi-modal framework to train transcript expression level prediction model, dubbed IsoFormer, that takes as input DNA sequence, an RNA transcript sequence, and its matching protein sequence to predict the expression of that transcript across tissues as measured by bulk RNA-seq (Fig. 2).
DNA sequence
RNA isoforms
Protein isoforms
Exoo
Transcription
Translation
DNA sequence transcribed different RNA isoforms depending on the context
Exon A T Intron
group nucleotidos translated into protein amino-acid
B
expression prediction head
An iscform will have specific expression levele per lissue
Expression lrvel
h
h
by
Multi-modsilty embeddings
Aggregation module
Tissue
ha
Single-modally embeddings
hmm
hyr
Pre-trained DNA encoder
Pre-trained RNA encoder
Pre-trained protein encoder
Xyut
Figure 2: a) Three types of biological sequences considered this work: DNA, RNA, proteins. These sequenees are composed of nuclcotides (DNA and RNA) or amino-acids (protcin). In single gene, several coding regions or exons can be used to create different RNA transcript isoforms and proteins. The abundance of each isoform is tissue-dependent and its measurement is called expression level. b) IsoFormer leverages pre-trained encoders that produce modality-specifie embeddings, which then aggregated into multi-modal embeddings. These are used predict the expression given RNA transcript isoform across multiple tissues.
Aggregation module To capture the local patterns and their relationship across modalities, introduce an embedding aggregation method based on cross-attention with residual connections (Fig. 1). The module applied each modality, performing cross-attention successively the other modalities to produce multi-modal embeddings that are specific to each modality by keeping their dimensionality and which can be also stacked. Note that this aggregation method is robust to the absence of specific modality as the cross-ates attenntotn term will be zeroed out. The final multi-modal embedding hmutt can then be used solve any task. Our model can be trained end-to-end and could be applied to different tasks by changing the prediction head.
4.3 Transferring from pre-trained biological encoders
Table I: Characteristics of cidgaityy encoders considered IsoFormer. t This column indicates the lokenization scheme (e.g., token corresponds to 6 nucleotides).
Modality
Model
Pre-training
Num. Params.
Tokens'
Sequence Length
DNA
Enformer [14]
Supervised
110M
Nucleotide
190,000
DNA
NT (y2) [3]
Self Self- Supervised
250M
6 Nucleotides
12,282
RNA
NT (v2)
Self Supervieed
250M
6 Nucleotides
12.282
Protein
ESM2 [2]
Self- Supervised
150M
Amino-acid
2,047
5

---

Encoders We describe here the encoders used to process each sequence modality (see also Table 1 and Fig. 2b). For protein sequences, we used ESM (5SM2-150M) [2], which handles any protein up to 2,048 amino-acids. The ESM models have been pre-trained through MLM on large corpuses of protein sequences and are considered state-of-the-an on multiple tasks including folding. Protein sequences were all within ESM input length, so no additional processing was required. For DNA sequences, we used the NT model [3], which has also been trained through self-supervision to reconstruct masked 6-mers within 12 kbp genomic regions from 850 species. Additionally to the NT, we also considered the Enformer model as DNA encoder. The Enformer is a different type of model that has been pre- trained through full supervision to predict multiple experiments related to chromatin accessibility and modifications, transcription factor binding, and more importantly gene expression. Enformer is strong candidate for our architecture's ' DNA encoder module it can process sequences up to 200kbp as well as one can expect to obtain transfer from its gene expression capabilities. Finally, while foundation models have been reported to be pre-trained been made publicly available at the moment and pre-training a foundation model from scratch is out of the scope of this study. As such, we re-used the NT model [3] to compute embeddings for RNA sequences as this model has been reported recently to also be able to solve RNA [57] and protein [59] tasks with simple adaptations, using the trick described in section 4. RNA transcripts longer than 12kb were left-cropped to 12kb to conserve the 3'UTR regions, which are crucial for mRNA stability and polyadenylations impacting isoform abundance. This adjustment affected only a small portion of the dataset.
RNA sequences, none of them has
Our architecture leverages three pre-trained encoders, one per bio-modality. well as the aggregation function defined above, to generate a multi-modality embedding mmuli (Fig. 2b). That embedding finally transformed by expression head fo with weights 1 make isoform expression level predictions across tissues. As the shape of the network output must be independent from the dimension of its inputs, we used a global average pooling over the length of the embedding. A linear layer then outputs one value per tissue. As long the encoders can take in biological sequence to produce an embedding, our method can function with different types of general- encoders. While this flexibility allows us leverage big part of the landscape of biological sequence encoders, for this work we chose the specific models described in the section below.
Architecture
4.4 Training
Objective We denote the IsoFormer model by fe.e. where θ is the concatenation of the weights of the encoder models. These weights are initialized to the values obtained after pre-training of the different encoders. Respectively e and & denote the weights of the aggregation module and expression prediction head. The IsoFormer is trained to minimize the following objective
LMSE Σ T
(hmatti 7)
[x(2
ma: ))
2
wherfe hmulti
Se. XXama=
x(:)
Xprot
(1)
where the summation is performed over a set of available tissues. Note that this framework can also accept only one or two of the three modalities as input. This is the case for instance when predicting expression of non-coding transcripts that do not translate into proteins.
Dataset We conducted our analysis of IsoFormer on RNA transcript expression data obtained from the GTEx² portal. We used Transcript TPMS measurements across 30 tissues, which come from more than 5,000 individuals. We followed a common process in gene expression datasets [56]: we averaged the expression levels for given tissue across individuals, and used the reference genome sequence as input. We mapped transcripts to their original genes and associated proteins using the Ensembl database [60]. Our resulting dataset is made of triplets of RNA transcript sequences, DNA sequences (centered on the Transcription Start Site (TSS) of the transcript), and proteins. In total, the dataset is made of ~10k unique transcripts, of which 90k are protein-coding and correspond 20k unique genes. Our dataset has fixed train and test set, divided by genes; all presented results correspond to the performance on the test set. We provide more details on the dataset Appendix A.
Hyperparameters We used the Adam optimizer with a learning rate of 3 10-5 and batch size of 64, and used early stopping on validation set comprised of 5% of the train set reduce training time. We also made our baseline model's weights available More details in Appendix B.
Pnttpp://www htwp:/rawal..ogg/o//////oooolooa http:p////gggggeooooooooeeeeeees https///wwwo//eacaaaae///////ooo
wtwp:/iteeeeeeeeeeeiiiio expression

---

5 Experiments
We present extensive experiments to assess the performance of our multi-modal approach on the transcript isoform expression prediction task and compare it with existing single-modality approaches. We show that (i) our architecture efficiently aggregates modalities to improve its performance on this task; (ii) by using a tailored model for expression prediction as a base DNA encoder, IsoFormer reaches satt--o-ttheaar performance; (iii) we provide an extensive ablation study on different aggregation approaches; (iv) we demonstrate that our approach achieves transfer learning both intra-modalities from their independent pre-training as well as inter-modalities.
5.1 Bridging three foundational models outperforms mono-modal approaches
Experiment We investigated the effect of adding the different modalities within our multi-modal framework for the prediction of expression of each RNA transcript across different human tissues. We used NT as the foundation model for both DNA and RNA and ESM as the protein encoder. We compared our multi-modal approach (DNA + RNA + protein) with models trained with different combinations of modalities as input: DNA only, RNA only, protein only, DNA protein and DNA + RNA. Results were obtained over 5 random seeds; for each random seed we change the validation set and randomly initialize the non pre-trained parameters (Ø, 1)) of our model. We report both R which measures how well each model predicts the actual values of expression, and Spearman correlation across tissues, which is metric for ranking transcripts based their expression in each tissue.
Table Performance of different variants IsoFormer for the prediction of transcript isoform expression. R² and Spearman correlation across tissues for 5 different random seeds is reported. NT is used as both DNA and RNA encoder while ESM is used to process protein sequences.
Model Imput
R
Spearman
DNA only
0.13 + 0.02
0.43 + 0.01
RNA only
0.36 + 0.03
0.61 + 0.01
Protein only
0.20 0.01
0.46 + 0.01
DNA Protein
0.28 0.01
0.52 1 0.01
DNA RNA
0.39 + 0.01
0.64 + 0.01
DNA RNA Protein
0.43 + 0.01
0.65 + 0.01
Results We observed that our approach benefits from adding more modalities as the performance increases from one modality alone to having two combined, and the best performance is achieved with the three (DNA, RNA, and protein) modalities together (Table 2). This is true for both Spearman correlation and R² metrics, with stronger improvement for the latter reflecting a more accurate prediction of the actual values of expression and not just the ranking of transcripts. This is a strong demonstration that our model can aggregate information across modalities improve performance on this isoform expression task. addition, we observe increased performance by using DNA together with RNA compared with DNA and protein information. This can be related to the strong importance of the UTR regions of the RNA sequence in the regulation of its degradation and stability [61]. which affect its final expression level in the cells, that are not captured at the protein level.
5.2 Enformer as DNA encoder module to obtain transfer between expression prediction tasks
Experiment To showcase the flexibility of IsoFormer towards different modality-specific encoders, we tested replacing NT by the Enformer model [14] as DNA encoder. Enformer has been trained over gene-level expression data obtained from CAGE assays (one value of expression per gene per tissue), while our model is trained to predict RNA transcript expression data obtained from bulk RNA-seq assay (one value of expression for each isoform of a given gene per tissue) and therefore represents a different challenge that cannot be tackled from the DNA sequence alone. Still, as the Enformer has been trained to predict gene-level expression across tissues directly from DNA sequences, a related task to predicting RNA isoform expression, one might expect to obtain transfer by using it DNA encoder. We give full details of this experiment in Appendix D.
We obtained superior performance using the Enformer instead of NT as pre-trained DNA encoder both as DNA-only but also when we combined with the RNA and protein encoders (Table 3). Importantly, also with the Enformer our framework benefits from bridging modalities. This improved
Results
7

---

Table 3: Comparison Enformer and NT DNA encoders used IsoFormer. It and Spearman correlation
across tissues on the transcript isoform expression prediction task. Standard deviation across 5 seeds is reported.
Model
R
Spearman
Enformer
0.21 + 0.01
0.46 + 0.00
IsoFormer (NT)
0,43 = 0.01
0.65 = 0.01
IsoFormer (Enformer)
0.53 = 0.01
0.72 = 0.00
IsoFormer (Borzoi)
0.48 + 0.01
0.69 + 0.00
SUTR
3UTR
CDS
R2 per tssue
5:00
Exlorer
875
035
475
Blood
530
038
Brain
025
Live
Stamach
R2
11ead
Meadd
1Tead
Figure 3: Left: Performance IsoFormer and Enformer [14] per tissue on selected subset of tissues. Right: Changes in attention in the RNA encoder during fine-tuning. These scores are reported for three genomics elements of interest for all heads and layers of the RNA encoder.
performance can be explained by the fact that Enformer has been pre-trained on the related task of gene expression prediction and thus its embeddings are better aligned with the isoform prediction task. Moreover, the Enformer is a model that can handle sequences of large context (up to 196k nucleotides), enabling it to capture long-range dependencies, known to be relevant for expression. We also tried the Borzoi model [] as an enocder for DNA model. Borzoi is a model pre-trained on sequences up to 51 nucleotides to predict RNA-seq coverage which directly relates to RNA transcript isoform exression. These results demonstrate that our multi-modal framework can be improved by leveraging more domain-specitn achieved using the Enformer as DNA encoder, we will use it as DNA encoder by default for all the following experiments. We make the weights of this IsoFormer model available on HuggingFace"
encoders. As our best model for isoform prediction is
Interpretation We report the performance of IsoFormer across selected tissues in Fig. 3-1eft (results across all tissues are presented in Appendix D). IsoFormer obtains similar performance across tissues despite tissues having different distributions of expression levels. To gain additional insights about the representations learned by IsoFormer, we analysed the attention layers inside the RNA encoder as it is the one providing stronger improvement on this task. Specifically, we compared how the attention distribution within each layer and head changes when we finetune the RNA encoder alone versus finetuning IsoFormer altogether. We report changes in attention scores at each layer and head of the RNA encoder for three genomics elements known to have a strong effect on the isoform splicing and gene expression processes, namely the 3UTR, 5UTR and CDS sequence, see Fig. 3-right (additional details on these scores are in Appendix Section C. The results show that, when finetuning using the three modalities, different layers specialize to capture specific features relative to isoform splicing and expression. Notably, the middle set of layers put higher attention weights to 3UTR regions whereas the top layers of NT attributes higher attention weights on CDS and SUTR. We assume that this RNA encoder specialization during finetuning is key to achieve strong representation towards the prediction of its tissue-specific expression.
5.3 Ablation studies the aggregation strategy
We compared the IsoFormer's aggregation module with alternative strategies from recent multi-modal literature (Fig. 4). Inspired by recent vision-text models [20, 52, 62, 63], we considered these three approaches: (i) Perceiver Resampler [50]: a variant of our cross-attention method using a Perceiver Resampler module. This block learns a fixed number of tokens for each modality, thus reducing the cost of the subsequent cross-attention layer. (ii) Linear Projection: a strategy that linearly projects the embeddings of the three modalities into common representation space and concatenates all tokens. This concatenated sequence is fed to Perceiver Resampler to
Experiment
8

---

Cross assss-tttonion
Resampler and cross attention
C- Abstractor
Linear projection
Croesvalleniion
Cross-attention
C-Abstractor
Resampler
Resempler
1-0 Resicet
Maar pool Rescet
Croes-ellention
Cross-altention
Rosamplor
C-Aostractor
Lineer projection and concatonation
Dm
Dgeect
RNA embeddings
DNA embedddings
Protein embeddings
Multi-motal DNA embeddinggs
Latent queries
Figure 4: Different aggregation strategies compared during the ablation studies. The figures show the spe-
cific case for obtaining multi-modal DNA embeddings h the same structure is used to obtain multi-modal RNA and protein embeddings (h/m and Blwee respectively). In all cases, the Resampler module is a Perceiver Resampler [50] and the Mean pooling block is the Adaptive mean pooling operator used in [20].
learn fixed number of tokens which are then used in the head of the model. (iii) C-Abstractor I-dimensional version the C-abstractor architecture that provides a compromise between flexibility -choosing arbitrary number of resampled tokens- and locality preservation [20].
Table Ablation study different aggregation strategies when considering the modalities together. experiments were run using Enformer as the DNA encoder. PR = Perceiver Resampler.
Aggregation Method
R
Spearman
Ours
0.53 1 0.01
0.72 1 0.00
PR + Cross-Attn
0.49 + 0.04
0.69 1 0.02
Linear Proj. PR
0.49 = 0.02
0.69 + 0.01
C-Abstractor
0.53 = 0.01
0.72 0.01
Results We performed hyperparameters search with the same budget used for the IsoFormer aggregation module for all ablations methods. We report in Table 4 the results obtained with the best set of hyperparameters for each method. We observe that, using an additional step of Perceiver Resampler always performs worse than only cross-attention -even after optimizing hyperparameters. Similarly, C-Abstractor does not confer any benefit over cross-aftention; therefore we consider this latter strategy the optimal aggregation strategy for our method. One advantage of the cross-attention mechanism we use is its interpretability, since it helps understanding which regions of the different modalities are being leveraged to make predictions. These conclusions align with recent multi-modal studies for other modalities [64].
5.4 IsoFormer leverages knowledge of pre-trained encoders
Experiment We showed in previous sections that IsoFormer is able to aggregate information from different biological sequence modalities to formulate high-quality predictions. Here, we investigated which extent IsoFormer's performance can be attributed to the transfer from each modality encoders" pre-training. We conduct extensive experiment where we compared the performance of IsoFormer trained using all three encoders pre-trained on their respective domains with different IsoFormer model variants with all possible combinations of pre-trained / non-pretrained modality- specific encoder.
Results Our results demonstrate that using pre-trained encoders confers a substantial advantage to IsoFormer, the R²2 is substantially larger (0.53) when compared with IsoFormer with none of the encoders pre-trained (0.10; Table 5). This demonstrates that IsoFormer is leveraging the knowledge acquired by each foundation model in the respective domains. However, we observed that when we randomly initialized only the DNA or RNA encoders, the drop in performance is smaller (0.41 for DNA and 0.48 for RNA). This suggests that IsoFormer not only leverages intra-modalities pre-training but also inter-modalities transfer. Altogether, these results underpin our approach of relying initializing IsoFormer with pre-traine trairre encoders, as the information learned during pre-training transferred and leveraged when considering multi-modal tasks.

---

Table 5: Comparing the use of pre-trained and non-pre- non-pre-trained encoders within IsoFormer. For this set of experiments the considered encoders are the Enformer for DNA, NT for RNA and ESM for proteins. indicates the use of pre-trained encoder whereas indicates the encoder trained from scratch (random initialization).
DNA
RNA
Protein
R
Spearman
X
X
X
0.10 1 0.03
0.31 + 0.01
X
X
0.45 1 0.01
0.67 1 0.00
1
X
0.39 0.01
0.61 0.00
X
V
0.34 1.0 0.01
0.59 = 0.01
X
0,52 + 0.01
0.71 = 0.00
X
V
0.48 + 0.01
0.69 = 0.00
X
J
V
0.41 0.01
0.64 - 0.01
0.53
0.72 = 0.00
Conclusion
IsoFormer the first model designed for multi-moda biological sequence modeling connecting DNA, RNA, and protein sequences. IsoFormer achieves state-of-the-an results by effectively leveraging and transferring knowledge from pre-trained DNA-, RNA-, and protein-specife encoders on one of the significant multi-modal problems in genomics: RNA transcript isoform expression prediction. As part of our efforts, we are are open-sourcing our model, and hope IsoFormer paves the way to new milestones in building multi-modal models for biology.
10

---

References
[1] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool. Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589 2021.
[2] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary--cale prediction atomic-level protein structure with language model. Science, 379(6633) 2023.
[3] Hugo Palla-Torre Liam Gonzalez, Javier Mendoza Revilla, Nicolas Lopez Carranza, Adam Hen- ryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop. Hassan Sirelkhatim, Guil- laume Richard, Marcin Skwark, Karim Beguir, Marie Lopez, and Thomas Pierrot. The nu- cleotide transformer: Building and evaluating robust foundation models for human genomics. bioRxiv, 2023.
[4] Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert- 2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15000 2023.
[5] Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch- Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024.
[6] Jun Cheng, Guido Novati, Joshua Pan, Clare Bycroft, Akvilè Zemgulyté, Taylor Applebaum, Alexander Pritzel, Lai Hong Wong, Michal Zielinski, Tobias Sargeant, Rosalia G. Schnei- der, Andrew W. Senior, John Jumper, Demis Hassabis, Pushmeet Kohli, and Ziga Avsec Accurate proteome-wide missense variant effect prediction with alphamissense. Science, 381 (8666))))dd77492 2023.
[7] Ahmara G Ross, Brahim Chaqour, Devin S McDougald. Kimberly E Dine, Thu T Duong, Ryan E Shindler, Jipeng Yue, Tehui Liu, and Kenneth Shindler. Selective upregulation of sirtl expression in retinal ganglion cells by aav-mediated gene delivery increases neuronal cell survival and alleviates axon demyelination associated with optic neuritis. Biomolecules, 12(6):830, 2022.
[8] Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature, 624(7990):80-88) 2023.
[9] Eric Nguyen, Michael Poli, Matthew G. Durrant, Armin W. Thomas, Brian Kang, Jeremy Sullivan, Madelena Ng. Ashley Lewis, Aman Patel, Aaron Lou, Stefano Ermon, Stephen A. Baccus, Tina Hermandez-Boussaae Christopher Ré, Patrick D. Hsu, and Brian L. Hie. Sequence modeling and design from molecular to genome scale with evo. bioRxiv, 2024.
[10] Ken Chen, Yue Zhou, Maolin Ding, Yu Wang, Zhixiang Ren, and Yuedong Yang. Self-supervised learning on millions of pre-mma sequences improves sequence-based ma splicing prediction. bioRxiv, 2023.
[11] Albi Celaj, Alice Jiexin Gao, Tammy T.Y. Lau, Erle M. Holgersen, Alston Lo. Varun Lodaya, Christopher B. Cole, Robert E. Denroche, Carl Spickett, Omar Wagih, Pedro O. Pinheiro, Parth Vora, Pedrum
-iiemmmmddd--heraaiiiii, Steve Chan, Zach Nussbaum, Xi Zhang, Helen Zhu, Easwaran Ramamurthy, Bhargav Kanuparthi, Michael lacocca, Diane Ly, Ken Kron, Marta Verby, Kahlin Cheung-Ong Zvi Shalev, Brandon Vaz, Sakshi Bhargava, Farhan Yusuf, Sharon Samuel, Sabriyeh Alibai, Zahra Baghestani, Xinwen He, Kirsten Krastel, Oladipo Oladapo, Amrudha Mohan, Arathi Shanavas, Magdalena Bugno, Jovanka Bogojeski, Frank Schmitges, Carolyn Kim, Solomon Grant, Rachana Jayaraman, Tehmina Masud, Amit Deshwar, Shreshth Gandhi, and Brendan Frey. An ma foundation model enables discovery of disease mechanisms and candidate therapeutics. bioRxiv, 2023.

---

[12] Sizhen Li, Saeed Moayedpour, Ruijiang Li, Michael Bailey, Saleh Riahi, Milad Miladi, Jacob Miner. Dinghai Zheng, Jun Wang. Akshay Balsubramani, Khang Tran, Minnie Zacharia. Monica Wu, Xiaobo Gu, Ryan Clinton, Carla Asquith, Joseph Skalesk, Lianne Boeglin, Sudha Chivukula, Anusha Dias, Fernando Ulloa Montoya, Vikram Agarwal, Ziv Bar-Joseph, and Sven Jager. Large language models for mrna design and optimization. bioRxiv, 2023.
Codonbert:
[13] Yekaterina Shulgina, Marena I Trinidad, Conner J Langeberg, Hunter Nisonoff, Seyone Chithrananda, Petr Skopintsev, Amos Nissley, Jaymin Patel, Ron Boger, Honglue Shi, et al. Rna language models predict mutations that improve ma function. bioRxiv, pages 2024-04 2024.
[14] Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska- Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley, Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10): (110)))116-12333 2021.
[15] Francis Crick. Central dogma of molecular biology. Nature, 227:561- 1970.
[16] Katarzyna Tomezak, Patrycja Czerwińska, and Maciej Wiznerowicz. Review the cancer genome atlas (tega): an immeasurable source knowledge. Contemporary OncologyWsspoczeena Onkologia, 2015(1):68-77, 2015.
[17] Patroklos Samaras, Tobias Schmidt, Martin Frejno, Siegfried Gessulat, Maria Reinecke, Anna Jarzab, Jana Zecha, Julia Mergner, Piero Giansanti, Cans-Christian Ehrlich, al. Proteomicsdb: a multi-omice and multi- oulti-organism resource for life science research. Nucleic acids research, 48(DI):DI153-DII 2020.
(18) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, PMLR, 2021.
pages
[19] Bapttste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736. 2022.
[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[21] Siddharth Srivastava and Gaurav Sharma. Omnivec: Learning robust representations with cross modal sharing. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1236-122 1248, 2024.
[22] Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-text generation for visual- semantic arithmetic. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17918-17928, 2022.
[23] Siyu Lu, Mingzhe Liu, Lirong Yin, Zhengtong Yin, Xuan Liu, and Wenfeng Zheng. The multi-modal fusion in visual question answering: review of attention mechanisms. PeerJ Computer Science, 9:e 1400, 2023.
[24] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas Montine, and James Zou. A visual-language foundation model for pathology image analysis using medical twitter. Nature Medicine, 29:2307-23 2023.
[25] Ming Y. Lu, Bowen Chen, Drew F. Williamson, Richard Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, Anil V. Parwani, Andrew Zhang, and Faisal Mahmood. A visual-language foundation model for computational pathology. Nature Medicine, 30:863-874 2024.
[26] Junkang Wei, Siyuan Chen, Licheng Zong, Xin Gao, and Yu Li. Protein-ma interaction prediction with deep learning: structure matters. Briefings in bioinformatics, 23)1 :bbbb5444 2022.
12

---

[27] Yehudit Hasin, Mareus Seldin, and Aldons Lusis. Multi-omics approaches to disease. Genome biology, 18:1-15, 2017.
[28] Sambit K Mishra, Viraj Muthye, and Gaurav Kandoi. Computational methods for predicting functions at the mma isoform level. International Journal of Molecular Sciences, 26616))668, 2020.
[29] Heng Li. Protein-to-genee alignment with miniprot. Bioinformatics, 39(1):btad014, 2023.
[30] Byung-Jun Yoon. Hidden markov models and their applications in biological sequence analysis. Current genomics, 10(6):402-415, 2009.
[31] Thomas D Wu, Jens Reeder, Michael Lawrence, Gabe Becker, and Matthew Brauer. Gmap and gsnap for genomic sequence alignment: enhancements to speed, accuracy, and functionality. Statistical genomics: methods and protocols, pages 283-334, 2016.
[32] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1870.04805, 2018.
[33] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[34] Nadav Brandes, Grant Goldman, Charlotte H Wang, Chun Jimmie Ye, and Vasilis Ntranos. Genome-wide prediction of disease variant effects with deep protein language model. Nature Genetics, 55(9): 1522, 2023.
[35] Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234, 2024.
[36] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: vl: A frontier large vision- laiguage model with versatile abilities. CoRR, abs/2308.1 12966, 2023.
[37] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang. Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschamnen, Arsha Nagrani. Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, A. J. Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim Alabdulmobsin, Lucas Beyer, Julien Amelot, Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali-x: On scaling up a multilingual vision and language model. CoRR, abs/2305. 18565, 2023.
[38] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato,, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 8469-8488, PMLR, 2023.
[39]
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv. Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Nils Johan Bertil Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
13

---

[40] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, 2023, 23-29 July 2023, Honolulu, Hawaii, U.A, volume 202 of Proceedings of Machine Learning Research, pages 19730-19742. PMLR, 2023.
[41] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 7331-7341. Computer Vision Foundation / IEEE, 2021.
[42] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[43] Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text modeling with zero-shot transfer from contrastive captioners. CoRR, abs/2212.04979, 2022.
[44] Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Trans. Mach. Learn. Res., 2022, 2022.
[45] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao. Hongjie Zhang. Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang. and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. CoRR, abs/2212.03199 2022.
[46] Muhammad Bilal Shaikh, Douglas Chai, Syed Mohammed Shamsul Islam, and Naveed Akhtar. Maivar-t: Multimodal audio-image and video action recognizer using transformers. In 11th European Workshop on Visual Information Processing, EUVIP 2023, Gjovik, Norway, September 11-14, 2023, pages 1-6. IEEE, 2023.
[47] Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. Avsegformer: Audio- visual segmentation with transformer. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eightn AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty- Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 12155-12163. AAAI Press, 2024.
[48] Yaoting Wang, Weisong Liu, Guangyao Li, Jian Ding, Di Hu, and Xi Li. Prompting segmentation with sound is generalizable audio-visus source localizer. In Michael Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Viihth-Eigtth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 5669-5677. AAAI Press, 2024.
[49] Yang Liu, Ying Tan, and Haoyuan Lan. Self-supervised contrastive learning for audio-visual action recognition. In IEEE International Conference on Image Processing, ICIP 2023, Kuala Lumpur, Malaysia, October 8-1 1, 2023, pages 1000-1004. IEEE, 2023.
[50] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651-4664. PMLR, 2021.
[51] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality- enhanced projector for multimodal 11m. arXiv preprint arXiv:2312.06772. 2023.
[52] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng. Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training arXiv preprint arXiv:2403.09611 2024.
14

---

[53] Chao Cheng, Roger Alexander, Rengiang Min, Jing Leng, Kevin Y Yip, Joel Rozowsky, Koon- Kiu Yan, Xianjun Dong, Sarah Djebali, Yijun Ruan, et al. Understanding transcriptional regulation by integrative analysis of transcription factor binding data. Genome research, 168661668- 2012.
[54] Alvaro J González, Manu Setty, and Christina S Leslie. Early enhancer establishment and regulatory locus complexity shape transcriptional programs in hematopoietic differentiation. 2015.
Nature genetics, 47(11):
[55] David R Kelley, Yakir A Reshef, Maxwell Bileschi, David Belanger, Cory Y McLean, and Jasper Snoek. Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome research, 28(5):73--750 2018.
[56] Jian Zhou, Chandra L Theesffll, Kevin Yao, Kathleen M Chen, Aaron K Wong, and Olga G Troyanskaya. Deep learning sequence- based ab initio prediction variant effects on expression 2018.
and disease risk. Nature genetics,
[57] Guillaume Richard, Bernardo P de Almeida, Hugo Dalla-Torre, Christopher Blum, Lorenz Hexemer, Priyanka Pandey, Stefan Laurent, Marie P Lopez, Alexander Laterre, Maren Lang, et al. Chatnt: A multimodal conversational agent for dna, raa and protein tasks. bioRxiv, pages 2024 04, 2024.
[58] Johannes Linder, Divyanshi Srivastava, Han Yuan, Vikram Agarwal, and David Kelley. Predicting ma-seq coverage from dna sequence unifying model of gene regulation. bioRxiv, pages 2023-08, 2023.
[59] Sam Boshar, Evan Trop, Bernardo P de Almeida, and Thomas Pierrot. Are genomic language models all you need? exploring genomic language models on protein downstream tasks. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations, 2024.
[60] Fergal J Martin, M Ridwan Amode, Alisha Aneja, Olanrewaju Austine- Orimoloye, Andrey G Azov, Barnes, Ame Becker, Ruth Bennett, Andrew Berry, Jyothish Bhai, et al. Ensembl 2023. Nucleic acids research, 51(DI):D933-D94) 2023.
[61] Woo Seuk Koh, Joshua R Porter, and Eric Batchelor. Tuning of mmna stability through altering 3' uutr sequences generates distinct output expression in a synthetic circuit driven by p53 oscillations. Scientific reports, 91)::997,, 2019.
[62] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[63] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision- understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022.
[64] Brandon McKinzie, Zhe Gan, Jean-philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzii Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang. Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong. Aonan Zhang, Jianyu Wang, Chong Wang. Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. MM1: methods, analysis & insights from multimodal L.M pre-training. CoRR, abs/2403.09611 2024.
15

---

A Dataset details
We based our dataset on the Genotype-Tissur Expression (GTEx) portal. Specifically, we use the 8th release of the Transcript TPMs table This table is made of expression measurements of 170k: transcripts across 30 different tissues from 5, 000 individuals. As the goal is to build a general model for expression prediction from biological sequences, we averaged measurements across individuals to get an average expression value for each transcript in each tissue.
We mapped each transcript to its corresponding gene and protein using the Ensembl' database. Using this database, we were able to retrieve associated DNA, RNA, and protein sequences, For DNA sequences, we used the latest release of the human reference genome GRCh38
To summarize, the steps re-create our training dataset are the following:
1. Download Transcript TPMs table from GTEx portal
2. Compute average measurement per transcript ID and tissue
3. Map RNA transcript isoform ID to its corresponding RNA sequence using Ensembl
4. Get associated protein isoform using Ensembl
5. Get chromosome and transcription start site position on the DNA sequence
Average Expression per Tissue
Expression Standard Deviation per Tissue
15093
10089
200
Sales Gest
Sah
Cent cont
Gast
Hod
Figure 5: Average and standard deviation of expression values across transcripts per tissuc.
B Training and architecture details
All experiments were carried out with 5 seeds on A 100 GPUs (80GB RAM). Depending on the model, training lasts between 1 and 5 hours. We provide model hyper-parameters in Table and the hyper-parameters of the encoders Table
In order to stabilize training procedure, expression values were processed in two steps: first, use a log-transform (val log(1 + val)) to minimize the effect of outliers; secondly, we normalized expression values per tissue it has been shown stabilize training when using mean squared error loss.
C Experimental details
C.1 Attention maps analysis
Section 5.2 introduces significant results on IsoFormer's ability to leverage pre-trained modality- specific encoders. When finetuning the pre-trained encoders together, there specialization on specific elements of the sequence, as observed in the RNA attention maps in Figure 3-right. These maps have been obtained through the following process:
hhttps://wv/ ensembl. org/index. html
Phttpa://wws .ncbi. nlm. nim..iihhhee attp.g/w//aataaaess/eemme///////
http:///wwwwwtetttaa/aao www.:oor///ooooooee hrt.r/allorrg//ooo///oomoooasaa http//ddllltggge htt.a///aao/uuuuu--geeee/l///l// tissue expressiio
16

---

Table 6: Model
Hyper-parameee
Value
Cross-Atto number of heads
8
Perceiver Resampler: number of layers
1
Perceiver Resampler: number resampled tokens
8
C-ahstractor: kernel size
3
C-abstractor: number of residual layers
2
C-abstractor: number of resampled tokens
8
Maximum number of nucleotides in DNA sequences (Enformer)
196,608
Maximum number of nuclcotides in DNA sequences (NT-v2) Maximum number of nucleotides in RNA sequences
12,288
12,288
Maximum number of amino-acids in protein sequences
1,200
Table 7: Encoder haper-parameters
Hyper-parameti
Value
NT: maximum number of tokens
2,048
NT: number of attention heads
16
NT: embedding dimension
768
NT: number of layers activation
24
swish
Enformer: number of parameters
110M
Enformer: embedding dimension
1,536
Enformer: number of Transformer layers
8
ESM-2-150M: number of attention heads
20
ESM-2-150MS embedding dimension
640
ESM-2-150M: number of layers
30
1. Focusing on the pre-trained RNA encoder, we take the attention weights for each layer and head after running the IsoFormer finetuning process. For each layer and head, we compute how much attention (percentage) is directed towards a specific region of interest of the RNA sequence. The regions we consider are 3UTR, 5UTR, and CDS, and we use the following equation to compute the exact ratio of attention:
Σ, (())i(((i.)) > p))
p(f)
Σ X 2EXm Σ,
1
(6)
p)
where Xma is the set of RNA sequences in the test set, a(i.)) is the attention coefficient between tokens and j. fii is an auxiliary function that equals I token belongs to the region of interest in the sequence (e.g. 3UTR), and is a threshold value (we choose 0.01). We denote these attention maps by Pir-
2. We repeat this process for the finetuning run in which only the RNA encoder is used (RNA only in Table 2). We denote these attention maps by PNT-
3. For each layer and head, compute the ratio Ap (PI: (PP==PNT/PPO) PNT)/PNT- For simplicity, cap these values to (i.e., attention rate is doubled in the IsoFormer case compared to finetuning RNA encoder alone).
4. In addition, for both PIF and PNT. we can consider the samples in the test set X and build distribution of attention rates per element of interest. Comparing both distributions (the one coming from PIF and the one coming from PNT), we can carry out a t-test per layer and head. We follow these t-tests and select the combinations of layers and heads in which there are statistically significant differences (i.e., p 0.05) between Pu and PNr-
5. Figure 3-right shows the ratio Ap for those pairs of layers and heads in which statistically significant differences are observed. The rest is set to zero.
D Additional results
D.1 Full results every tissue
Figure shows the performance of TsoFormer using Enformer as DNA encoder for each of the 30 tissues. IsoFormer outperforms the DNA-only Enformer model on every tissue.
17

---

R2 per tissue
Isoformer
Enformer
0.6
0.0
Adipose Tiesue
Adrenal Gland
Bladder
Blond
Vessel
Brain
Breast
Colon
Esophagees
Heart
Kidney
Liver
Lung
Muscle
Nerve
Ovary
Pancreas
Pituttry
prostate
sallvery Gland
Skin
Small Intestine
Spleen
Stomare
Tests
Thyrold
Uierus
Vagina
Cervix
Figure 6: Performance of IsoFormer and Enformer [14] per tissue on all tissues.
D.2 Full ablation studies on modalities
Table 8: Performance of different variants of IsoFormer for the prediction of transcript isoform expression using three different DNA encoders: Nucleotide Transformer, Enformer, and Borzoi (input sequence length in parentheses). R²² and Spearman correlation across tissues for 5 different random seeds is reported. is used as RNA encoder while
used process protein sequences.
NT (12k)
Enformer (196k)
Borzoi (512k)
Base DNA Model
R
Spearman
R
Spearman
R
Spearman
DNA Only
0.13
0.43 0.01
0.21 = 0.02
0.46 0.00
0.12 0.02
0.35 0:35:000 0.01
RNA Only
0.36 - 0.03
0.61 0.01 0.01
0.36 - 0.03
0.61 + 0.01
0.36
0.61 0.01
Protein Only
0.20 + 0.01
0.46 0.01
0.20 + 0.01 0.46 + 0.01
0.20
0,46 0.01
RNA Protein
0.40 0.01
0.63
0.40 0.63
0.01
0.00
0.63 0.01
DNA + Protein
0.28
0.52 0.01
0.39
0.61 1.1::000
0.30
0.56 0.01
DNA RNA
0.39 = 0.01
0.64 0.01
0.52 + 0.01
0.72
0.47 -47
0.69 0.00
DNA + RNA + Prot.
0.43 1 0.01
0.65 0.01
0.53 0.01
0.71 0.00
0.48 1 0.01
0.69 1.00
18

---

E
Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim in the abstract that propose novel way to integrate different pre-trained biological sequence encoders to solve the RNA isoform expression task. This is proven by experimental results.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: While we do provide specific limitations section, we discuss limitations in the corresponding sections.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not claim novel theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
We provide extensive details on the dataset creation and processing as well architectural choices and training procedure. We also will open-source our model publication.
Justification:
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
At the current stage, we do not provide training data and code but will open- source our model at publication time. However, we provide extensive details for anyone willing to gather our training dataset.
Justification:
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We give full details of how defined our dataset, hyperparameters and training procedure. We also acknowledge that some of those hyperparameters were taken arbitrarily.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yss]
Justification: We ran each experiment over several seeds that account for variability between different initialization and train/validation splits. While we do not directly report error bars report standard deviations of our metrics over those seeds.
19

---

8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We give those details in the Appendix, where we specify the amount of compute as well as training times for each experiment.
9. Code Of Ethies
Question: Does the research conducted in paper conform, in every respect, with the / httpppp.....pp.c..tttiicccceeee. NeurlPS Code of Ethics https / http//ewuriipppeeeeee
Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
This paper is applied to RNA transcript isoform expression, an important biological process. Our approach can be of the tools used in pipeline to accelerate drug or vaccine design and as such have a positive societal impact.
Justification:
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used internal reproduction of existing assets but we properly credit authors for their original work.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We fully detail all architecture choices and experiments and also open-source our model weights.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
20


Reviews:
[{"review_id": "90BY5oyMeL", "reviewer": "Reviewer_p67W", "summary": "The paper models isoform relative abundance across tissues with a multimodal approach based on 3 pretrained encoders for DNA, RNA, and AA sequences.  DNA encoder uses a sequence centered on the gene’s TSS, RNA encoder uses the known isoform sequence from RNAseq and the protein encoder uses corresponding AA sequence.  They perform multiple ablations on the utility of having all 3 separate encoders and, given separate encoders, how to aggregate them into a single isoform specific embedding/prediction, and look at attention layers of RNA module to find biologically meaningful regions of attention.", "questions": "Some comments and questions:\n\nCentering on TSS will only capture regulatory elements within the chosen sequence length.  There could be distal or trans CREs outside\n\nGTEx database has less than 1000 donors, confirming the 5000 individuals claim?\n\nLooking at equation 5 in 4.4, how does f_psi depend on the tissue T? Is it a separate head per tissue, or are you predicting the vector of isoform abundance across tissues with one pass?  And is f_theta and f_phi the same f but different weights as f_psi?  Where does the summation over i take place?\n\n5.1 does ablations with one DNA encoder, then 5.2 shows superior performance with Enformer as the DNA encoder.  So the ablations in 5.1 may not be accurate with respect to this new encoder.  It also begs the question of how would the Enformer do as the RNA encoder as well.\n\nHow are RNA and protein sequence lengths handled when they’re longer than the model input sequence size?", "limitations": "The authors should be more explicit about the limitation of using reference genome and known isoform sequences and how this kind of sweeps splicing as a function of dna sequence under the rug.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Isoform level analysis using 3 separate pretrained encoders for DNA, RNA, and AA sequences is a good strategy.  The authors provide useful ablations on the utility of the multi modal approach and on modern strategies for combining those into a single embedding.  Looking for biolgoically meaningful interpretations of attention layers is useful.", "weaknesses": "I don’t think the authors can claim this is the first attempt to combine DNA, RNA, and AA modalities with techniques from NLP.   See the recent Evo work here https://www.biorxiv.org/content/10.1101/2024.02.27.582234v2 .  While they evaluate their performance against Enformer, that’s a large part of their own model.  So the evaluations have an intramural feel to them.  It’d be interesting to see how their strategy compares to other multi modal models such as Evo, and more RNA centric work like Borzoi, which looks at a more fine grained look of variant effects on the DNA to RNA relationship.  Looking at average isoform abundance across individuals is all well and good, but GTEx also has individual genomes, and genomic variation across individuals will also of course affect splicing patterns and which isoforms come from what individuals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the constructive comments and positive assessment of our work.\n\n> I don’t think the authors can claim this is the first attempt to combine DNA, RNA, and AA modalities with techniques from NLP. See the recent Evo work\n\nEvo is a model based solely on DNA sequences and has been applied to RNA and protein modalities using their respective DNA sequences, a technique used in recent models [1,2]. Evo was trained on prokaryotic data, while ours uses eukaryotic data. For eukaryotes, models trained on genomes show limitations due to data distribution differences arising from the presence of intronic regions [2]. Our approach uses different vocabularies per modality and models pre-trained on corresponding datasets. Therefore, Evo and our models are not directly comparable and have different strengths. Ours is the first to combine DNA, RNA, and AA modalities with their respective alphabets.\n\n> It’d be interesting to see how their strategy compares to other multi modal models such as Evo, and more RNA centric work like Borzoi\n\nWe have now compared our multi-modal approach using Enformer and Borzoi in addition to NT. Interestingly, using Borzoi as DNA encoder leads to the same conclusions on the positive effect of aggregating more modalities (see attached PDF, Table 1), maintaining the ranking between the different ablations. This observation supports the robustness of our approach. Additionally, Borzoi outperformed NT as a DNA encoder, which is expected since Borzoi is tailored for expression tasks. However, despite the Borzoi authors reporting similar performance to Enformer in modeling gene expression, we observed slightly lower performance for Borzoi.\n\n>  Looking at average isoform abundance across individuals is all well and good, but GTEx also has individual genomes, and genomic variation across individuals will also of course affect splicing patterns and which isoforms come from what individuals.\n\n> The authors should be more explicit about the limitation of using reference genome and known isoform sequences and how this kind of sweeps splicing as a function of dna sequence under the rug.\n\nGenomic variation across individuals will does affect splicing patterns, we are not accounting for that for now. GTEx also has individual genome data but we do not have access to it yet. We will highlight this point in the paper and propose as future experiments to incorporate genetic variants. Still, this should not impact the comparison between using the different modalities alone or in combination, as in all cases they use the reference genome sequences.\n\n> Centering on TSS will only capture regulatory elements within the chosen sequence length. There could be distal or trans CREs outside.\n\nIt is true that centering on TSS will only capture regulatory elements within the chosen sequence length. But when using Enformer we use a context window of ~190kb, thus capturing most distal enhancers, being the state-of-the-art method to capture such long interactions. Similarly, when using Borzoi in our new experiments, we use a context window of ~512kb.\n\n> GTEx database has less than 1000 donors, confirming the 5000 individuals claim?\n\nWe indeed had a typo and will correct it in the revised version, thanks for pointing this out.\n\n> Looking at equation 5 in 4.4, how does f_psi depend on the tissue T? Is it a separate head per tissue, or are you predicting the vector of isoform abundance across tissues with one pass? And is f_theta and f_phi the same f but different weights as f_psi? Where does the summation over i take place?\n\nf_psi returns indeed the vector of isoform abundance across tissues in one pass, returning a vector of n_t floating numbers where n_t is the number of tissues. The reviewer is right, the notation f is confusing. f_theta, f_phi and f_psi are different functions parametrized with different weights; we'll rename them f^{encoders}_theta, f^{aggregation}_phi and f^{expression}_psi to improve clarity. \n\nFor the sake of simplicity, we removed the summation over batches of data in the loss function. The summation over i takes place implicitly as we sample randomly isoforms and their expression in the dataset. We do not necessarily have all isoforms from the same gene within the same batch. We will add a sentence in the text to clarify this.\n\n> 5.1 does ablations with one DNA encoder, then 5.2 shows superior performance with Enformer as the DNA encoder. So the ablations in 5.1 may not be accurate with respect to this new encoder. It also begs the question of how would the Enformer do as the RNA encoder as well.\n\nWe have also now performed all encoder ablations for NT, Enformer, and Borzoi models, providing more insights about the benefits of combining multiple encoders (see attached PDF, Table 1). Many thanks for the suggestion.\n\nRegarding using Enformer as the RNA encoder, since it is long-range but most RNAs are shorter than 10kb, we favored the use of a more general RNA encoder. This could be done as future work, or rather using recently published RNA pre-trained models.\n\n> How are RNA and protein sequence lengths handled when they’re longer than the model input sequence size?\n\nWe will clarify in the revised version how we handle RNA and protein sequences that exceed the model's input size. Protein sequences were all within the model's maximum input length, so no additional processing was required. RNA transcripts longer than 12kb were left-cropped to 12kb to conserve the 3’UTR regions, which are crucial for mRNA stability and polyadenylation, impacting isoform abundance. This adjustment affected only a small portion of the dataset. We will include the sequence length distributions in the paper's supplementary material.\n\n[1] Richard, G., et al. \"ChatNT: A Multimodal Conversational Agent for DNA, RNA and Protein Tasks.\" bioRxiv (2024): 2024-04.\n\n[2] Outeiral, C., and C. M. Deane. \"Codon language embeddings provide strong signals for protein engineering. bioRxiv.\" preprint (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AdoAS6mcOS", "reviewer": "Reviewer_pNzb", "summary": "The paper introduces a new framework for the multi-modality pretrain model according to the Central dogma of biology. The method encode DNA, protein and RNA at the same time. The proposed method can transfer knowledge from the encoders pretraining and modalities.", "questions": "The authors should include more motivation about the proposed method. For example, why can the changed DNA influence the RNA seq? Why not directly predict RNA expression from RNA seq?", "limitations": "The biological system is more complex. The proposed method only include the direct map from DNA to RNA. However, in real world, RNA can effect the expression of DNAs. More details should be discuss in the future work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is well-organized and easy to follow\nThe authors have proved that the multi-modality of single cell data can help model predictions.", "weaknesses": "Lack of experiments. 1. More ablation studies should be conducted about removing different modalities of the model in Table 2 ( e.g. we observe only RNA can achieve a high performance, what about protein+RNA? ).\n\n 2. More dataset details should be included. The split of training/validation/test sets is not clear. If the authors do the experiments on the same dataset, they should split the dataset according to the tissues to validate the transferability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate this reviewer's comments and suggestions that will improve the revised manuscript.\n\n> More ablation studies should be conducted about removing different modalities of the model in Table 2 ( e.g. we observe only RNA can achieve a high performance, what about protein+RNA? ).\n\nWe have now performed the suggested, and missing, ablation studies for NT, Enformer and Borzoi (added now) models (see attached PDF, Table 1). We now can compare the results of using any combination of one, two, or three modalities with three different DNA encoders. These results reinforce the advantage of combining multiple modalities to solve the isoform expression prediction task. The new version of the paper will incorporate these results and their appropriate interpretation. Many thanks for the suggestion.\n\n> More dataset details should be included. The split of training/validation/test sets is not clear. If the authors do the experiments on the same dataset, they should split the dataset according to the tissues to validate the transferability of the proposed method.\n\nRegarding the data split in training/validation/test sets, we will provide more details on the methods. Briefly, we have followed standard approaches for sequence-based models where different chromosomes are held out for the different sets (chromosomes 1-19 for training, 20 for validation and 21-22 for test). This ensures no data leakage due to overlapping or recombinant sequences. Since the only input are DNA/RNA/AA sequences, we do not expect transferability between tissues and can only do tissue-specific predictions for the tissues used during training.\n\n> The authors should include more motivation about the proposed method. For example, why can the changed DNA influence the RNA seq? Why not directly predict RNA expression from RNA seq?\n\nWe will also include more motivation about the proposed method. For further clarification, here we are predicting isoform expression which is measured by RNA-seq. Thus, it would be trivial to predict it from RNA-seq and the main challenge is to predict it from DNA sequence. Transcript isoform expression is influenced by DNA regulatory elements in addition to RNA regulatory elements, and that is the main motivation for combining multiple encoders to improve performance on this task.\n\n> The biological system is more complex. The proposed method only includes the direct map from DNA to RNA. However, in the real world, RNA can affect the expression of DNAs. More details should be discussed in the future work.\n\nWe will also improve the discussion of the complexity of gene regulation and the system. Indeed, the fact that RNA can also affect in turn the expression of genes from DNA is one of the main motivations to mix DNA and RNA sequences in our approach, modeling interactions between biological modalities based on their sequence alone.\n\nWe hope our revisions address your concerns. We would be happy to expand on any of these points throughout the discussion period if further clarification is needed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PVGiTOdUKD", "reviewer": "Reviewer_Z3Wd", "summary": "The paper introduces a novel multi-modal model, IsoFormer, designed to integrate DNA, RNA, and protein sequences for predicting RNA transcript isoform expression across different tissues. It utilizes pre-trained modality-specific encoders to generate embeddings that are then combined using a sophisticated aggregation method. The model demonstrates significant improvements in prediction accuracy compared to single-modality approaches.\n\nContriburion:\n\n1. Developed the first general-purpose multi-modal model integrating DNA, RNA, and protein sequences.\n\n2. Demonstrated successful application of transfer learning from modality-specific encoders.\n\n3. Provided a new robust framework for advancing the prediction of RNA transcript isoform expression.", "questions": "(1)\tFor Tab 5., wonder what’s the performance for “DNA and RNA encoder not pre-trained”\n\n(2)\tCould authors provide evaluation results for DNA, RNA and protein encoder separately on their own popular benchmarking tasks? Ideally, these models should improve on those downstream tasks as well. Would give 7’ or even 8’ if authors conduct these experiments. It could be a great and exciting work in this AI4Bio field.", "limitations": "1. Data Requirements: The effectiveness of the model is contingent on the availability of comprehensive and high-quality multi-modal datasets.\n\n2. Generalizability: While promising, the results are primarily validated on specific types of gene expression data, and its performance across broader biological applications remains to be fully assessed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Innovative Integration of Modalities: The paper presents the first attempt to integrate three biological sequence modalities (DNA, RNA, and proteins) in a unified model, providing a comprehensive approach reflective of natural biological processes.\n\n2. Effective Transfer Learning: IsoFormer effectively leverages pre-trained encoders to enhance its predictive power, benefiting from both intra-modal and inter-modal transfer learning.\n\n3. Robust Evaluation: Experiments demonstrate the model's capability, outperforming existing methods in predicting transcript isoform expression, which is a challenging task due to its multi-modal nature.", "weaknesses": "1. Complexity and Computation: The model's complexity and the computational demands might limit its accessibility and use, particularly in environments with restricted resources.\n\n2. More Comprehensive Evaluation for PLM's representation learning capability would make this paper better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback and positive comments of our work.\n\n> For Tab 5., wonder what’s the performance for “DNA and RNA encoder not pre-trained”\n\nWe have now completed this ablation study with full evaluation of the effect of pre-training for each encoder (detailed in the attached PDF, Table 2). Interestingly, we observe in every setting that using a pre-trained encoder against its randomly initialized counterpart consistently leads to improved performance. This further demonstrates that IsoFormer leverages modality-specific pre-training.\n\n> Could authors provide evaluation results for DNA, RNA and protein encoder separately on their own popular benchmarking tasks? Ideally, these models should improve on those downstream tasks as well. Would give 7’ or even 8’ if authors conduct these experiments. It could be a great and exciting work in this AI4Bio field.\n\nWe evaluated the downstream performance of the Nucleotide Transformer model before and after fine-tuning it within the Isoformer on the isoform prediction task (when considering it as a base DNA model). We selected the prediction of gene expression measured by bulkRNA-seq from Chia Hsiang et al. [1] as it aligns with the fine-tuning task of the IsoFormer while relying on a different data source. We show a clear increase in performance of the NT model before and after fine-tuning within the IsoFormer framework (see attached PDF, Table 3). We will add these results to the paper. This highlights the potential of our approach to jointly pre-train models for different modalities. We will discuss in the conclusion section how this could be extended to a larger set of downstream tasks across modalities, which will probably require to expend the tasks and datasets used to train the IsoFormer. \n\n> Data Requirements: The effectiveness of the model is contingent on the availability of comprehensive and high-quality multi-modal datasets.\n\nThe reviewer brings up one of the key motivating aspects of our work. One of the biggest bottlenecks in the development of models integrating multiple biological sequences has been the lack of available matched data between DNA, RNA, and proteins. These multi-modal datasets are becoming more available but the amount of data is not enough to train models from scratch (as we demonstrate in Table 5 of the paper). Our approach bridges this gap by leveraging uni-modal encoders that have been trained on high-quality uni-modal datasets and achieving transfer learning across modalities. Our work demonstrates that this limitation can be overcome.\n\n[1] Kao, Chia Hsiang, et al. \"Advancing dna language models: The genomics long-range benchmark.\" ICLR 2024 Workshop on Machine Learning for Genomics Explorations. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their time reading our manuscript and for providing constructive feedback in their reviews.

We are glad the reviewers value positively our approach to combine different biological modalities together and emphasize that the experimental results support the paper’s contributions. We also acknowledge certain areas of the manuscript can be strengthened and are grateful to the reviewers for providing improvement suggestions on those areas. Below, we summarize the main concerns that have been raised during the review and the changes we have made to address them. Concrete concerns and questions are further discussed in the reviewer-specific responses.

**Full enumeration of modality combinations and use of RNA-centric models like Borzoi**

Reviewers suggested expanding the results presented in Tables 2 and 3 of the paper to include 1) any combination of one, two, or three modalities, and 2) the use of Borzoi as one of the encoders. We agree with the suggestion and present exhaustive results in Table 1 of the attached PDF. These results reinforce the advantage of combining multiple modalities to solve the isoform expression prediction task. In addition, we can observe that when using Borzoi as a DNA encoder the ranking between the different ablations is maintained. We will include this updated table in the final version of the paper.

**Full enumeration of using the pre-trained and non-pre-trained versions of each encoder**

Table 5 of our paper compares different cases in which only a subset of the encoders is pre-trained. This table demonstrates the effectiveness of transfer learning from pre-trained uni-modal encoders to achieve a strong performance in the multi-modal task. Reviewers suggested extending this table to include the 8 possible combinations of pre-trained vs. non-pretrained. We provide the complete ablation in Table 2 in the attached PDF. The extended results show in every setting that using a pre-trained encoder against its randomly initialized counterpart consistently leads to improved performance. We will add this table to the final version of the paper.

**Clarification of the data processing approach**

The reviewers have inquired about our specific use of the GTEx data. First, an interesting point has been made on using reference genome vs. individual genomes. We have opted to use the reference genome in our work because of the availability of the data. While we agree it would be interesting to use individual genome data, we believe that the reference genome is already a strong support for proving the main goal of the paper: combining multiple biological sequence modalities together to obtain superior performance in multi-modal tasks. We will discuss the use of individual genome data as future work in the final version of the paper. 

Second, reviewers have asked to provide further details on the train/validation/test splits. In our work we have created these splits based on chromosomes, which is a standard approach in sequence-based models and ensures there is no data leakage across splits. We add the specific splits in our response and in the final version of the paper. Finally, we have also addressed other specific points about the dataset such as the treatment of long sequences (we crop them, not necessary when using Enformer) and the motivation to look into DNA for a RNA-seq-related task (DNA and RNA are intrinsically linked and the expression of one modality is affected by the other and vice versa). We will reemphasize these discussion points in the final version of the paper.

---------

Overall, we hope these changes address the different concerns that have been raised and reflect our willingness to improve the paper in all possible ways. Please, do not hesitate to provide further feedback.

Meta-review: This work introduces IsoFormer, a multimodal approach that leverages pre-trained models on DNA, RNA, and protein sequences to predict transcript isoforms across multiple human tissues. The paper employs a modified cross-attention mechanism and concatenation scheme to create multi-modal embeddings, achieving state-of-the-art performance in isoform prediction. Throughout the review process, the benchmarking experiments and presentations have been refined and significantly improved. While the methods themselves may lack novelty, the introduction of the new challenge of predicting isoforms, combined with the availability of relevant data, will be of strong interest to NeurIPS audience.
Decision: Accept (poster)