Title: Multi-modal Transfer Learning between Biological Foundation Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multi-Modal

Transformers

BioAI
Abstract: Biological sequences encode fundamental instructions for the building blocks of life, in the form of DNA, RNA, and proteins. Modeling these sequences is key to understand disease mechanisms and is an active research area in computational biology. Recently, Large Language Models have shown great promise in solving certain biological tasks but current approaches are limited to a single sequence modality (DNA, RNA, or protein). Key problems in genomics intrinsically involve multiple modalities, but it remains unclear how to adapt general-purpose sequence models to those cases. In this work we propose a multi-modal model that connects DNA, RNA, and proteins by leveraging information from different pre-trained modality-specific encoders. We demonstrate its capabilities by applying it to the largely unsolved problem of predicting how multiple \rna transcript isoforms originate from the same gene (i.e. same DNA sequence) and map to different transcription expression levels across various human tissues. We show that our model, dubbed IsoFormer, is able to accurately predict differential transcript expression, outperforming existing methods and leveraging the use of multiple modalities. Our framework also achieves efficient transfer knowledge from the encoders pre-training as well as in between modalities. We open-source our model, paving the way for new multi-modal gene expression approaches.

Multi-modal Transfer Learning between Biological Foundation Models

Abstract

Biological sequences encode fundamental instructions for the building blocks of life, in the form of DNA, RNA, and proteins. Modeling these sequences is key to understand disease mechanisms and is an active research area in computational biology. Recently, Large Language Models have shown great promise in solving certain biological tasks but current approaches are limited to a single sequence modality (DNA, RNA, or protein). Key problems in genomics intrinsically involve multiple modalities, but it remains unclear how to adapt general-purpose sequence models to those cases. In this work we propose a multi-modal model that connects DNA, RNA, and proteins by leveraging information from different pre-trained modality-specific encoders. We demonstrate its capabilities by applying it to the largely unsolved problem of predicting how multiple RNA transcript isoforms originate from the same gene (i.e. same DNA sequence) and map to different transcription expression levels across various human tissues. We show that our model, dubbed IsoFormer, is able to accurately predict differential transcript expression, outperforming existing methods and leveraging the use of multiple modalities. Our framework also achieves efficient transfer knowledge from the encoders pretraining as well as in between modalities. We open-source our model, paving the way for new multi-modal gene expression approaches.

1 Introduction

Foundation models have ignited a revolution in numerous scientific fields, starting in NLP and computer vision, and more recently in several domains within the life sciences. Within the biological sciences, these models have enabled predicting protein structures from sequences [1, 2], deciphering the genome functions [3, 4, 5, 6] and interactions of biomolecules [7], and crafting new molecules not found in nature [8]. Specifically, significant progress has been made with foundation models tailored to DNA [3, 4, 5, 9], RNA [10, 11, 12, 13], and protein [2] sequences. These foundation models have typically been developed and trained separately, using either self-supervised learning techniques

such as Masked Language Modeling (MLM), as seen in models like ESM [2] and the Nucleotide Transformer (NT) [3], or supervised learning approaches on large datasets, as in AlphaFold [1] and Enformer [14]. These models have been instrumental in advancing our understanding of biology by accurately predicting the structures and functions of biological sequences.

While existing methods provide relevant insights, they are still limited by the fact that they only consider a single sequence modality. In biology, the central dogma describes the flow of genetic information from DNA to RNA to proteins [15]. This fundamental concept underscores the interconnectedness of these three types of biological sequences and highlights the potential for a unified modeling approach. An architecture that integrates DNA, RNA, and protein modalities should provide a comprehensive model for biology that mirrors the natural processes within cells. Furthermore, by enabling transfer learning across modalities, the model can capitalize on the vast amounts of pre-training already performed on individual DNA, RNA, and protein datasets.

Developing deep learning models using multiple biological sequence modalities has been mainly limited by the lack of matched available data; existing databases usually isolate a specific modality and thus relationships between modalities are not easily obtainable. As more multi-modal datasets are made available [16, 17], it is becoming possible to develop models that extract and combine the information from DNA, RNA, and protein sequences to better model the different cellular processes. Such multi-modal models have already been successful in other domains, such as mixing language and visual inputs [18, 19, 20, 21, 22, 23, 24, 25], but until now there are no models that can handle multiple biological sequence modalities.

In our work, we propose the first multi-modal architecture to connect DNA, RNA, and proteins (Fig. 1). Our approach is based on three main components: (i) pre-trained modality-specific encoders that produce one embedding per modality, (ii) aggregation layers that combine information from the encoders and create a multi-modal representation, and (iii) a task-specific head that predicts the desired output. We show that our multi-modal approach transfers and aggregates knowledge of pre-trained mono-modal encoders and outperforms previous single-modality baselines (Enformer [14], NT [3], and ESM [2]). We also demonstrate the flexibility of our approach by comparing different encoders for a specific modality and different aggregation techniques. While some previous approaches have modeled specific interactions between modalities, such as protein-to-DNA interaction using structure information and modules [26], our approach is general-purpose and can be adapted to any task involving one or more biological sequence types.

Cross-attention

+

Uni-modal

DNA embeddings

Uni-modal

RNA embeddings

Uni-modal

Protein embeddings

Multi-modal

DNA embeddings

Multi-modal

RNA embeddings

Multi-modal

Protein embeddings

Cross-attention

+

Cross-attention

+

Cross-attention

+

Cross-attention

+

Cross-attention

+

Significant problems in genomics intrinsically involve multiple sequence modalities [27, 26], and it is still unclear how to adapt general-purpose sequence models to those cases. In order to validate our multi-modal approach, we focused on the study of a crucial task in genomics that has been challenging to tackle using a single sequence modality - namely the prediction of RNA transcript isoform expression across different tissues [28]. When gene DNA sequences are transcribed into mRNA molecules to produce proteins, they generally do not express it in just a single way producing a single protein isoform. After DNA sequences are transcribed, pre-mRNA transcripts undergo a process called RNA alternative splicing where they are cut and re-assembled to form different variants of mature mRNA molecules that can be translated into proteins. This process allows for the generation of multiple RNA and protein isoforms from a single gene DNA sequence, that can differ in structure, function, localization, or other properties (Fig. 2a). Therefore, predicting which isoforms are expressed in a given cell or under specific conditions is key to understand gene regulation and disease mechanisms.

Isof

This task is multi-modal in nature, since only looking at the DNA sequence present in the cell does not provide a complete picture of the different RNA isoform landscape. Focusing on this single important task in opposition to tackling more but less challenging tasks should provide a stronger evidence for the effectiveness of our approach to advance biological knowledge.

In summary, our contributions are the following: (i) We built the first multi-modal model for the integration of DNA, RNA, and protein sequences. (ii) We show that our model achieves efficient transfer learning between the three modalities, not only leveraging intra-modalities pre-training but also inter-modalities transfer. (iii) We use our architecture to tackle a new central task in biology that requires a multi-modal approach, namely RNA transcript isoform expression prediction , and obtain state-of-the-art results, overcoming limitations of existing gene expression models such as Enformer. (iv) Finally, we performed ablation studies to validate our different architectural choices and release our trained isoform expression prediction model IsoFormer, providing a new framework and baseline to the community and opening the door to future research on multi-modal sequence modeling and multi-modal biological problems.

2 Related Work

Biological sequence modeling Researchers have explored different ways to process DNA, RNA, and protein sequences for multiple applications. Initial approaches included dynamic programming [29], hidden Markov models [30], and genomic hash tables [31]. Recently, deep learning, via supervised [1] and unsupervised frameworks [3, 4, 13, 2], has gained thrust in the community. These methods, influenced by advancements in Large Language Models [32, 33], have focused on DNA [3, 4, 5, 9], RNA [10, 11, 12, 13], and protein sequences [2]. They capture complex biological patterns and perform tasks like protein structure prediction [2] and variant effect prediction [34]. These efforts have become more targeted given the different challenges of each sequence modality. For instance, models like HyenaDNA [5, 9] or Caduceus [35] tackled the long-range dependencies in DNA. Few models consider multiple modalities, mainly focusing on structural information (e.g. predicting DNA interacting residues in proteins [26]). Our work, IsoFormer, is the first general-purpose model integrating three biological sequence modalities.

Multi-Modal Integration Efforts to address multi-modality in deep learning models have been prominent in NLP [36, 37, 38, 39, 40] and computer vision [41, 42, 43, 44, 45]. In computer vision, integrating image and audio for video classification [46] or audio-visual segmentation [47] is common, often using cross-attention [48]. Unsupervised approaches using contrastive learning also integrate image, audio, and other modalities [49]. Large Language Models have driven multi-modal efforts for text and image with methods like Flamingo [19] and CLIP [18], leading to various integration architectures, including the Perceiver Resampler [50, 19] and C-Abstractor [51, 52]. IsoFormer integrates DNA, RNA, and protein sequences, leveraging past integration architectures.

Gene Expression Prediction Transcript isoform expression prediction is a more refined and challenging task within gene expression prediction. Traditionally, gene expression has been addressed through tailored approaches and experimental annotations [53, 54]. Recently, deep learning models have demonstrated improved results by predicting gene expression directly from DNA sequence [55, 56]. Enformer [14] leveraged dilated convolutions and attention layers to extend the context window to 190 kilo base pairs (kbp), achieving new state-of-the-art results on gene expression. However, these models are limited to gene-level expression levels and cannot predict isoform-specific expression as DNA sequence information is not sufficient to solve this task. We introduce IsoFormer, the first multi-modal model that is able to predict transcript isoform expression.

3 Background

Central dogma of biology In this work, we consider three biological sequence types: DNA, RNA, and proteins. These sequences (composed of nucleotides or amino-acids) are fundamental to biological processes in living organisms. They are also intricately intertwined: DNA dictates RNA synthesis during transcription , and RNA guides protein synthesis through translation (see Fig. 2; [15]). Thus, changes in DNA can alter RNA and protein sequences, impacting an organism's phenotype (i.e. function). DNA and RNA sequences consist of four nucleotides (ACGT and ACGU, respectively), while proteins are made of 20 amino-acids. Thus, these sequences are commonly modelled as linear strings.

Obtaining data in this format is becoming more available thanks to recent advances in high-throughput sequencing technologies.

Gene expression and isoforms Gene expression is the process by which a gene's DNA sequence is transcribed into various RNA molecules that code for specific proteins. This process is complex, as a single gene can produce different RNA and protein isoforms with varying abundances across tissues (Fig. 2a). RNA isoforms are mRNA molecules of different exon compositions derived from the same gene, produced via processes like alternative splicing. The expression level of each isoform is commonly measured by counting the amount of its RNA molecules in cells. Accurate isoform expression prediction across tissues can aid in understanding genetic variants' effects on cellular processes and phenotypes. However, this task cannot be tackled solely using DNA sequences as the same DNA sequence produces different isoforms in different cellular contexts and types. In addition, both DNA and RNA sequence features are crucial as DNA sequences contain regulatory elements that control transcription levels, while RNA isoform sequences have features affecting their stability and degradation.

4 Method

4.1 Multi-modal framework

We consider respectively DNA, RNA, and protein sequences x dna ∈ A dna , x rna ∈ A rna and x prot ∈ A prot where A dna is the DNA base alphabet ACGT, A rna is the RNA base alphabet ACGU, and A prot is the set of 20 amino-acids. We then consider three associated modality-encoders f with respective weights ( θ dna , θ rna , θ prot ) that encode sequences x into corresponding embeddings h :

We assume that the weights ( θ dna , θ rna , θ prot ) have been obtained through independent pre-training processes that can involve supervised or self-supervised training techniques over large corpus of biological data. Note that in practice, as commonly used in recent works [57], models pre-trained over DNA sequences can be re-used to produce embeddings for RNA sequences, replacing artificially the uracil base (U) by thymine (T) in the input. In this work, we aim to connect these encoders and train them jointly to learn a multi-modal embedding h multi. We start by learning a multi-modal embedding per modality defined as

where f agg ϕ is an aggregation function with weights ϕ . Then, we define the multi-modal embedding as the concatenation of the per modality multi-modal embeddings:

Note that this definition is general as it allows the use of any aggregation function over the modality embeddings h . The multi-modality embeddings per modality h ′ are introduced to solve tasks that are "modality-centered". For instance, h ′ dna could be used to solve a task that involves nucleotide-level annotation over a DNA sequence while requiring other modalities as input. We rely otherwise on the concatenated multi-modal embedding h multi to solve any other task.

4.2 Genes and isoforms expression

We now introduce our notations specific to the task of RNA isoform expression prediction (Fig. 2a). We consider a DNA sequence x dna of length L to contain a gene g . In practice, given the input size limitation of the existing foundation models, the sequence length L might be shorter than the full length of the gene. In this case, we choose the DNA sequence x dna to be centered on the start of the gene, i.e., where transcription begins, which also surrounds the promoter regions known to be important for transcription. This way, we also capture all the enhancer regulatory elements upstream of the transcription site.

We denote by x (1) rna , ..., x ( n ) rna the n existing transcripts for gene g across all tissues. Coding transcripts are translated into proteins and we denote by x ( i ) prot the amino-acid sequence of the protein associated

to the transcript x ( i ) rna. We define the expression e of genes and transcripts across tissues T as:

While deep learning models have been trained to predict the overall expression of genes across tissues with great accuracy [55, 56, 14, 58], to our knowledge no model can predict the expression of the different RNA transcripts across tissues directly from the sequence. In this work, we leverage our multi-modal framework to train a transcript expression level prediction model, dubbed IsoFormer, that takes as input a DNA sequence, an RNA transcript sequence, and its matching protein sequence to predict the expression of that transcript across tissues as measured by bulk RNA-seq (Fig. 2).

A U C G A U

I

L

DNA sequence

RNA isoforms

Protein isoforms

Transcription

Translation

Exon

Intron

A T C T G A

Each group of 3 RNA

nucleotides is translated

into a protein amino-acid

Pre-trained DNA encoder

Pre-trained RNA encoder

Pre-trained protein encoder

Aggregation module

Isoform expression prediction head

Multi-modality

embeddings

Single-modality

embeddings

A

B

The same DNA sequence can be

transcribed into different RNA

isoforms depending on the context

Expression

level

Tissue

…

An isoform will

have specific

expression

levels per tissue

Exon 1

Exon 2

Exon 3

Aggregation module To capture the local patterns and their relationship across modalities, we introduce an embedding aggregation method based on cross-attention with residual connections (Fig. 1). The module is applied to each modality, performing cross-attention successively to the other modalities to produce multi-modal embeddings that are specific to each modality by keeping their dimensionality and which can be also stacked. Note that this aggregation method is robust to the absence of a specific modality as the cross-attention term will be zeroed out. The final multi-modal embedding h multi can then be used to solve any task. Our model can be trained end-to-end and it could be applied to different tasks by changing the prediction head.

4.3 Transferring from pre-trained biological encoders

Encoders We describe here the encoders used to process each sequence modality (see also Table 1 and Fig. 2b). For protein sequences, we used ESM (ESM2-150M) [2], which handles any protein up to 2,048 amino-acids. The ESM models have been pre-trained through MLM on large corpuses of protein sequences and are considered state-of-the-art on multiple tasks including folding. Protein sequences were all within ESM input length, so no additional processing was required. For DNA sequences, we used the NT model [3], which has also been trained through self-supervision to reconstruct masked 6-mers within 12 kbp genomic regions from 850 species. Additionally to the NT, we also considered the Enformer model as DNA encoder. The Enformer is a different type of model that has been pretrained through full supervision to predict multiple experiments related to chromatin accessibility and modifications, transcription factor binding, and more importantly gene expression. Enformer is a strong candidate for our architecture's DNA encoder module as it can process sequences up to 200kbp as well as one can expect to obtain transfer from its gene expression capabilities. Finally, while foundation models have been reported to be pre-trained on RNA sequences, none of them has been made publicly available at the moment and pre-training a foundation model from scratch is out of the scope of this study. As such, we re-used the NT model [3] to compute embeddings for RNA sequences as this model has been reported recently to also be able to solve RNA [57] and protein [59] tasks with simple adaptations, using the trick described in section 4.1. RNA transcripts longer than 12kb were left-cropped to 12kb to conserve the 3'UTR regions, which are crucial for mRNA stability and polyadenylation, impacting isoform abundance. This adjustment affected only a small portion of the dataset.

Architecture Our architecture leverages three pre-trained encoders, one per bio-modality, as well as the aggregation function defined above, to generate a multi-modality embedding h multi (Fig. 2b). That embedding is finally transformed by an expression head f exp ψ with weights ψ to make isoform expression level predictions across tissues. As the shape of the network output must be independent from the dimension of its inputs, we used a global average pooling over the length of the embedding. A linear layer then outputs one value per tissue. As long as the encoders can take in a biological sequence to produce an embedding, our method can function with different types of general-purpose encoders. While this flexibility allows us to leverage a big part of the landscape of biological sequence encoders, for this work we chose the specific models described in the section below.

4.4 Training

Objective We denote the IsoFormer model by f θ,ϕ,ψ where θ is the concatenation of the weights of the encoder models. These weights are initialized to the values obtained after pre-training of the different encoders. Respectively ϕ and ψ denote the weights of the aggregation module and expression prediction head. The IsoFormer is trained to minimize the following objective

where the summation is performed over a set of available tissues. Note that this framework can also accept only one or two of the three modalities as input. This is the case for instance when predicting expression of non-coding transcripts that do not translate into proteins.

Dataset We conducted our analysis of IsoFormer on RNA transcript expression data obtained from the GTEx 3 portal. We used Transcript TPMs measurements across 30 tissues, which come from more than 5,000 individuals. We followed a common process in gene expression datasets [56]: we averaged the expression levels for a given tissue across individuals, and used the reference genome sequence as input. We mapped transcripts to their original genes and associated proteins using the Ensembl database [60]. Our resulting dataset is made of triplets of RNA transcript sequences, DNA sequences (centered on the Transcription Start Site (TSS) of the transcript), and proteins. In total, the dataset is made of ∼ 170k unique transcripts, of which 90k are protein-coding and correspond to ∼ 20k unique genes. Our dataset has a fixed train and test set, divided by genes; all presented results correspond to the performance on the test set. We provide more details on the dataset in Appendix A.

Hyperparameters We used the Adam optimizer with a learning rate of 3 · 10 -5 and batch size of 64 , and used early stopping on a validation set comprised of 5% of the train set to reduce training time. We also made our baseline model's weights available 4 . More details in Appendix B.

5 Experiments

We present extensive experiments to assess the performance of our multi-modal approach on the transcript isoform expression prediction task and compare it with existing single-modality approaches. We show that (i) our architecture efficiently aggregates modalities to improve its performance on this task; (ii) by using a tailored model for expression prediction as a base DNA encoder, IsoFormer reaches state-of-the-art performance; (iii) we provide an extensive ablation study on different aggregation approaches; (iv) we demonstrate that our approach achieves transfer learning both intra-modalities from their independent pre-training as well as inter-modalities.

5.1 Bridging three foundational models outperforms mono-modal approaches

Experiment We investigated the effect of adding the different modalities within our multi-modal framework for the prediction of expression of each RNA transcript across different human tissues. We used NT as the foundation model for both DNA and RNA and ESM as the protein encoder. We compared our multi-modal approach (DNA + RNA + protein) with models trained with different combinations of modalities as input: DNA only, RNA only, protein only, DNA + protein and DNA + RNA. Results were obtained over 5 random seeds; for each random seed we change the validation set and randomly initialize the non pre-trained parameters ( ϕ, ψ ) of our model. We report both R 2 , which measures how well each model predicts the actual values of expression, and Spearman correlation across tissues, which is a metric for ranking transcripts based on their expression in each tissue.

Results We observed that our approach benefits from adding more modalities as the performance increases from one modality alone to having two combined, and the best performance is achieved with the three (DNA, RNA, and protein) modalities together (Table 2). This is true for both Spearman correlation and R 2 metrics, with stronger improvement for the latter reflecting a more accurate prediction of the actual values of expression and not just the ranking of transcripts. This is a strong demonstration that our model can aggregate information across modalities to improve performance on this isoform expression task. In addition, we observe increased performance by using DNA together with RNA compared with DNA and protein information. This can be related to the strong importance of the UTR regions of the RNA sequence in the regulation of its degradation and stability [61], which affect its final expression level in the cells, that are not captured at the protein level.

5.2 Enformer as DNA encoder module to obtain transfer between expression prediction tasks

Experiment To showcase the flexibility of IsoFormer towards different modality-specific encoders, we tested replacing NT by the Enformer model [14] as DNA encoder. Enformer has been trained over gene-level expression data obtained from CAGE assays (one value of expression per gene per tissue), while our model is trained to predict RNA transcript expression data obtained from bulk RNA-seq assay (one value of expression for each isoform of a given gene per tissue) and therefore represents a different challenge that cannot be tackled from the DNA sequence alone. Still, as the Enformer has been trained to predict gene-level expression across tissues directly from DNA sequences, a related task to predicting RNA isoform expression, one might expect to obtain transfer by using it as DNA encoder. We give full details of this experiment in Appendix D.

Results We obtained superior performance using the Enformer instead of NT as a pre-trained DNA encoder both as DNA-only but also when we combined with the RNA and protein encoders (Table 3). Importantly, also with the Enformer our framework benefits from bridging modalities. This improved

0

0.6

R2

Stomach

Lung

Liver

Kidney

Brain

Blood

R2 per tissue

Isoformer

Enformer

0

5

10

15

Head

0

5

10

15

20

Layer

5UTR

0

5

10

15

Head

0

5

10

15

20

Layer

3UTR

0

5

10

15

Head

0

5

10

15

20

Layer

CDS

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

performance can be explained by the fact that Enformer has been pre-trained on the related task of gene expression prediction and thus its embeddings are better aligned with the isoform prediction task. Moreover, the Enformer is a model that can handle sequences of large context (up to 196k nucleotides), enabling it to capture long-range dependencies, known to be relevant for expression. We also tried the Borzoi model [] as an enocder for DNA model. Borzoi is a model pre-trained on sequences up to 512kb nucleotides to predict RNA-seq coverage which directly relates to RNA transcript isoform exression. These results demonstrate that our multi-modal framework can be improved by leveraging more domain-specific encoders. As our best model for isoform prediction is achieved using the Enformer as DNA encoder, we will use it as DNA encoder by default for all the following experiments. We make the weights of this IsoFormer model available on HuggingFace 5 .

Interpretation We report the performance of IsoFormer across selected tissues in Fig. 3-left (results across all tissues are presented in Appendix D). IsoFormer obtains similar performance across tissues despite tissues having different distributions of expression levels. To gain additional insights about the representations learned by IsoFormer, we analysed the attention layers inside the RNA encoder as it is the one providing stronger improvement on this task. Specifically, we compared how the attention distribution within each layer and head changes when we finetune the RNA encoder alone versus finetuning IsoFormer altogether. We report changes in attention scores at each layer and head of the RNA encoder for three genomics elements known to have a strong effect on the isoform splicing and gene expression processes, namely the 3UTR, 5UTR and CDS sequence, see Fig. 3-right (additional details on these scores are in Appendix Section C.1). The results show that, when finetuning using the three modalities, different layers specialize to capture specific features relative to isoform splicing and expression. Notably, the middle set of layers put higher attention weights to 3UTR regions whereas the top layers of NT attributes higher attention weights on CDS and 5UTR. We assume that this RNA encoder specialization during finetuning is key to achieve a strong representation towards the prediction of its tissue-specific expression.

5.3 Ablation studies on the aggregation strategy

Experiment We compared the IsoFormer's aggregation module with alternative strategies from recent multi-modal literature (Fig. 4). Inspired by recent vision-text models [20, 52, 62, 63], we considered these three approaches: (i) Perceiver Resampler [50]: a variant of our cross-attention method using a Perceiver Resampler module. This block learns a fixed number of tokens for each modality, thus reducing the cost of the subsequent cross-attention layer. (ii) Linear Projection : a strategy that linearly projects the embeddings of the three modalities into a common representation space and concatenates all tokens. This concatenated sequence is fed to a Perceiver Resampler to

Resampler and cross-attention

+

+

+

+

+

+

1-D Resnet

1-D Resnet

Mean pool.

Resampler

Cross-attention

Resampler

Cross-attention

Cross-attention

Cross-attention

C-Abstractor

C-Abstractor

Resampler

Linear projection and

concatenation

C-Abstractor

Linear projection

Cross-attention

RNA embeddings

DNA embeddings

Protein embeddings

Multi-modal

DNA

embeddings

Latent queries

learn a fixed number of tokens which are then used in the head of the model. (iii) C-Abstractor a 1-dimensional version of the C-abstractor architecture that provides a compromise between flexibility -choosing an arbitrary number of resampled tokens- and locality preservation [20].

Results We performed hyperparameters search with the same budget used for the IsoFormer aggregation module for all ablations methods. We report in Table 4 the results obtained with the best set of hyperparameters for each method. We observe that, using an additional step of Perceiver Resampler always performs worse than only cross-attention -even after optimizing hyperparameters. Similarly, C-Abstractor does not confer any benefit over cross-attention; therefore we consider this latter strategy as the optimal aggregation strategy for our method. One advantage of the cross-attention mechanism we use is its interpretability, since it helps understanding which regions of the different modalities are being leveraged to make predictions. These conclusions align with recent multi-modal studies for other modalities [64].

5.4 IsoFormer leverages knowledge of pre-trained encoders

Experiment We showed in previous sections that IsoFormer is able to aggregate information from different biological sequence modalities to formulate high-quality predictions. Here, we investigated to which extent IsoFormer's performance can be attributed to the transfer from each modality-specific encoders' pre-training. We conduct extensive experiment where we compared the performance of IsoFormer trained using all three encoders pre-trained on their respective domains with different IsoFormer model variants with all possible combinations of pre-trained / non-pretrained modalityspecific encoder.

Results Our results demonstrate that using pre-trained encoders confers a substantial advantage to IsoFormer, as the R 2 is substantially larger (0.53) when compared with IsoFormer with none of the encoders pre-trained (0.10; Table 5). This demonstrates that IsoFormer is leveraging the knowledge acquired by each foundation model in the respective domains. However, we observed that when we randomly initialized only the DNA or RNA encoders, the drop in performance is smaller (0.41 for DNA and 0.48 for RNA). This suggests that IsoFormer not only leverages intra-modalities pre-training but also inter-modalities transfer. Altogether, these results underpin our approach of relying on initializing IsoFormer with pre-trained encoders, as the information learned during pre-training is transferred and leveraged when considering multi-modal tasks.

6 Conclusion

IsoFormer is the first model designed for multi-modal biological sequence modeling connecting DNA, RNA, and protein sequences. IsoFormer achieves state-of-the-art results by effectively leveraging and transferring knowledge from pre-trained DNA-, RNA-, and protein-specific encoders on one of the significant multi-modal problems in genomics: RNA transcript isoform expression prediction . As part of our efforts, we are are open-sourcing our model, and hope IsoFormer paves the way to new milestones in building multi-modal models for biology.

References

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science , 379(6637):1123-1130, 2023.

Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Hassan Sirelkhatim, Guillaume Richard, Marcin Skwark, Karim Beguir, Marie Lopez, and Thomas Pierrot. The nucleotide transformer: Building and evaluating robust foundation models for human genomics. bioRxiv , 2023.

Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15006 , 2023.

Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum BirchSykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems , 36, 2024.

Jun Cheng, Guido Novati, Joshua Pan, Clare Bycroft, Akvil˙ e Žemgulyt˙ e, Taylor Applebaum, Alexander Pritzel, Lai Hong Wong, Michal Zielinski, Tobias Sargeant, Rosalia G. Schneider, Andrew W. Senior, John Jumper, Demis Hassabis, Pushmeet Kohli, and Žiga Avsec. Accurate proteome-wide missense variant effect prediction with alphamissense. Science , 381(6664):eadg7492, 2023.

Ahmara G Ross, Brahim Chaqour, Devin S McDougald, Kimberly E Dine, Thu T Duong, Ryan E Shindler, Jipeng Yue, Tehui Liu, and Kenneth S Shindler. Selective upregulation of sirt1 expression in retinal ganglion cells by aav-mediated gene delivery increases neuronal cell survival and alleviates axon demyelination associated with optic neuritis. Biomolecules , 12(6):830, 2022.

Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature , 624(7990):80-85, 2023.

Eric Nguyen, Michael Poli, Matthew G. Durrant, Armin W. Thomas, Brian Kang, Jeremy Sullivan, Madelena Y. Ng, Ashley Lewis, Aman Patel, Aaron Lou, Stefano Ermon, Stephen A. Baccus, Tina Hernandez-Boussard, Christopher Ré, Patrick D. Hsu, and Brian L. Hie. Sequence modeling and design from molecular to genome scale with evo. bioRxiv , 2024.

Ken Chen, Yue Zhou, Maolin Ding, Yu Wang, Zhixiang Ren, and Yuedong Yang. Self-supervised learning on millions of pre-mrna sequences improves sequence-based rna splicing prediction. bioRxiv , 2023.

Albi Celaj, Alice Jiexin Gao, Tammy T.Y. Lau, Erle M. Holgersen, Alston Lo, Varun Lodaya, Christopher B. Cole, Robert E. Denroche, Carl Spickett, Omar Wagih, Pedro O. Pinheiro, Parth Vora, Pedrum Mohammadi-Shemirani, Steve Chan, Zach Nussbaum, Xi Zhang, Helen Zhu, Easwaran Ramamurthy, Bhargav Kanuparthi, Michael Iacocca, Diane Ly, Ken Kron, Marta Verby, Kahlin Cheung-Ong, Zvi Shalev, Brandon Vaz, Sakshi Bhargava, Farhan Yusuf, Sharon Samuel, Sabriyeh Alibai, Zahra Baghestani, Xinwen He, Kirsten Krastel, Oladipo Oladapo, Amrudha Mohan, Arathi Shanavas, Magdalena Bugno, Jovanka Bogojeski, Frank Schmitges, Carolyn Kim, Solomon Grant, Rachana Jayaraman, Tehmina Masud, Amit Deshwar, Shreshth Gandhi, and Brendan J. Frey. An rna foundation model enables discovery of disease mechanisms and candidate therapeutics. bioRxiv , 2023.

Sizhen Li, Saeed Moayedpour, Ruijiang Li, Michael Bailey, Saleh Riahi, Milad Miladi, Jacob Miner, Dinghai Zheng, Jun Wang, Akshay Balsubramani, Khang Tran, Minnie Zacharia, Monica Wu, Xiaobo Gu, Ryan Clinton, Carla Asquith, Joseph Skalesk, Lianne Boeglin, Sudha Chivukula, Anusha Dias, Fernando Ulloa Montoya, Vikram Agarwal, Ziv Bar-Joseph, and Sven Jager. Codonbert: Large language models for mrna design and optimization. bioRxiv , 2023.

Yekaterina Shulgina, Marena I Trinidad, Conner J Langeberg, Hunter Nisonoff, Seyone Chithrananda, Petr Skopintsev, Amos J Nissley, Jaymin Patel, Ron S Boger, Honglue Shi, et al. Rna language models predict mutations that improve rna function. bioRxiv , pages 2024-04, 2024.

Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka GrabskaBarwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods , 18(10):1196-1203, 2021.

Francis Crick. Central dogma of molecular biology. Nature , 227:561--563, 1970.

Katarzyna Tomczak, Patrycja Czerwi´ nska, and Maciej Wiznerowicz. Review the cancer genome atlas (tcga): an immeasurable source of knowledge. Contemporary Oncology/Współczesna Onkologia , 2015(1):68-77, 2015.

Patroklos Samaras, Tobias Schmidt, Martin Frejno, Siegfried Gessulat, Maria Reinecke, Anna Jarzab, Jana Zecha, Julia Mergner, Piero Giansanti, Hans-Christian Ehrlich, et al. Proteomicsdb: a multi-omics and multi-organism resource for life science research. Nucleic acids research , 48(D1):D1153-D1163, 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:23716-23736, 2022.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

Siddharth Srivastava and Gaurav Sharma. Omnivec: Learning robust representations with cross modal sharing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1236-1248, 2024.

Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17918-17928, 2022.

Siyu Lu, Mingzhe Liu, Lirong Yin, Zhengtong Yin, Xuan Liu, and Wenfeng Zheng. The multi-modal fusion in visual question answering: a review of attention mechanisms. PeerJ Computer Science , 9:e1400, 2023.

Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J. Montine, and James Zou. A visual-language foundation model for pathology image analysis using medical twitter. Nature Medicine , 29:2307-2316, 2023.

Ming Y. Lu, Bowen Chen, Drew F. K. Williamson, Richard J. Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, Anil V. Parwani, Andrew Zhang, and Faisal Mahmood. A visual-language foundation model for computational pathology. Nature Medicine , 30:863-874, 2024.

Junkang Wei, Siyuan Chen, Licheng Zong, Xin Gao, and Yu Li. Protein-rna interaction prediction with deep learning: structure matters. Briefings in bioinformatics , 23(1):bbab540, 2022.

Yehudit Hasin, Marcus Seldin, and Aldons Lusis. Multi-omics approaches to disease. Genome biology , 18:1-15, 2017.

Sambit K Mishra, Viraj Muthye, and Gaurav Kandoi. Computational methods for predicting functions at the mrna isoform level. International Journal of Molecular Sciences , 21(16):5686, 2020.

Heng Li. Protein-to-genome alignment with miniprot. Bioinformatics , 39(1):btad014, 2023.

Byung-Jun Yoon. Hidden markov models and their applications in biological sequence analysis. Current genomics , 10(6):402-415, 2009.

Thomas D Wu, Jens Reeder, Michael Lawrence, Gabe Becker, and Matthew J Brauer. Gmap and gsnap for genomic sequence alignment: enhancements to speed, accuracy, and functionality. Statistical genomics: methods and protocols , pages 283-334, 2016.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Nadav Brandes, Grant Goldman, Charlotte H Wang, Chun Jimmie Ye, and Vasilis Ntranos. Genome-wide prediction of disease variant effects with a deep protein language model. Nature Genetics , 55(9):1512-1522, 2023.

Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234 , 2024.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR , abs/2308.12966, 2023.

Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, A. J. Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali-x: On scaling up a multilingual vision and language model. CoRR , abs/2305.18565, 2023.

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 8469-8488. PMLR, 2023.

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Nils Johan Bertil Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping languageimage pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 19730-19742. PMLR, 2023.

Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 7331-7341. Computer Vision Foundation / IEEE, 2021.

Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023.

Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text modeling with zero-shot transfer from contrastive captioners. CoRR , abs/2212.04979, 2022.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Trans. Mach. Learn. Res. , 2022, 2022.

Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. CoRR , abs/2212.03191, 2022.

Muhammad Bilal Shaikh, Douglas Chai, Syed Mohammed Shamsul Islam, and Naveed Akhtar. Maivar-t: Multimodal audio-image and video action recognizer using transformers. In 11th European Workshop on Visual Information Processing, EUVIP 2023, Gjovik, Norway, September 11-14, 2023 , pages 1-6. IEEE, 2023.

Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. Avsegformer: Audiovisual segmentation with transformer. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, ThirtySixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada , pages 12155-12163. AAAI Press, 2024.

Yaoting Wang, Weisong Liu, Guangyao Li, Jian Ding, Di Hu, and Xi Li. Prompting segmentation with sound is generalizable audio-visual source localizer. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada , pages 5669-5677. AAAI Press, 2024.

Yang Liu, Ying Tan, and Haoyuan Lan. Self-supervised contrastive learning for audio-visual action recognition. In IEEE International Conference on Image Processing, ICIP 2023, Kuala Lumpur, Malaysia, October 8-11, 2023 , pages 1000-1004. IEEE, 2023.

Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning , pages 4651-4664. PMLR, 2021.

Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Localityenhanced projector for multimodal llm. arXiv preprint arXiv:2312.06742 , 2023.

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611 , 2024.

Chao Cheng, Roger Alexander, Renqiang Min, Jing Leng, Kevin Y Yip, Joel Rozowsky, KoonKiu Yan, Xianjun Dong, Sarah Djebali, Yijun Ruan, et al. Understanding transcriptional regulation by integrative analysis of transcription factor binding data. Genome research , 22(9):1658-1667, 2012.

Alvaro J González, Manu Setty, and Christina S Leslie. Early enhancer establishment and regulatory locus complexity shape transcriptional programs in hematopoietic differentiation. Nature genetics , 47(11):1249-1259, 2015.

David R Kelley, Yakir A Reshef, Maxwell Bileschi, David Belanger, Cory Y McLean, and Jasper Snoek. Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome research , 28(5):739-750, 2018.

Jian Zhou, Chandra L Theesfeld, Kevin Yao, Kathleen M Chen, Aaron K Wong, and Olga G Troyanskaya. Deep learning sequence-based ab initio prediction of variant effects on expression and disease risk. Nature genetics , 50(8):1171-1179, 2018.

Guillaume Richard, Bernardo P de Almeida, Hugo Dalla-Torre, Christopher Blum, Lorenz Hexemer, Priyanka Pandey, Stefan Laurent, Marie P Lopez, Alexander Laterre, Maren Lang, et al. Chatnt: A multimodal conversational agent for dna, rna and protein tasks. bioRxiv , pages 2024-04, 2024.

Johannes Linder, Divyanshi Srivastava, Han Yuan, Vikram Agarwal, and David R Kelley. Predicting rna-seq coverage from dna sequence as a unifying model of gene regulation. bioRxiv , pages 2023-08, 2023.

Sam Boshar, Evan Trop, Bernardo P de Almeida, and Thomas Pierrot. Are genomic language models all you need? exploring genomic language models on protein downstream tasks. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations , 2024.

Fergal J Martin, M Ridwan Amode, Alisha Aneja, Olanrewaju Austine-Orimoloye, Andrey G Azov, If Barnes, Arne Becker, Ruth Bennett, Andrew Berry, Jyothish Bhai, et al. Ensembl 2023. Nucleic acids research , 51(D1):D933-D941, 2023.

Woo Seuk Koh, Joshua R Porter, and Eric Batchelor. Tuning of mrna stability through altering 3 ′ -utr sequences generates distinct output expression in a synthetic circuit driven by p53 oscillations. Scientific reports , 9(1):5976, 2019.

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 , 2023.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pretraining for unified vision-language understanding and generation. In International conference on machine learning , pages 12888-12900. PMLR, 2022.

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. MM1: methods, analysis & insights from multimodal LLM pre-training. CoRR , abs/2403.09611, 2024.

A Dataset details

We based our dataset on the Genotype-Tissue Expression (GTEx) portal. Specifically, we use the 8th release of the Transcript TPMs table 6 . This table is made of expression measurements of 170 k transcripts across 30 different tissues from 5 , 000 individuals. As the goal is to build a general model for expression prediction from biological sequences, we averaged measurements across individuals to get an average expression value for each transcript in each tissue.

We mapped each transcript to its corresponding gene and protein using the Ensembl 7 database. Using this database, we were able to retrieve associated DNA, RNA, and protein sequences. For DNA sequences, we used the latest release of the human reference genome GRCh38 8 .

To summarize, the steps to re-create our training dataset are the following:

Download Transcript TPMs table from GTEx portal 9

Compute average measurement per transcript ID and tissue

Map RNA transcript isoform ID to its corresponding RNA sequence using Ensembl

Get associated protein isoform using Ensembl

Get chromosome and transcription start site position on the DNA sequence

Average Expression per Tissue

Adipose Tissue

Adrenal Gland

Bladder

Blood

Blood Vessel

Brain

Breast

Cervix Uteri

Colon

Esophagus

Fallopian Tube

Heart

Kidney

Liver

Lung

Muscle

Nerve

Ovary

Pancreas

Pituitary

Prostate

Skin

Spleen

Stomach

Testis

Thyroid

Uterus

Vagina

0

100

200

300

400

500

Adipose Tissue

Adrenal Gland

Bladder

Blood

Blood Vessel

Brain

Breast

Cervix Uteri

Colon

Esophagus

Fallopian Tube

Heart

Kidney

Liver

Lung

Muscle

Nerve

Ovary

Pancreas

Pituitary

Prostate

Skin

Spleen

Stomach

Testis

Thyroid

Uterus

Vagina

0

5000

10000

15000

Expression Standard Deviation per Tissue

B Training and architecture details

All experiments were carried out with 5 seeds on 4 A100 GPUs (80GB RAM). Depending on the model, a training run lasts between 1 and 5 hours. We provide model hyper-parameters in Table 6 and the hyper-parameters of the encoders in Table 7.

In order to stabilize our training procedure, expression values were processed in two steps: first, we use a log-transform ( val = log(1 + val ) ) to minimize the effect of outliers; secondly, we normalized expression values per tissue as it has been shown to stabilize training when using mean squared error loss.

C Experimental details

C.1 Attention maps analysis

Section 5.2 introduces significant results on IsoFormer's ability to leverage pre-trained modalityspecific encoders. When finetuning the pre-trained encoders together, there is specialization on specific elements of the sequence, as observed in the RNA attention maps in Figure 3-right. These maps have been obtained through the following process:

Focusing on the pre-trained RNA encoder, we take the attention weights for each layer and head after running the IsoFormer finetuning process. For each layer and head, we compute how much attention (percentage) is directed towards a specific region of interest of the RNA sequence. The regions we consider are 3UTR, 5UTR, and CDS, and we use the following equation to compute the exact ratio of attention:

where X rna is the set of RNA sequences in the test set, α ( i, j ) is the attention coefficient between tokens i and j , f ( i ) is an auxiliary function that equals 1 if token i belongs to the region of interest in the sequence (e.g. 3UTR), and µ is a threshold value (we choose 0.01). We denote these attention maps by ρ IF .

We repeat this process for the finetuning run in which only the RNA encoder is used (RNA only in Table 2). We denote these attention maps by ρ NT.

For each layer and head, we compute the ratio ∆ ρ = ( ρ IF -ρ NT ) /ρ NT. For simplicity, we cap these values to 1 (i.e., attention rate is doubled in the IsoFormer case compared to finetuning a RNA encoder alone).

In addition, for both ρ IF and ρ NT, we can consider the samples in the test set X rna and build a distribution of attention rates per element of interest. Comparing both distributions (the one coming from ρ IF and the one coming from ρ NT), we can carry out a t-test per layer and head. We follow these t-tests and select the combinations of layers and heads in which there are statistically significant differences (i.e., p < 0.05) between ρ IF and ρ NT.

Figure 3-right shows the ratio ∆ ρ for those pairs of layers and heads in which statistically significant differences are observed. The rest is set to zero.

D Additional results

D.1 Full results over every tissue

Figure 6 shows the performance of IsoFormer using Enformer as DNA encoder for each of the 30 tissues. IsoFormer outperforms the DNA-only Enformer model on every tissue.

Adipose Tissue

Adrenal Gland

Bladder

Blood

Blood Vessel

Brain

Breast

Cervix Uteri

Colon

Esophagus

Fallopian Tube

Heart

Kidney

Liver

Lung

Muscle

Nerve

Ovary

Pancreas

Pituitary

Prostate

Skin

Spleen

Stomach

Testis

Thyroid

Uterus

Vagina

0.0

0.6

R2 per tissue

Isoformer

Enformer

D.2 Full ablation studies on modalities

Table 8: Performance of different variants of IsoFormer for the prediction of transcript isoform expression using three different DNA encoders: Nucleotide Transformer, Enformer, and Borzoi (input sequence length in parentheses). R 2 and Spearman correlation across tissues for 5 different random seeds is reported. NT is used as RNA encoder while ESM is used to process protein sequences.

E Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim in the abstract that we propose a novel way to integrate different pre-trained biological sequence encoders to solve the RNA isoform expression task. This is proven by our experimental results.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: While we do not provide a specific limitations section, we discuss limitations in the corresponding sections.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not claim novel theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide extensive details on the dataset creation and processing as well as architectural choices and training procedure. We also will open-source our model at publication.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: At the current stage, we do not provide training data and code but will opensource our model at publication time. However, we provide extensive details for anyone willing to gather our training dataset.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We give full details of how we defined our dataset, hyperparameters and training procedure. We also acknowledge that some of those hyperparameters were taken arbitrarily.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We ran each experiment over several seeds that account for variability between different initialization and train/validation splits. While we do not directly report error bars we report standard deviations of our metrics over those seeds.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We give those details in the Appendix, where we specify the amount of compute as well as training times for each experiment.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper is applied to RNA transcript isoform expression, an important biological process. Our approach can be one of the tools used in a pipeline to accelerate drug or vaccine design and as such have a positive societal impact.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We used internal reproduction of existing assets but we properly credit authors for their original work.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We fully detail all architecture choices and experiments and also open-source our model weights.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel multi-modal model, IsoFormer, designed to integrate DNA, RNA, and protein sequences for predicting RNA transcript isoform expression across different tissues. It utilizes pre-trained modality-specific encoders to generate embeddings that are then combined using a sophisticated aggregation method. The model demonstrates significant improvements in prediction accuracy compared to single-modality approaches.\n\nContriburion:\n\n1. Developed the first general-purpose multi-modal model integrating DNA, RNA, and protein sequences.\n\n2. Demonstrated successful application of transfer learning from modality-specific encoders.\n\n3. Provided a new robust framework for advancing the prediction of RNA transcript isoform expression.\n\nSTRENGTHS:\n1. Innovative Integration of Modalities: The paper presents the first attempt to integrate three biological sequence modalities (DNA, RNA, and proteins) in a unified model, providing a comprehensive approach reflective of natural biological processes.\n\n2. Effective Transfer Learning: IsoFormer effectively leverages pre-trained encoders to enhance its predictive power, benefiting from both intra-modal and inter-modal transfer learning.\n\n3. Robust Evaluation: Experiments demonstrate the model's capability, outperforming existing methods in predicting transcript isoform expression, which is a challenging task due to its multi-modal nature.\n\nWEAKNESSES:\n1. Complexity and Computation: The model's complexity and the computational demands might limit its accessibility and use, particularly in environments with restricted resources.\n\n2. More Comprehensive Evaluation for PLM's representation learning capability would make this paper better.\n\nQUESTIONS:\n(1)\tFor Tab 5., wonder what’s the performance for “DNA and RNA encoder not pre-trained”\n\n(2)\tCould authors provide evaluation results for DNA, RNA and protein encoder separately on their own popular benchmarking tasks? Ideally, these models should improve on those downstream tasks as well. Would give 7’ or even 8’ if authors conduct these experiments. It could be a great and exciting work in this AI4Bio field.\n\nLIMITATIONS:\n1. Data Requirements: The effectiveness of the model is contingent on the availability of comprehensive and high-quality multi-modal datasets.\n\n2. Generalizability: While promising, the results are primarily validated on specific types of gene expression data, and its performance across broader biological applications remains to be fully assessed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

comments: SUMMARY:
The paper introduces a new framework for the multi-modality pretrain model according to the Central dogma of biology. The method encode DNA, protein and RNA at the same time. The proposed method can transfer knowledge from the encoders pretraining and modalities.

STRENGTHS:
The paper is well-organized and easy to follow
The authors have proved that the multi-modality of single cell data can help model predictions.

WEAKNESSES:
Lack of experiments. 1. More ablation studies should be conducted about removing different modalities of the model in Table 2 ( e.g. we observe only RNA can achieve a high performance, what about protein+RNA? ).

 2. More dataset details should be included. The split of training/validation/test sets is not clear. If the authors do the experiments on the same dataset, they should split the dataset according to the tissues to validate the transferability of the proposed method.

QUESTIONS:
The authors should include more motivation about the proposed method. For example, why can the changed DNA influence the RNA seq? Why not directly predict RNA expression from RNA seq?

LIMITATIONS:
The biological system is more complex. The proposed method only include the direct map from DNA to RNA. However, in real world, RNA can effect the expression of DNAs. More details should be discuss in the future work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper models isoform relative abundance across tissues with a multimodal approach based on 3 pretrained encoders for DNA, RNA, and AA sequences.  DNA encoder uses a sequence centered on the gene’s TSS, RNA encoder uses the known isoform sequence from RNAseq and the protein encoder uses corresponding AA sequence.  They perform multiple ablations on the utility of having all 3 separate encoders and, given separate encoders, how to aggregate them into a single isoform specific embedding/prediction, and look at attention layers of RNA module to find biologically meaningful regions of attention.\n\nSTRENGTHS:\nIsoform level analysis using 3 separate pretrained encoders for DNA, RNA, and AA sequences is a good strategy.  The authors provide useful ablations on the utility of the multi modal approach and on modern strategies for combining those into a single embedding.  Looking for biolgoically meaningful interpretations of attention layers is useful.\n\nWEAKNESSES:\nI don’t think the authors can claim this is the first attempt to combine DNA, RNA, and AA modalities with techniques from NLP.   See the recent Evo work here https://www.biorxiv.org/content/10.1101/2024.02.27.582234v2 .  While they evaluate their performance against Enformer, that’s a large part of their own model.  So the evaluations have an intramural feel to them.  It’d be interesting to see how their strategy compares to other multi modal models such as Evo, and more RNA centric work like Borzoi, which looks at a more fine grained look of variant effects on the DNA to RNA relationship.  Looking at average isoform abundance across individuals is all well and good, but GTEx also has individual genomes, and genomic variation across individuals will also of course affect splicing patterns and which isoforms come from what individuals.\n\nQUESTIONS:\nSome comments and questions:\n\nCentering on TSS will only capture regulatory elements within the chosen sequence length.  There could be distal or trans CREs outside\n\nGTEx database has less than 1000 donors, confirming the 5000 individuals claim?\n\nLooking at equation 5 in 4.4, how does f_psi depend on the tissue T? Is it a separate head per tissue, or are you predicting the vector of isoform abundance across tissues with one pass?  And is f_theta and f_phi the same f but different weights as f_psi?  Where does the summation over i take place?\n\n5.1 does ablations with one DNA encoder, then 5.2 shows superior performance with Enformer as the DNA encoder.  So the ablations in 5.1 may not be accurate with respect to this new encoder.  It also begs the question of how would the Enformer do as the RNA encoder as well.\n\nHow are RNA and protein sequence lengths handled when they’re longer than the model input sequence size?\n\nLIMITATIONS:\nThe authors should be more explicit about the limitation of using reference genome and known isoform sequences and how this kind of sweeps splicing as a function of dna sequence under the rug.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}