Multi-modal Transfer Learning between Biological Foundation Models

Authors: juan jose garau-luis
instadeep  patrick bordes
instadeep liam gonzalez
instadeep masa roller
instadeep  bernardo
p  de almeida instadeep
lorenz  hexemer
christopher  blum biontech
stefan  laurent
jan  grzegorzewski
maren  lang
thomas  pierrot
instadeep  guillaume
richard † instadeep
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:49:22

Abstract: Biological sequences encode fundamental instructions for the building blocks of life, in the form of DNA, RNA, and proteins. Modeling these sequences is key to understand disease mechanisms and is an active research area in computational biology. Recently, Large Language Models have shown great promise in solving certain biological tasks but current approaches are limited to a single sequence modality (DNA, RNA, or protein). Key problems in genomics intrinsically involve multiple modalities, but it remains unclear how to adapt general-purpose sequence models to those cases. In this work we propose a multi-modal model that connects DNA, RNA, and proteins by leveraging information from different pre-trained modality-specific encoders. We demonstrate its capabilities by applying it to the largely unsolved problem of predicting how multiple \rna transcript isoforms originate from the same gene (i.e. same DNA sequence) and map to different transcription expression levels across various human tissues. We show that our model, dubbed IsoFormer, is able to accurately predict differential transcript expression, outperforming existing methods and leveraging the use of multiple modalities. Our framework also achieves efficient transfer knowledge from the encoders pre-training as well as in between modalities. We open-source our model, paving the way for new multi-modal gene expression approaches.

Main paper: 
 introduction: 
 Foundation models have ignited a revolution in numerous scientific fields, starting in NLP and computer vision, and more recently in several domains within the life sciences. Within the biological sciences, these models have enabled predicting protein structures from sequences [1, 2] , deciphering the genome functions [3, 4, 5, 6] and interactions of biomolecules [7] , and crafting new molecules not found in nature [8] . Specifically, significant progress has been made with foundation models tailored to DNA [3, 4, 5, 9] , RNA [10, 11, 12, 13] , and protein [2] sequences. These foundation models have typically been developed and trained separately, using either self-supervised learning techniques such as Masked Language Modeling (MLM), as seen in models like ESM [2] and the Nucleotide Transformer (NT) [3] , or supervised learning approaches on large datasets, as in AlphaFold [1] and Enformer [14] . These models have been instrumental in advancing our understanding of biology by accurately predicting the structures and functions of biological sequences.While existing methods provide relevant insights, they are still limited by the fact that they only consider a single sequence modality. In biology, the central dogma describes the flow of genetic information from DNA to RNA to proteins [15] . This fundamental concept underscores the interconnectedness of these three types of biological sequences and highlights the potential for a unified modeling approach. An architecture that integrates DNA, RNA, and protein modalities should provide a comprehensive model for biology that mirrors the natural processes within cells. Furthermore, by enabling transfer learning across modalities, the model can capitalize on the vast amounts of pre-training already performed on individual DNA, RNA, and protein datasets.Developing deep learning models using multiple biological sequence modalities has been mainly limited by the lack of matched available data; existing databases usually isolate a specific modality and thus relationships between modalities are not easily obtainable. As more multi-modal datasets are made available [16, 17] , it is becoming possible to develop models that extract and combine the information from DNA, RNA, and protein sequences to better model the different cellular processes. Such multi-modal models have already been successful in other domains, such as mixing language and visual inputs [18, 19, 20, 21, 22, 23, 24, 25] , but until now there are no models that can handle multiple biological sequence modalities. Significant problems in genomics intrinsically involve multiple sequence modalities [27, 26] , and it is still unclear how to adapt general-purpose sequence models to those cases. In order to validate our multi-modal approach, we focused on the study of a crucial task in genomics that has been challenging to tackle using a single sequence modality -namely the prediction of RNA transcript isoform expression across different tissues [28] . When gene DNA sequences are transcribed into mRNA molecules to produce proteins, they generally do not express it in just a single way producing a single protein isoform. After DNA sequences are transcribed, pre-mRNA transcripts undergo a process called RNA alternative splicing where they are cut and re-assembled to form different variants of mature mRNA molecules that can be translated into proteins. This process allows for the generation of multiple RNA and protein isoforms from a single gene DNA sequence, that can differ in structure, function, localization, or other properties (Fig. 2a ). Therefore, predicting which isoforms are expressed in a given cell or under specific conditions is key to understand gene regulation and disease mechanisms. This task is multi-modal in nature, since only looking at the DNA sequence present in the cell does not provide a complete picture of the different RNA isoform landscape. Focusing on this single important task in opposition to tackling more but less challenging tasks should provide a stronger evidence for the effectiveness of our approach to advance biological knowledge.In summary, our contributions are the following: (i) We built the first multi-modal model for the integration of DNA, RNA, and protein sequences. (ii) We show that our model achieves efficient transfer learning between the three modalities, not only leveraging intra-modalities pre-training but also inter-modalities transfer. (iii) We use our architecture to tackle a new central task in biology that requires a multi-modal approach, namely RNA transcript isoform expression prediction, and obtain state-of-the-art results, overcoming limitations of existing gene expression models such as Enformer. (iv) Finally, we performed ablation studies to validate our different architectural choices and release our trained isoform expression prediction model IsoFormer, providing a new framework and baseline to the community and opening the door to future research on multi-modal sequence modeling and multi-modal biological problems. 
related work: 
 Biological sequence modeling Researchers have explored different ways to process DNA, RNA, and protein sequences for multiple applications. Initial approaches included dynamic programming [29] , hidden Markov models [30] , and genomic hash tables [31] . Recently, deep learning, via supervised [1] and unsupervised frameworks [3, 4, 13, 2] , has gained thrust in the community. These methods, influenced by advancements in Large Language Models [32, 33] , have focused on DNA [3, 4, 5, 9] , RNA [10, 11, 12, 13] , and protein sequences [2] . They capture complex biological patterns and perform tasks like protein structure prediction [2] and variant effect prediction [34] . These efforts have become more targeted given the different challenges of each sequence modality. For instance, models like HyenaDNA [5, 9] or Caduceus [35] tackled the long-range dependencies in DNA. Few models consider multiple modalities, mainly focusing on structural information (e.g. predicting DNA interacting residues in proteins [26] ). Our work, IsoFormer, is the first general-purpose model integrating three biological sequence modalities.Multi-Modal Integration Efforts to address multi-modality in deep learning models have been prominent in NLP [36, 37, 38, 39, 40] and computer vision [41, 42, 43, 44, 45] . In computer vision, integrating image and audio for video classification [46] or audio-visual segmentation [47] is common, often using cross-attention [48] . Unsupervised approaches using contrastive learning also integrate image, audio, and other modalities [49] . Large Language Models have driven multi-modal efforts for text and image with methods like Flamingo [19] and CLIP [18] , leading to various integration architectures, including the Perceiver Resampler [50, 19] and C-Abstractor [51, 52] . IsoFormer integrates DNA, RNA, and protein sequences, leveraging past integration architectures.Transcript isoform expression prediction is a more refined and challenging task within gene expression prediction. Traditionally, gene expression has been addressed through tailored approaches and experimental annotations [53, 54] . Recently, deep learning models have demonstrated improved results by predicting gene expression directly from DNA sequence [55, 56] . Enformer [14] leveraged dilated convolutions and attention layers to extend the context window to 190 kilo base pairs (kbp), achieving new state-of-the-art results on gene expression. However, these models are limited to gene-level expression levels and cannot predict isoform-specific expression as DNA sequence information is not sufficient to solve this task. We introduce IsoFormer, the first multi-modal model that is able to predict transcript isoform expression. 
background: 
 Central dogma of biology In this work, we consider three biological sequence types: DNA, RNA, and proteins. These sequences (composed of nucleotides or amino-acids) are fundamental to biological processes in living organisms. They are also intricately intertwined: DNA dictates RNA synthesis during transcription, and RNA guides protein synthesis through translation (see Fig. 2 ; [15] ). Thus, changes in DNA can alter RNA and protein sequences, impacting an organism's phenotype (i.e. function). DNA and RNA sequences consist of four nucleotides (ACGT and ACGU, respectively), while proteins are made of 20 amino-acids. Thus, these sequences are commonly modelled as linear strings.Obtaining data in this format is becoming more available thanks to recent advances in high-throughput sequencing technologies.Gene expression is the process by which a gene's DNA sequence is transcribed into various RNA molecules that code for specific proteins. This process is complex, as a single gene can produce different RNA and protein isoforms with varying abundances across tissues (Fig. 2a ). RNA isoforms are mRNA molecules of different exon compositions derived from the same gene, produced via processes like alternative splicing. The expression level of each isoform is commonly measured by counting the amount of its RNA molecules in cells. Accurate isoform expression prediction across tissues can aid in understanding genetic variants' effects on cellular processes and phenotypes. However, this task cannot be tackled solely using DNA sequences as the same DNA sequence produces different isoforms in different cellular contexts and types. In addition, both DNA and RNA sequence features are crucial as DNA sequences contain regulatory elements that control transcription levels, while RNA isoform sequences have features affecting their stability and degradation. 
multi-modal framework: 
 We consider respectively DNA, RNA, and protein sequences x dna ∈ A dna , x rna ∈ A rna and x prot ∈ A prot where A dna is the DNA base alphabet ACGT, A rna is the RNA base alphabet ACGU, and A prot is the set of 20 amino-acids. We then consider three associated modality-encoders f with respective weights (θ dna , θ rna , θ prot ) that encode sequences x into corresponding embeddings h:EQUATIONWe assume that the weights (θ dna , θ rna , θ prot ) have been obtained through independent pre-training processes that can involve supervised or self-supervised training techniques over large corpus of biological data. Note that in practice, as commonly used in recent works [57] , models pre-trained over DNA sequences can be re-used to produce embeddings for RNA sequences, replacing artificially the uracil base (U) by thymine (T) in the input. In this work, we aim to connect these encoders and train them jointly to learn a multi-modal embedding h multi . We start by learning a multi-modal embedding per modality defined ash ′ dna = f agg ϕ (h dna , h rna , h prot ) , h ′ rna = f agg ϕ (h rna , h dna , h prot ) , h ′ prot = f agg ϕ (h prot , h dna , h rna ) (2)where f agg ϕ is an aggregation function with weights ϕ. Then, we define the multi-modal embedding as the concatenation of the per modality multi-modal embeddings:EQUATIONNote that this definition is general as it allows the use of any aggregation function over the modality embeddings h. The multi-modality embeddings per modality h ′ are introduced to solve tasks that are "modality-centered". For instance, h ′ dna could be used to solve a task that involves nucleotide-level annotation over a DNA sequence while requiring other modalities as input. We rely otherwise on the concatenated multi-modal embedding h multi to solve any other task.We now introduce our notations specific to the task of RNA isoform expression prediction (Fig. 2a ). We consider a DNA sequence x dna of length L to contain a gene g. In practice, given the input size limitation of the existing foundation models, the sequence length L might be shorter than the full length of the gene. In this case, we choose the DNA sequence x dna to be centered on the start of the gene, i.e., where transcription begins, which also surrounds the promoter regions known to be important for transcription. This way, we also capture all the enhancer regulatory elements upstream of the transcription site.We denote by x rna . We define the expression e of genes and transcripts across tissues T as:EQUATIONWhile deep learning models have been trained to predict the overall expression of genes across tissues with great accuracy [55, 56, 14, 58] , to our knowledge no model can predict the expression of the different RNA transcripts across tissues directly from the sequence. In this work, we leverage our multi-modal framework to train a transcript expression level prediction model, dubbed IsoFormer, that takes as input a DNA sequence, an RNA transcript sequence, and its matching protein sequence to predict the expression of that transcript across tissues as measured by bulk RNA-seq (Fig. 2 ). Aggregation module To capture the local patterns and their relationship across modalities, we introduce an embedding aggregation method based on cross-attention with residual connections (Fig. 1 ). The module is applied to each modality, performing cross-attention successively to the other modalities to produce multi-modal embeddings that are specific to each modality by keeping their dimensionality and which can be also stacked. Note that this aggregation method is robust to the absence of a specific modality as the cross-attention term will be zeroed out. The final multi-modal embedding h multi can then be used to solve any task. Our model can be trained end-to-end and it could be applied to different tasks by changing the prediction head. Encoders We describe here the encoders used to process each sequence modality (see also Table 1 and Fig. 2b ). For protein sequences, we used ESM (ESM2-150M) [2] , which handles any protein up to 2,048 amino-acids. The ESM models have been pre-trained through MLM on large corpuses of protein sequences and are considered state-of-the-art on multiple tasks including folding. Protein sequences were all within ESM input length, so no additional processing was required. For DNA sequences, we used the NT model [3] , which has also been trained through self-supervision to reconstruct masked 6-mers within 12 kbp genomic regions from 850 species. Additionally to the NT, we also considered the Enformer model as DNA encoder. The Enformer is a different type of model that has been pretrained through full supervision to predict multiple experiments related to chromatin accessibility and modifications, transcription factor binding, and more importantly gene expression. Enformer is a strong candidate for our architecture's DNA encoder module as it can process sequences up to 200kbp as well as one can expect to obtain transfer from its gene expression capabilities. Finally, while foundation models have been reported to be pre-trained on RNA sequences, none of them has been made publicly available at the moment and pre-training a foundation model from scratch is out of the scope of this study. As such, we re-used the NT model [3] to compute embeddings for RNA sequences as this model has been reported recently to also be able to solve RNA [57] and protein [59] tasks with simple adaptations, using the trick described in section 4.1. RNA transcripts longer than 12kb were left-cropped to 12kb to conserve the 3'UTR regions, which are crucial for mRNA stability and polyadenylation, impacting isoform abundance. This adjustment affected only a small portion of the dataset.Architecture Our architecture leverages three pre-trained encoders, one per bio-modality, as well as the aggregation function defined above, to generate a multi-modality embedding h multi (Fig. 2b ).That embedding is finally transformed by an expression head f exp ψ with weights ψ to make isoform expression level predictions across tissues. As the shape of the network output must be independent from the dimension of its inputs, we used a global average pooling over the length of the embedding. A linear layer then outputs one value per tissue. As long as the encoders can take in a biological sequence to produce an embedding, our method can function with different types of general-purpose encoders. While this flexibility allows us to leverage a big part of the landscape of biological sequence encoders, for this work we chose the specific models described in the section below.Objective We denote the IsoFormer model by f θ,ϕ,ψ where θ is the concatenation of the weights of the encoder models. These weights are initialized to the values obtained after pre-training of the different encoders. Respectively ϕ and ψ denote the weights of the aggregation module and expression prediction head. The IsoFormer is trained to minimize the following objectiveEQUATIONwhere the summation is performed over a set of available tissues. Note that this framework can also accept only one or two of the three modalities as input. This is the case for instance when predicting expression of non-coding transcripts that do not translate into proteins.Dataset We conducted our analysis of IsoFormer on RNA transcript expression data obtained from the GTEx3 portal. We used Transcript TPMs measurements across 30 tissues, which come from more than 5,000 individuals. We followed a common process in gene expression datasets [56] : we averaged the expression levels for a given tissue across individuals, and used the reference genome sequence as input. We mapped transcripts to their original genes and associated proteins using the Ensembl database [60] . Our resulting dataset is made of triplets of RNA transcript sequences, DNA sequences (centered on the Transcription Start Site (TSS) of the transcript), and proteins. In total, the dataset is made of ∼170k unique transcripts, of which 90k are protein-coding and correspond to ∼20k unique genes. Our dataset has a fixed train and test set, divided by genes; all presented results correspond to the performance on the test set. We provide more details on the dataset in Appendix A.Hyperparameters We used the Adam optimizer with a learning rate of 3 • 10 -5 and batch size of 64, and used early stopping on a validation set comprised of 5% of the train set to reduce training time. We also made our baseline model's weights available 4 . More details in Appendix B. 
experiments: 
 We present extensive experiments to assess the performance of our multi-modal approach on the transcript isoform expression prediction task and compare it with existing single-modality approaches. We show that (i) our architecture efficiently aggregates modalities to improve its performance on this task; (ii) by using a tailored model for expression prediction as a base DNA encoder, IsoFormer reaches state-of-the-art performance; (iii) we provide an extensive ablation study on different aggregation approaches; (iv) we demonstrate that our approach achieves transfer learning both intra-modalities from their independent pre-training as well as inter-modalities.Experiment We investigated the effect of adding the different modalities within our multi-modal framework for the prediction of expression of each RNA transcript across different human tissues. We used NT as the foundation model for both DNA and RNA and ESM as the protein encoder. We compared our multi-modal approach (DNA + RNA + protein) with models trained with different combinations of modalities as input: DNA only, RNA only, protein only, DNA + protein and DNA + RNA. Results were obtained over 5 random seeds; for each random seed we change the validation set and randomly initialize the non pre-trained parameters (ϕ, ψ) of our model. We report both R 2 , which measures how well each model predicts the actual values of expression, and Spearman correlation across tissues, which is a metric for ranking transcripts based on their expression in each tissue. We observed that our approach benefits from adding more modalities as the performance increases from one modality alone to having two combined, and the best performance is achieved with the three (DNA, RNA, and protein) modalities together (Table 2 ). This is true for both Spearman correlation and R 2 metrics, with stronger improvement for the latter reflecting a more accurate prediction of the actual values of expression and not just the ranking of transcripts. This is a strong demonstration that our model can aggregate information across modalities to improve performance on this isoform expression task. In addition, we observe increased performance by using DNA together with RNA compared with DNA and protein information. This can be related to the strong importance of the UTR regions of the RNA sequence in the regulation of its degradation and stability [61] , which affect its final expression level in the cells, that are not captured at the protein level.Experiment To showcase the flexibility of IsoFormer towards different modality-specific encoders, we tested replacing NT by the Enformer model [14] as DNA encoder. Enformer has been trained over gene-level expression data obtained from CAGE assays (one value of expression per gene per tissue), while our model is trained to predict RNA transcript expression data obtained from bulk RNA-seq assay (one value of expression for each isoform of a given gene per tissue) and therefore represents a different challenge that cannot be tackled from the DNA sequence alone. Still, as the Enformer has been trained to predict gene-level expression across tissues directly from DNA sequences, a related task to predicting RNA isoform expression, one might expect to obtain transfer by using it as DNA encoder. We give full details of this experiment in Appendix D.We obtained superior performance using the Enformer instead of NT as a pre-trained DNA encoder both as DNA-only but also when we combined with the RNA and protein encoders (Table 3 ). Importantly, also with the Enformer our framework benefits from bridging modalities. This improved Changes in attention in the RNA encoder during fine-tuning. These scores are reported for three genomics elements of interest for all heads and layers of the RNA encoder.performance can be explained by the fact that Enformer has been pre-trained on the related task of gene expression prediction and thus its embeddings are better aligned with the isoform prediction task. Moreover, the Enformer is a model that can handle sequences of large context (up to 196k nucleotides), enabling it to capture long-range dependencies, known to be relevant for expression.We also tried the Borzoi model [] as an enocder for DNA model. Borzoi is a model pre-trained on sequences up to 512kb nucleotides to predict RNA-seq coverage which directly relates to RNA transcript isoform exression. These results demonstrate that our multi-modal framework can be improved by leveraging more domain-specific encoders. As our best model for isoform prediction is achieved using the Enformer as DNA encoder, we will use it as DNA encoder by default for all the following experiments. We make the weights of this IsoFormer model available on HuggingFace 5 .Interpretation We report the performance of IsoFormer across selected tissues in Fig. 3 -left (results across all tissues are presented in Appendix D). IsoFormer obtains similar performance across tissues despite tissues having different distributions of expression levels. To gain additional insights about the representations learned by IsoFormer, we analysed the attention layers inside the RNA encoder as it is the one providing stronger improvement on this task. Specifically, we compared how the attention distribution within each layer and head changes when we finetune the RNA encoder alone versus finetuning IsoFormer altogether. We report changes in attention scores at each layer and head of the RNA encoder for three genomics elements known to have a strong effect on the isoform splicing and gene expression processes, namely the 3UTR, 5UTR and CDS sequence, see Fig. 3 -right (additional details on these scores are in Appendix Section C.1). The results show that, when finetuning using the three modalities, different layers specialize to capture specific features relative to isoform splicing and expression. Notably, the middle set of layers put higher attention weights to 3UTR regions whereas the top layers of NT attributes higher attention weights on CDS and 5UTR. We assume that this RNA encoder specialization during finetuning is key to achieve a strong representation towards the prediction of its tissue-specific expression.Experiment We compared the IsoFormer's aggregation module with alternative strategies from recent multi-modal literature (Fig. 4 ). Inspired by recent vision-text models [20, 52, 62, 63] , we considered these three approaches: (i) Perceiver Resampler [50] : a variant of our cross-attention method using a Perceiver Resampler module. This block learns a fixed number of tokens for each modality, thus reducing the cost of the subsequent cross-attention layer. (ii) Linear Projection: a strategy that linearly projects the embeddings of the three modalities into a common representation space and concatenates all tokens. This concatenated sequence is fed to a Perceiver Resampler to rna and h ′ prot , respectively). In all cases, the Resampler module is a Perceiver Resampler [50] and the Mean pooling block is the Adaptive mean pooling operator used in [20] . learn a fixed number of tokens which are then used in the head of the model. (iii) C-Abstractor a 1-dimensional version of the C-abstractor architecture that provides a compromise between flexibility -choosing an arbitrary number of resampled tokens-and locality preservation [20] . We performed hyperparameters search with the same budget used for the IsoFormer aggregation module for all ablations methods. We report in Table 4 the results obtained with the best set of hyperparameters for each method. We observe that, using an additional step of Perceiver Resampler always performs worse than only cross-attention -even after optimizing hyperparameters. Similarly, C-Abstractor does not confer any benefit over cross-attention; therefore we consider this latter strategy as the optimal aggregation strategy for our method. One advantage of the cross-attention mechanism we use is its interpretability, since it helps understanding which regions of the different modalities are being leveraged to make predictions. These conclusions align with recent multi-modal studies for other modalities [64] .Experiment We showed in previous sections that IsoFormer is able to aggregate information from different biological sequence modalities to formulate high-quality predictions. Here, we investigated to which extent IsoFormer's performance can be attributed to the transfer from each modality-specific encoders' pre-training. We conduct extensive experiment where we compared the performance of IsoFormer trained using all three encoders pre-trained on their respective domains with different IsoFormer model variants with all possible combinations of pre-trained / non-pretrained modalityspecific encoder.Our results demonstrate that using pre-trained encoders confers a substantial advantage to IsoFormer, as the R 2 is substantially larger (0.53) when compared with IsoFormer with none of the encoders pre-trained (0.10; Table 5 ). This demonstrates that IsoFormer is leveraging the knowledge acquired by each foundation model in the respective domains. However, we observed that when we randomly initialized only the DNA or RNA encoders, the drop in performance is smaller (0.41 for DNA and 0.48 for RNA). This suggests that IsoFormer not only leverages intra-modalities pre-training but also inter-modalities transfer. Altogether, these results underpin our approach of relying on initializing IsoFormer with pre-trained encoders, as the information learned during pre-training is transferred and leveraged when considering multi-modal tasks. 1. Focusing on the pre-trained RNA encoder, we take the attention weights for each layer and head after running the IsoFormer finetuning process. For each layer and head, we compute how much attention (percentage) is directed towards a specific region of interest of the RNA sequence. The regions we consider are 3UTR, 5UTR, and CDS, and we use the following equation to compute the exact ratio of attention:EQUATIONwhere X rna is the set of RNA sequences in the test set, α(i, j) is the attention coefficient between tokens i and j, f (i) is an auxiliary function that equals 1 if token i belongs to the region of interest in the sequence (e.g. 3UTR), and µ is a threshold value (we choose 0.01).We denote these attention maps by ρ IF . 2. We repeat this process for the finetuning run in which only the RNA encoder is used (RNA only in Table 2 ). We denote these attention maps by ρ NT . 3. For each layer and head, we compute the ratio ∆ρ = (ρ IF -ρ NT )/ρ NT . For simplicity, we cap these values to 1 (i.e., attention rate is doubled in the IsoFormer case compared to finetuning a RNA encoder alone). 4. In addition, for both ρ IF and ρ NT , we can consider the samples in the test set X rna and build a distribution of attention rates per element of interest. Comparing both distributions (the one coming from ρ IF and the one coming from ρ NT ), we can carry out a t-test per layer and head. We follow these t-tests and select the combinations of layers and heads in which there are statistically significant differences (i.e., p < 0.05) between ρ IF and ρ NT . 5. Figure 3 -right shows the ratio ∆ρ for those pairs of layers and heads in which statistically significant differences are observed. The rest is set to zero.Figure 6 shows the performance of IsoFormer using Enformer as DNA encoder for each of the 30 tissues. IsoFormer outperforms the DNA-only Enformer model on every tissue. https://www.gtexportal.org/home/downloads/adult-gtex/bulk_tissue_expressionhttps://huggingface.co/InstaDeepAI/isoformerhttps://huggingface.co/InstaDeepAI/isoformer.https://www.gtexportal.org/home/downloads/adult-gtex/bulk_tissue_expressionhttps://www.ensembl.org/index.htmlhttps://www.ncbi.nlm.nih.gov/datasets/genome/GCF_000001405.26/https://www.gtexportal.org/home/downloads/adult-gtex/bulk_tissue_expression 
 
 Appendix:  


Decision: This work introduces IsoFormer, a multimodal approach that leverages pre-trained models on DNA, RNA, and protein sequences to predict transcript isoforms across multiple human tissues. The paper employs a modified cross-attention mechanism and concatenation scheme to create multi-modal embeddings, achieving state-of-the-art performance in isoform prediction. Throughout the review process, the benchmarking experiments and presentations have been refined and significantly improved. While the methods themselves may lack novelty, the introduction of the new challenge of predicting isoforms, combined with the availability of relevant data, will be of strong interest to NeurIPS audience.

Reviews: review: paper_summary: paper_summary: The paper models isoform relative abundance across tissues with a multimodal approach based on 3 pretrained encoders for DNA, RNA, and AA sequences.  DNA encoder uses a sequence centered on the gene’s TSS, RNA encoder uses the known isoform sequence from RNAseq and the protein encoder uses corresponding AA sequence.  They perform multiple ablations on the utility of having all 3 separate encoders and, given separate encoders, how to aggregate them into a single isoform specific embedding/prediction, and look at attention layers of RNA module to find biologically meaningful regions of attention.
strength_weakness: strength_weakness: I don’t think the authors can claim this is the first attempt to combine DNA, RNA, and AA modalities with techniques from NLP.   See the recent Evo work here https://www.biorxiv.org/content/10.1101/2024.02.27.582234v2 .  While they evaluate their performance against Enformer, that’s a large part of their own model.  So the evaluations have an intramural feel to them.  It’d be interesting to see how their strategy compares to other multi modal models such as Evo, and more RNA centric work like Borzoi, which looks at a more fine grained look of variant effects on the DNA to RNA relationship.  Looking at average isoform abundance across individuals is all well and good, but GTEx also has individual genomes, and genomic variation across individuals will also of course affect splicing patterns and which isoforms come from what individuals.
questions: questions: Some comments and questions:

Centering on TSS will only capture regulatory elements within the chosen sequence length.  There could be distal or trans CREs outside

GTEx database has less than 1000 donors, confirming the 5000 individuals claim?

Looking at equation 5 in 4.4, how does f_psi depend on the tissue T? Is it a separate head per tissue, or are you predicting the vector of isoform abundance across tissues with one pass?  And is f_theta and f_phi the same f but different weights as f_psi?  Where does the summation over i take place?

5.1 does ablations with one DNA encoder, then 5.2 shows superior performance with Enformer as the DNA encoder.  So the ablations in 5.1 may not be accurate with respect to this new encoder.  It also begs the question of how would the Enformer do as the RNA encoder as well.

How are RNA and protein sequence lengths handled when they’re longer than the model input sequence size?
limitations: limitations: The authors should be more explicit about the limitation of using reference genome and known isoform sequences and how this kind of sweeps splicing as a function of dna sequence under the rug.
paper_summary: paper_summary: The paper introduces a new framework for the multi-modality pretrain model according to the Central dogma of biology. The method encode DNA, protein and RNA at the same time. The proposed method can transfer knowledge from the encoders pretraining and modalities.
strength_weakness: strength_weakness: Lack of experiments. 1. More ablation studies should be conducted about removing different modalities of the model in Table 2 ( e.g. we observe only RNA can achieve a high performance, what about protein+RNA? ).

 2. More dataset details should be included. The split of training/validation/test sets is not clear. If the authors do the experiments on the same dataset, they should split the dataset according to the tissues to validate the transferability of the proposed method.
questions: questions: The authors should include more motivation about the proposed method. For example, why can the changed DNA influence the RNA seq? Why not directly predict RNA expression from RNA seq?
limitations: limitations: The biological system is more complex. The proposed method only include the direct map from DNA to RNA. However, in real world, RNA can effect the expression of DNAs. More details should be discuss in the future work.
paper_summary: paper_summary: The paper introduces a novel multi-modal model, IsoFormer, designed to integrate DNA, RNA, and protein sequences for predicting RNA transcript isoform expression across different tissues. It utilizes pre-trained modality-specific encoders to generate embeddings that are then combined using a sophisticated aggregation method. The model demonstrates significant improvements in prediction accuracy compared to single-modality approaches.

Contriburion:

1. Developed the first general-purpose multi-modal model integrating DNA, RNA, and protein sequences.

2. Demonstrated successful application of transfer learning from modality-specific encoders.

3. Provided a new robust framework for advancing the prediction of RNA transcript isoform expression.
strength_weakness: strength_weakness: 1. Complexity and Computation: The model's complexity and the computational demands might limit its accessibility and use, particularly in environments with restricted resources.

2. More Comprehensive Evaluation for PLM's representation learning capability would make this paper better.
questions: questions: (1)	For Tab 5., wonder what’s the performance for “DNA and RNA encoder not pre-trained”

(2)	Could authors provide evaluation results for DNA, RNA and protein encoder separately on their own popular benchmarking tasks? Ideally, these models should improve on those downstream tasks as well. Would give 7’ or even 8’ if authors conduct these experiments. It could be a great and exciting work in this AI4Bio field.
limitations: limitations: 1. Data Requirements: The effectiveness of the model is contingent on the availability of comprehensive and high-quality multi-modal datasets.

2. Generalizability: While promising, the results are primarily validated on specific types of gene expression data, and its performance across broader biological applications remains to be fully assessed.

Comments: title: Clarification on the additional gene expression task
comment: Thanks for your response. I have raised my score. Hope your next version can be more clear
For clarification, the additional experiment is the prediction of gene expression measured by bulkRNA-seq from Chia Hsiang et al. [1], which is different from the isoform expression prediction task that was used to train IsoFormer. In the transcript isoform expression prediction task, the goal is to predict the expression of each RNA transcript isoform of a given gene, whereas in the gene expression prediction task, the goal is to predict the expression of a gene from its DNA sequence - and this is one of the key benchmarking tasks for DNA language models nowadays. Those tasks are different but correlated, hence we chose it as we expect transferability between them. Our experiment demonstrates that training Nucleotide Transformer as a DNA encoder within our multi-modal transcript expression framework increases its performance on this gene expression task.

We did not have the time to run extensive experiments on existing benchmarks for protein and RNA language models before the end of discussion period but we agree that those would be valuable insights. This is an interesting step for future iterations of this work that we will add to the conclusion of the paper.

[1] Kao, Chia Hsiang, et al. "Advancing dna language models: The genomics long-range benchmark." ICLR 2024 Workshop on Machine Learning for Genomics Explorations. 2024.
thank you for addressing my questions.  i think those corrections and clarifications will benefit the paper’s presentation.
Hi thanks for all the efforts and detailed response. For my question (2), we refer to benchmarking tasks used in protein language models, RNA language models, and DNA language models, not only on transcript isoform expression prediction task. Not sure if the remaining time would be enough. Sorry that I didn't response promptly because of work during weekdays.