DOI: 10.5281/zenodo.17130428
title: UltraSynBERT_All_Archives

description: Decoding the RNA interactome by UltraSynBERT

The official repository for the article "Decoding the RNA interactome by UltraSynBERT".

Overview

RNA encodes complex molecular behaviors that underlie binding, regulation, and cellular fate. Current RNA language models have been shaped primarily by endogenous transcripts, leaving the potential of exogenous RNAs to extend model boundaries unexplored. We present UltraSynBERT, an RNA language model pretrained on synthetic RNA binding landscapes to capture RNA binding properties across in vitro and in vivo contexts. Fine-tuned on in vitro SELEX datasets, the model identifies RNA aptamers for diverse targets, including small molecules, proteins, cells, and tissues. The model also predicts tissue specificity for millions of RNA species across 22 human organs based on their 3’-UTR sequences, captures humanpathogenic viral RNA tropism, recognizes RNA base modifications, and characterizes SARS-CoV-2 replicase binding with single-base resolution features.

 

UltraSynBERT Archives

1. Pre-training datasets and models

In this repository, you will find the following pre-training datasets and pre-trained model checkpoints:




Model Name
Parameters
Pre-training Data
RNA Species Number
Description
Checkpoint




UltraSynBERT
33.5M
UltraSelex SiR-B
10 million
pre-training from scratch
UltraSynBERT.pkl


UltraSynBERTsource_SiR
33.5M
SELEX SiR
10 million
pre-training from scratch
UltraSynBERT_source_SiR.pkl


UltraSynBERTsource-nsp12
33.5M
SELEX nsp12
10 million
pre-training from scratch
UltraSynBERT_source_nsp12.pkl


UltraSynBERTmolecules
33.5M
12 SELEX targets’ training sets
3.24 million
continued pre-training the basic UltraSynBERT model
UltraSynBERT_molecule.pkl


UltraSynBERT3UTR
33.5M
The preliminary 3’-UTR full training datasets
1.69 million
continued pre-training the basic UltraSynBERT model
UltraSynBERT_3UTR.pkl


UltraSynBERTplus
33.5M
12 SELEX targets’ training sets and the preliminary 3’-UTR full training datasets
4.93 million
continued pre-training the basic UltraSynBERT model
UltraSynBERT_plus.pkl


UltraSynBERTRIP
33.5M
Human RIP-Seq in ENCODE
6.88 million
continued pre-training the basic UltraSynBERT model
UltraSynBERT_RIP.pkl




NOTE:





The pretraining data is located in the Dataset Availability\1. Dataset for pre-training folder.




The pretrained models can be found in the Model Availability\1. Pre-training models and Model Availability\2. Continued pre-training models folders.



 

2. Downstream task datasets and fine-tuned models

Additionally, this official repository also provides all datasets for downstream tasks and supervised fine-tuned models based on the base UltraSynBERT for downstream applications, including: RNA aptamer identification models targeting small molecules, proteins, cells, and tissues (12 specialized datasets), and tissue-specificity prediction for human mRNA 3'-UTR regions.




Task Type
Data Type
Dataset Number
Dataset




SiR-binding prediction
In vitro
3
(1) UltraSelex-SiR-B.csv (2) SELEX-SiR-whole.csv (3) SELEX-SiR-exclusive.csv


In vitro RNA-target binding prediction
In vitro
12
(1) SELEX-small-molecule-DAse.csv; (2) SELEX-small-molecule-BC.csv: (3) SELEX-small-molecule-PR.csv (4) SELEX-small-molecule-MI.csv: (5) SELEX-protein-TARDBP.csv: (6) SELEX-protein-RT.csv (7) SELEX-protein-RBM24.csv; (8) SELEX-protein-S15.csv; (9) SELEX-(multi)cellular-ISLETS.csv (10) SELEX-(multi)cellular-MDSC.csv; (11) SELEX-(multi)cellular-CHO-K1.csv;(12) SELEX-(multi)cellular-TNBC.csv


In vivo RNA-protein interactions prediction (human source)
In vivo
11
(1) 16_ICLIP_hnRNPC_Hela_iCLIP_all_clusters_sequences.csv; (2) 17_ICLIP_HNRNPC_hg19_sequences.csv;(3) 18_ICLIP_hnRNPL_Hela_group_3975_all-hnRNPL-Hela-hg19_sum_G_hg19--ensembl59_from_2337-2339-741_bedGraph-cDNA-hits-in-genome_sequences.csv (4) 19_ICLIP_hnRNPL_U266_group_3986_all-hnRNPL-U266-hg19_sum_G_hg19--ensembl59_from_2485_bedGraph-cDNA-hits-in-genome_sequences.csv; (5) 20_ICLIP_hnRNPlike_U266_group_4000_all-hnRNPLlike-U266-hg19_sum_G_hg19--ensembl59_from_2342-2486_bedGraph-cDNA-hits-in-genome_sequences.csv; (6) 22_ICLIP_NSUN2_293_group_4007_all-NSUN2-293-hg19_sum_G_hg19--ensembl59_from_3137-3202_bedGraph-cDNA-hits-in-genome_sequences.csv (7) 27_ICLIP_TDP43_hg19_sequences.csv; (8) 28_ICLIP_TIA1_hg19_sequences.csv; (9) 29_ICLIP_TIAL1_hg19_sequences.csv (10) 30_ICLIP_U2AF65_Hela_iCLIP_ctrl_all_clusters_sequences.csv; (11) 31_ICLIP_U2AF65_Hela_iCLIP_ctrl+kd_all_clusters_sequences.csv


In vivo RNA-protein interactions prediction (mouse source)
In vivo
11
(1) CLIP-mouse-EZH2-sequences.csv; (2) CLIP-mouse-FUS-sequences.csv; (3) CLIP-mouse-HNRNPR-sequences.csv (4) CLIP-mouse-LIN28A-sequences.csv; (5) CLIP-mouse-RBFOX2-sequences.csv; (6) CLIP-mouse-RBM10-sequences.csv (7) CLIP-mouse-SRSF2-sequences.csv; (8) CLIP-mouse-SRSF3-sequences.csv; (9) CLIP-mouse-TARDBP-sequences.csv (10) CLIP-mouse-U2AF2-sequences.csv; (11) CLIP-mouse-YTHDC2-sequences.csv


modification 
In vivo
13


m6A: (1) m6A-A549.csv; (2) m6A-CD8T.csv; (3) m6A-ESC.csv (4) m6A-HCT116.csv; (5) m6A-HEK293.csv; (6) m6A-HEK293T.csv (7) m6A-Hela.csv; (8) m6A-HepG2.csv; (9) m6A-MOLM13.csv

m1A: m1A_dataset_with_split.csv

m5C: m5c_dataset_with_split.csv

m6Am: m6Am_dataset_with_split.csv

pseudouridine: pseudouridine_dataset_with_split.csv



3'UTR tissue-specific recognition
In vivo
1
human_22_tissue_three_terminal_UTR.csv


Zero-shot RNA mutation effect prediction targeting SARs-CoV-2 replicase nsp12
In vitro
The wild-type sequence and seven mutated sequences
RNA-mutation.fasta


RNA post-transcriptional regulation–related tasks 
in vitro/In vivo
4


(1) miRNA _interactions.csv; (2) mRNA_subcellular_localization.csv; (3) Polyadenylation_signal.csv; (4) RNA_stability.csv

 



human-pathogenic RNA viruses
In vivo
1


virus.csv





NOTE:





The datasets for downstream tasks are in the Dataset Availability\2. Dataset for fine-tuning folder.




The fine-tuned models for downstream tasks are available in the Model Availability\3. Fine-tuned models folder.