Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types
Authors: Shentong Mo
Xi Fu
Chenyang Hong
Yizhen Chen
Yuxuan Zheng
Xiangru Tang
Yanyan Lan
Zhiqiang Shen
Eric Xing
Venue: ICLR 2022 Submitted (2022)

Abstract: In the genome biology research, regulatory genome modeling is an important topic for many regulatory downstream tasks, such as promoter classification, transaction factor binding sites prediction. The core problem is to model how regulatory elements interact with each other and its variability across different cell types. However, current deep learning methods often focus on modeling genome sequences of a fixed set of cell types and do not account for the interaction between multiple regulatory elements, making them only perform well on the cell types in the training set and lack the generalizability required in biological applications. In this work, we propose a simple yet effective approach for pre-training genome data in a multi-modal and self-supervised manner, which we call $\textbf{\texttt{GeneBERT}}$. Specifically, we simultaneously take the 1d sequence of genome data and a 2d matrix of (transcription factors × regions) as the input, where three pre-training tasks are proposed to improve the robustness and generalizability of our model. We pre-train our model on the ATAC-seq dataset with 17 million genome sequences. We evaluate our GeneBERT on regulatory downstream tasks across different cell types, including promoter classification, transaction factor binding sites prediction, disease risk estimation, and splicing sites prediction. Extensive experiments demonstrate the effectiveness of multi-modal and self-supervised pre-training for large-scale regulatory genomics data. 

Under review as a conference paper at ICLR 2022
MULTI-MODAL SELF-SUPERVISED PRE-TRAINING FOR REGULATORY GENOME ACROSS CELL TYPES
Anonymous authors Paper under double-blind review
ABSTRACT
In the genome biology research, regulatory genome modeling is an important topic for many regulatory downstream tasks, such as promoter classification, transaction factor binding sites prediction. The core problem to model how regulatory ele- ments interact with each other and its variability across different cell types. How- ever, current deep learning methods often focus on modeling genome sequences of fixed set of cell types and do not account for the interaction between multiple regulatory elements, making them only perform well on the cell types in the train- ing set and lack the generalizability required in biological applications. In this work, we propose a simple yet effective approach for pre-training genome data in multi-modal and self-supervised manner, which we call GeneBERT. Specif- ically, we simultaneously take the 1d sequence of genome data and a 2d matrix of (transcription factors regions) the input, where three pre-training tasks are proposed to improve the robustness and generalizability of our model. We pre-train our model on the ATAC-seq dataset with 17 million genome sequences. We evaluate our GeneBERT on regulatory downstream tasks across different cell types, including promoter classification, transaction factor binding sites predic- tion, disease risk estimation, and splicing sites prediction. Extensive experiments demonstrate the effectiveness of multi-modal and self-supervised pre-training for large-scale regulatory genomics data.
INTRODUCTION
In the gene biology research, many important regulatory problems including promoter (Li et al. 2006) classification and transaction factor binding sites (Stewart et al.) 2012) prediction requires a well-designed approach for modeling regulatory genome sequences. As multi-cellular organ- ism, the human body is formed by different cell types, with each has its own gene expression pat- terns (Schena et al. 1995 Ross et al. 2000). Therefore, modeling regulatory genome, especially across different cell types, crucial for both the understanding of this fundamental biological pro- cess and the development of RNA-level therapeutic intervention of a vast among of diseases. The main challenge is how model the interaction between regulatory elements and its variability across different cell types.
In recent years, some works (Chen al. 2016 Kelley et al.) 2016 Romero al. 2017] Qiu al. 2018 Torada et al.] 2019: Chen et al. [2000a] Avsec et al. [202 Ji et al.) 2021) have been proposed apply deep learning architectures on modeling the genome data. For example, Enformer (Avsec) et al. 2021) combines dilated CNN and transformer architecture as well as multi-head output to predict gene expression and epigenomic marks, and gain some performance improvements against traditional methods. Most of these models are based on supervised lcarning paradigm, which limits their abilities to learn the general interaction representations between different regulatory elements. As result, each model is only useful for some specific downstream tasks. More recently, DNABERT (Ji et al. 2021 is introduced to formulate the whole DNA sequence as a sentence of nucleotide k-mers and utilize BERT to model the sequence generatively. In this way, the interactions between different regulatory elements could well captured, like the language modeling approach in natural language processing (NLP) to capture the co-occur information between different words, and could be employed for modeling several related downstream tasks. However, DNABERT only performs well on the cell types in the training set, and generalizes poorly unseen cell types. This

---

Under review as a conference paper at ICLR 2022
is mainly because the whole DNA sequence is common for different cells, and pre- pre-training on such data cannot well reflect various interaction patterns across different cell types.
Integration of genome data modalities across different cell types could help to build a more holistic model of gene expression regulation and benefit downstream applications such as mutation impact evaluation and disease risk prediction, as well as promote understanding of the cell-yype-peecfiic regulatory programs and various development processes and disease etiology. Inspired by this idea, in this work, we present simple yet effective method called GeneBERT, for pre-training large-scale genome data in multi-modal and self-supervised manner. Specifically, we simultaneously take the ID modality (i.e. sequence) and 2D modality (i.e. regulatory region) of genome data as the input, where three pre-training tasks are proposed to improve the robustness and generalizability of our model. 1) masked genome modeling: we randomly mask some parts of the input k-mers with a special token (i.e., [MASK]), and the model is trained to predict the masked k-mers. 2) next genome segment prediction: we train the model using the embedding [CLS] to classify whether a pair of given sequences are two consecutive sequences in a cell. 3) sequence-region matching: a sequence- region matching mechanism is proposed to capture the multi-modal alignment between sequence and regulatory region of genome data.
We pre-train the GeneBERT on the ATAC-seq dataset (Domcke et al. 2020) with 17 million gene sequences. Furthermore, conduct extensive experiments to evaluate our GeneBERT vari- ous downstream tasks, including promoter prediction, transaction factor binding sites prediction, gene mutation localization, and personalized diseases prediction. Comprehensive ablation studies demonstrate the effectiveness of multi-modal self-supervised pre-training for large-scale genome data across different cell types.
The main contributions of this work lie in the proposal of a simple yet effective method named GeneBERT, for large-scale genome data pre-training in a multi-modal and self-supervised manner. To the best of our knowledge, we are the first to incorporate different genome data modalities across various cell types into the pre-training for large-scale genome data, to tackle the regulatory genome modeling problem. Except for meaningful biological improvements, our model makes an important contribution to the machine learning community, by introducing a novel multi-modality construc- tion. Different from existing multi-modal learning tasks, such as visual question answering, image caption, and image-text retrieval, the 'visual' modality in our work is constructed based on the reg- ulatory property of the 'language' sequential units. This idea brings some inspiration to building new 'visual' modality, based on text matching, part-of- tagging, and named entity recognition etc., for the study of NLP.
2 RELATED WORK
Uni-modal language pre-training, Self-supervised pre-training models such as GPT (Radford eet al. 2018), BERT (Devlin et al., 2018), RoBERTa (Liu et al. 2019). and ERNIE (Sun et al. 2019 have led to dramatic improvement on variety of natural language processing tasks in the past few years, significantly surpassing the traditional context-independeer as Werd2Vec. Obviously, BERT (Devlin al. 2018) uses the masked language model (MLM) and next sentence prediction (NSP) for pre-training which represent the dynamic vector of words according the context. RoBERTa (Liu al. 2019) uses dynamic MLM and discards NSP to train the model with a long time. ERNIE (Sun et al. 2019) masks entities and phrases to learn more context relations. These self-supervised pre-training models are capable of performing natural language processing tasks more effectively by improving MLM.
language model such
Multi-modal pre-training. Multi-modal pre-training has recently addressed researchers' atten- tion for learning meaningful representations. Typically, Previous methods (Radford et al., 202T Huo et al., 2021 learn visual representations from text paired with images unsupervised, self- supervised, weakly supervised, and supervised ways. Since language and vision can share a sim- ilar semantic meaning, CLIP (Radford et al.] (021) a commonly-used neural network trained on variety of (image, text) pairs for learning transferable visual representations from natural lan- guage supervision. WenLan (Huo et al. 2021) applies a cross-mode modal contrastive learning frame- work called BriVL for image-tex pre- tre-iraining. However, in this work, leverage the modli- self-supervised pre-training on the genome data to improve the robustness and generalizability of pre-trained models used for data-scarce scenarios.
2

---

Under review as a conference paper at ICLR 2022
Genome data pre-training. Transformer models have been recently established to better under- stand the genotype- peetpppypp relationships (Avsec et al. 2021 Ji et al. 2021). DNABERT used the human genome pre-traane BERT- model, trying to decipher the regulatory code related to gene expression (Ji et al. 2021). In order to adapt the DNA scenario, sequences were split into 510 base-pair long and tokenized 3- 6-mer representations. The model was then pre-trained using those k-mers by predicting around 15% randomly masked regions of the sequence. After the pre-training, the model was fine-tuned on three downstream tasks related gene regulation: pre- diction of promoters, transcription factor binding sites (TFBSs), and splice sites. Furthermore, by analyzing the attention maps, DNABERT could visualize the important regions contributing to the model decision, which improved the interpretability of the model.
METHOD: GENEBERT
In this section, propose simple yet effective approach for pre-training genome data in multi- modal and self-supervised manner, as illustrated in Figure Specifically, we introduce two main ob- jectives for sequence pre-training, including Masked Genome Modeling (MGM) and Next Genome- Segment Prediction (NGSP). Next, we extract features of the region for region pre-training. Fi- nally, present the Sequence-Region Matching mechanism explicitly align features of the quence and region in the input genome data in a contrastive self- self-supervised manner. Therefore, our GeneBERT consists of three main components: sequence pre-training, region pre-training, and sequence-region matching.
Sequence Pre-training
MGM
Teacs
Sequence Requence-Region Matching
Transformer
Bing
F
E/g
Eb
(Sequence)
Rive
triboking
Ects
EIMAiRS
Eoaca
Fsr
Bix
O
CACG
GPCA
AATC
(Region Regulstor)
Region Transformer
E2
3,
Region Pre-training
Figure 1: The overall framework of our proposed GeneBERT model.
3.1 INPUT DATA
We use potential regulatory elements mapped with ATAC-seq which measures the chromatin ac- cessibility across the genome. The regions with open accessibility could be bound by transcription factors, which together with regulatory elements and RNA polymerase determines the expression level. In one cell type, the list of all regulatory elements can be viewed sub-regions across the genome. Each region or regulatory element can be viewed as string like "AATTCCT.." consist of 4 elements: [A, T, C. GJ. The binding site of different transcription factors has different sequence patterns, some of which have been determined using experimental methods and have been modeled using probabilistic models like position weight matrix (PWM). Using PWM, one can get the predicted affinity of sequence segment given transcription factor. Each regu- latory region can then be associated with a 'trancription factor binding vector" which contains the in silico prediction of their affinity fixed number of transcription factors. By scanning through regulatory element sequence with PWM of multiple transcription factors, we can get a transcription factor binding affinity vector for that regulatory element.
set of non-overlapped
3

---

Under review as a conference paper at ICLR 2022
When the collection of transcription factor binding models is comprehensive and precise enough, we can get good representation of regulatory potential for given sequence. In different cell types, the accessibility of regulatory elements is different. Some regulatory elements are shared across different cell types while some are more specific to one cell type. As a result, in a local region of the genome, the accessible transcription factor binding sites are determined by the combination of accessible regulatory regions. This combinatorial interaction between regulatory regions and tran- scription factors varies across different cell types such that precise control of expression in different cell types is possible. To capture the combinatorial interaction and the implicit cell-type specific in- formation, we then group 10 consecutive regulatory elements into one train sample by concatenating their string (ID sequence) and stacking their binding vector into 2D region, as shown in Figure]
3.2 SEQUENCE PRE-TRAINE
Similar to DNABERT, we take k-mer tokens as the sequence units. The k-mer refers to a sequence with length k, i.e. for a sequence AGTCAG, the 3-mers are {AGT, GTC, TCA, CAG), and the 4-mers are {AGTC, GTCA, TCAG]. Therefore for sequence pre-training, we input three types of embeddings: 1) k-mer embedding x* for each k-mer in sequence; 2) segment embedding x" indicating which part of the sequence the k-mer is from; 3) a position embedding x" for the position of the k-mer in the sequence. Then we sum up all three embeddings to obtain contextual representation e", n E 21.2.... N), where N denotes the number of k-mers in the sequence. After being fed into BERT-based transformer (-ea()). (.) those contextual embeddings become Exey i.e. the obtained representation for [CLS]. Inspired by BERT (Devlin et al., 2018), we adopt two objectives including Masked Genome Modeling (MGM) Lmgm. and Next Genome Segment Prediction (NGSP) Engep- For mmnn we randomly mask some parts of the input k-mers with special token (i.e., [MASK]), and the model is trained predict the masked kmer. The objective is formulated as:
mggm = max P(K [K)
where K and mask denote the unmasked and masked k-mers, respectively. As for Engep, we train the model using the embedding [CLS] to classify whether a pair of given sequences are consecutive in cell. The loss function of Engsp is calcualted as:
Lngsp = CrossEntropy (P(CCL)))
(2)
where P([CLS)) and y represent the predicted probability and target for classification, respectively.
3.3 REGION PRE-TRAINING
For the region features in the pre-training, we consider strong backbone (i.e. Swin (Luu et al. 2021 transformer as the region encoder (-eg()) to extract representations Ere. Specifically, we apply the Swin transformer pre-trained on ImageNet to the region input directly to generate Ereg- During the pre-training, we unfreeze the parameters of the Swin transformer and update them for learning better regional representations. In the pre- pre-training setting, each region input corresponds to each sequence such that we can capture the multi-mode alignment between sequence and region of genome data.
3.4 SEQUENCE-REGION MATCHING
In order to learn the alignments between sequence and region of genome data, we propose a sequence-region matching mechanism based sequence embeddings Exeq and region embeddings Ery-- Firstly, we calculate the similarity (i.e. inner product) between each pair of 'linguistic' em- beddings Eie, and 'visual' embeddings Eieg in a batch of size b where E 1,,2..., b. Then, those similarities are jointly learned for alignments between the whole sequence and each region in contrastive learning manner, where we maximize the similarity of the sequential and regional em- beddings of the b correct pairs in the batch while minimizing the similarity of the embeddings of the b(b 1) false pairs. Specifically, an info-NCE like loss defined as follows is applied over these similarities scores for optimization:
Ei=1 50g
E
Corm - -log
TOB
(3)
Σ
E
Ei=1 2j=1
Bay
For other transformer strcutures, we explore them in the ablation study.
4

---

Under review as a conference paper at ICLR 2022
Algorithm Gen multi-modal pre-training algorithm
Input: Sequence transformer (.) Region transformer freg())
1: ID-data: k-mer embedding x segment embedding x position embedding x
2: 2D-data: Region matrix X
3: Initialize the parameters (scg()) fre(() with pre-trained weights
4: for each iteration step do
5: Sequence Pre-training
6: Feed x* x², x2 into (+) to generate Egeq
7: calculate Lmgm and Lngep in Eq. T and 2
8:
Region Pre-training
9:
Feed 2D region X into f to generate Ereg
Sequence-Regee Region Matching
11:
Apply embeddings Exeq, Ere to calculate the loss in Eq. 3 Compute the total loss Eq.
13:
Update the parameters of fseg ()) freg(
Output: [seg()), (re(())
In this way, alignments between the whole sequence and each region are learned via our GeneBERT in the pre-training process.
Combining the above three losses together, obtain the overall objective function for GeneBERT:
[ Lanmm + Lugsp + A Earm
where we vary A in [0.01, I] to perform the parameter study for X, and observe that the performance of our model is stable when À lies in the region of [0.5, 1]. Therefore in our experiments, we directly set À = 0.5. The overall algorithm is summarized in Algorithm
4
EXPERIMENTS
4.1 PRE-TNAINING DATA & SETTINGS
For pre-training data, process public human fetal cerebrum single-cell chromatin accessibility data in the Descartes database (Domcke et al. 2020) to generate pseudo-bulk accessibility tracks for each cell type (Seurat cell clustering provided by the original paper). Specifically, we take the provided "Peak Count Sparse Matrices" and summed up columns (cells) according to cell type defi- nition, producing a regions X cell-types matrix. Then we binarize the matrix and use only non-zero entries (accessible regions) for each cell type. The corresponding sequence for each region is then retrieved from the HG19 human reference genome, a single representation of multiple genomes. While the motif scanning for each region is either retrieved from the Descartes database or scanned following the same approach using JASPAR 2018 (Khan et al., 2018) vertebrate transcription fac- tor binding site motifs. In total, cludes 1,000,029 accessible regions across the genome, covering 504,657,45 7,56 base pairs. For the 1D modality, we group 10 consecutive accessible regions into sample, which corresponds to (10 x number of TFFs) matrix for the 2D modality. Following previous works (Ji et al.] (021). we pre-train the model for 120k steps with warm-up learning rate of 4c-4 and batch size of 2000. 15% of k-mers in each sequence are masked in the first 100k steps and 20% for the last 20k steps.
17 cell types and the union of all accessibility track in-
4.2 DOWNSTREAM TASKS & RESULTS
Promoter Classification. Promoters are the elements responsible for regulating the initial transerip- tion of the gene, which is located near the transcription start site (TSS). As the promoters play an important role in gene regulation, using machine learning methods to predict promoter sites accu- rately is one of the most popular problems in bioinformatics. Here we first used the promoter core dataset from (Ji et al. 2021), which are the 70bp sequences centered around TSS. The promoter core is the key part of the promoter flanking region which is sufficient direct accurate initiation of transcription (Oubounyt et al. 2019). Here we fine-tune our GeneBERT model to predict the promoter core sequences and compared them with DNABERT. We report the experimental results
5

---

Under review as a conference paper at ICLR 2022
in [ For fair comparison with DNABERT, reproduce the results of DNABERT by fine- tuning the pre-trained models provided by the authors. During the reproduction process, we observe that it is hard to get the same results reported in the original paper their model is very sensitive to the hyper-parameter settings. From the comparison results, we can see that our model can predict promoter core more precisely with improved performance when compared DNABERT. This is due to the inclusion of regulatory genome information in our 2D pre-training data, which guides the pre-training process to focus on those binding site motifs and their interactions with each other.
Table I: Comparison results on promoter classification.
Method
Precision
Recall
AUC
DNABERT
0.675
0.637
0.693
GeneBERT (ours)
0.805
0,803
0.894
Table Comparison results on Transcription Factor Binding Sites classification.
Transcription Factor & Call Line
Method
Precision
Recall
AUC
DNABERT
0.250
0,500
0.501
AFF_A559_CCTT UT-AA
GeneBERT (ours)
0.908
0,899
0.983
DNABERT
0,250
0,500
0.542
CTCF ACC9 www..41C99CC.....
UUW
GeneBERT (ours)
0.925
0.921
0.983
DNABERT
0.250
0.500
0.523
CTCF SCCF_AGG4449.CCTOO -TW.449CCCTFF_WWW
GeneBERT (ours)
0.907
0,894
0.983
DNABERT
0.250
0,500
0.501
TTCFFAGG4450.CTCF.UM
GeneBERT (ours)
0.929
0.925
0.987
DNABERT
0.250
0.500
0.545
CTCF AG09309
GeneBERT (ours)
0.931
0.927
0.987
DNABERT
0.250
0,500
0.529
STCFLAG09319.CO
GeneBERT (ours)
0.924
0,919
0.983
DNABERT
0.250
0.500
0.535
EE510803_CTCE.
GeneBERT (ours)
0.934
0.932
0.981
DNABERT
0.250
0.500
0.531
CTCF AOAF CTCF UW
GeneBERT (ours)
0.917
0,913
0.982
DNABERT
0.250
0,500
0.540
CTCF BE(2-C CTCF UW
GeneBERT (ours)
0.937
0,935
0.989
Transcription Factor Binding Sites (TFBS) Classification. Predicting TFBS is an important step in studying gene regulation. Sequencing technologies like ChIP-seq can provide information on the in vivo binding sequences, which improve the identification of gene regulatory regions. There are several previous studies that tried to predict TFBSs using traditional machine learning (Hong & Yip, 2020) and deep learning methods (Alipanahi et al. 2013). By incorporating the multi-modal pre- training. the prediction of TFBSs can be further improved. Although we utilize the motif information during the region pre- pre-training, we do not provide any matching information to the model, which avoids leaking information about the actual motif of specific TE. Here fine-tune our model for predicting TFBSs from the ChIP-seq data, using 497 TF ChIP-seq uniform peak profiles from ENCODE Consortium (Consortium al. 2012). Note that the data in this task is different from that reported in DNABERT, since they did not provide the data for this task. We reproduce the results of DNABERT this TFBS data with the same setting as our GeneBERT. We take the peak sequences of each TF as the positive set and generated corresponding negative set by randomly shuffling the nucleotides in each positive sequence while preserving dinucleotide frequencies. Again, our model can accurately predict the CTCF binding sites in all tested cell lines.
Disease Risks Estimation. GeneBERT could provide more interpretations complex genetic dis- eases. On the one hand, while the disease status and genomic mutations were available, by inte- grating the 2D data, the relationships among regulatory regions of genes could be captured, which allowed us to estimate the disease risk more accurately. As shown in Table 3 GeneBERT can pre- cisely predict Hirschsprung Disease (HSCR), which is known as a genetic disorder with complex patterns of inheritance. Particularly, terms of all metrics (Precision, Recall, AUC), which further demonstrates the effectiveness of our approach in the treatment target sites and proceeded the medical experimental validation.
GeneBERT outperforms the baseline by a large margin in

---

Under review as a conference paper at ICLR 2022
Table 3: Comparison results on disease risks estimation.
Data
Method
Precision
Recall
AUC
DNABERT
0.265
0.500
0.500
GeneBERT (ours)
0.770
0.519
0.562
DNABERT
0.252
0,500
0.462
HSCR-RET-Long
GeneBERT (ours)
0.768
0.513
0.541
RNA-Splicing Sites Prediction. RNA splicing is important for post-transcription processing to remove introns from pre-mRNA sequences and generate mature mRNA for protein translation. Pre- viously, Dilated CNN models have been used to predict splice junction across the genome and evaluate the impact of genomics variants on splicing sites (Jaganathan al., 2019). In particular, for each nucleotide in a given sequence for splicing site prediction, we follow the previous approach and include a context sequence around the nucleotide, which could potentially capture the sequence specificity features of RNA RNA-binding proteins and splicing machinery. Since open chromatin regions and splicing sites do not always overlap with each other, among all 548,000 splicing sites in the GTEx pre-mRNA transcripts data, our pre-training sequence only fully covers the entire (in the 256nt context setting) sequence of 72,500 sites. The experimental results are reported in Table In total, 26.7% of nucleotides in context and splicing site sequence are included in the open chromatin region that we use for pre-training. Following the same training/testing split scheme and classifica- tion metric (Top-k Accuracy and PR-AUC) in the SpliceAl study (Jaganathan et al. 2019), achieve comparable or better results in different context settings without an extremely long con- text sequence. These results clearly demonstrate the generalizability of our GeneBERT pre-trained representations. By integrating sequence binding features of RNA binding proteins, we are able to further expand our model enable cell-type specific splicing junction prediction in the future.
Table 4: Comparison results on Splicing datasets.
Data
Method
Top-k Accuracy
PR-AUC
dilated CNN
0.57
0.60
SpliceAI-80nt
GeneBERT (ours)
0.83
0.89
dilated CNN
SpliceA1-256nt
GeneBERT (ours)
0.93
0.95
dilated CNN
0.90
0.95
SpliccAI-400nt
GeneBERT (ours)
0.95
0.98
dilated CNN
0.93
0.97
SpliceAI-2k
GeneBERT (ours)
0.97
0.99
5 ABLATION STUDY
In this section, we perform extensive ablation studies on the effect each module on the final performance of GeneBERT, the effectiveness of modeling different cell types, the effect of 2D modality on pre-training, and the effect of 2D transformers on the final performance of our approach. Unless specified, we conduct all ablation studies on the promoter classification task and report the mean value of all results with 5 random seeds.
Effect of each loss. In order explore the effect of each loss on the final performance of our BERT we ablate each loss in our method and report the comparison results of promoter classi- fication in Table As can be seen, pre-training without three proposed losses achieves the worst re- sults in terms of all metrics. This means that pre-training without any objectives causes embeddings with insignificant genome information. Adding Lungm first boosts the performance by 0.659, 0.625, 0.672 for precision, recall, and AUC. By combining fmng and Ligsp- we achieve the performance gain of 0.028, 0.024, 0.032, which validates the rationality of Emgm and Engep on modeling the meaningful embeddings of genome sequences. In the meanwhile. adding Carm to the final objective further improves the results by 0.103, 0. 041, and 0.169, in terms of all metrics. This further demon- strates the effectiveness of our sqquence-region matching loss in learning the alignments between sequence and region of genome data. In this case, the sequence-regiee matching loss is helpful for multi-modal supe-supe pre-training on large-scale genome data, which boosts the performance of downstream tasks.
7

---

Under review as a conference paper at ICLR 2022
Table 5: Ablation study each loss.
Lagge
Eugap
Derm
Precision
Recall
AUC
X
x
X
0.016
0.012
0.021
x
x
0.675
0.637
0.693
/
x
0.703
0.661
0.725
/
/
0.805
0.803
0.894
Table Exploration study on different cell/tissue types.
Protein
Cell type
Tissue type
Precision
Recall
AUC
AG04449
Fibroblast
Skin
0,907
0.894
0.983
AG04450
Fibroblast
Lung
0.929
0.925
0.987
AG09309
Fibroblast
Skin
0.931
0.927
0.987
AG09319
Fibroblast
Gingival
0.924
0.919
0.983
AG10803
Fibroblast
Skin
0.934
0.932
0.981
AoAF
Fibroblast
Heart
0.917
0.913
0.982
BE(2)-C
Neuroblast
Brain
0.937
0.935
0.989
Generalization different cell/tissue types. To validate the generalizability of our approach to various cell or tissue types, we evaluate our GeneBERT on Transcription Factor Binding Sites (TFBS) classification. Specifically, we choose different proteins from various cells tissues in the human body, such as skin, lung, heart, brain, etc. The experimental results of TFBS classification on different cell/tissue types are reported Table In terms modeling different cell/tissue types, DNABERT (Ji et al., 2021) performs poorly on the generalizability. as we reported in Table 2 En- former Avsec et al. (2021) only performs well on the gene expression prediction for the single cell. various From the results, we observe that our GeneBERT achieves superior performance cell types and tissue types in terms of all metrics, including precision, recall, and AUC. This indeed learning generalizable embeddings for different cell types
shows the advantage of our GeneBERT and tissue types in the human body. Particularly, we achieve the best performance on the brain tissue and neuroblast cell, which might be caused by the pre-training data from the human brain.
50
10
20
0-
-20
400
-20
-40
-00
-40
-20
40
-00
40
-20
1585-1
Figure 2: Visualization results of representations pre-trained by the uni-modal (left) and multi-modal (right) manner. Different colors denotes the various cell types.
Effect of 2D modality on pre-training. In order to further evaluate how the 2D modality data affect the quality of pre-training on large-scale genome data, we visualize the pre-trained representations pre-trained by the uni-modal and multi-modal manner in Figure Typically, we project the embed- dings pre-trained from BERT onto 2 dimensions using tSNE tools, and select randomly 10 cell types in the pre-training data for visualization. We can observe that our GeneBERT pre-trained rep- resentations form more separated clusters that are distributed more uniformly on the space in terms of all cell types when compared to the pre-trained representations generated by the uni-modal model. Furthermore, the representations pre-trained by our multi-modal self-supervised method have much smaller distances inside each cluster of cell type. This further demonstrates the effectiveness of our 2D modality in improving the quality of pre-trained representations.
Effect of different transformer streutures. In order to figure out why choose the strong Swin transformer to take the 2D modality data as input, we explore the effect of different 2D transformers on the final performance of GenT Specifically, apply four commonly-used backbones
8

---

Under review as a conference paper at ICLR 2022
(VIT Dosoviiskii et al. 2021). T2T- VIT Yuaa et al. 2021), DeiT (Touvron et al. 2020). Swin Liu
et al. 2021)) as the 2D transformer to extract the region features, and report the comparison results GeneBERT with Swin-B-23 B-384 backbone achieves the best perfor-
Table 7 As can be seen, mance in terms of all metrics (precision, recall, AUC). In the meanwhile, with the increase input size, the performance of our GeneBERT improves, which demonstrates the effectiveness of features extraction of the 2D modality data in learning discriminative representations of large-scale genome data. As the 2D transformer gets powerful, our GeneBERT achieves better results in terms of all metrics on the promoter classification task. This further validates the importance of the 2D trans- former in multi-modal seff-supervised learning. In this case, we choose the Swin transformer as the 2D backbone since we achieve the best results with the Swin transformer.
Table 7: Exploration study 2D transformers with different input sizes.
2D Transformer
Input Size
Precision
Recall
AUC
ViT-B-384
384
0.758
0.755
0.853
VIT-L-384
384
0.752
0.753
0.851
T2T- VIT-
224
0.805
0.803
0.892
T2T-ViT-24
224
0.808
0.806
0.896
DeiT-S-224
224
0.788
0.795
0.873
DeiT-B 224
224
0.806
0,805
0.895
DeiT-B-384
384
0.809
0.808
0.897
Swin-T- T-224
224
0.805
0.803
0.894
Swin-S-224
224
0.813
0.811
0.905
Swin-B-224
224
0.822
0.823
0.915
Swin-B B-384
384
0.824
0.826
0.919
Effect of batch size. Batch size is always a significant hyper-panee parameter in the contrastive learning literature. To explore how the batch size affects the final performance of our multi-moda self- supervised pre-training, we vary the batch size from 16, 32, 64, 128, 256, and 512. The experi- mental results reported in Table X We observe that with the increase of the batch size, GeneBERT achieves better performance due to the effectiveness of more negative samples in the sequence-region matching loss. This demonstrates the importance of a larger batch size and more negative samples on aaage genome data pre-training, which complies with the conclusion of vision tasks (Chen al. 2020b c
He et al., 2020 Chen al. 2020dh.
Table Ablation study on the paramters, batch size and X.
Batch Size
Precision
Recall
AUC
Precision
Recall
AUC
16
0.642
0.651
0.733
0.01
0.633
0.621
0.683
32
0.751
0.752
0.853
0.05
0.703
0.702
0.772
64
0.783
0.792
0,871
0.1
0,781
0.762
0,851
128
0,805
0,803
0,894
0.5
0.805
0.803
0.894
256
0,812
0.813
0.906
1.0
0.802
0.801
0.892
512
0.815
0.817
0.908
5.0
0.785
0.782
0.852
Effect of A. Furthermore, explore the effect of À the final performance of our GeneBERT. Specifically, we vary À from 0.01, 0.05, 0.1, 0.5, 1.0 and 5.0, and report the comparison results in Table g Our GeneBERT achieves stable performance when À is set 0.5 and With the decrease of X, the performance of our approach deteriorates in terms of all metrics, which validates the importance of our sequence-region loss in our GeneBERT. To receive the stable performance of multi-modal self-supervised pre-training, we set À = 0.5 in all experiments.
6 CONCLUSION
In this work, we present the GeneBERT, multi-modal self-supervised framework for large-scale genome data pre-training. Specifically, we leverage sequence pre-training, region pre-training, and sequence-region matching together to improve the generalizability our model across different cell types for regulatory genome modeling. Extensive experiments on four main regulatory downstream tasks demonstrate the effectiveness of our GeneBERT, Besides, comprehensive ablation studies have been conducted show the important role of multi-modal modeling.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Babak Alipanahi, Andrew Delong, Matthew Weirauch, and Brendan J Frey. Predicting the se- quence specificities of dna-and ma-binding proteins by deep learning. Nature biotechnology, 33 (8):831-838, 2015.
Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R. Ledsam, Agnieszka Grabska-Barwinske Kyle R. Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R. Kelley. Effective gene expression prediction from sequence by integrating long-ram ong-range interactions. Nature Methods, 18: 1196-1203, 2021.
Rumpu Chen, Le Yang, Steve Goodison, and Yijun Sun. Deep-learning approach to identifying can- cer subtypes using high-dimensional genomic data. Bioinformatics, 36(5); 10 2020a. ISSN 1367-4803, doi: 10. 00:10//bioinfommaticc////// /
bioinformaticas Dtttmaatiic//t225555
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning (ICML), 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020c.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint (rxXi::000304229,, 2020d.
Yifei Chen, Yi Li, Rajiv Narayan, Aravind Subramanian, and Xiaohui Xie. Gene expression in- ference with deep learning. Bioinformatics, 32(12))1832-1833 02 2016. ISSN 1367-4803, doi: 10. 10.1099/bioonfommaiiic//t/ow/ URL httpt:tts///oo.orrrrer http://.....r..oooo.oooo/oooooo. btw074
ENCODE Project Consortium et al. An integrated encyclopedia of dna elements in the human genome. Nature, 489(7414)557, 2012.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018.
Silvia Domeke, Andrew J. Hill, Riza M. Daza, Junyue Cao, Diana R. O'Day, Hannah A. Pliner, Kim- berly A. Aldinger, Dmitry Pokholok, Fan Zhang, Jennifer H. Milbank, Michael A. Zager, Ian A. Glass, Frank J. Steemers, Dan Doherty, Cole Trapnell, Darren A. Cusanovich, and Jay Shendure. November
A human cell atlas fetal chromatin accessibility. Science, 37066518)cebba' 2020. doi: 10.1126/scienee..be abb711.. httl::ttt////ett science. orq/ lookup /doi/ 10.1126/ 2/26//cieee wwciennce.... aecceenceabbb7612 Science.
Publisher: American Association for the Advancement of
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. Proceedings of International Conference on Learning Representations (ICLR), 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729 9938, 2020.
Chenyang Hong and Kevin Y Yip. Flexible k-mers with variable-length indels for identifying bind- ing sequences of protein dimers. Briefings in Bioinformatics, 117777777717777 2020.
Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng, Zongzheng Xi, Yueqian Yang, Anwen Hu, Jimming Zhao, Ruichen Li, Yida Zhao, Liang Zhang, Yuqing Song, Xin Hong, Wanqing Cui, Danyang Hou, Yingyan Li, Junyi Li, Peiyu Liu, Zheng Gong, Chuhao Jin, Yuchong Sun, Shizhe Chen, Zhiwu Lu. Zhicheng Dou, Qin Jin, Yanyan Lan, Wayne Xin Zhao, Ruihua Song, and Ji-Rong Wen. WenLan: Bridging vision and language by large-scale molli pre-training. arXiv preprint arXiv:2 103. 103.06561, 2021.
10

---

Under review as a conference paper at ICLR 2022
Kishore Jaganathan, Sofia Kyriazopoulou Panagiotopoulon, Jeremy F. McRae, Siavash Fazel Dar- bandi, David Knowles, Yang Li, Jack A. Kosmicki, Juan Arbelaez, Wenwu Cui, Grace B. Schwartz, Eric D. Chow, Efstathios Kanterakis, Hong Gao, Amirali Kia, Serafim Batzoglou, Stephan Sanders, and Kyle Kai-How Farh. Predicting Splicing from Primary Sequence with Deep Learning. Cell, (76(3):535-548.e2) January 2019. ISSN 00928674. doi: 10. 10.1006/j.ccll1200000 12.00........ 12.015. pii/ 051/S009228674180
URL
www.whww..eeeei........... com wttp://ieeeeeeeeee///eeee
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, (1115):21222122 2021.
David R Kelley, Jasper Snoek, and John Rinn. Basset: Learning the regulatory code of the accessible genome with deep convolutional neural networks. bioRxiv, 2016. doi: 10.1101/2016.0..56 028399.
Aziz Khan, Oriol Fornes, Araud Stigliani, Marius Gheorghe, Jaime A Castro-Mondrragn, Robin van der Lee, Adrien Bessy, Jeanne Chêneby, Shubhada R Kulkarni, Ge Tan, Damir Baranasic, David J Arenillas, Albin Sandelin, Klaas Vandepoele, Boris Lenhard, Benoît Ballester, Wyeth W Wasserman, François Parey, and Anthony Mathelier. JASPAR 2018: update of the open-access database of transcription factor binding profiles and its web framework. Nucleic Acids Research, doi: 10.1093/nar/gkx1 188.
46(D1):D1284, January 2018 ISSN 0305-1048.
URL https:
//doi. 1093 //aar//ara /ttp////g/x11/1./////
Yunhai Li, Kee Khoon Lee, Sean Walsh, Caroline Smith, Sophie Hadingham, Karim Sorefan, Gavin C. Cawley, and Michael W. Bevan. Establishing glucose- and aba-regulated transcrip- tion networks in arabidopsis by microarray analysis and promoter classification using a relevance vector machine. Genome research, 163:414 14-111427777 2006.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
Mhaned Oubounyt, Zakaria Louadi, Hilal Tayara, and Kil To Chong. Deepromoter: robust promoter predictor using deep learning. Frontiers in genetics, 10:286, 2019.
Yeping Lina Qiu, Hong Zheng, and Olivier Gevaert. A deep learning framework for imputing missing values in genomic data. bioRxiv, 2018. doi: 2..10/22188.990044066666
Alec Radford, Karthik Narasimhan, Tim Salimans, and 1lya Sutskever. Improving language under-standing by generative pre-training. URL http:://s3-uswwest amazonaws. com/openai- assets/research-
understanding paper.pdf. 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2/03.00020, 2021.
Adriana Romero, Pierre Luc Carrier, Akram Erraqabi, Tristan Sylvain, Alex Auvolat, Etienne De- joie, Marc-André Legault, Marie-Pierre Dubé, Julie G. Hussin, and Yoshua Bengio. Diet net- works: Thin parameters for fat genomic. In Proceedings of International Conference on Learning Representations (ICLR), 2017.
Douglas Ross, Uwe Scherf, Michael B Eisen, Charles M Perou, Christian Rees, Paul Spell- man, Vishwanath Iyer, Stefanie S Jeffrey, Matt Van de Rijn, Mark Waltham, Alexander Perga- menschikov, Jeffrey C.F. Lee, Deval Lashkari, Dari Shalon, Timothy G Myers, John N We- instein, David Botstein, and Patrick O Brown. Systematic variation in gene expression pat- terns in human cancer cell lines. Nature Genetics, 24(3). 2000. doi: 10.1038/73432. URL https: www.wwww.. osti wtt.:osst......ooootttttto

---

Under review as a conference paper at ICLR 2022
Mark Schena, Dari Shalon, Ronald W. Davis, and Patrick O. Brown. Quantitative monitoring of gene expression patterns with a complementary dna microarray. Science, 27065235):467-770
1995. doi: 10.1 00.11266sieemc222055555....4.. 10. 10.1126/
stt//sciinccccccc 270. 5235.467
Alexander Stewart, Sridhar Hannenhalli, and Joshua B Plotkin. Why Transcription Factor Bind- ing Sites Are Ten Nucleotides Long. Genetics, 192(3)9973 985, 11 2012. ISSN 1943-2631. doi: 10. 12344/eentiis.112...44..
/RLhttss:://dl http::///ooooog////////o// wtt.///.....///eeeeeee 112.
143300
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223, 2019.
Luis Torada, Lucrezia Lorenzon, Alice Beddis, Ulas Isildak, Linda Pattini, Sara Mathieson, and Matteo Fumagalli. Imagene: convolutional neural network quantify natural selection from genomic data. BMC Bioinformatiis, 20, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. arXiv preprint ar arXiv:2012.12877, 2020.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arX1v:2101.11986, 2021.
12


Reviews:
[{"review_id": "xErcZDaZB6t", "reviewer": "Reviewer_FiEi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper\n* The simple-yet-effective approach for pre-training of genome sequence data works remarkably well for very diverse tasks \n* The wide availability of multi-modal data provides an opportunity for adaptation by the field. \n* Benchmarking and ablation studies are well designed. \n* The paper is well written and makes effort to explain diverse concepts from biology and machine learning to a wide audience\n\nWeaknesses \n* While the formulation is indeed a novel multi-modal construction as claimed by the authors, the generalization beyond genome data is not clear and hence I am not sure if this will be of broad interest to the ICLR attendees. \n* An important feature of gene regulation is that the genomic distance between regulatory elements matter - the fact that 10 consecutive accessible regions are grouped into one sample breaks this relationship and might affect interpretability. \n* The 2D representation is certainly interesting but it is not a true image representation. This might be a case of defining a representation to fit the model but the performance certainly shows there is value to such a representation. \n", "summary_of_the_paper": "The manuscript describes GeneBERT, a self-supervised and multi-modal pre-training approach for genomic data. GeneBERT combines 1D genome sequence data with a 2D representation of regulatory elements in different cell-types in three different pre-training tasks. A large-scale single-cell ATAC dataset is used to identify pseudo-bulk ATAC profiles for 17 cell-types spanning > 1M genomic locations. The sequence of these genomic locations provides the 1D representation of the data which are used in self-supervised training with masked genome modeling and next genome segment prediction loss functions, both inspired by the BERT framework. A 2D representation is derived using accessibility per regulatory element and cell type, and is self-supervised using a infoNCE loss. The pre-training framework is then compared in a variety of biological tasks including promoter prediction, TFBS prediction, splice site prediction. The authors then present ablation studies highlighting the importance of the different components of the loss function.", "summary_of_the_review": "The problem tackled by the manuscript address an important need in repressing genome data using both sequences and additional measurements. The results presented both in performance and diversity of tasks measured is impressive. However, I am not completely sure if the submission will be of broad interest to the ICLR audience. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uLDo1nhfDjL", "reviewer": "Reviewer_omfC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors’ description of the method does not contain sufficient details to understand it. Specifically, the input data and neural network layers that process it are difficult to decipher. For example, why are 10 sequences concatenated? Where does the ATAC-seq data come in? A caption for Figure 1 would be very helpful.\n\nThe authors transform the 1D nucleotide sequence to a 1D sequence of recognized PWM vectors. As I understand it, the sequence is then treated as a 2D object and processed with transformer layers. Isn’t each position in the sequence depth 1? The authors suggest that they used a model pre-trained on ImageNet. How could it be that this pre-trained model has learned anything useful for this very different task?\n\nClassifying promoters from DNA is a very easy task that the field has had high performance on for many years and does not solve a useful problem. If the authors can demonstrate that their model can predict gene expression level, particularly in diverse cell types, it would be far more compelling. In addition, there are strong competing methods based on neural networks to which the authors could compare their different approach.\n\nClassifying TF binding sites is an important task, but the authors do not compare to any of the leading methods, such as DeepBind, DeepSea, Basset, DanQ and many successors. Furthermore, CTCF has a long informative PWM, making it arguably the easiest TF to predict. The authors should show results from more diverse TFs.\n\nThe disease risk estimation task is impossible to understand because there are so few details. What is an example in this task? For example, are genetic variants the examples? Where did the labels as disease-related or not come from? What procedure did you use to include negative examples? How did you choose Hirschprung disease among the many options? Since the model focuses on gene regulation rather than protein coding sequence, how should we think about coding gene mutations in your task?\n\nThe authors observation that their model outperforms SpliceAI using shorter sequences is very surprising. Could the authors include more details about the full model that they used for this task, and how their pre-trained model is used within it? When the authors write “dilated CNN” in their Table 4, are they referring to SpliceAI itself or to the author’s own implementation of a dilated CNN?\n\nNew methods in the space are typically compared to competing alternatives, but also interpreted. Can the authors interpret what their model has learned that alternative approaches have not? Referring to Figure 2, the authors state, “We can observe that our GeneBERT pre-trained representations form more separated clusters that are distributed more uniformly on the space in terms of all cell types when compared to the pre-trained representations generated by the uni-modal model. Furthermore, the representations pre-trained by our multi-modal self-supervised method have much smaller distances inside each cluster of cell type.” This doesn’t appear to be true to my eye. The authors should compute summary statistics to make this claim convincing. However, this visualization also doesn't really help interpret the model; mutagenesis or saliency methods applied to DNA sequences would be better.", "summary_of_the_paper": "The authors describe a pipeline to perform self-supervised learning on genome sequences, guided by accessible chromatin peaks. Their learning procedure is inspired by the NLP method BERT and uses several tasks from that work and its successors. After pre-training transformer layers, they fine-tune on several regulatory sequence classification tasks and demonstrate high performance.", "summary_of_the_review": "The authors explore a new method for self-supervised pre-training before tackling several regulatory sequence analysis tasks, but are not able to deliver a clear method description or compelling empirical results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "fR62tYeGN8J", "reviewer": "Reviewer_9J4F", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The idea of using 2d information of (transcription factor by regions) in transformer models to learn meaningful genomic embeddings is cool. Using such information is a win over DNABERT which is not so informative. \n\n- The authors have used the recent advanced models from NLP and have employed three losses whose optimization all together leads to significant improvement over the DNABERT. \n\n- The authors has thoroughly ablated the model and analyzed the impact of each component of the model as well as hyper parameters and batch size. It is convincing that it is all three loss together that gives rise to such improvement. \n\nWeaknesses:\n\n- Some of the claims in the paper regarding the biological importance are not so true. For example: \"As the promoters play an important role in gene regulation, using machine learning methods to predict promoter sites accurately is one of the most popular problems in bioinformatics.\" Finding the promoters are not a challenge. The genes and their transcription start sites (TSS) are nearly annotated in the genome and it is known that the promoters are next to the TSS's. Instead, the most challenging part is to find functional enhancers in the genome which is a fundamental question in biology. The majority of enhancers are distal regulatory elements, meaning that they are very far from their target genes (could be 1Mb away). So, one interesting experiment for GeneBERT is to check and see if it can find enhancers or not (functional enhancers can be validated by CRISPRi perturbation data that are available in some cell lines such as K562).\n\n- For transcription factor binding site (TFBS), the paper only shows the results for CTCF in Table 2. I was expecting to see the results on many TFs. Why did the authors not show the other TF's results? The CTCF motif is not so complicated and maybe that is the reason why we see such great performance. I think the authors should provide the performance on all the TFs and discuss which ones the model finds well and which ones not and why.\n\n- On the effect of 2D modality on pre-training, the t-SNE plots in Figure 2 are not so informative. They look similar and it is hard to get any conclusion from that. The authors can provide a quantitative metric to show how 2D modality helps the embeddings. \n\n- It is not clear in Table 6, what are the classification tasks. The authors only mentioned they chose different proteins from various cells or tissues in the human body. But it should be stated clearly what TFs are used in each cell.\n\n- Typo: several times in the paper including the abstract, \"transaction factor\" has been used instead of \"transcription factor\". \n", "summary_of_the_paper": "This work proposes an approach, called GeneBERT, for pre-training genome data in a multi-modal and self-supervised manner. They take the 1d sequence of genomic data and a 2d matrix of (transcription factors × regions) as the input and optimize three pre-training tasks to improve the robustness and generalizability the model. Specifically, they introduce two main objectives for sequence pre-training, Masked Genome Modeling (MGM) and Next Genome-Segment Prediction (NGSP). GeneBERT consists of three main components: sequence pre-training, region pre-training, and sequence-region matching. The main contribution of the paper is that they use transcription factor information in genomic regions which makes the model more generalizable to other cell types than the previous DNABERT that only uses the sequence information. ", "summary_of_the_review": "The paper tries to address an important problem in genomics: how to effectively embed DNA sequence for downstream tasks. They have used some recent transformer architectures and loss function to effective do this. Their main contribution, which is an important one, is to use the transcription factor information in the accessible genomic regions which are cell type specific. Therefore, the model would be more generalizable than the previous methods like DNABERT. The ablation study of the model and hyperparameters are comprehensive. However, some biological experiments and their importance are overstated. I provided some comments for the authors to enrich the experiments. I think the paper is interesting and if some comprehensive and important experiments are coupled with it to show that it can really find interesting biological events, it would be a great paper. As such, I choose the score 6 for the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-O3t3kLw4X-", "reviewer": "Reviewer_9xx4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n•\tA novelty is that GeneBERT combines the losses for representation learning and alignment and simultaneously estimate the model parameters.\n•\tLarge-scale multi-modal data integration\n\nWeaknesses\n•\tThe authors need to elaborate on the details about datasets and data processing such as how to select TFs and binding motifs to construct 2D modality, how to merge 17 cell types, which cell types in fetal data were used. Fig 2 only shows cell cluster numbers, for instance.\n•\tscATAC-seq is sparse and noisy. Binarizing data by non-zero can also introduce many false positive open regions.\n•\tMany regulatory regions for various tissue types and cell types are yet open in fetal stage, so they are likely missed.\n•\tCTCF is a general TF with strong binding activities. The authors should predict other TFs, especially cell-type-specific TFs. Also, ENCODE TF ChIP-seq data were for cell lines, which may not match the cell types that the authors used.\n•\tThe authors also should try more disease types for generality. ", "summary_of_the_paper": "In this paper, the authors developed a transformer-based model, GeneBERT to align DNA sequences with regulatory elements. In particular, GeneBERT first applies transformers to learn representations of sequencing data and regulatory regions (e.g., open chromatin), and then aligns the representations of two modalities for identifying region-aligned sequences. The authors applied GeneBERT to recent scATAC-seq data in fetal and use aligned sequences to predict promoters, CTCF binding sites, a disease type and RNA splicing sites.", "summary_of_the_review": "The paper was organized logically. However, the applications were not well presented. Many details are unclear and missing, especially on datasets, data processing, feature selection, cell types. Also, it seems that the authors don’t fully understand cell-type gene regulation for misusing different datasets (train orange to predict apple).\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: While several reviewers acknowledge that the paper contains potentially useful ideas related to multi-modal self-training applied to genomic data, they also point out a number of weaknesses and room for improvement that the discussion with authors did not fully address. This includes in particular the need to better explain the details of what is done in the paper; the choice of experiments which is not relevant (eg, predicting promoter regions) or complete (eg, showing results on only one transcription factor); the lack of comparison with existing methods, etc... We therefore consider that the paper is not ready for publication in its current form, but hope that the reviews will help the authors work on a revision addressing the issues.
Decision: Reject