Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: In this paper, the authors developed a transformer-based model, GeneBERT to align DNA sequences with regulatory elements. In particular, GeneBERT first applies transformers to learn representations of sequencing data and regulatory regions (e.g., open chromatin), and then aligns the representations of two modalities for identifying region-aligned sequences. The authors applied GeneBERT to recent scATAC-seq data in fetal and use aligned sequences to predict promoters, CTCF binding sites, a disease type and RNA splicing sites.
main_review: Strengths
•	A novelty is that GeneBERT combines the losses for representation learning and alignment and simultaneously estimate the model parameters.
•	Large-scale multi-modal data integration

Weaknesses
•	The authors need to elaborate on the details about datasets and data processing such as how to select TFs and binding motifs to construct 2D modality, how to merge 17 cell types, which cell types in fetal data were used. Fig 2 only shows cell cluster numbers, for instance.
•	scATAC-seq is sparse and noisy. Binarizing data by non-zero can also introduce many false positive open regions.
•	Many regulatory regions for various tissue types and cell types are yet open in fetal stage, so they are likely missed.
•	CTCF is a general TF with strong binding activities. The authors should predict other TFs, especially cell-type-specific TFs. Also, ENCODE TF ChIP-seq data were for cell lines, which may not match the cell types that the authors used.
•	The authors also should try more disease types for generality. 
summary_of_the_review: The paper was organized logically. However, the applications were not well presented. Many details are unclear and missing, especially on datasets, data processing, feature selection, cell types. Also, it seems that the authors don’t fully understand cell-type gene regulation for misusing different datasets (train orange to predict apple).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript describes GeneBERT, a self-supervised and multi-modal pre-training approach for genomic data. GeneBERT combines 1D genome sequence data with a 2D representation of regulatory elements in different cell-types in three different pre-training tasks. A large-scale single-cell ATAC dataset is used to identify pseudo-bulk ATAC profiles for 17 cell-types spanning > 1M genomic locations. The sequence of these genomic locations provides the 1D representation of the data which are used in self-supervised training with masked genome modeling and next genome segment prediction loss functions, both inspired by the BERT framework. A 2D representation is derived using accessibility per regulatory element and cell type, and is self-supervised using a infoNCE loss. The pre-training framework is then compared in a variety of biological tasks including promoter prediction, TFBS prediction, splice site prediction. The authors then present ablation studies highlighting the importance of the different components of the loss function.
main_review: Strengths of the paper
* The simple-yet-effective approach for pre-training of genome sequence data works remarkably well for very diverse tasks 
* The wide availability of multi-modal data provides an opportunity for adaptation by the field. 
* Benchmarking and ablation studies are well designed. 
* The paper is well written and makes effort to explain diverse concepts from biology and machine learning to a wide audience

Weaknesses 
* While the formulation is indeed a novel multi-modal construction as claimed by the authors, the generalization beyond genome data is not clear and hence I am not sure if this will be of broad interest to the ICLR attendees. 
* An important feature of gene regulation is that the genomic distance between regulatory elements matter - the fact that 10 consecutive accessible regions are grouped into one sample breaks this relationship and might affect interpretability. 
* The 2D representation is certainly interesting but it is not a true image representation. This might be a case of defining a representation to fit the model but the performance certainly shows there is value to such a representation. 

summary_of_the_review: The problem tackled by the manuscript address an important need in repressing genome data using both sequences and additional measurements. The results presented both in performance and diversity of tasks measured is impressive. However, I am not completely sure if the submission will be of broad interest to the ICLR audience.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors describe a pipeline to perform self-supervised learning on genome sequences, guided by accessible chromatin peaks. Their learning procedure is inspired by the NLP method BERT and uses several tasks from that work and its successors. After pre-training transformer layers, they fine-tune on several regulatory sequence classification tasks and demonstrate high performance.
main_review: The authors’ description of the method does not contain sufficient details to understand it. Specifically, the input data and neural network layers that process it are difficult to decipher. For example, why are 10 sequences concatenated? Where does the ATAC-seq data come in? A caption for Figure 1 would be very helpful.

The authors transform the 1D nucleotide sequence to a 1D sequence of recognized PWM vectors. As I understand it, the sequence is then treated as a 2D object and processed with transformer layers. Isn’t each position in the sequence depth 1? The authors suggest that they used a model pre-trained on ImageNet. How could it be that this pre-trained model has learned anything useful for this very different task?

Classifying promoters from DNA is a very easy task that the field has had high performance on for many years and does not solve a useful problem. If the authors can demonstrate that their model can predict gene expression level, particularly in diverse cell types, it would be far more compelling. In addition, there are strong competing methods based on neural networks to which the authors could compare their different approach.

Classifying TF binding sites is an important task, but the authors do not compare to any of the leading methods, such as DeepBind, DeepSea, Basset, DanQ and many successors. Furthermore, CTCF has a long informative PWM, making it arguably the easiest TF to predict. The authors should show results from more diverse TFs.

The disease risk estimation task is impossible to understand because there are so few details. What is an example in this task? For example, are genetic variants the examples? Where did the labels as disease-related or not come from? What procedure did you use to include negative examples? How did you choose Hirschprung disease among the many options? Since the model focuses on gene regulation rather than protein coding sequence, how should we think about coding gene mutations in your task?

The authors observation that their model outperforms SpliceAI using shorter sequences is very surprising. Could the authors include more details about the full model that they used for this task, and how their pre-trained model is used within it? When the authors write “dilated CNN” in their Table 4, are they referring to SpliceAI itself or to the author’s own implementation of a dilated CNN?

New methods in the space are typically compared to competing alternatives, but also interpreted. Can the authors interpret what their model has learned that alternative approaches have not? Referring to Figure 2, the authors state, “We can observe that our GeneBERT pre-trained representations form more separated clusters that are distributed more uniformly on the space in terms of all cell types when compared to the pre-trained representations generated by the uni-modal model. Furthermore, the representations pre-trained by our multi-modal self-supervised method have much smaller distances inside each cluster of cell type.” This doesn’t appear to be true to my eye. The authors should compute summary statistics to make this claim convincing. However, this visualization also doesn't really help interpret the model; mutagenesis or saliency methods applied to DNA sequences would be better.
summary_of_the_review: The authors explore a new method for self-supervised pre-training before tackling several regulatory sequence analysis tasks, but are not able to deliver a clear method description or compelling empirical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes an approach, called GeneBERT, for pre-training genome data in a multi-modal and self-supervised manner. They take the 1d sequence of genomic data and a 2d matrix of (transcription factors × regions) as the input and optimize three pre-training tasks to improve the robustness and generalizability the model. Specifically, they introduce two main objectives for sequence pre-training, Masked Genome Modeling (MGM) and Next Genome-Segment Prediction (NGSP). GeneBERT consists of three main components: sequence pre-training, region pre-training, and sequence-region matching. The main contribution of the paper is that they use transcription factor information in genomic regions which makes the model more generalizable to other cell types than the previous DNABERT that only uses the sequence information. 
main_review: Strengths:

- The idea of using 2d information of (transcription factor by regions) in transformer models to learn meaningful genomic embeddings is cool. Using such information is a win over DNABERT which is not so informative. 

- The authors have used the recent advanced models from NLP and have employed three losses whose optimization all together leads to significant improvement over the DNABERT. 

- The authors has thoroughly ablated the model and analyzed the impact of each component of the model as well as hyper parameters and batch size. It is convincing that it is all three loss together that gives rise to such improvement. 

Weaknesses:

- Some of the claims in the paper regarding the biological importance are not so true. For example: "As the promoters play an important role in gene regulation, using machine learning methods to predict promoter sites accurately is one of the most popular problems in bioinformatics." Finding the promoters are not a challenge. The genes and their transcription start sites (TSS) are nearly annotated in the genome and it is known that the promoters are next to the TSS's. Instead, the most challenging part is to find functional enhancers in the genome which is a fundamental question in biology. The majority of enhancers are distal regulatory elements, meaning that they are very far from their target genes (could be 1Mb away). So, one interesting experiment for GeneBERT is to check and see if it can find enhancers or not (functional enhancers can be validated by CRISPRi perturbation data that are available in some cell lines such as K562).

- For transcription factor binding site (TFBS), the paper only shows the results for CTCF in Table 2. I was expecting to see the results on many TFs. Why did the authors not show the other TF's results? The CTCF motif is not so complicated and maybe that is the reason why we see such great performance. I think the authors should provide the performance on all the TFs and discuss which ones the model finds well and which ones not and why.

- On the effect of 2D modality on pre-training, the t-SNE plots in Figure 2 are not so informative. They look similar and it is hard to get any conclusion from that. The authors can provide a quantitative metric to show how 2D modality helps the embeddings. 

- It is not clear in Table 6, what are the classification tasks. The authors only mentioned they chose different proteins from various cells or tissues in the human body. But it should be stated clearly what TFs are used in each cell.

- Typo: several times in the paper including the abstract, "transaction factor" has been used instead of "transcription factor". 

summary_of_the_review: The paper tries to address an important problem in genomics: how to effectively embed DNA sequence for downstream tasks. They have used some recent transformer architectures and loss function to effective do this. Their main contribution, which is an important one, is to use the transcription factor information in the accessible genomic regions which are cell type specific. Therefore, the model would be more generalizable than the previous methods like DNABERT. The ablation study of the model and hyperparameters are comprehensive. However, some biological experiments and their importance are overstated. I provided some comments for the authors to enrich the experiments. I think the paper is interesting and if some comprehensive and important experiments are coupled with it to show that it can really find interesting biological events, it would be a great paper. As such, I choose the score 6 for the paper.
review_title: null

Meta-review: While several reviewers acknowledge that the paper contains potentially useful ideas related to multi-modal self-training applied to genomic data, they also point out a number of weaknesses and room for improvement that the discussion with authors did not fully address. This includes in particular the need to better explain the details of what is done in the paper; the choice of experiments which is not relevant (eg, predicting promoter regions) or complete (eg, showing results on only one transcription factor); the lack of comparison with existing methods, etc... We therefore consider that the paper is not ready for publication in its current form, but hope that the reviews will help the authors work on a revision addressing the issues.
Decision: Reject