Hyperbolic Genome Embeddings

Authors: Raiyan R. Khan, Philippe Chlenski, Itsik Pe'er
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: genomics, representation learning, hyperbolic geometry

Abstract:
Current approaches to genomic sequence modeling often struggle to align the inductive biases of machine learning models with the evolutionarily-informed structure of biological systems. To this end, we formulate a novel application of hyperbolic CNNs that exploits this structure, enabling more expressive DNA sequence representations. Our strategy circumvents the need for explicit phylogenetic mapping while discerning key properties of sequences pertaining to core functional and regulatory behavior. Across 37 out of 42 genome interpretation benchmark datasets, our hyperbolic models outperform their Euclidean equivalents. Notably, our approach even surpasses state-of-the-art performance on seven GUE benchmark datasets, consistently outperforming many DNA language models while using orders of magnitude fewer parameters and avoiding pretraining. Our results include a novel set of benchmark datasets---the Transposable Elements Benchmark---which explores a major but understudied component of the genome with deep evolutionary significance. We further motivate our work by exploring how our hyperbolic models recognize genomic signal under various data-generating conditions and by constructing an empirical method for interpreting the hyperbolicity of dataset embeddings. Throughout these assessments, we find persistent evidence highlighting the potential of our hyperbolic framework as a robust paradigm for genome representation learning. Our code and benchmark datasets are available at https://github.com/rrkhan/HGE.

Decision: accept (poster)
Reviewer scores: 5, 5, 8, 8

Reviews and discussion:
<thread>
creation date:
1730092662476

thread_type:
official_review

<thread_object>
creator: reviewer_knja

<content>
summary:
The paper proposes the application of fully hyperbolic CNNs (HCNNs) for representation of biological data, in specific DNA sequences. The authors note that genomic data is often generated by evolutionary processes, and result in phylogenetic hierarchies that may be better represented in hyperbolic space as opposed to Euclidean space. The authors apply HCNNs on several genomic element classification benchmarks, and additionally propose a new dataset, the Transposable Elements Benchmark, in which models are tasked with classifying the type of transposable element that exists in a context window. The authors find that hyperbolic CNNs outperform their Euclidean counterparts in most of the benchmark tasks, and can occasionally outperform genomic foundation model performance. They also perform an empirical analysis on the hyperbolicity of the selected datasets.

NOTE: Score raised from 5 -> 8 in response to rebuttal.

strengths:
- Paper presents a creative application of HCNNs to the field of genomics, and serves as a useful preliminary paper to explore the benefits of representing genomic data in hyperbolic space. Given the current community interest in representation learning for genomics, this new approach could be of high interest and significance. Authors also provide new dataset for benchmarking a relevant task.

- The authors make an effort to construct high quality baselines for their experiments. In addition to demonstrating the benefits of an HCNN vs a Euclidean CNN, the method is also benchmarked against genomic foundation models. The authors also take care during their empirical evaluation of dataset hyperbolicity to account for dataset dimensionality by calibrating against simulated data.

- Experimental results indicate the hyperbolic representation offers benefit on some classes of tasks. This shows that the HCNN or other hyperbolic representations could have promise in genomic representation learning. Visually, it appears that the tasks where HCNNs offer improvement also correlate with the underlying data hyperbolicity, supporting the core hypothesis of the work.

- Experiments are well executed, with replication across random seeds and confidence intervals + statistical significance reported. Experimental setup is well described, and code is available, indicating that results are likely to be reproducible.

weaknesses:
- I believe the analysis could be improved by investigating exactly how the HCNN represents sequences related through evolution. A common intuition I have seen for hyperbolic embeddings in computer vision is that datum closer to the origin tend to represent high level features and vice versa. It would be convincing if some analogy could be made for the genomic sequences through the experiments. For example, while Figure 2 demonstrates that hierarchical structure exists in the HCNN representation, it would be interesting to see whether the sequences placed at the "top" of the learned hierarchy reconcile with biological intuition.

- It's surprising that the HCNN representation is not consistently better for datasets that seem highly related to evolutionary processes. For example, it is strange that the covid variant classification is so poor given that the observed variation is likely driven by selection, and that the HCNN-M is outperformed by the Euclidean CNN on the synthetic dataset which is generated using a evolutionary simulator. Similarly, it is interesting that the HCNN does not significantly outperform the Euclidean models on promoter classification tasks despite the results in the empirical study in Figure 3. I would appreciate any deeper insights from the authors on these issues.

- Tying into the last point, given the lack of consistent improvement and correlation in improvement with seemingly evolutionary data generating processes, its unclear under what circumstance the HCNN should be used compared to existing approaches.

- To my knowledge, there is very limited methodological novelty in the paper, although I believe the current paper may be sufficient for an applied paper.

questions:
- I would appreciate if the authors could address the above points. 

- Could selection of GUE as a benchmark cause underperformance? From my understanding, the benchmark does not control for homology in the train / test splits. Conceivably, the benefit of the HCNN would be improved generalization in evolutionary / phylogenetic contexts, and potential data leakage in the GUE benchmark could mask the performance benefit in this aspect. 

- I would appreciate if the colour of the DNABERT-2 distribution in supplemental figure 12 could be changed to be more distinct for folks with colour vision deficiency!
</content>
</thread_object>

</thread>
<thread>
creation date:
1730314405333

thread_type:
official_review

<thread_object>
creator: reviewer_h56n

<content>
summary:
The authors propose an architecture that is better suited for genomic property prediction from DNA sequences. They implement hyperbolic convolutional neural networks (HCNN) and demonstrate improvements across a number of tasks. They compare against a matched CNN in Euclidean space. They motivate the hyperbolic embeddings due to the tree like underlying generative process of genomic sequences. In addition the authors propose an additional Transposable Elements Benchmark. In addition they use a measure of $\delta$ hyperbolicity or a measure of tree-likeness in genetic data.

strengths:
- The authors analyze an existing problem through a novel lens suggesting a new possible inductive bias to an important problem. 
- The authors perform a comprehensive evaluation across a wide variety of tasks
- Authors propose a new dataset which can evaluate ability of models to classify transposable elements. This class of sequences are widely abundant however it is unclear as to their biological significance compared to sequences encoding mature RNAs.

weaknesses:
- This work can benefit from additional  justification for why the hierarchical tree structure is a good inductive bias for genomic data. Although the original data generating process can take on a tree structure is it necessarily the case that we would want to add this as an inductive bias to the model?
- Is there an experiment that the authors can do that would confirm the validity of this inductive bias? Do the authors find it strange that the synthetic task that was explicitly designed to test performance of the inductive bias the baseline falls within the confidence interval of the best performing model? 
- In genomic sequence modeling it is common to do homology splitting where sequences related through their homologous relationships are fully left out and the model's capacity is assessed on whether it's able to generalize to unseen homology branches. Do the authors expect that this inductive bias helps or hurts the performance of the model in this domain? Can they perform an experiment validating this?

questions:
- If the hierarchical relationship is between sequences then why do the authors note that a single point would be an individual nucleotide and a sequence would be a set? Isn't a sequence then exponential in the number of individual nucleotides thus resulting in an exponential growth of possibility space even without the hyperbolic space inductive bias?
- There is no mention regarding the parameter count of the eucledian CNN. I assume it is the same as the hyperbolic version but would be good to get a confirmation. Can the authors also include FLOP count per forward / backward pass for each of the models?
- The work can benefit from an expanded more thorough explanation in the appendix of mapping between tangent space and Lorentz manifold especially when describing the hypebolic layers. 
- Can the authors mention how the confidence intervals were generated in table 1. 
- Do the authors find it strange that for covid variant classification, arguably in a very clear cut case where there is abundant information regarding evolution of the virus the hyperbolic architecture doesn't perform so well? Do the authors have a hypothesis for why that is the case?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730599609474

thread_type:
official_review

<thread_object>
creator: reviewer_q1g1

<content>
summary:
This paper adopts the hyperbolic CNNs to genome representation learning. Empirical results suggest the advantage of hyperbolic CNN to its Euclidean equivalents in genome learning. Moreover, the hyperbolic CNNs for genome sequences can outperform existing genome foundation models on 7 out of 27 datasets on the GUE benchmark.

strengths:
1. The paper is well-written, with good backgrounds and clear visualization.
2. The introduced Transposable Elements Benchmark is valuable to the research community.
3. The hyperbolic CNN demonstrates consistent improvements over the Euclidean CNN. It could serve as a baseline choice for genome classification models.

weaknesses:
1. Comparing the number of parameters between CNN-based models and Transformer-based models is not convincing. Due to the underlying difference between these two model architectures, fewer parameters do not guarantee efficiency. A more suitable comparison would be the time and memory usage of training the models on the same dataset.
2. There is a lack of baselines. As a CNN-based model, the model should also be compared with HyenaDNA and Caduceus.
3. The empirical study is not well-presented. In the main text, this paper dedicates most of the experimental sections to the comparison of hyperbolic and Euclidean CNN, which is less relevant to the genomics audience. On the one hand, this conclusion has already been drawn in the HCNN paper [1]. On the other hand, the Euclidean CNN is not the SOTA in genome classification. Compared with existing genomics foundation models, as shown in Table 4, it is more important, in my opinion.
4. The contribution is limited from both methodology and application perspectives.
    - Methodologically, HCNN is introduced in ICLR 2024, and the advantage of hyperbolic over Euclidean equivalents is also shown in that paper. This work adopts the same model (with tiny modifications on two versions ) and validates the same observations in genome embedding. 
    - From an application perspective, this work's contribution is limited. Though the authors do not report the wall-clock time / memory usage of the training the model, given that GFMs like NT and DNABERT only require 3-5 epochs to converge and this model is trained for 100 epochs, this model is likely to be more costly than existing ones. Moreover, it does not deliver better performances than GFMs. From an application perspective, users do not care much about whether a model is pre-trained / randomly initialized. The cost of fitting a model to a specific dataset is more meaningful.

[1] Bdeir, Ahmad, Kristian Schwethelm, and Niels Landwehr. "Fully Hyperbolic Convolutional Neural Networks for Computer Vision." arXiv preprint arXiv:2303.15919 (2023).

questions:
1. Can you compare the cost of different models by the wall-clock time of fitting it on the same dataset with the same devices?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666143583

thread_type:
official_review

<thread_object>
creator: reviewer_zknz

<content>
summary:
The authors introduce a hyperbolic CNN for genomic sequences. They compare their model to a standard CNN on both synthetic and real benchmarks for genomic sequence classification (GUE, GE, and their transposable element benchmark). The authors find that the HCNN model outperforms the standard CNN on most tasks.

strengths:
Hyperbolic architectures is a promising avenue for training models on genomic sequences, as they are inherently related through phylogenetic tree structures. This paper provides interesting initial results in this direction.

weaknesses:
- How does the CNN/HCNN performance scale with number of parameters? Figure 5 seems to indicate that performance gets worse with increasing hidden dim? Additionally figure 5 should include error bars. Also the cumulative improvement y-axis is unclear. It should just be the average MCC value. 

- Figure 1 shows that sequences which obey a phylogenetic tree structure are embedded into a hyperbolic space which learns this structure. However I do not see any results which indicate that the phylogenetic structure is being learned. The authors could provide an experiment to validate this by looking at correlation between embedding distances and phylogenetic distances (for example EDS task from https://www.biorxiv.org/content/10.1101/2024.07.10.602933v1)

- The results would be stronger if the authors provided a baseline column for a standard transformer architecture in Table 1, as CNNs are used less frequently for genomic tasks. Additionally, the DNABERT/NT performance should be included in Table 1, not in supplementary Table 4, and would be much more clear by averaging a single value for each task. 

- Figure 2 shows the decision boundary for a 2D model, and shows better separation for the hyperbolic embeddings. However, it is not clear that this result extends to higher dimensions. Instead, the figure would be stronger by using a PCA/UMAP plot with a higher dimensionality model.

questions:
- Is the CNN/HCNN model pretrained with a MLM task similar to DNABERT models? If so, on what dataset? If not, why?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733279126843

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for an engaging and productive discussion. While we’ve addressed all of the reviewers’ main concerns, reviewers Q1G1 and Zknz raised an additional point about pretraining a genome foundation model (GFM) to compare its performance against existing GFMs.  Although a pretrained GFM presents an interesting line of research as a follow up to this paper, our findings comprise a necessary standalone foundational work for the development of hyperbolic GFMs. Still, recognizing the importance of scaling for future GFMs, we used the rebuttal period to scale HCNN-S and HCNN-M to a parameter count more consistent with GFMs (increasing channel dimension to 256 and kernel size to 15, without hyperparameter tuning due to time constraints). While these larger HCNNs are still smaller than their competitors, they reach SOTA performance on more GUE datasets than DNABERT-2 and NT-2500M-multi, the two GFMs that attain SOTA performance on GUE. These findings, presented below, demonstrate the capabilities of the hyperbolic framework.  

| Model Class           |SOTA performance on GUE (number of datasets) |  
|-------------------|--------------------:|
| NT-2500M-multi |  5 |       
| DNABERT-2 | 11 | 
| HCNNs (Large) |     **12** | 


Full GUE results are shown here:

|          | NT-2500M-multi | DNABERT-2 | DNABERT-2-PT | HCNN-S (Large) | HCNN-M (Large) |
|-------------------|----------------:|----------:|-------------:|---------------:|---------------:|
| Parameters            |           2.5B |       117M |         117M |            **86M** |            **86M** |
| H3                |          78.77 |      78.27 |        **80.17** |          72.17 |          72.21 |
| H3K14ac           |          56.20 |      52.57 |        57.42 |          **70.56** |          69.87 |
| H3K36me3          |          61.99 |      56.88 |        61.90 |          **68.06** |          67.92 |
| H3K4me1           |          55.30 |      50.52 |        53.00 |          53.33 |          **55.45** |
| H3K4me2           |          36.49 |      31.13 |        39.89 |          **54.67** |          52.50 |
| H3K4me3           |          40.34 |      36.27 |        41.20 |          **67.25** |          64.61 |
| H3K79me3          |          64.70 |      67.39 |        65.46 |          70.49 |          **70.65** |
| H3K9ac            |          56.01 |      55.63 |        57.07 |          **63.36** |          60.66 |
| H4                |          81.67 |      80.71 |        **81.86** |          76.54 |          74.78 |
| H4ac              |          49.13 |      50.43 |        50.35 |          62.16 |          **67.30** |
| promoter all               |          **91.01** |      86.77 |        88.31 |          83.35 |          83.73 |
| promoter notata            |          94.00 |      94.27 |        **94.34** |          88.36 |          90.67 |
| promoter tata              |          79.43 |      71.59 |        68.79 |          **81.86** |          79.19 |
| Human TF 0                 |          66.64 |      **71.99** |        69.12 |          65.09 |          62.85 |
| Human TF 1                 |          70.28 |      **76.06** |        71.87 |          67.59 |          69.91 |
| Human TF 2                 |          58.72 |      66.52 |        62.96 |          **70.73** |          63.79 |
| Human TF 3                 |          51.65 |      **58.54** |        55.35 |          42.12 |          46.26 |
| Human TF 4                 |          69.34 |      **77.43** |        74.94 |          71.95 |          70.33 |
| core promoter all          |          **70.33** |      69.37 |        67.50 |          61.77 |          62.56 |
| core promoter notata       |          **71.58** |      68.04 |        69.53 |          66.01 |          65.71 |
| core promoter tata         |          72.97 |      74.17 |        76.18 |          80.20 |          **80.26** |
| Mouse TF 0        |          63.31 |      56.76 |        **64.23** |          48.84 |          49.24 |
| Mouse TF 1        |          83.76 |      84.77 |        **86.28** |          81.07 |          79.43 |
| Mouse TF 2        |          71.52 |      79.32 |        **81.28** |          80.51 |          75.61 |
| Mouse TF 3        |          69.44 |      66.47 |        73.49 |          **81.57** |          78.77 |
| Mouse TF 4        |          47.07 |      **52.66** |        50.80 |          41.79 |          43.70 |
| Covid             |          **73.04** |      71.02 |        68.49 |          45.06 |          31.09 |
| Splice            |          **89.35** |      84.99 |        85.93 |          79.23 |          78.84 |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an interesting method to represent genomic sequences. It is well-motivated, as it takes into account not only the sequence itself but the broader context of its phylogenetic relations -- a type of structure more easily captured by hyperbolic geometry. 

As it stands, there was no consensus to be had: two Reviewers were suggesting acceptance and two Reviewers suggesting rejection. This is a borderline situation, in which I needed to make a judgement call.

From what I was able to see, the key remaining quip of the rejecting Reviewers (Q1G1 and Zknz) appears to be that the Authors did not attempt any inter-operability study of their method with foundation-level genomic models.

While I agree this would be a useful experiment to have in the arsenal, after further championing from the accepting Reviewers, it is evident that HCNN is performant on relevant genomic tasks without any integration into a pre-trained method. This speaks clear volumes about the practical applicability of the work and, especially considering it is an applied paper, and the associated costs of training foundation models, perhaps this is reason enough to let slide that more thorough foundational model integrations had not been attempted.

A such, I believe the paper is in a good shape to be presented at ICLR, and will recommend acceptance. Congratulations to the Authors!