Hyperbolic Genome Embeddings
Authors: Raiyan R. Khan
Philippe Chlenski
Itsik Pe'er
Venue: ICLR 2025 Poster (2025)
Keywords: genomics
representation learning
hyperbolic geometry
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: A hyperbolic geometry-based approach for genomic sequence representation learning

Abstract: Current approaches to genomic sequence modeling often struggle to align the inductive biases of machine learning models with the evolutionarily-informed structure of biological systems. To this end, we formulate a novel application of hyperbolic CNNs that exploits this structure, enabling more expressive DNA sequence representations. Our strategy circumvents the need for explicit phylogenetic mapping while discerning key properties of sequences pertaining to core functional and regulatory behavior. Across 37 out of 42 genome interpretation benchmark datasets, our hyperbolic models outperform their Euclidean equivalents. Notably, our approach even surpasses state-of-the-art performance on seven GUE benchmark datasets, consistently outperforming many DNA language models while using orders of magnitude fewer parameters and avoiding pretraining. Our results include a novel set of benchmark datasets---the Transposable Elements Benchmark---which explores a major but understudied component of the genome with deep evolutionary significance. We further motivate our work by exploring how our hyperbolic models recognize genomic signal under various data-generating conditions and by constructing an empirical method for interpreting the hyperbolicity of dataset embeddings. Throughout these assessments, we find persistent evidence highlighting the potential of our hyperbolic framework as a robust paradigm for genome representation learning. Our code and benchmark datasets are available at https://github.com/rrkhan/HGE.

Published as a conference paper at ICLR 2025
HYPERBOLIC GENOME EMBEDDINGS
Raiyan R. Khan, Philippe Chlenski, Itsik Pe'er Columbia University
(raiyan, pac, itsik @cs. colcoolmmma. a. edu
ABSTRACT
Current approaches to genomic sequence modeling often struggle to align the inductive biases of machine learning models with the evolutionarily-iniei structure of biological systems. To this end, we formulate a novel application of hyperbolic CNNs that exploits this structure, enabling more expressive DNA sequence representations. Our strategy circumvents the need for explicit phylo- genetic mapping while discerning key properties of sequences pertaining to core functional and regulatory behavior. Across 37 out of 42 genome interpretation benchmark datasets, our hyperbolic models outperform their Euclidean equiva- lents. Notably, our approach even surpasses state-of- ofte------h-tttt the- performance seven GUE benchmark datasets, consistently outperforming many DNA language mod- els while using orders of magnitude fewer parameters and avoiding pretraining. Our results include a novel set of benchmark datasets-tth the Transposable Ele- ments Benchmark genome with deep evolutionary significance. We further motivate our work by exploring how our hyperbolic models recognize genomic signal under various data- generating conditions and by constructing an empirical method for interpreting the hyperbolicity of dataset embeddings. Throughout these assessments, we find persistent evidence highlighting the potential of our hyperbolic framework as a robust paradigm for genome representation learning. Our code and benchmark datasets are available at
explores major but understudied component of the
com/ rrrhaar/////// /HGR HGR.
INTRODUCTION
Representation learning of genome sequences has enabled the exploration of critical unsolved problems in biology. particularly the understanding of genome function and organization (Avsec) ctt al. 2021 Chen et al. 2022a| Dudnyk et al.) 2024). Many effective approaches used for genomic sequence modeling have arisen from the same machine learning methods that have powered natural language and image embeddings Yuue et al. 2023 Zhou 2022; Consens et al., 2023). While the field has made progress by utilizing these methods, the inductive biases of these models not usually bespoke to genomic data, limiting the expressive power of the resulting sequence representations. Given the tremendous amount of information sequestered within DNA sequences encoding cellular and molecular activity, an efficient and nuanced representation is necessary for genome interpretation and downstream analyses.
Genome organization is complex, and much of this complexity is the product of evolutionary processes. Any single genome represents the culmination of information diffusion across generations. However, this information transfer occurs through noisy channels, as background mutation rates may degrade the sequence signal (Lu et al.) 2020). Accounting for phylogenetic relationships may therefore contextualize the content of the genome and ultimately benefit genome interpretation attempts. The shared influence of common ancestor across all genomes imbues DNA sequence data with underlying hierarchical structure. These hierarchical relationships emerge through variety of mechanisms, such as orthology and paralogy, which both codify homologous sequences but occur under different circumstances. Further compounding these interdependencies are the multiple overlapping grammatical structures for different regulatory pathways that characterize the language of the genome. Altogether, these nested levels of latent hierarchies confound genome interpretation.
In developing a modeling paradigm better suited to handling the hierarchical nature of DNA sequences, considering the geometry of the embedding spaces is essential. While most embeddings are Fuclidean

---

Published as a conference paper at ICLR 2025
by default, non- aunlldeid spaces may offer compelling alternative. Specifically, hyperbolic spaces, which have the representational capacity to capture tree-structured data with high fidelity, are well- equipped to manage the hierarchical patterns ubiquitous in genomic sequences. The negative curvature of hyperholic spaces facilitates the continuous embedding of exponentially growing structures like phylogenetic trees with relatively low distortion.
In this work, we contend that hyperbolic spaces may be appropriate for learning meaningful rep- resentations of the genome. We leverage fully hyperbolic framework to embed DNA sequences, implicitly handling the latent hierarchies present in the data. Our main contributions are:
1. We adopt the machinery of fully hyperbolic convolutional neural networks (HCNNs), building two classes of HCNNs for genome sequence learning. We contrast hyperbolic and Euclidean approaches to sequence representation.
2. We introduce a novel, curated set of datasets the Transposable Elements Benchmark designed investigate transposable elements, which remain an underexplored area of the genome with deep evolutionary roots.
3. We demonstrate the performance potential of our HCNNs across 42 real-world datasets addressing foundational challenges in genomics.
4. We elucidate the underlying mechanism by which HCNNs parse genomic signal by simulat- ing and testing plausible mata-generating processes for biological sequences.
5. We further motivate our work by formulating an empirical method for interpreting the hyperbolicity of dataset embeddings and this technique to interrogate properties of genome representations generated by our models.
PRELIMINARIES
2.1 RELATED WORK
Driven by the limitations of traditional Euclidean- approaches in capturing relationships within complex data structures, hyperbolic deep learning methods have materialized as promising research area. Early iterations of these methods introduced formalizations for performing the core opera- tions of neural networks in hyperbolic space (Ganea et al. 2018 Nickel & Kiela 2018). alongside optimization techniques generalized to Riemannian manifolds (Bécigneul & Ganea 2019). These approaches have been further extended to a variety of frameworks, including fully hyperbolic neural networks (Chen et al.) 2022b), hyperbolic graph convolutional networks (Chami et al.) 2019), hyper- bolic attention networks (Gulcehre al., 2018), and hyperbolic variational auto-encoders (Mathieul et al., (019). These models, among others, have proven effective across a variety of real-world domains, including vision (Liu et al.
2020 Hsu al. 2021: Mathieu al., [2019), natural lan- guage (Tifrea et al. 2019: Chen et al.,
2024). and computational biology (Zhou & Sharpee 2021
Tian et 2023).
In genomics, hyperbolic methods have correctly modeled established phylogenies. showcasing their supremacy in representing tree-structured data (Chami et al., [020a]] Diang et al.) 2022b] Hughes et al. 2004. Chen et al., 2025). These methods assume that the phylogenetic tree is known priori; thus, the scope of these techniques is limited by the availability of evolutionary metadata. A subset of these methods produces representations of DNA sequences but relies on explicit mapping of phylogenetic relationships (Corso et al. 202 Jiang et al., 2022a) in the form of pairwise edit distances incomplete phylogenies.
2.2 BACKGROUND
The nz-dimensional hyperbolic space Hi is homogeneous, simply connected Riemannian manifold, (M", o ). consisting of smooth manifold M with Riemannian metric g and described by 0. Several equivalent formulations of hyperbolic space exist, including the Lorentz. model, the Poincaré disk model, and the (Beltrami-) model. Here, we use the Lorentz model, L (L", o with manifold L" Riemannian metric tensor g diag(-1, 1, I,.., 1), and origin = |V /V-1/Ks .,,..,""""" The Lorentz model describes points by their configurations the forward sheet of two-sheeted hyperboloid L in (n -)-dimensional
constant negative curvature
2

---

Published as a conference paper at ICLR 2025
Minkowski space, defining the manifold as:
L" = ( E Ru+1 (0)
K Ji > 0}.
(1)
where the Lorentzian inner product follows:
(x. y) = - F.te + x. ys = x² diag 1,.,,... 1)y-
(2)
Utilizing special relativity conventions, the zeroth element in is denoted as the timelike component 2y and the remaining we can further define the timelike component 2t = vI 2 1/K.
elements form the spacelike components s.s giving x: [1t, x.. ) where
Exponential and logarithmic maps used to map between the manifold M and tangent space Tn- with a E M. For mapping a tangent vector & E TeLk onto the Lorentz manifold, we can use the exponential map which is defined as:
K (z) cosh (a x sinh(a) where a V-K||z|le. || (zz,2) exp
(3)
Conversely, to map a point y E LK to the tangent space, we use the logarithmic map:
cosh (B) (y - Bx), B = K Kx y)
log (y)
3
Furthermore, in order to move points along geodesics, the parallel transport operation PT (v) maps a vector v E T&M from the tangent space of æ M the tangent space of y E M. The Lorentzian formula for parallel transport is:
n- (
(y,v)c
PT 1 (v) =
(x + y)-
= (x. y)
METHODS
3.1 FULLY HYPERBOLIC CNN
We leverage the HCNN methodology proposed by Bdeir et al. (2024) in the development of our fully hyperbolic genome sequence model. Under this framework, the elements of the traditional CNN model are reinterpreted in the context of the Lorentz model of hyperbolic space. Briefly, we describe the main Lorentzian components utilized in our model.
Lorentz Convolutional Layer. In Euclidean setting, convolutional layer constitutes matrix multiplication between linearized kernel and input feature maps. In the hyperbolic analogue, each channel is defined separate point on the hyperboloid, with the input to each layer forming an ordered set of n-dimensional hyperbolic vectors I. This formulation enforces the constraint that operations points remain the hyperboloid, Lg C g=+1 In the context of this work, each sequence is thus an ordered set of n-dimensional hyperbolic vectors, where each position describes a nucleotide in the sequence.
For one-oinnnee dinensiienii hyperbolic convolutional layer with input feature map x = f E L the features contained in the receptive field of kernel G E |g are (0r+d IL 7 in which I' marks the starting position and is the stride. Given this parameterization, convolution layer as the output of two transformations:
express the
yi = LFC(HCat( (( E 1
(6)
where HCat is an operation concatenating hyperbolic vectors, and LFC is Lorentz fully-connected layer performing the affine transformation of the kernel (refer to A.I.II. Next, Lorentz batch
normalization (LBN) reframes the underlying operations of batch normalization by using Fréchet mean (Lou et al. 2020) for re-centering points and Fréchet variance (Kobler et al. (022) for re-scaling them. The algorithm is expressed as:
PT pnn
(log (ar)
LBN(x) = expx K
PT
-0
7
1
+
3

---

Published as a conference paper at ICLR 2025
Finally, Lorentz multinomial logistic regression (MLR) builds upon the original formulation of
Euclidean MLR (Lebanon & Lafferty] 2004), which is defined using input æ E I" and C' classes:
ply c/a) x exp(Uur (z)), Uap, (a)) (we, w))](welld((;;, Hw.);
We E RR"
(8)
in which H m. is the decision hyperplane of class Bdeir al. (2024) replace component operations with their Lorentzian interpretations to produce the Lorentz MLR formulation. Using parameters
ag R and ze E R", the Lorentz MLR's output logit for class given input æ: E I. is expressed as:
1 V-K Uxc (x)
sign(a)
sinh
3
a coshly V (K) sinh( (inh((V-a))
B V /l ( <|))=|2 (sin((V-Ke)ll=)3>
For further details, including Lorentz formulations of residual connections and nonlinear activation, we refer the reader to Bddii et al. (2024).
3.2 MODEL OVERVIEW
As our goal is to distill the difference between using Euclidean versus hyperbolic embedding spaces, we employ relatively simple model design. The HCNN architecture consists of three major components: (1) hyperbolic convolutional blocks, (2) a flattening layer, and (3) MLR (Figure I Each input DNA sequence is one-hot encoded at the nucleotide level, and then projected channel- wise onto a hyperbolic manifold (p R4x1. + L4x1n The result of this transformation serves as the input to the hyperbolic convolutional blocks, which produce output feature maps x E T, where C' is the channel dimension. After a flattening step, the model performs classification using Lorentz MLR find the hyperbolic decision hyperplanes splitting the sequences by label.
For each hyperbolic component in our models, there exists an equivalent Euclidean counterpart, ensuring architectural parity across models for fair comparison (Appendix Figure D. However, the layers in the HCNNs also include a learnable K parameter corresponding to the curvature of the hyperboloid on which the points reside. For our downstream experiments, we evaluate two versions of the HCNN model: HCNN-S (single K) and HCNN-M (multiple Ks). In HCNN-S, a single manifold with a fixed curvature is used across all layers, offering more direct comparison with CNNs. In contrast, HCNN-M assigns distinct curvatures [K,..., K.,) to each of the designated blocks, with intermediary steps mapping points between manifolds (see 1.2. By constructing two classes of HCNN models, we analyze the trade-off's between the enhanced representational flexibility of multiple curvatures and the potential instability introduced by the additional exponential and logarithmic mapping steps required for projecting points onto different manifolds. Additional
modeling details are provided in A.2
3.3 S-HYPERBOLICITY
Gromov introduces the notion of 6-hyperbolicity as measure of the deviation of a metric space from perfect tree-like structure (Gromov 1987). We can define metric space (M, d). in which the Gromov product of 2, y E M with respect to x E M is:
1
(z, v)
(d(x, + d(y. z) d(z, v))
(10)
Then, the metric space is characterized 5-hyperbolic for some 8 2 if it satisfies the four point condition, which states that for any four points F, y, 2, w E M:
(x, 2 min ((x, 2) (y. 2) 2)
(11)
The smallest 5 for which this inequality holds is the Gromov 5-hyperbolicity of (M,d).
5-hyperbolicity has been an important tool in elucidating innate properties of metric spaces (Fournier
al. 2015 Albert al. 2014). Recently, this measure has been extended explore the hyperbolic
4

---

Published as a conference paper at ICLR 2025
CAGTCTA
RAXL
L
G
x3
L
9: 9:x x 1
L
CAGTC..TA
CAGTG..CG
Hyperbolic
CAGTG..GG
Convolutional
CAATC-TA
Module
GTAGA..A
GGAAA-CT
MLR
Figure 1: Overview of our HCNNs. Model inputs are sequences with latent phylogenetic structure (bottom left). As sequences pass through the hyperbolic convolutional module, they are projected onto hyperboloid before the model's 's convolutional and flattening steps (top insert). Using hyperbolic MLR, each sequence classified according to the hyperplane boundaries (bottom right).
behavior of specific datasets and their respective embeddings within the domains of computer vision (Khrulkov et al. (220),, and natural language processing (Yang et al. 2024). While the original Gromov's s ò (which we will denote as 6worst hereinafter) is designed to represent the upper bound in terms of deviation from tree-like structure, other approaches have argued favor utilizing an average Gromov hyperbolicity. davg. the grounds that worst case analysis of space may not ultimately be representative of the true hyperbolic capacity of the space (Chatterjee & Sloman 2021 Albert al., 2014 Tifrea et al. 2019). We further develop these ideas in the context of the genomic datasets used in this paper.
As in previous approaches, examine the behavior of &weest and 5agg in high dimensional feature space, As a comparative measure, we compute a scale-invariant value of 6, defined as ére := 25
(Borassi et al. 2015), where Dmax denotes the maximal pairwise distance, or set diameter. d,t
Das
constrained [0, IJ. with a value of denoting complete hyperbolicity, or perfect tree structure. Unless otherwise specified, all ôs referred to in this work the scale-invariaa value.
Ultimately, both Sworst and dayg are point estimates over what may be a complex landscape of values. To offer more comprehensive evaluation, we examine the entire distribution of values across each dataset to thoroughly assess the hyperbolic underpinnings of DNA sequence data. By appraising the full landscape of 6-hyperbolicity in embedding space, gain richer understanding of the intrinsic tree structure across each dataset. We provide further details on 5 computation and other experimental configurations in A.9.
DATA
Synthetic Datasets. order to rigorously interrogate the applicability hyperbolic architectures in genomics, we create several synthetic datasets to illuminate the underlying biological processes being captured by our models. Our approach considers various plausible data-generating processes for biological sequences, and defines three potential cases of biological signal transmission learned by the models. Additionally, given prior evidence from Corso et al. (2021) that purely artificial sequences may not always be indicative of performance real-world datasets, explore this phenomenon by creating two sets of data for each case: one where sequences are completely randomly generated and where sequences randomly sampled from existing genomes.
5

---

Published as a conference paper at ICLR 2025
A
B
C
Figure 2: The various plausible evolutionary scenarios informing genomic sequence learning. Leaf coloring (blue vs. red) represents label assignments for A) intra-tree differentiation, B) inter-tree differentiation, and tree identification scenarios.
We mimic evolutionary dynamics in our synthetic datasets by perturbing input sequences based on phylogenetic tree structure. After establishing an initial input sequence, we simulate sequence evolution along tree branches with the generalized time-reversible (GTR) nucleotide model (Tavaré 1984). Each scenario, visualized in Figure 2 is defined as follows:
(A) Intra-tree differentiation:
sequences generated from single phylogenetic tree, with labels assigned based on clade membership.
(B) Inter-tree differentiation: sequences generated from different phylogenetic trees, with labels derived from phylogeny membership.
(C) Tree identification: sequences are labeled based on the generating process: phylogenetic tree generation or non-phylogenetic (random) generation.
We leverage these scenarios to better understand the specific advantages hyperbolic models and identify the conditions under which they demonstrate the greatest effectiveness. For full details regarding dataset generation, see A.6
benchmark for exploring how transposable elements (TEs) are codified in sequence. TEs are highly abundant, mobile elements of genomic sequence that represent specific evolutionary trajectories within organisms (Hayward Gilbert 2022. Wells Feschotte, 2020). Due to their ability to move within genomes, TEs drive genomic plasticity and have been identified as key players in the evolution of genomic com- plexity (Schrader Schmitz, 2019 Bowen & Jordan (002).. TEs can influence gene expression and regulation by acting as alternative promoters (Faulkner et al. 2009), providing transcription factor binding sites (Sundaram et al.) 2014), introducing alternative splicing (Shen et al. (011). and mediating epigenetic modifications (Drongitis al. (2019). As such, TEs have also been implicated in disease pathogenesis (Jönsson et al., 2020 Hancks & Kazazian 2016). Overall, TEs represent a powerful force in evolutionary biology, continually shaping the genetic landscape.
Transposable Elements Benchmark. We introduce
variety of TES exist across genomes and can be categorized into several subclasses. The genetic structure of TE types follows regular patterns of structural features and motifs, and thus represents an interesting learning opportunity for sequence models. The Transposable Elements Benchmark (TEB) presents a novel resource for investigating TEs, which represent an area of genome organization that remains underexplored in the genomics deep learning literature. TEB surveys several different TE classes across plant and human genomes. Specifically, TEB offers binary classification datasets for identifying seven specific elements across three different TE classes: retrotransposons, DNA transposons, and pseudogenes. Detailed data preprocessing and dataset statistics are further presented in A.3
Genome Understanding Evaluation. The Genome Understanding Evaluation (GUE) benchmark is recently published tool that contains seven biologically significant genome analysis tasks that span 28 datasets. Designed to scrutinize the capabilities genome foundation models, GUE prioritizes genomic datasets that challenging enough to discern differences between models. The datasets contain sequences ranging from 70-1000 base pairs in length and originating from yeast, mouse, human, and virus genomes. Further details be found in Zhou et al. (2024).

---

Published as a conference paper at ICLR 2025
Genomic Benchmarks. We utilize the Genomic Benchmarks (GB) resource, which consists of eight separate classification datasets that spotlight regulatory elements across three different model organisms: human, mouse, and roundworm. Datasets were carefully constructed from published data repositories and consist of input sequences of length 200-500, with the exception of the drosophila enhancers stark dataset, in which sequences have median length of 2,142. Full details data preprocessing and dataset summary statistics can be found in Grešová et al. (2023). As the human non-tata promoters dataset GB was compiled using data that was also used create the promoter detection datasets in GUE (Dreos et al. 2013), we handle this redundancy by only counting non- overlapping datasets when discussing model performance.
5
EXPERIMENTS
5.1 GENOMIC CLASSIHICATION
Data-Cenerating Scenarios. The synthetic dataset experiments offer deeper insight into how the hyperbolic inductive bias operates in a genomic learning context. Table T suggests that this bias is particularly beneficial in Scenario C, where it aids in uncovering the underlying phylogenetic tree structure in the presence of noise. While this learning mechanism may also help disentangle distinct evolutionary patterns (Scenario B), the results further indicate that discernment in Scenario C may be unrelated to discernment in Scenario A. This distinction likely arises because sequence differentiation occurs at the tree level rather than the clade level.
In evaluating predictive models for biological sequence data, homology splitting is commonly used to assess a model's ability to generalize by excluding homologous sequences. We investigate how this partitioning impacts HCNNs by measuring their zero-shot capability in distinguishing sequences from an unseen phylogenetic tree against randomly sampled background sequences. This experiment, detailed in A.7 demonstrates that hyperbolic models outperform Euclidean models in generalizing to unseen homology branches. These findings suggest that the inductive biases of hyperbolic models offer an even greater advantage than previously estimated, most genomic datasets overlook this effect and may thus overestimate the performance of predictive methods (Teufel et al. 2023).
Classification Tasks. The results from the three classification benchmarks are sum- marized in Table 2 Across the 42 dis- tinct datasets, the hyperbolic models out- perform the equivalent Euclidean model on 37 tasks, as measured by the Matthews cor- relation coefficient (MCC). In 29 of these datasets, the improvement in score by hy- perbolic model is statistically significant when accounting for variance across dif- ferent model initializations, whereas the Fuclidean CNN statistically outperforms HCNNs in only two datasets.
Table 1: Model performance (MCC) under different synthetic data-generating scenarios, averaged over five random seeds (mean + standard deviation). The highest- scoring model is in bold, while denotes statistically significant improvement over the opposite geometry model(s) with p 0.05, Wilcoxon rank-sum test.
Model Hyperbolic HCNN-S
Scenario
Sequence
Euclidean CNN
Hyperbolic HCNN-M
Artificial
62.38
65.25
$9.25-260
A
Real 61.72a108
66.44:314
61.26x259
Artificial
38.502
60.532n80
59,75
Further examination of the results suggests that HCNNs confer particularly strong advantage in distinguishing transcription factor binding sites, epigenetic marks, and TEs in sequence. Across promoter detec- tion tasks, hyperbolic embeddings provide no apparent benefit. Since promoters likely function through more complex combinatorial interactions, these dynamics may be more challenging for HCNNs to effectively represent. HCNNs also seem to be significantly disadvantaged in the Covid variant prediction task, which requires distinguishing nine different COVID variants based on their sequences. These findings appear consistent with the synthetic dataset results: the most significant performance gains are observed in scenarios where an evolutionary signal (e.g.. transcription factor binding sites, epigenetic marks, TEs) is distinguished from background noise (e.g., non-functional or background sequences). In contrast, the Covid task closely resembles Scenario A, in which a single ancestral sequence evolves along multiple paths.
Real
57,50a0.8
62.531694
19.12an.4
Artificial
62.05
67.65...m
67.43at.5*
Real
66.22.0.44
73.62an02
69.30
7

---

Published as a conference paper at ICLR 2025
Tandseson
cuc@ddpm
LiVE TATAA
SNF
Badlgrroun
Pronaceed proustogene
Figure 3: Decision boundaries learned by 2-dimensional HCNNs (circles) and CNNs (squares) for differentiating genomic sequence classes. Boundaries for transposon sequences and processed pseudogenes are visualized on the Poincaré disk and Euclidean plane. Regions are colored by predicted class labels, while points are colored based on their true class labels.
Notably, when comparing the best scoring model across runs, HCNNs outperform DNA language models (LMs) in seven of the 28 GUE datasets (A.4 and Appendix Table 5). Across the majority of tasks, HCNNs outpace HyenaDNA (Nguyen et al., 2024), Caduceus- Ph (Schiff et al. (2024). DNABERT (5 mer), DNABERT (6-mer) (Ji et al. 2021), NT-500M human, NT-500M- 1000g, and NT-25000M- 1 100g (Dalla-Torre et al., 2024). The performance gap between HCNNS and the aforementioned LMSs especially striking given the immense scale of these LMs, which contain 1.7x to 543x more trainable parameters than HCNNs and have undergone pretraining on the entire human genome and 1000 Genomes Project sequences (Byrska-Bishon et al. 2022). HCNNs appear have a consistent advantage over Euclidean models across many of the core deep learning genomics tasks.
By directly comparing the embeddings and decision boundaries learned by each class of model, we can begin to infer differences in their expressiveness. Figure 3 visualizes the distinctive class boundaries and sequence relationships learned by HCNNs and CNNs, following the setup in (Chlenski et al. 2024). We observe far better separation of classes in the hyperbolic embeddings than in the Fuclidean case, lending further credence to the appropriateness of hyperbolic embeddings in genomic setting.
Expressive Power.
Additional experiments, detailed in A0 develop intuition for factors informing the positioning of genomic sequence embeddings in the latent space.
Embedding Dimensionality, Prior work on hyperbolic neural networks has demonstrated that the effectiveness of hyperbolic embeddings is especially pronounced in lower dimensions (Chami et al. 20206 Chamberlain et al.) 2017). We probed whether this trend holds under our study conditions by varying the number of channels in the convolutional blocks in both the CNNs and HCNNs. Each distinct model was then trained and evaluated on TEB. The results Appendix Figure Shosow that HCNN-S exhibits marginal increase in improvement over CNNs at lower channel dimensions and HCNN-M shows no gains.
Next, evaluated the potential of the HCNN model class foundational framework for DNA LMs. To align more closely with the parameter scales of DNA LMs, we expanded HCNN-S and HCNN-M. Benchmarking these larger models against the two leading model classes that achieve state-of-the-art (SOTA) performance on GUE, DNABERT-2 (Zhou et al. 2024) and NT-2500-ma mutti,, suggests that the hyperbolic framework holds promise for DNA LM adoption. Table despite having fewer parameters than their competitors, the larger HCNNs achieve SOTA performance on 12 GUE datasets datasets-ouuperffonnt outperforming DNABERT-2 (11 datasets) and NT-2500-min (five datasets).
shown in Appendix
Learned Curvature. The curvature of the hyperbolic manifold a learnable parameter. Exploration of this parameter in TEB (detailed in A.5 illustrates that the value of K does not deviate significantly from its default initialization value of I. However, the HCNN-S and HCNN-M models gravitate towards different curvature values (K -1 and K < 1,, respectively), with small adjustments in the curvature of the embedding spaces for each block of the model.
Hybrid Models. We construct hybrid models that combine Lorentzian and Euclidean components (see for details). Our results indicate that Euclidean embeddings may still benefit from hyperbolic decision boundaries.
8

---

Published as a conference paper at ICLR 2025
5.2 6-HYPERBOLICIT ESTIMATION
As presented in Appendix Figure TO investigation reveals several notable characteristics of 5-hyperbolicity values in finite datascts. The (Appendix Figure I0) and óworst (Appendix Table 9 values computed from the final embedding layer are ostensibly hyperbolic; all values are closer to O than I, indicating tree-like tendencies. However, we observe that the increase in values of Swarst are only weakly anticorrelated with relative improvements
performance on learning tasks (rs = 0.35, TM = -0221, Appendix Tigure(TT) An outlier to this pattern appears to be the Covid dataset, which has low hyperbolicity and poor performance from HCNNs.
Table 2:
Model performance (MCC) on all real-world genomics datasets, averaged over five random seeds (mean standard deviation). The highest-scoring model in bold, while denotes statistically significant improvement over the opposite geometry model(s) with < 0.05, Wilcoxon rank-sum test. Note: the GB human non-tata promoters dataset and GUE promoter detection datasets overlap.
Model Hyperbolic HCNN-S
Benchmark
Task
Dataset
Euclidean CNN
Hyperbolic HICNNMM
LTR Copia
54.73at.45
64.58a307
68.05=2.0
Retrotransposons
LINEs
70.63at.24
16.12a216
77.10.292
SINEs
85. 15.15e1..6
85.45:1.16
81,85-1.9
DNA transposons
CMC-FnnSm
18
80.98-148*
80.65+1.30
hAT-Ac
87.45.030
89.61+1.34
91,04-158
Pseudogenes
processed
60.66an.82
68.30=a.5
65 35.4145.54
unprocessed
51.94
13.03
58.3621.00
H3
64.83
68.14s1.4
68.32a2.12
H3K14ac
34.27a6.14
50.37:s.14
45.69s1.95
H3K36me3
43.74a2.32
53.28:1.94
43,41=2.00
H3K4mel
28. 28.76=a.00
40.84-1.15
34.71=3.0
Epigenetic Marks
H3K4me2
25. 25.38a5.40
39.74s1.11
29.5321.97
Prediction
H3K4mm3
21.77
49.51
30.39u3.32
73K79me3
54 54.88
62.39.2.44
58.48a1.85
H3K9ac
40.37s3.8
12.90
50.21
H4ac
11.59.8.45
92.29.099
44.88
H4
74.8 1.0.92
75.43
76.20
0
58 65
62.842064
60.92a1.72
Human
1
61.4121.0
67.1342.99
66.76.1.25
Transcription Factor
2
49.79an.51
67.1745.2
68.3622.70
GUB
Prediction
3
35.67
41.96x295
42.93a2.00
57 37.68
66.01
67.99.2.00
Splice Site Prediction
reconstructed
78.64a043
80.32a124
80.76e1.05
0
22.51
46.09x217*
47.96=5.1
Mouse
1
76.56v0.51
78.93
76.68-0s1
Transcription Factor Prediction
2
62.69
74.76x3.7
74.78=2.95
3
36.93 36.93
68.61.4.24
66.58a3.24
4
30.23
40.074083
40.57
Covid Variant Classification
Covid
66.43.o.48
16.71
14.81
tata
78.26x2.85
19.54+1.1
79.87-1.0
Core Promoter Detection
notata
66.60.1.00
66.52.028
65.95a051
66.4720.34
65.26s1.11
67.1620.55
tafa
78. 18.58a3.0
79.7442.66
78.77-078
Promoter Detection
notata
90.81
89.86 89.862775
90.28s0.97
all
88.00
87.60+asi
87.93.0.6
Demo
coding vs intergenomic seqs
14.0.5
80.044a28
80.25.024
human worm
89.89
89.89.a.15
92.65.a111
92.71s007
drosophila enhancers stark
7.9923.01
10.77a2.4
10.8721.2
Enhancers
human enhancers cohn
30.7622.08
46.632oms
46.68
human enhancers ensembl
79.48ss.10
44.48a294
72.99s0.3
Regulatory
human ensembl regulatory
89.73
89.91
90.21+1.97
human non-lata promutery
64.98
83,57
79.90s144
Open Chromatin Regions
human ensembl
39.92s0.85
56.2220.2
55.3622.52

---

Published as a conference paper at ICLR 2025
Previous studies have attempted to calibrate their reported dwerst values by comparing them to empirical estimates of ówosst for the Poincaré disk DD2 and the 2-sphere 5² (Khrulkov et al. 2020 Yang et al. 2024), however we note that these empirical estimates are for metric spaces that are categorically much lower in dimensionality than the feature spaces used for the dataset emheddings, leading to potentially incongruous comparisons. Indeed, we find that high-dimensional data produces "emergent hyperbolicity". with points at higher dimensions producing smaller ówerst and óaeg values (detailed in p.9.2). Our results highlight pronounced disparity: the difference in empirical ò values between embeddings sampled on H and those sampled on
hyperbolic spaces (H* where E [200. 1000))- with comparable magnitudes to the sequence embeddings can be as large as 0.2 (Appendix Figure 12 This disparity becomes even more pronounced on Euclidean (R") and hyperspherical (&d) manifolds. Such significant differences
values may largely determine whether the estimated 6 indicates a more hyperbolic nature of the underlying space or otherwise.
To provide a more equitable calibration of hyperbolicity, we compare the d distributions from our genomic datasets to those from simulated datasets of matching dimensionalityy We generate these simulated datasets on both Euclidean and hyperbolic (K = -1) manifolds. Appendix Figure TO illustrates the 5 distributions for each set of dataset embeddings, where each embedding G R with de as the final embedding layer size. Our results reveal that the majority of the genomic dataset embeddings exhibit greater hyperbolicity (lower å values) compared embeddings simulated from a baseline Gaussian distribution on a Euclidean manifold of the same dimensionality. To quantify this differenne,, we employ the Wilcoxon rank-sum test between the baseline and the genome dataset distributions. This analysis shows that 25 out of 43 sequence datasets have significantly lower d' values than the baseline (p- < 0.05). These findings support the hypothesis that genomic sequence data may possess an innate hyperbolicity, making them better suited to hyperbolic representations.
Our approach of examining the entire distribution of 5 values, rather than relying on single scalar measure, reveals nuanced insights into the hyperbolic tendencies of different datasets. This comprehensive view allows us to capture subtleties that might otherwise be overlooked. For instance, the H3K36me3 dataset exhibits a 5 distribution that is significantly lower in hyperbolicity compared to the baseline. However, its high Swrsst estimate suggests that may be less hyperbolic than the baseline when considering only this single metric. Similarly, while the TEB datasets show relatively large owass estimates, their 8 distributions are notably right-skewed. These characteristics appear more consistent with the superior performance of HCNN models these datasets.
The discrepancies between single-poin point estimates (Sweest. davg) and full distributions underscore the importance of a more holistic approach. By considering the entire spectrum of å values across the feature space, we gain a more accurate characterization of the data's tree-like properties. This comprehensive perspective not only provides a richer understanding of the dataset's geometric structure but also offers better insights into why models like HCNNs perform well. Expanding this analysis to DNA LMs (Section A.9.3) reveals that these characteristics generalize across a broader range of models. Still, while moving beyond scalar metrics and calibrating against dimensionality- matched geometries has uncovered hyperbolic tendencies in genomic data that point estimates miss, critical challenges persist: formalizing the behavior of d-distributions statistically, particularly in the face of emergent hyperbolicity, and exploring their robustness to the choice of metric would significantly clarify in which situations hyperbolic representation learning is applicable.
CONCLUSION
We introduce novel application of HCNNs for genomic sequence modeling, critically evaluating their strengths and limitations. Our findings show that hyperbolic embeddings offer a distinct performance advantage in key genomics tasks, particularly under resource constraints. Additionally, our analysis of dataset embeddings uncovers significant correlations between dimensionality and d-hyperbolicity, reinforcing the value of hyperbolic space for genome representation.
HCNNs are lightweight, modular models with the scalability to produce competitive DNA LMs, offering additional performance gains through pretraining and complementary techniques. Moreover, this work drives future research toward developing robust metrics for evaluating dataset hyperbolicity and formalizing its relationship with curvature and dimensionality. By advancing the understanding and optimization of hyperbolic models in genomics, our study encourages deeper exploration of this promising paradigm.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Réka Albert, Bhaskar DasGupta, and Nasim Mobasheri. Topological implications of negative curvature for biological and social networks. Physical Review E, 89(3):032811, 2014.
Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska. Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18 (10):1196-120) 1203, 2021.
Ahmad Bdeir, Kristian Schwethelm, and Niels Landwehr. Fully hyperbolic convolutional neu- ral networks for computer vision. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
https://cpenree//
net Forum 1o
Gary Bécigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
rleiq109K7
Michele Borassi, Alessandro Chessa, and Guido Caldarelli. Hyperbolicity measures democracy in real-world networks. Physical Review E, 92(3):032212,, 2015.
Nathan J Bowen and King Jordan. Transposable elements and the evolution of eukaryotic complexity. Current issues in molecular biology, 4(3):65-76, 2002.
Marta Byrska-Bishan Uday S Evani, Xuefang Zhao, Anna O Basile, Haley Abel, Allison A Regier, André Corvelo, Wayne E Clarke, Rajeeva Musunuri, Kshithija Nagulapalli, et al. High-coverage whole-genome sequencing of the expanded 1000 genomes project cohort including 602 trios. Cell, 185(18):446-3444, 2022.
Benjamin Paul Chamberlain, James R. Clough, and Marc Peter Deisenroth. Neural embeddings of graphs in hyperbolic space. CoRR, 1035999 2017. BtL/httppp/axxxv.org
1705. 10359
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. Advances in neural information processing systems, 32, 2019.
Ines Chami, Albert Gu, Vaggos Chatziafratis, and Christopher Ré. From trees to continuous embed- dings and back: Hyperbolic hierarchical clustering. Advances in Neural Information Processing Systems, 33:15065-15076, 2020a.
Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. Low- dimensional hyperbolic knowledge graph embeddings. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5- 2020, pp. 6901 6914. Association httpss
for Computational Linguistics, 2020b. main. 617
Sourav Chatterjee and Leila Sloman. Average gromov hyperbolicity and the parisi ansatz, Advances in Mathematics, 107417, 2021.
Alex Chen, Philipe Chlenski, Kenneth Munyuza, Antonio Khalil Moretti, Christian Naesseth, and Itsik Pe er. Variational combinatorial sequential monte carlo for bayesian phylogenetics in hyperbolic space, 2025. UtL/tttss://axxit org/abs abs 2500 17965
Kathleen M Chen, Aaron K Wong, Olga G Troyanskaya, and Jian Zhou. A sequence-based global map of regulatory activity for deciphering human genetics. Nature genetics, 54(7):940-949 2022a.
Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Fully hyperbolic neural networks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 5672-5686 5686. Association for Computational Linguistics, 2022b. 10. 111.1...866//1120222LLLLLL http://oo..........1/. 00w.11..16553///
URL
ong.

---

Published as a conference paper at ICLR 2025
Weize Chen, Xu Han, Yankai Lin, Kaichen He, Ruobing Xie, Jie Zhou, Zhiyuan Liu, and Maosong Sun. Hyperbolic pre-trained language model. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Philippe Chlenski, Ethan Turok, Antonio Khalil Moretti, and Itsik Pe' 'r. Fast hyperboloid decision tree algorithms. In The Twelfth International Conference Learning Representattins, 2024. URL
ew net forum? feeet//oommmdddToeooo
Philippe Chlenski, Kaizhu Du, Dylan Satow, and Itsik Pe Manify: A python library for learning non-euclidean representations, 2025. URL htttst:::: :ttpppss///xiii..o org/ abs /2003. 09576
Nathann Cohen, David Coudert, and Aurélien Lancin. On computing the gromov hyperbolicity.
Journal of Experimental Algorithmics (JEA), 20: 1-18, 2015.
Micaela E Consens, Cameron Dufault, Michael Wainberg, Duncan Forster, Mehran Karimzadeh, Hani Goodarzi, Fabian J Theis, Alan Moses, and Bo Wang. To transformers and beyond: large language models for the genome. arXiv preprint arXiv:231 1.76222 2023.
Gregory M Cooper, Eric A Stone, George Asimenos, Eric Green, Serafim Batzoglou, and Arend Sidow. Distribution and intensity of constraint in mammalian genomic sequence. Genome research, 15(7):901-913, 2005.
Gabriele Corso, Zhitao Ying, Michal Pándy, Petar Veličković, Jure Leskovee, and Pietro Liò. Neural distance embeddings for biological sequences. Advances in Neural Information Processing Systems, 34:18539-18855 18551, 2021.
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, et al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, pp. 1-11, 2024.
René Dreos, Giovanna Ambrosini, Rouayda Cavin Périer, and Philipp Bucher. Epd and epdnew, high-quality promoter resources the next-generation sequencing era. Nucleic acids research, 41 (DI):D157-D144, 2013.
Denise Drongitis, Francesco Aniello, Laura Fucci, and Aldo Donizetti. Roles of transposable elements in the different layers of gene expression regulation. International Journal of Molecular Sciences, 20(22):5755, 2019.
Kseniaa Dudnyk, Donghong Cai, Chenlai Shi, Jian Xu, and Jian Zhou, Sequence basis of transcription initiation in the human genome. Science, 384(6694) 16, 2024.
Geoffrey J Faulkner, Yasumasa Kimura, Carsten O Daub, Shivangi Wani, Charles Plessy, Katharine M Irvine, Kate Schroder, Nicole Cloonan, Anita L Steptoe, Timo Lassmann, al. The regu- lated retrotransposon transcriptome of mammalian cells. Nature Genetics, 41(5):563-577 2009, 10.33/..038/
Hervé Fournier, Anas Ismail, and Antoine Vigneron. Computing the gromov hyperbolicity of discrete metric space. Information Processing Letters, 115(6-8)5576-5777 2015.
Adam Frankish, Mark Diekhans, Anne-Maud Ferreira, Rory Johnson, Irwin Jungreis, Jane Loveland, Jonathan M Mudge, Cristina Sisu, James Wright, Joel Armstrong, al. Gencode reference annotation for the human and mouse genomes. Nucleic acids research, 47(D1):D766-DD DD73, 2019.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018
Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimeček, and Panagiotis Alexiou. Ge- nomic benchmarks: collection of datasets for genomic sequence classification. BMC Genomic Data, 24(1):25, 2023.
M Gromov. Hyperbolic groups. Essays in Group Theory, pages/Springer--eraag 1987.
12

---

Published as a conference paper at ICLR 2025
Albert Gu and Tri Dao. Mamba: Linear-ting sequence modeling with selective state spaces. In Finst Conference on Language Modeling, 2024. URL httpps://operreview..ett www.netet/oooiiiieo
tEYskwIVYZ
Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, et al. Hyperbolic attention networks. arXiv preprint arXiv: /805009786 2018.
Aric Hagberg, Pieter Swart, and Daniel A Schult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Laboratory (LANL), Los Alamos, NM (United States), 2008.
Dustin C Hancks and Haig H Kazazian. Mobilization of transposable elements by envi- ronmental and endogenous factors. Human Molecular Genetics, 25(R2): R50, 2016. dbi:10.1093/mm//dwwwoo
Alexander Hayward and Clément Gilbert. Transposable elements. Current Biology. 32(17))R904 R909, 2022.
Joy Hsu, Jeffrey Gu, Gong Wu, Wah Chiu, and Serena Yeung. Capturing implicit hierarchical structure in 3d biomedical images with self-supervised hyperbolic representations. Advances in neural information processing systems, 34:51 12-5123,, 2021.
Jaime Huerta-Cepas, François Serra, and Peer Bork. Ete 3: reconstruction, analysis, and visualization of phylogenomic data. Molecular biology and evolution, 33(6): 2016.
Timothy Hughes, Young Hyun, and David Liberles. Visualising very large phylogenetic trees in three dimensional hyperbolic space. BMC bioinformatics, 5:1-6, 2004.
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert:
pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37 (15):2112-21200 2021.
Yueyu Jiang, Puoya Tabaghi, and Siavash Mirarab. Learning hyperbolic embedding for phylogenetic tree placement and updates. Biology, 11(9):1256, 2022a.
Yueyu Jiang, Puoya Tabaghi, and Siavash Mirarab. Phylogenetic placement problem: A hyperholic embedding approach. In RECOMB International Workshop on Comparative Genomics, pp. 68-85. Springer, 2022b.
Martin Jönsson, Rebecca Garza, Per Johansson, and Johan Jakobsson. Transposable ele- ments: common feature of neurodegenerative disorders. Mobile DNA, 11(1):1-15, 2020. 10000000006//3311000000000000100
Valentin Khrulkov. Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6418-6428, 2020.
Reinmar J Kobler, Jun-ichiro Hirayama, and Motoaki Kawanabe. Controlling the fréchet variance improves batch normalization on the symmetric positive definite manifold. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3863-3867. IEEE, 2022.
Guy Lebanon and John Lafferty. Hyperplane margin classifiers on the multinomial manifold. In Proceedings of the twenty-fist international conference on Machine learning, pp. 66, 2004.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong-War Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. Hyperbolic visual embedding learning for zero-shot recogmition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9273-9281, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
http://wwwwe...o....eeeeeeeooo wdw.kkdddd---eee. CqY7
13

---

Published as a conference paper at ICLR 2025
Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. Differentiating through the fréchet mean. International conference on machine learning, pp- 6393 6393-6403. PMLR. 2020.
Amy X Lu, Alex X Lu, and Alan Moses. Evolution is all you need: phylogenetic augmentation for contrastive learning. arXiv preprint arXiv:2012.13475, 2020.
Xizhi Luo, Shiyu Chen, and Yu Zhang. Plantrep: database of plant repetitive elements. Plant cell reports, pp. 1-4, 2022.
Emile Mathieu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee Whye Teh. Contin- uous hierarchical representations with poincaré variational auto-encoders. Advances in neural information processing systems, 32, 2019.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped normal distribution on hyperbolic space for gradient-based learning. In International Conference on Machine Learning, pp. 4693-4702. PMLR, 2019.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel. Clayton Rabideau, Yoshua Bengio, al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024.
Maximiliaa Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In International conference on machine learning, pp. 3779-3788. PMLR, 2018.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp. 28043-28078, PMLR, 2023.
Eric Qu and Dongmian Zou. Autoencoding hyperbolic representation for adversarial generation. arXiv preprint arXiv:2201. 2022.
Yair Schiff, Chia Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range DNA sequence modeling. In First Workshop on Long-Coonee Foundation Models @ ICML 2024, 2024. URL (tLLhttps:// httt:::s//peenrvieeeee net/
Eorum? 1in:2ii--YYJJJJT
Lukas Schrader and Jürgen Schmitz. The impact of transposable elements adaptive evolution. Molecular Ecology, 2836)11577-1549 2019.
Shihao Shen, Lan Lin, James Cai, Peng Jiang, Emily Kenkel, Miranda R Stroik, Shigeo Sato, Beverly I. Davidson, and Yi Xing. Widespread establishment and regulatory impact of alu exons in human genes. Proceedings of the National Academy of Sciences, 108(7)2837-2842, 2011. 110110.0077//naa.001223233
Ondrej Skopek, Octavian-Euee Eugen Ganea, and Gary Bécigneul. Mixed-curvature variational au- toencoders. In 8th international conference on learning representations (ICLR virttv), International Conference on Learning Representations, 2020.
Stephanie J Spielman and Claus O Wilke. Pyvolve: a flexible python module for simulating sequences along phylogenies. PloS one, 1019):e0139047, 2015.
Vasavi Sundaram, Yong Cheng, Zhihai Ma. Daofeng Li, Xiaoyun Xing, Peter Edge, Michael P Snyder, and Ting Wang. Widespread contribution of transposable elements to the innovation of gene regulatory networks. Genome Research, 24(12):1963-197 2014. doi 10. 10.1101//r.
Simon Tavaré. Line-of-descent and genealogical processes, and their applications in population genetics models. Theoretical population biology, 26(2):119-164, 1984.
14

---

Published as a conference paper at ICLR 2025
Felix Teufel, Magnús Halldór Gíslason, José Juan Almagro Armenteros, Alexander Rosenberg Johansen, Ole Winther, and Henrik Nielsen. Graphpart: homology partitioning for biological sequence analysis. NAR genomics and bioinformatics, 5(4):lqad088, 2023.
Tian Tian, Cheng Zhong, Xiang Lin, Zhi Wei, and Hakon Hakonarson. Complex hierarchical
structures in single-cell genomics data unveiled by deep hyperbolic manifold learning. Genome Research, 33(2):232-245 246, 2023.
Alexandru Tifrea, Gary Bécigneul, and Octavian- Eugen Ganea. Poincare glove: Hyperbolic word embeddings. In 7th International Conference Learning Representations, ICLR 2019, New Orleuns, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL /nttps://ppnrrrviee
net/ Int// Eorum?ormm222 5tt.:mmmiisSSk5533. 3Ack?
Jonathan N Wells and Cédric Feschotte. A field guide to eukaryotic transposable elements. Annual review of genetics, 54(1):539-561, 2020.
Menglin Yang, Aosong Feng, Bo Xiong, Jiahong Liu, Irwin King, and Rex Ying. Enhancing 1lm complex reasoning capability through hyperbolic geometry. ICML 2024 Workshop on LLMs and Cognition, 2024.
Tianwei Yue, Yuanxin Wang, Longxiang Zhang, Chumming Gu, Haoru Xue, Wenping Wang. Qi Lyu, and Yujie Dun. Deep learning for genomics: From early neural nets to modern large language models. International Journal of Molecular Sciences, 24(21):15858, 2023.
Jian Zhou. Sequence-hased modeling of three-dimensional genome architecture from kilobase chromosome scale. Nature genetics, 54(5):725-734, 2022.
Yuansheng Zhou and Tatyana O Sharpee. Hyperbolic geometry of gene expression. Iscience, 24(3). 2021.
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V. Davuluri, and Han Liu. DNABERT-2: efficient foundation model and benchmark for multi speties genomes. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OvenRevi, 2024. URL https: httl:httts://oeeeeeetttt /ttp//ppeevvvveeeeee ew, net. foot/furm?? d=oMI.QR4EZE1
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 LORENTZ CONVOLUTIONAL LAYER
A.1.1 LAYER COMPONENTS
We further break down the Lorentz convolutional layer by defining each separate transformation. First, given hyperbolic points {a; ] the Lorentz direct concatenation (HCat) (Qu & Zou 2022 is defined as:
N
y HCat( (2:))
(12)
N.
R=N+1 This manipulation provides numerically stable way concatenate hyperbolic representations. Next, Chen et al. (2022b) introduced Lorentz fully-connected layer. Given the input vector and the weight parameters W m=xx+1 y R for the fully connected layer, the transformation matrix is defined as:
with y LK
T W
1)
W 2 I/K P
fm
(13)
Then, adding other layer components (except internal layer normalization) results in following formula:
(l b) 2- 1/K (W +b
y LFC(x)=
(14)
where b E IR" and ve denote the bias and activation, respectively.
A.1.2 LAYER MAPPING
HCNN-M models leverage multiple manifolds with corresponding curvatures [K,,.., Ku) for each of u designated blocks. Therefore, we define the mapping between manifolds as follows, using the definitions of exponential and logarithmic maps defined equations 3 and 4 respectively. For mapping of point Mi (where Mj has corresponding curvature K2) to the manifold M2 (with curvature K2), we must first apply a logarithmic map at the origin to bring x to the tangent space Mo Then, apply exponential map at the origin of the resulting point to the new manifold M2. The layer map operation LMM (x) can therefore be defined as follows:
Ky LM,,-M,,xx) expa (log Ki (x))
(15)
A.2 MODELING DETAILS
A.2.1 MODEI.
detailed breakdown of the CNN/HCNN model architecture is visualized in Figure The HCNNs the Lorentz formulation of each model component. For HCNN-M, we show the partition of each manifold across each segment of the architecture. We use cross-entropy loss as our objective and train each model end-to-end on each dataset.
16

---

Published as a conference paper at ICLR 2025
HCNN-- Partitions
CNNNCNN
Architecture
Manitoid K
Conselutional Biock
Coreciatinal Layer
Mantfold K2
Convelatonal Block
ReLU
Convolutional Block
Manifold K
Balthorn
Fallen
Manifold K
Danse Ral.U
MLR
Figure 4: The generalized block architecture for the CNNs/HCNNs. On the left, we delineate the manifold partitions used in our HCNN-M models.
A.2.2 HYPERPARAMETERS
When possible, we keep the hyperparameters constant across the different model types (Table However, we train the Euclidean CNN using the AdamW optimizer (Loshchilov & Hutter 2019 and the HCNNs using RiemannianAdam (Bécigneul & Ganea 2019).
Table 3: Hyperparameter settings for CNN/HCNN training.
Euclidean CNN
HCNN-S
HCNN-M
Optimizer
Adam
Riemannian
RiemannianAdan
Learning Rate (TEB/GUE/GB)
le-4, le-4, le-5
le-4, le-4, le-5
le-4, le-4, le-5
Manifold Learning Rate
N/A
2e-2
2e-2
Batch size
100
100
100
Weight decay
0.1
0.1
0.1
Epochs
100
100
100
B1. B2
0.9, 0.999
0.999
0.9, 0.999
A.3 TRANSPOSABLE ELEMENTS BENCHMARK
TEB presents seven distinct sequence classification datasets categorized within three prediction tasks. An overview of the datasets is presented in Table Sequence and annotation data were integrated from both human and plant genome datasets.
For the retrotransposon and DNA transposon tasks, we crafted a dataset by employing annotations from PlantRep (Luo et al. 2022). a database that provides comprehensive annotations of plant repetitive elements across 459 plant genomes. We narrowed the number of candidate species to those that had an appropriate number of TEs of interest to power deep learning tasks, as well as an average TE sequence length of similar magnitude to the other benchmark datasets (200- 1000 bp). Then, we randomly selected Oryza glumipatula from the set of candidate species use as the plant species for our benchmark. Annotations were downloaded from PlantRep, while the Oryza glumipatula genome (v1.5) was downloaded from the NCBI genome browser (https: //ft:: ncbi n 1lm. nih. gov). Within the retrotransposon group, there are three datasets: LTR Copia, LINES, and SINEs. LTR Copia are a type of retrotransposon characterized by a pair of identical flanking repetitive regions called long terminal repeats (LTRs). Conversely, long interspersed nuclear elements (LINEs), and short interspersed nuclear elements (SINEs) are retrotransposons that do not contain LTRs, and generally contain promoter while varying by length. Next, within the DNA transposon group, we target two
17

---

Published as a conference paper at ICLR 2025
of the most ubiquitous subfamilies: CMC-EnSpm and hAT-Ac, each of which is distinguished by specific short terminal inverted repeats.
While pseudogenes themselves not type of TE, they often the result of TE activity. Therefore, we examine the presence of pseudogenes in the human reference genome (GRCh38.pl) pl2), using gene/transeript biotype annotations from GENCODE and Ensembl (Frankish al. 2019). Pseudo- genes are classified as processed and unprocessed, each of which results from a different mechanism of action. A processed pseudogene lacks introns and arises from reverse transcription of mRNA followed by reinsertion of DNA into the genome, while an unprocessed pseudogene may contain introns and is the product of gene duplication event.
For dataset construction, we created positive set of sequences spanning each TE of interest. We then generated negative set by randomly sampling non-overlapping, remaining portions of the genome (without replacement) until we had matching number of negative sequences. We used a chromosome level train/validationneestt split for our sequences, separating out chromosomes 8/9 and 20-22/17-19 for validation/test sets in Oryza glumipatula and human, respectively, while the remaining chromosomes were used for the training sets.
Table 4: Summary statistics for TEB, including the specific type of TE and the number of training, validation, and test samples in each dataset.
Prediction Task
Species
Max Length
Datasets
Train Dev Test
500
LTR Copia
7666 / 682 568
Retrotransposons
Plant
1000
LINEs
22502 / 2030 / 1782
500
SINES
21152 / 1836 / 1784
DNA Transposons
Plant
200
CMC-EnSpm
19912 / 1872 / 1808
1000
hAT-Ac
17322 / 1822 / 1428
Pseudogenes
Human
1000
processed
17956 / 1046 / 1740
1000
unprocessed
12938 / 766 / 884
A.4 DNA LANGUAGE MODELS
We compare the best classification performance of our HCNN models to that of several DNA LMs. Table documents the performance of ten large DNA LMs on the GUE datasets, along with the number of trainable parameters present in each model. We benchmarked HyenaDNA and Caduceus- Ph, while for other models, we used the benchmarking results reported in (Zhou et al.) 2024). For the HCNN-S and HCNN-M models, report the average number of model parameters used across all GUE datasets. Below, we provide short description of each model class:
NYA: A long-context DNA LM that uses the Hyena operator as a basic building block (Poli al.) 2023), which is a subquadratic alternative to attention. HyenaDNA utilizes extended convolutions and data-controlled gating mechanisms to identify long-range genomic effects (Nguyen al. 2024).
Caduceus-Ph: A bidirectional DNA LM for long-range sequence modeling that builds on the Mamba module (Gu & Dao 2024 Schiff et al. 2024).
DNABERT
(5-mer, 6-mer): An early iteration of a pretrained transformer model for the genome, DNABERT (Ji et al. 2022) uses the BERT architecture and is trained on human DNA sequences. There are four variants of the model, and here we list the results for the 5-mer and 6-mer versions, which use overlapping 5-mer and 6-mer tokenization of sequences.
Nucleotide Transformer (500M human, 500M 1000g, 2500M 1000g, 2500M multi): Nu- cleotide Transformer (NT) represents the largest class models in terms of parameters and training data. There are four variants of NT. The labels "500M" and "2500M" correspond to the number of trainable parameters in the model (Dalla- Torre et al., 2024). For the training data, the categories "human", "1000g", and "multi" refer to the human reference genome,
18

---

Published as a conference paper at ICLR 2025
the 3203 human genomes from the 1000 Genome project, and genomes from 850 different species, respectively.
DNABERT-2, DNABERT-2-PT Byte-Pair Encoding and several architectural upgrades for improved learning capabilities. DNABERT-2 pretrained on the human reference genome, whereas DNABERT-2-PT further pretrained on the training sets of the 28 GUE datasets (Zhou et al. 2024
refinement of DNABERT, DNABERT-2 incorporates
Table 5: The performance (F1-score for Covid, MCC for all other datasets) of several prominent DNA LMs in comparison to the HCNNs on GUE. The best-performing score for each GUE dataset is bolded.
DNA
DNA
NT
NT
NT
NT
DNA
Caduceus
Hyena
DNA BERT-2
HCNN
HCNN
BERT
BERT
-500M
-500M
-2500M
-2500M
BERT-2
-Ph
DNA
-S
-M
(5-mer)
(6-mer)
human
1000g
1000g
multi
-PT
Parameters
7.7M
28.2M
87M
89M
500M
500M
2.5B
2.5B
117M
117M
4.6M
4.6M
H3
77.09
67.17
73.40
73.10
69.67
72.52
74.61
78.77
78.27
80.17
69.42
69.95
H3K 14ac
41.44
31.98
40.68
40.06
33.55
39.37
44.08
56.20
52.57
57.42
56.03
48.25
H3K36me3
46.49
48.27
48.29
47.25
44.14
45.58
50.86
61.99
56.88
61.90
55.27
45.76
H3K4me1
37.76
35.83
40.65
41.44
37.15
40.45
43.10
55.30
50.52
53.00
41.86
39.78
H3K4me2
28.16
25.81
30.67
32.27
30.87
31.05
30.28
36.49
31.13
39.89
43.88
31.27
H3K4me3
24.40
23.15
27.10
27.81
24.06
26.16
30.87
40.34
36.27
41.20
50.58
33.59
H3K79me3
60.31
54.09
59.61
61.17
58.35
59.33
61.20
64.70
67.39
65.46
64.62
63.35
Н3K9ac
52.70
50.84
51.11
51.22
45.81
49.29
$2.36
56.01
55.63
57.07
54.09
52.25
H4
79.91
73.69
77.27
79.26
76.17
76.29
79.76
81.67
80.71
81.86
77.24
76.94
H4ac
40.90
38.44
37.48
37.43
33.74
36.79
41.46
49.13
50.43
50.35
52.94
51.86
prom all
85.87
47.38
90.16
90.48
87.71
89.76
90.95
91.01
86.77
88.31
88.23
88.83
prom notata
93.23
52.24
92.45
93.05
90.75
91.75
93.07
94.00
94.27
94.34
90.92
90.74
prom tata
66.07
5.34
69.51
61.56
78.07
78.23
75.80
79.43
71.59
68.79
82.70
79.80
Human TF
67.32
62.30
66.97
66.84
61.59
63.64
66.31
66.64
71.99
69.12
63.56
63.35
Human
72.10
67.86
69.98
70.14
66.75
70.17
68.30
70.28
76.06
71.87
69.39
68.48
Human TF2
58.92
46.85
59.03
61.03
53.58
52.73
58.70
58.72
66.52
62.96
73.80
71.40
Human TF 54.85
41.78
52.95
51.89
42.95
45.24
49.08
51.65
58.54
55.35
44.08
43.66
Human TF4
69.45
61.23
69.26
70.97
60.81
62.82
67.59
69.34
77.43
74.94
68.43
70.01
prom all
67.28
36.95
69.48
68.90
63.45
66.70
67.39
70.33
69.37
67.50
66.33
67.84
c. prom notata
66.07
35.38
69.81
70.47
64.82
67.17
67.46
71.58
68.04
69.53
66.78
66.48
prom tata
72.94
72.87
76.79
76.06
71.34
73.52
69.66
72.97
74.17
76.18
81.34
82.07
Mouse TFQ 0
56.18
35.62
42.45
44.42
31.04
39.26
48.31
63.31
56.76
64.23
48.41
52.31
Mouse TF 1
80.31
80.50
79.32
78.94
75.04
75.49
80.02
83.76
84.77
86.28
79.26
77.41
Mouse TF 2
75.89
65.34
62.22
71.44
61.67
64.70
70.14
71.52
79.32
77.86
77.51
Mouse TF3 3
73.47
54,20
49.92
44,89
29.17
33.07
42.25
69.44
66.47
73.49
73.51
69.73
Mouse TF4 4
47.98
19.17
40.34
42.48
29.27
34.01
43.40
47.07
52.66
50.80
41.27
43.62
Covid
45.19
23.27
50.46
55.50
50.82
52.06
66.73
73.04
71.02
68.49
46.43
24.74
Splice
81.59
72.67
84.02
84.07
79.71
80.97
85.78
89.35
84.99
85.93
81.96
82.23
A.5
MANIFOLD CURVATURE
Figure depicts the learned curvatures for models trained TEB. In the HCNN-M models, blocks 1-3 represent the hyperbolic convolutional blocks in the model, each associated with a corresponding manifold that has its own curvature. Block represents the portion of the model that involves flattening step, dense layer, and MLR, operations that all occur on a single hyperbolic manifold (Figure 4p. the HCNN-S models, the value of K is fixed, as single manifold is used across the entire model.
A.6 SYNTHETIC DATASETS
We construct each synthetic dataset by randomly sampling a phylogenetic tree using the Environment for Tree Exploration (ETE) toolkit Sluerta-Cepas et al. (2016). To simulate nucleotide sequence evolution along the tree's branches, we use the PYVOLVE package (Spielman & Wilke 2013). specifically for its implementation of the Generalized Reversieer (GTR) model (Tavaré 1984)
19

---

Published as a conference paper at ICLR 2025
HOMNS
Mean Inpiovementt in MCC
0.08
0.7
0.5
0.06
0.04
0.3
0.02
CNN
HCNN-5
Dimension
Dimension
Figure 5: On the left, we show the average improvement in performance (MCC) on TEB datasets for HCNNs compared to CNNs the channel dimension in the convolutional layers varies. On the right, we present the mean MCC achieved by the models across TEB datasets, for each channel dimension.
CMC-EnSam
-0.00
INTAAc
Copla
SINES
processed
-0.95
unprocessed
Rullic
Shilk K
-1.5
Book
2
Figure Average values of K, the curvature parameter in the HCNNs, as they vary across each block of the model. Values are reported for models trained on each of the seven classification tasks in TEB.
with default parameters. Four types of fixed-length sequences generated and used across Scenarios A, B, and C:
Artificial tree: The starting ancestral (root) sequence is randomly generated.
Real tree: The starting ancestral sequence is sampled from the human genome.
Artificial background sequence: Sequences generated randomly and independently by sampling nucleotides.
Real background sequence: Sequences sampled from independent (different chromosome) regions of the human genome relative to the starting ancestral sequence.
We define the task for each scenario follows:
(A) Intra-tree differentiation:
single tree is sampled, with clade membership determining class labels. The model's task is to differentiate clades.
(B) Inter-tree differentiation: A different tree (with different starting ancestral sequence) is sampled for each label. The model's task is to differentiate trees.
(C) Tree identification:
A single tree is sampled, and all sequences from this tree share the same label. Independently sampled background sequences are assigned a separate label. The model's task is to differentiate the tree from the background sequences.
20

---

Published as a conference paper at ICLR 2025
Table 6: The performance (Fl-score for Covid, MCC for all other datasets) SOTA DNA LMs and scaled HCNNs GUE benchmark datasets. The best- best-perfer perforreef score for each dataset is bolded. For the scaled HCNN-S and HCNN-M models, we report the average number of model parameters used across all GUE datasets.
NT
DNABERT-2
HCNN-S (Large)
HCNN-M (Large)
-2500M
DNABERT-2
-PT
-multi
Parameters
2.5B
117M
117M
43M
43M
H3
78.77
78.27
80.17
72.17
72.21
13K14ac
56.20
52.57
57.42
70.56
69.87
H3K36me3
61.99
56.88
61.90
68.06
67.92
H3K4me1
55.30
50.52
53,00
53.33
55.45
H3K4me2
36.49
31.13
39.89
54.67
52.50
H3K4me3
40.34
36.27
41.20
67.25
64.61
H3K79me3
64.70
67.39
65.46
70.49
70.65
H3K9ac
56.01
55.63
57.07
63.36
60.66
H4
81.67
80.71
81.86
76.54
74.78
H4ac
49.13
50.43
50.35
62.16
67.30
promoter all
91.01
86.77
88.31
83.35
83.73
promoter notata
94.00
94.27
94.34
88.36
90.67
promoter tata
79.43
71.59
68.79
81.86
79.19
Human TF 0
66.64
71.99
69.12
65.09
62.85
Human TF 1
70.28
76.06
71.87
67.59
69.91
Human
58.72
66.52
62.96
70.73
63.79
Human TF 3
51.65
58.54
55.35
42.12
46.26
Human TF4 4
69.34
77.43
74.94
71.95
70.33
promoter all
70.33
69.37
67.50
61.77
62.56
core promoter notata
71.58
68.04
69.53
66.01
65.71
core promoter tata
72.97
74.17
76.18
80.20
80.26
Mouse TF
63.31
56.76
64.23
48.84
49.24
Mouse TF
83.76
84.77
86.28
81.07
79.43
Mouse TF2 2
71.52
79.32
81.28
80.51
75.61
Mouse TF
69.44
66.47
73.49
81.57
78.77
Mouse TF 4
47.07
52.66
50.80
41.79
43.70
Covid
73.04
71.02
68.49
45.06
31.09
Splice
89.35
84.99
85.93
79.23
78.84
Representative simulated phylogenetic trees and their corresponding labels are visualized in Figures We introduce noise into the datasets by randomly swapping 10% of the labels in the training
7 and 8 and validation sets.
A.7 HOMOLOGY SPLITTING
The experimental setup for homology splitting is visualized Figure For the training and validation data, generate synthetic dataset in Scenario C, where sequences generated from the tree share the same label, and background sequences not originating from the tree share a different label. However, instead of creating test from this dataset, we create the test set by generating completely new phylogenetic tree and sampling sequences from it. The geeereeed sequences in the test dataset thus originate from entirely unseen homology branches.
Results of this experiment are presented in Table 7 Hyperbolic models show significantly improved generalization the Euclidean model in evolutionary and phylogenetic context.
21

---

Published as a conference paper at ICLR 2025
Figure 7: Leaf node sequence classifications (with added noise) in Scenario A for the simulated phylogenetic tree (structure visible on the left).
Figure& 8: Hamming distance matrix for all leaves in the simulated phylogenetic tree for Scenario A.
Table 7: Model performance (MCC) on the homology splitting experiment, averaged over five random seeds (mean + standard deviation). The highest-scoring model is in bold, while denotes statistically significant improvement over the opposite geometry model(s) with < 0.05, Wilcoxon rank-sum test.
CNN
HCNN-S
HCNN-M
45.73
40.87=8.93
22

---

Published as a conference paper at ICLR 2025
Train + Validation
Test
Figure 9: Overview of the homology splitting experiment. A training and validation dataset (left) generated in the same manner as the synthetic dataset used for Scenario C. For the test dataset (right), completely new tree and ancestral sequence are used generate the class.
A.8 HYBRID MODELS
Following Bdeir al. (2024), we evaluate hybrid CNN models, in which we substitute components of our models across different manifolds. We construct two hybrid model variants: E2H-CNN and H2E-CNN. E2H-CNN, we use Euclidean CNN head and a Lorentzian MLR, whereas H2E-CNN employs HCNN head and a Euclidean MLR. We compare the performance of the two hybrid models to the other three models Table X On TEB datasets, we observe that incorporating a Lorentzian component generally improves performance over fully Euclidean model, with larger gains from E2H-CNN. These results suggest that using hyperbolic hyperplanes to separate classes may be beneficial, even for Euclidean embeddings. Overall, the results highlight the potential of hybrid models.
Table 8: Model performance (MCC) in TEB, averaged over five random seeds. The best-performing model is bolded.
Dataset
CNN
HCNN-S
HCNN-M
E2H-CNN
H2E-CNN
LTR Copia LINEs
54.73+1.45
64.58=3.07
68.05+2.80
61.82+221
63.95+3.52
70.63a1.24
76. 16:221
77.10x2.92
75.65
79.15x2.6
SINES
15.1611
85,45=1.16
81,85+2.95
89.65:2.13
79.49+3.40
CMC-EnSpm
12.18
80.98=1.48
80.65+1.30
76.75
77. 153.44
hAT-Ac
87. 87.4520.90
89.61=1.34
91.04
89.76x0.85
85.6 1..4.41144
processed
60.6620.2
68.30=0.93
=5.54
66.68 6..1.11
66. 12x0.4
unprocessed
51.94+2.99
56. 1 3=0.56
58.36=1.80
58.09=0.96
18.16
A.9
J-HYPERBOLICITY
A.9.1 ESTIMATION PROCEDURF
Computing Swerst naively is an On*)) operation for set of n points, therefore we use the efficient approach introduced in Khrulkov al. (2020) and und[Cohen et al.] (2015). Specifically, we incorporate sampling procedure to estimate hyperbolicity in computationally tractable manner. The steps are as follows:
1. Sample N. points from the dataset (we set N, 1000).
2. Compute the matrix A of pairwise Gromov products using equation 10 and a fixed point
z 2g (detailed in Cohen al, (2015)).
3. Determine the matrix C' (A A) A, where @ represents the min-max matrix product:
(A @ B)y max min; (A,,. Bay}.
4. For &werat we take the maximum value from C, and for dagg- we compute the expected
value the unique elements of C pertaining to valid tuples. We apply the scale-invariant
23

---

Published as a conference paper at ICLR 2025
transformation mentioned in the main text to the ó values determine the final values reported. However, for the davg values, we instead transform the raw values using the scale-invariant ratio introduced in Borassi al. (2015): 26ws where Davg is the average Dos"
distance between two randomly selected points.
Results are averaged across multiple runs, and we provide the resulting mean and standard deviation. For the genomic datasets, we use the test set of sequence embeddings generated from the final embedding layer of the trained Euclidean CNN models (Table
A.9.2 METRIC SPACE CALIBRATIONS
In order to calibrate our 9-hyperbolicity measurements, scrutinize the behavior of ó approximations at various fixed curvatures (K) and dimensionalities (d). We use the MANIFY package, introduced in Chlenski et al. (025), to randomly sample data points from Gaussian distribution across different manifolds, using the wrapped normal distribution in the hyperbolic (K = -1, -2) (Nagano et al.) 2019) and the hyperspherical (K 1, 2) (Skopek al. 2020) cases. We then compute ó estimates according to the procedure in A.9.1 We use the geodesic distance of each manifold to determine the
distance matrix between points.
The results of the simulations are visualized in Figure 12 The decreasing trend in both &worst and davg estimates across curvatures suggests that higher dimenstonality of data points may lead increased hyperbolicity in datasets. For discrete metric spaces, we confirm that for trees, &warst Oavg 0 by using the NETWORKX package (Hagberg et al.) 2008) to generate random tree graphs and compute the distance matrix based on shortest paths within each graph.
A.9.3 DNA LANGUAGE MODELS
We explore the hyperbolicity of sequences embedded by large DNA LMs. Our analysis encompasses a diverse range of pretrained models, selected to represent various architectural approaches and scales. The models under examination are HyenaDNA, DNABERT-2, and NT-500M human.
As case study, we probe subset of sequences that likely reflect strongly conserved evolutionary relationships. We therefore generate LM embeddings for a randomly sampled set of SINE sequences from TEB. The embeddings are derived by applying mean pooling over the final layer embedding output of each model. To establish a comparative baseline, we juxtapose the underlying distribution of each LM with distribution generated from randomly sampled points from Gaussian of equivalent dimensionality, following the procedure outlined in Section 5.2
The results of our analysis are presented in Figure 13 Notably, the embeddings produced by HyenaDNA and DNABERT-2 exhibit significantly higher hyperbolicity compared to the baseline embeddings (p < 0.01, Wilcoxon rank-sum test). In contrast, the representations generated by the NT-500M human display substantially lower hyperbolicity than the bascline. This disparity may stem from the higher dimensionality of the NT-500M human embeddings, suggesting that hyperbolicity may become less critical at sufficiently large embedding scales.
A.10 HYPERBOLIC SEQUENCE REPRESENTATIONS
In exploring the sequence representations learned by HCNNs, we build the intuition introduced by Khrulkov et al. (2020), where hyperbolic image embeddings of MNIST show that ambiguous digits tend to cluster near the center of the Poincaré disk, while clearer, more confidently classified digits lie closer to the boundary. Similarly, in Figure 14 we observe that in the processed pseudogene dataset from TEB, sequence embeddings located near the center of the Poincaré disk (representing the top of the hierarchy) correspond to low-confidence predictions by HCNNs, approximated using model loss. In contrast, embeddings near the boundary exhibit the highest classification confidence. This pattern supports the notion that well-defined sequences occupy lower regions in the hierarchy, where the increased representational capacity of hyperbolic space allows for grained separation based on distinctive sequence features.
To systematically investigate the underlying sequence features informing these hyperbolic genome embeddings, we performed an in silico mutagenesis experiment using the processed pseudogene
24

---

Published as a conference paper at ICLR 2025
LTR Copla
LINES
SINES
CMC-EnSSpm
pseudogenes
Unprocessed peeudogeness
H3K14ac*
H3K35mme3
H
H3KAmo2"
H3K4mc3
HaK79mme3"
HaKSac
H4ac
H4
Human TF 0
Human TF
Human TF
Human TF 3
Human TF 4
Spice
Mouse TF 0
Mouse TF
Mouse TF 2
Mouse TF 3
Mouse TF.4
Covd
Core Promoter
Core Framoter notata
Care Promcler tata
Promater 300
Promoter 300 notata
Promoter 300 tata"
Coding Intergerionic seqs
Human or Worm*
Dresophla enhancers Stark
Human onhsnears Cohn
enhoncers
Human Ensembl regulatory"
promoters
Simulated (K=0)
0.1
0.2 0.3 0.4 0.5
9-hyperbolicity
Figure 10: Distribution of scaled 6-hyperbolicity values across each genomic dataset. Colors delineate different task categories, while the bottom rows provide reference distributions for d values computed from a set of points sampled from the normal distribution on a Euclidean (K = 0, red) and hyperbolic (K 1, blue) manifold. Dashed lines indicate the dayg values for the hyperbolic reference (blue) and the Euclidean reference (red). An asterisk (*) denotes that the corresponding distribution constitutes smaller 5 values (i.e., is more hyperbolic) than the Euclidean reference based on the Wilcoxon rank-sum test (p < 0.01).
25

---

Published as a conference paper at ICLR 2025
Table 9: o-Hyperbolicity values of the final embeddings for CNNs trained on each genomic dataset. Results are averaged 10 sampling (mean + standard deviation).
Benchmark
Task
Dataset
8
LTR Copia LINEs
0.36soe175
0.145 0.145an0010
Retnorransposons
0.40108110
0.164
SINEs
0.0820e006
0.1 0.170a00016
DNA transposons
CMC-EaSpu
0.18s0c18t
0.163
lAT--Ac
0.37x06220
0.215a0m0
Pseudogenes
processed
0.36x00204
0.
unprocessed
0.35
0.157
H3
0. 1.10noccoz
0.098
H3K 14ac
0.09
0.101
H3K36me3
0.26s00541
0.251
H3K4me1
0.21
0.225aum25
Epigenetic Marks
H3K4me2
0.13200112
0.125
Prediction
H3K4me3
0. 0.14g00068
0.169
H3K 79me3
0.15200255
0.122s00007
H3K9ac
1.21
0.265
H4ac
O. 188001156
8.6
0.10co.ess
0.082.00041
0.20s00114
0.160l.0m02
Human Transcription Factor Prediction
0. 19
0.1 101
GUE
0.19:0
0.141 m.000
18
0. 0.140=00000
Splice Site Prediction
splice
0. 0.29
0.256
0.21s00147
0.140=0.0433
Mouse
0. 0.35u6es0:
0.249 .0.0332
Transcription Prediction
0. 192002257
0.13 500000
0.100m
0.1 0.002
Covid Variant Classification
covid
0.50ioe38s
0.417
0.29100ms
0.229 00003
Core Promoter Detection
0.28-00184
0.212-0001
tala
0.22:60082
0.138.n.015
0.29g00144
0.260.n.022
Promoter Detection
notata
0.31-00210
0.257-0004
tata
0.16600127
0.138
Demo
coding intergenomic seqs
0.21s00100
0.118
human or worm
0. 19
=0020
drosophila enhancers stark
0.30so0174
0.209
Enhancers
human eahancers cohn
0.19200137
0.092s0.00c
GB
human enhancers ensembl
0.1 0.19200108
0.109
Regulatory
human ensembl regulatory
0.234oezs77
0.1488mmm1
human non-tata promocets
0. 1.
0.1033x0x00
Open Chromatin Regions
human ensembl
0.24s00400
0.189
26

---

Published as a conference paper at ICLR 2025
HONN-S performance performance (MCC)
HCNN-M performace- CNN performance (MCC)
20
-10
-20
fousiers = -0.38
r= -0.35
-40
Touslers [= -0.21
-30
0.5
0.2
Ovent
Óvarst
Figure 11: Correlation between Swarst and the performance differential between HCCN-S and CNN models. Toutlees includes outliers in the Pearson correlation coefficient calculation, while excludes them (p < 0.05, except for HCNN-M
dataset. Our methodology involves structured approach to dissecting sequence representations. For a given sequence, we:
1. Retrieve the Genomic Evolutionary Rate Profiling (GERP) Cooper et al. (2005) score for each nucleotide along the sequence. GERP scores quantify evolutionary constraints at specific genomic positions, identifying which positions are functionally important based on selective pressure. GERP uses multiple sequence alignments across species identify conserved regions.
2. Strategically mutate subset of nucleotides under the highest selective pressure, generating multiple perturbed sequence variants.
3. Generate embeddings for each perturbed sequence using the trained HCNN.
Figure 15 visualizes this experiment using both processed pseudogene sequence and background sequence. As mutations progressively erode the evolutionary signal associated with strong selec- tion (corresponding the nucleotides with the highest GERP scores), the features rendering the pseudogene "gene-like" may deteriorate. This degradation increases sequence ambiguity from the HCNN's perspective, manifesting as a shift of the perturbed representations toward the top of the hierarchy- near the center of the Poincaré wweke low-confidence sequences typically reside. The loss of these evolutionary features actively hinders the model's 's ability recognize pseudogenes.
Critically, this effect is sequence-specifie perturbing conserved regions within noisy background sequences fails to produce an equivalent shift, suggesting the model prioritizes features consistently associated with the pseudogene class.
To validate the generalizability of this phenomenon, we conducted comprehensive analysis on randomly sampled set of 10,000 sequences from the processed pseudogene dataset, ensuring balanced class representation. For each sequence, we applied our mutagenesis protocol (steps 1-3), generating 10 perturbed sequence variants and corresponding HCNN representations.
We quantify this effect by measuring the embedding shifts these perturbed sequences relative to their original embeddings, specifically tracking their movement toward the representation space's s origin. This directional analysis provides insight into the HCNN's representational sensitivity: a trajectory toward the origin likely indicates increased representational ambiguity for the perturbed sequence. The distance between each sequence (perturbed and original) and the origin in the embedding space computed using Poincaré geodesics.
27

---

Published as a conference paper at ICLR 2025
10
Guaamr
0.0
0.6
0.4
0.0
600
2000
Dimandoo
Curveture
0.8
0.6
0.
Iimensios
Figure 12: Estimates for &worst (top) and dag (bottom) using simulated data points from wrapped normal distribution on manifolds with varying curvatures (K) and dimensionalities.
When comparing representational shifts between perturbed processed pscudogene and background sequences, observe significantly greater movement toward the Poincaré disk's origin for perturbed pseudogene sequences. This difference, statistically validated by the Wilcoxon rank-sum test (p < 0.05), demonstrates the robustness of our findings. We tested range of mutation rates, altering 10 - 30% of nucleotides per sequence, and found that the effect remained consistent and statistically significant across all rates.
28

---

Published as a conference paper at ICLR 2025
HyenaDNA> Try d=128
DNABERT-2* d=768
Mucleaiie Transformer
500k-human--ot
0.05
0.15
0.2
0.25
0.8
0.35
0.4
0.45
d-hyperbolicity
Figure 13: Distribution scaled 9-hyperbolicity values using embeddings from various DNA LMs. The distribution of each model is overlaid with the distribution of randomly sampled points from Gaussian equal dimensionality (red). An asterisk (*) denotes that the corresponding distribution has significantly smaller smaller ó values (i.e., is more hyperbolic) than the Euclidean reference, based on the Wilcoxon rank-sum test (p < 0.01).
Table 10: Mean model performance (MCC) aggregated by genomics task (mean + standard error).
Model Hyperbolic HCNN-S
Benchmark
Task
Euclidean CNN
Hyperbolic HCNN-M
Retrotransposon Prediction
70.4848.0
74.80+1348
75.98=12448
TEB
DNA transposon Prediction
71.91
85.30+1382
85.77=20.37
Pscudogene Prediction
56.30+7.0
62.22410.26
Epigenetic Marks Prediction
40.76x4.07
55.31
48. 18+2.81
Human Transcription Factor Prediction
52.52+3.63
61.1223.12
61.25
Splice Site Prediction
78. 78.6440.19
80.3240.55
80.7610.77
GUE
Mouse Transcription Factor Prediction
45.79+4.72
61.9345.52
61.5245.08
Core Promoter Detection
70. 13+2.6
70. 12
70.9942.39
Promoter Delection
85,80+1.75
85.73+1.37
55.66+1.82
Covid Variant Classification
66.43x0.21
1
14.8 8
Demo
82.52
86.3442.8
86.4812.83
Enhancers
39.4119.00
34.18 14..884.46
28.7742.8
GB
Regulatory
77.36x4.73
86.74+1.35
85.05+2.34
Open Chromatin Regions
39.92+0.38
56.22+0.13
55.36+1.23
29

---

Published as a conference paper at ICLR 2025
pro CoOoO pre.drogeee
0.8
8.
-0
Procebiitty that
00
Figure 14: HCNN embeddings for the processed pseudogene dataset, colored by model confidence on the left and by the probability that sequence is processed pseudogene (vs. background sequence) on the right. Sequence embeddings are visualized on the Poincaré disk.
Sequence Type
Oricinal
aequeenta
Perlubed pscudlgene seqquenee
Original
Perlabed background
aequeetca
Figure 15: HCNN embeddings for processed pseudogene sequence and background sequence. Each sequence has been perturbed multiple times, with different instances shown on the Poincaré disk.
Bechy
Piocoosad dagens
Figure 16: UMAP visualizations of the embeddings generated by the HCNN (left) and CNN (right)
trained on the processed pseudogene dataset TEB.
30


Reviews:
[{"review_id": "R0UH7dFdKI", "reviewer": "Reviewer_Zknz", "summary": "The authors introduce a hyperbolic CNN for genomic sequences. They compare their model to a standard CNN on both synthetic and real benchmarks for genomic sequence classification (GUE, GE, and their transposable element benchmark). The authors find that the HCNN model outperforms the standard CNN on most tasks.", "questions": "- Is the CNN/HCNN model pretrained with a MLM task similar to DNABERT models? If so, on what dataset? If not, why?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Hyperbolic architectures is a promising avenue for training models on genomic sequences, as they are inherently related through phylogenetic tree structures. This paper provides interesting initial results in this direction.", "weaknesses": "- How does the CNN/HCNN performance scale with number of parameters? Figure 5 seems to indicate that performance gets worse with increasing hidden dim? Additionally figure 5 should include error bars. Also the cumulative improvement y-axis is unclear. It should just be the average MCC value. \n\n- Figure 1 shows that sequences which obey a phylogenetic tree structure are embedded into a hyperbolic space which learns this structure. However I do not see any results which indicate that the phylogenetic structure is being learned. The authors could provide an experiment to validate this by looking at correlation between embedding distances and phylogenetic distances (for example EDS task from https://www.biorxiv.org/content/10.1101/2024.07.10.602933v1)\n\n- The results would be stronger if the authors provided a baseline column for a standard transformer architecture in Table 1, as CNNs are used less frequently for genomic tasks. Additionally, the DNABERT/NT performance should be included in Table 1, not in supplementary Table 4, and would be much more clear by averaging a single value for each task. \n\n- Figure 2 shows the decision boundary for a 2D model, and shows better separation for the hyperbolic embeddings. However, it is not clear that this result extends to higher dimensions. Instead, the figure would be stronger by using a PCA/UMAP plot with a higher dimensionality model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2hKBdFEuzD", "reviewer": "Reviewer_Q1G1", "summary": "This paper adopts the hyperbolic CNNs to genome representation learning. Empirical results suggest the advantage of hyperbolic CNN to its Euclidean equivalents in genome learning. Moreover, the hyperbolic CNNs for genome sequences can outperform existing genome foundation models on 7 out of 27 datasets on the GUE benchmark.", "questions": "1. Can you compare the cost of different models by the wall-clock time of fitting it on the same dataset with the same devices?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written, with good backgrounds and clear visualization.\n2. The introduced Transposable Elements Benchmark is valuable to the research community.\n3. The hyperbolic CNN demonstrates consistent improvements over the Euclidean CNN. It could serve as a baseline choice for genome classification models.", "weaknesses": "1. Comparing the number of parameters between CNN-based models and Transformer-based models is not convincing. Due to the underlying difference between these two model architectures, fewer parameters do not guarantee efficiency. A more suitable comparison would be the time and memory usage of training the models on the same dataset.\n2. There is a lack of baselines. As a CNN-based model, the model should also be compared with HyenaDNA and Caduceus.\n3. The empirical study is not well-presented. In the main text, this paper dedicates most of the experimental sections to the comparison of hyperbolic and Euclidean CNN, which is less relevant to the genomics audience. On the one hand, this conclusion has already been drawn in the HCNN paper [1]. On the other hand, the Euclidean CNN is not the SOTA in genome classification. Compared with existing genomics foundation models, as shown in Table 4, it is more important, in my opinion.\n4. The contribution is limited from both methodology and application perspectives.\n    - Methodologically, HCNN is introduced in ICLR 2024, and the advantage of hyperbolic over Euclidean equivalents is also shown in that paper. This work adopts the same model (with tiny modifications on two versions ) and validates the same observations in genome embedding. \n    - From an application perspective, this work's contribution is limited. Though the authors do not report the wall-clock time / memory usage of the training the model, given that GFMs like NT and DNABERT only require 3-5 epochs to converge and this model is trained for 100 epochs, this model is likely to be more costly than existing ones. Moreover, it does not deliver better performances than GFMs. From an application perspective, users do not care much about whether a model is pre-trained / randomly initialized. The cost of fitting a model to a specific dataset is more meaningful.\n\n[1] Bdeir, Ahmad, Kristian Schwethelm, and Niels Landwehr. \"Fully Hyperbolic Convolutional Neural Networks for Computer Vision.\" arXiv preprint arXiv:2303.15919 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ia7wLZAot0", "reviewer": "Reviewer_H56n", "summary": "The authors propose an architecture that is better suited for genomic property prediction from DNA sequences. They implement hyperbolic convolutional neural networks (HCNN) and demonstrate improvements across a number of tasks. They compare against a matched CNN in Euclidean space. They motivate the hyperbolic embeddings due to the tree like underlying generative process of genomic sequences. In addition the authors propose an additional Transposable Elements Benchmark. In addition they use a measure of $\\delta$ hyperbolicity or a measure of tree-likeness in genetic data.", "questions": "- If the hierarchical relationship is between sequences then why do the authors note that a single point would be an individual nucleotide and a sequence would be a set? Isn't a sequence then exponential in the number of individual nucleotides thus resulting in an exponential growth of possibility space even without the hyperbolic space inductive bias?\n- There is no mention regarding the parameter count of the eucledian CNN. I assume it is the same as the hyperbolic version but would be good to get a confirmation. Can the authors also include FLOP count per forward / backward pass for each of the models?\n- The work can benefit from an expanded more thorough explanation in the appendix of mapping between tangent space and Lorentz manifold especially when describing the hypebolic layers. \n- Can the authors mention how the confidence intervals were generated in table 1. \n- Do the authors find it strange that for covid variant classification, arguably in a very clear cut case where there is abundant information regarding evolution of the virus the hyperbolic architecture doesn't perform so well? Do the authors have a hypothesis for why that is the case?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The authors analyze an existing problem through a novel lens suggesting a new possible inductive bias to an important problem. \n- The authors perform a comprehensive evaluation across a wide variety of tasks\n- Authors propose a new dataset which can evaluate ability of models to classify transposable elements. This class of sequences are widely abundant however it is unclear as to their biological significance compared to sequences encoding mature RNAs.", "weaknesses": "- This work can benefit from additional  justification for why the hierarchical tree structure is a good inductive bias for genomic data. Although the original data generating process can take on a tree structure is it necessarily the case that we would want to add this as an inductive bias to the model?\n- Is there an experiment that the authors can do that would confirm the validity of this inductive bias? Do the authors find it strange that the synthetic task that was explicitly designed to test performance of the inductive bias the baseline falls within the confidence interval of the best performing model? \n- In genomic sequence modeling it is common to do homology splitting where sequences related through their homologous relationships are fully left out and the model's capacity is assessed on whether it's able to generalize to unseen homology branches. Do the authors expect that this inductive bias helps or hurts the performance of the model in this domain? Can they perform an experiment validating this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qyW2RqXL70", "reviewer": "Reviewer_KNja", "summary": "The paper proposes the application of fully hyperbolic CNNs (HCNNs) for representation of biological data, in specific DNA sequences. The authors note that genomic data is often generated by evolutionary processes, and result in phylogenetic hierarchies that may be better represented in hyperbolic space as opposed to Euclidean space. The authors apply HCNNs on several genomic element classification benchmarks, and additionally propose a new dataset, the Transposable Elements Benchmark, in which models are tasked with classifying the type of transposable element that exists in a context window. The authors find that hyperbolic CNNs outperform their Euclidean counterparts in most of the benchmark tasks, and can occasionally outperform genomic foundation model performance. They also perform an empirical analysis on the hyperbolicity of the selected datasets.\n\nNOTE: Score raised from 5 -> 8 in response to rebuttal.", "questions": "- I would appreciate if the authors could address the above points. \n\n- Could selection of GUE as a benchmark cause underperformance? From my understanding, the benchmark does not control for homology in the train / test splits. Conceivably, the benefit of the HCNN would be improved generalization in evolutionary / phylogenetic contexts, and potential data leakage in the GUE benchmark could mask the performance benefit in this aspect. \n\n- I would appreciate if the colour of the DNABERT-2 distribution in supplemental figure 12 could be changed to be more distinct for folks with colour vision deficiency!", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Paper presents a creative application of HCNNs to the field of genomics, and serves as a useful preliminary paper to explore the benefits of representing genomic data in hyperbolic space. Given the current community interest in representation learning for genomics, this new approach could be of high interest and significance. Authors also provide new dataset for benchmarking a relevant task.\n\n- The authors make an effort to construct high quality baselines for their experiments. In addition to demonstrating the benefits of an HCNN vs a Euclidean CNN, the method is also benchmarked against genomic foundation models. The authors also take care during their empirical evaluation of dataset hyperbolicity to account for dataset dimensionality by calibrating against simulated data.\n\n- Experimental results indicate the hyperbolic representation offers benefit on some classes of tasks. This shows that the HCNN or other hyperbolic representations could have promise in genomic representation learning. Visually, it appears that the tasks where HCNNs offer improvement also correlate with the underlying data hyperbolicity, supporting the core hypothesis of the work.\n\n- Experiments are well executed, with replication across random seeds and confidence intervals + statistical significance reported. Experimental setup is well described, and code is available, indicating that results are likely to be reproducible.", "weaknesses": "- I believe the analysis could be improved by investigating exactly how the HCNN represents sequences related through evolution. A common intuition I have seen for hyperbolic embeddings in computer vision is that datum closer to the origin tend to represent high level features and vice versa. It would be convincing if some analogy could be made for the genomic sequences through the experiments. For example, while Figure 2 demonstrates that hierarchical structure exists in the HCNN representation, it would be interesting to see whether the sequences placed at the \"top\" of the learned hierarchy reconcile with biological intuition.\n\n- It's surprising that the HCNN representation is not consistently better for datasets that seem highly related to evolutionary processes. For example, it is strange that the covid variant classification is so poor given that the observed variation is likely driven by selection, and that the HCNN-M is outperformed by the Euclidean CNN on the synthetic dataset which is generated using a evolutionary simulator. Similarly, it is interesting that the HCNN does not significantly outperform the Euclidean models on promoter classification tasks despite the results in the empirical study in Figure 3. I would appreciate any deeper insights from the authors on these issues.\n\n- Tying into the last point, given the lack of consistent improvement and correlation in improvement with seemingly evolutionary data generating processes, its unclear under what circumstance the HCNN should be used compared to existing approaches.\n\n- To my knowledge, there is very limited methodological novelty in the paper, although I believe the current paper may be sufficient for an applied paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)