Autoencoding Hyperbolic Representation for Adversarial Generation
Authors: Eric Qu
Dongmian Zou
Venue: Accepted by TMLR ()

Abstract: With the recent advance of geometric deep learning, neural networks have been extensively used for data in non-Euclidean domains. In particular, hyperbolic neural networks have proved successful in processing hierarchical information of data. However, many hyperbolic neural networks are numerically unstable during training, which precludes using complex architectures. This crucial problem makes it difficult to build hyperbolic generative models for real and complex data. In this work, we propose a hyperbolic generative network in which we design novel architecture and layers to improve stability in training. Our proposed network contains a hyperbolic autoencoder (AE) that produces hyperbolic embedding for input data and a hyperbolic generative adversarial network (GAN) for generating the hyperbolic latent embedding of the AE from simple noise. Our generator inherits the decoder from the AE and the generator from the GAN. Our architecture fosters expressive and numerically stable representation in the hyperbolic space. Theoretically, we validate the training of GAN in the hyperbolic space, and prove stability of our hyperbolic layers used in the AE. Experiments show that our model is capable of generating tree-like graphs as well as complex molecular data with comparable structure-related performance.

Published in Transactions on Machine Learning Research (09/2024)
Autoencoding Hyperbolic Representation for Adversarial Generation
Eric Qu
erioqu@berkeeey..du
Deportment of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley 94720, CA, U.S.
Dongmian Zou
Kongmnian.zouuhdhuee edu
Division of Natural and Applied Sciences, CMCS and DSRC Duke Kunshan University
Jinnguu 215316, China
Reviewed on OpenReview: https: openreview. net/ forum? id= NQ.9U0YLM3
Abstract
With the recent advance of geometric deep learning, neural networks have been extensively used for data in non-Euclidean domains. In particular, hyperbolic neural networks have proved successful in processing hicrarchical information data. However, many hyperbolic neural networks are numerically unstable during training, which precludes using complex architectures. This crucial problem makes it difficult to build hyperbolic generative mod- els for real and complex data. In this work, we propose hyperbolic generative network in which we design novel architecture and layers to improve stability in training. Our pro- posed network contains hyperbolic autoencoder (AF) that produces hyperbolic embedding for input data and a hyperbolic generative adversarial network (GAN) for generating the hyperbolic latent embedding of the AE from simple noise. Our generator inherits the de- coder from the AE and the generator from the GAN. Our architecture fosters expressive the hyperbolic space. Theorctically, we validate
and numerically stable representation the training of GAN in the hyperbolic space, and prove stability of our hyperbolie layers used in the AE. Experiments show that model is capable of generating tree-like graphs as well complex molecular data with comparable structure-related performance.
1 Introduction
High-dimenssinal data often show underlying geometric structure, which cannot be easily captured by neural networks designed for Euclidean spaces. Recently, there is intense interest in learning good representa- tion for hierarchical data, for which the most natural underlying geometry is hyperbolic. A hyperbolic space is a Riemannian manifold with a constant negative curvature (Anderson) 2006). The exponential growth of the radius of the hyperbolic space provides high capacity, which makes particularly suitable for modeling troc-like hicrarchical structures. Hyperbolic representation has been successfully applicd to, for instance, social network data in product recommendation (Wang al. 2019)) molecular data in drug discovery Yu
et al. 2020 Wu et al. 2021), and skeletal data in action recognition PPng et al. 2020
Many recent works (Ganca et al.) 2018 Shimizu et al.
2021 Chen et al. 2021 have successfully designed
hyperbolic neural operations. These operations have been used for generaling samples in the hyperbolic
space, For instance, several recent works (Nagano et al. 2019 Mathieu et al.) 2019 Dai et al. 2021b) have built hyperbolic variational autocncoders (VAE) (Kingma Welling 2014]. On the other hand. Lazcano
et al. (2021) have generalized generative adversarial networks (GAN) (Goodfellow al. 2014 Arjovsky
et al. 2017 to the hyperbolic space. However, the above hyperbolic generative models are known to suffer

---

Published in Transactions on Machine Learning Research (09/2024)
from gradient explosion when the networks deep. In order to build hyperbolic networks that can generate real data, it is desired to have a framework that has both representation power and numerical stability.
In recent advancements, there has been a notable shift towards the manipulation of the latent space of autoencoders (AEs), including models like Latent Diffusion (Van Den Oord et al.) 2017 Rombach et al. 2022), which has opened up plethora of benefits across various domains. This manipulation has allowed for the latent representations serve not just a means for dimensionality reduction but as foundational elements in generative modeling. Researchers and practitioners have leveraged these representations for generating new, synthetic instances of data that retain the complexity and diversity of their training sets Razavi et al. 2019 Esser et al., 2020). Beyond serving as powerful tool for learning reprosentations, the latent representation is increasingly utilized for its inherent smoothness (Rubanova et al. 2019 and mumerically stable gradient properties (Li et al. 2020).
To leverage these insights, we design a novel stable hybrid model which learns complex structures and hyperbolic embeddings from data, and then generates examples by sampling from random noises in the hyperbolic space. Altogether, our model contains three parts: first, it uses a hyperbolic AE to learn the embedding of training data in the latent hyperbolic space; second, we use hyperbolic GAN to learn the mapping from wrapped normal noise to the latent distribution; third, we generate samples by sequentially applying the GAN's generator and the AE's decoder. The use of AE latent space in GAN was pioneered by Prrykhokko et al. (2019) but our work considers the hyperbolic space and different neural operations. We name our model as Hyperbolic AE-GAN, or HAEGAN for short. The advantage of this architecture is twofold: boasts expressivity by processing noise through both generator and decoder layers, and it allows for the AE's flexible design without compromising the GAN's sampling efficiency. In addition, HAEGAN avoids the complicated form of ELBO in hyperbolic VAE, which is one source of numerical instability, The main contributions of this paper are as follows:
We design a hybrid framework for learning hyperbolic distributions with novel neural operations. To the best of our knowledge, similar architectures have not been explored in the hyperbolic space. Compared with other models, our model HAEGAN enjoys both expressivity and numerical stability.
We theoretically validate the Wasserstein GAN formulation the hyperbolic space, especially the way of sampling from the geodesic connecting a real sample and a generated sample.
We design novel concatenation operation in the hyperbolic space. We theoretically investigate its numerical stability and empirical comparisons with other hyperbolic networks.
In the experiments part, we illustrate that HAEGAN has the capacity not only to faithfully generate synthetic hyperbolic data, but also to generate real data with sound quality. In particular, we show that HAEGAN achieves comparable performance in molecular generation, especially metrics related to structural properties.
2 Background
2.1 Hyperbolic Geometry
Hyperbolic geometry is a special kind of Riemannian geometry with a constant negative curvature (Cannon et al., 1997 Anderson] 2006). To extract hyperbolic representations, it necessary to choose "model". or coordinate system, for the hyperbolic space. Popular choices include the Poincaré ball model and the Lorentz model, where the latter is found to be more numerically stable (Nickel & Kiela, 2018). We work with the Lorentz model L3 gL) with a constant negative curvature K, which is an re-iienssional manifold L embedded in the (n + 1)-dimcnsional Minkowski space, together with the Riemannian metric tensor = diag([-1, 1T),,, where 1, denotes the n-dimensional vector whose entries are all I's. Every point in LK is represented by x = [x,, x1 1 > O, E R", and satisfies (x, x) 1/K, where (:))c is the Lorentz inner product induced by g* :<x y) = x paper, we will refer to 2, as the "time component" and æ, as the "spatial component". Extensive details are provided in Appendix A.1
T gy -1,y: + x; ye: x, y E Lg. In the rest of the
2

---

Published in Transactions on Machine Learning Research (09/2024)
Notation We use dc(2; y) to denote the length of geodesic ("distance" along the manifold) connecting E k- For each point a E k, the langent space al a is denoted by T.T.kk The norm -1-1l== ((-)) For 2, y E R and v E TaLk, we use exp (v) to denote the exponential map of u at æ; on the other hand, we use log LK + +T to denote the logarithmic map such that log (exp xK (v)) = v. For two points
n, y L we use PT denote the parallel transport map which "transports" vector from L to T along the geodesic from a to y-
2.2 Hyperbolic Neural Operations
In this section, we introduce the fundamental linear layers designs Hyperbolic space. Additional intro- ductions of Graph Neural Network layers and Sypeebblii-EBulldeen conversion layers can be found in the Appendix A.2
Fully connected linear layers are the fundamental building block of Euclidean deep learning. In Euclidean space, each linear layer can be represented by a linear transformation add bias. Specifically, suppose the input a E R²", an Euclidean linear layer is
Wx + b
where y E R is the output, W E R is learnable weight matrix, b E Rn is the learnable bias.
To define a linear layer in hyperbolic space, the crucial part is to define hyperbolic linear transformation. However, we have an a E T.g and use an arbitrary matrix W E ((++1)x(m+1) as weight matrix, the output y is not guaranteed in the Lg To constraint the output in the hyperbolic space, there are two lines of works,
the langent linear layers Ganea et al.) 2018 Shimizu et al. 2021 Chami et al. 2019) and fully hyperbolic linear layers CChen et al., 2021).
Tangent Linear Layers The idea of this kind of linear layers is to perform linear transformation in the tangent space. Since the tangent space of any Riemannian manifold is an Euclidean subspace, it is possible to perform linear transformation in such space. Chami et al. (2019) defined the linear transformation of æ E Lg to y E Lg by first logmap x to the tangent space of the origin, perform Euclidean linear transformation by matrix W E RaxM and expmap it back to L,=:
= W & x = expo K (Wlogt (a:)
(2)
However, cascading many exponential and logarithmic maps may lead to numerical instability (Chami et al. 2019: Chen et al. 2021). Thus, people are interested in developing hyperbolic linear layers without the use of exponential and logarithmic maps.
Fully Hyperbolic Linear Layers In Lorentz model, the constraint for the point [ E &L are
(x,x)c = - |2,
where | is the Euclidean 2-norm. Thus, whatever the spacial component x, changes, we could always calculate the Ta = / 1/ 1/K to make in the Lorentz space.
Using this idea, Chen al. (2021) proposed the fully hyperbolic linear layer, which applies a genreal linear transformation to the spatial component of the input, and calculate the time component so that the whole
VIW [ 2 1/K where W E (1)(m1))) Wx
vector lies in the hyperboloid. Specifically, it maps E L to
With additional activation, bias and normalization, a general expressive lincar layer transforms an input x E Lg to
[1 1/K
y = HLinear(x) =
E LR.
(4)
h(x:
Here,
<(x) = Ao (v + b')
| (Wr(w> bl| (Wr(Ø) +b,,
(5)
3

---

Published in Transactions on Machine Learning Research (09/2024)
where E R'+1 and W [=x((+1) are trainable weights, b R" and d' R are trainable biases, o is the sigmoid function, I is the activation function, and the trainable parameter À > 0 scales the range. We may also write h(2;; W, b) to emphasize its dependence on W and b.
We remark that using Lorentz model comes with many advantage in numerical stability. We will demonstrate this the next section, and in the experiments in 5.2
3 Lorentz Concatenation
3.1 Motivation and Definition
Concatenation and split essential operations in neural networks for feature combination, parallel com- pulation, etc. (Shinizu et al. (2021) proposed Poincaré A-concatenation and B-split in the Poincaré model. Specifically, they first use the logarithmic map to lift hyperbolic points to the tangent plane of the origin, then perform Euclidean concatenation and split in this tangent space, and finally apply / regularization and apply the exponential map to bring it back to the Poincaré ball.
Since we use the Lorentz model, the above operations are not useful and we need define concatenation and split in the Lorentz space. One could define operations in the tangent space similarly to the Poincaré 9- concatenation and 8-split. More specifically, if we want to concatenate the input vectors {x; J. where each
E T we could follow a "Lorentz langent concatenation". first lift each o; to -logo (ax) = [ E Rn++1 and then perform the Euclidean concatenation to get == [o. vT.1 y exp_" (v) concatenated vector in the hyperbolic space. We denote y H Similarly, we could perform the "Lorentz tangent split" on an input n; E T with split sub-dimensions Σ n = n to get v log K (a:) oT, R RRI,...... ...., UN. [R=N]] T 8, = [[0 Toll and the split vectors y: = expo (w:)
Finally, we would get
succossively.
Unfortunately, there are two problems with both the Lorentz tangent concatenation and the Lorentz tangent split. First, they are not "regularized", which means that the norm of the spatial component will increase after concatenation, and decrease after split. This will make the hidden emboddings numerically unstable. This problem could be partially solved by adding hyperbolic linear layer after each concatenation and split, similarly to Ganea al. (2018); so that we have a trainable scaling factor X to regularize the norm of the output. The second and more important problem is that if we the Lorentz tangent concatenation and split in deep neural network, there would be too many exponential and logarithmic maps. It one hand sulfers from severe precision issue due to the inaccurate float representation Yuu & De Sa 2019 2021), and on the other hand easily suffers from gradient explosion. Moreover, the tangent space is chosen at o. If the points to concatenate are not close to o, their hyperbolic relation may not be captured very well. Therefore, we abandon the use of the langent space and propose more direct and numerically stable operations, which we call the "Lorentz direct concatenation and split", defined as follows.
Given the imput vectors { where each x, L2 and M Emm ni. ni, the Lorentz direct concatenation of {x, J.W is defined to be vector y LM given by
y HC;
Σ 2 t
T + +NN (LN-1)/K, 1 NNN,
(6)
Note that each x,, is the spatial component of x; If we consider æ; L a point in R the projection of z; onto the Euclidean subspace (0) (R", or the closest point there, is 2; The Lorentz direct concatenation can thus be considered as Euclidean concatenation of projections, where the Euclidean concatenated point is mapped back to M. by the inverse map of the projection. We remark that this concatenation directly inherits from the Lorentz model.
We also define the Lorentz split for completeness, though our main focus is on concatenation: Given an input E T. the Lorentz direct split of æ, with sub-dimensions m,."" BN where ΣN n4 n, will be
4

---

Published in Transactions on Machine Learning Research (09/2024)
(al)
(22)
40
(b1)
(b2)
(b3)
100
Figure 1: Illustration of the gradients of both concatenation methods (a) Lorentz tangent concatenation (b) 82s0
Lorentz direct concatenation. (al) (b1) 0z, dx. (a2) (b2) dr.
0z.1 Ox.
(a3) & (b3)
(y: where each y: Li is given by first splitting x as x 22 Vi
.yN.] and then calculating
the corresponding time dimension as y /ly. 1| 1/-1
yi
3.2 Advantage of Lorentz Direct Concatenation
Firstly, we state the following thcorctical result regarding the exploding gradient of the Lorentz tangent concalenation.
Theorem 3.1. Let {xi where x; E L denote the input features. Let y HCat(fx;) denote the output of the Lorentz direct concatenation and z = HTCat({2x, ) denote the output of the Lorenta tangent concatenation. Fix j E NN}. The following results hold:
For any {a: 1 and any entry y" of y. |[yy" /dx,, lan... .ay 1. For any M > 0, there exist {x.) and an entry z" of z for which ||0=" /0x, ler... xx ll M.
This theorem shows that while the Lorentz direct concatenation has bounded gradients, there no control on the gradients of Lorentz tangent concatenation. The proof can be found in Appendix C.1 We provide a
numerical validation in $3.2.1 and an empirical validation in $3.2.2 We also include additional analysis on the effect of the two concatenation methods to the hyperbolic distance in Appendix C.2
3.2.1 Numerical Validation of Theorem 3.1
In this scction, we numerically validate Theorem 3.1 by showing the gradients of the Lorentz tangent con- calenation and the Lorentz direct concalenation under the following simple setting. We concalenate two I-dimensional Lorentz vectors to obtain 22 2-dimensional Lorentz vector.
LL to be the two input vectors under the Lorentz model with K- -1 For both the Lorentz fangent concatenation and the Lorentz direct
Denote x V LL and y V
5

---

Published in Transactions on Machine Learning Research (09/2024)
(a) 160
(b)
Lorentz Direct Concatenation
Lorentz Direct Concatenation
140
Lorentz Tangent Concatenation
800
Lorentz Tangent Concatenation
Gradient Norm
Gradient Norm
120
600
100
400
200
40
20
0
10
20
30
40
50
60
0
20
40
60
80
100
120
Blocks
Blocks
Figure 2: Average gradient norm of each block in training, (a) 64 blocks. (b) 128 blocks.
concatenation, let = [ze, 200, 2s1
T E T_2, denote the concatenated vector of a and Numerically, we [-100, 100] and calculate the gradients of each entry of z with respect to Za.
consider the range of Ta E
Oz
and azm: Ox.
that is, du.
We plot them in Figure I We 1,,0,E(-100.00 0
F..
E. 100)
remark that due to symmetry, the graphs are the same for
ay.
From Figure D we observe unbounded gradients each component of the gradient of the Lorentz tangent concalenation. On the other hand, all the components of the gradient of the Lorentz direct concalenation has absolute value bounded by 1. The results of this numerical experiment have validated the conclusion in Thcorem 3.1
3.2.2 Empirical Validation of Theorem 3.1
In this section, we design the following simple experiment to show the advantage of our Lorentz direct concatenation over the Lorentz tangent concatenation when they used in simple neural networks. The hyperbolic neural network in this simple experiment consists of cascading of L blocks, and the architecture of each block is illustrated in Figure A d-dimensional input is fed into two different hyperbolic linear layers, whose outputs are then concatenated by the Lorentz direct concatenation and the Lorentz tangent concatenation, respectively. Then, the concatenated output further goes through another hyperbolic linear layer whose output is again d-dimensional. Specifically, for I= 0, LL-1,
h1 (
h Hlinear, (9);; (1) HCat(h (1) ())); x (411 Hlinearaxd,
(7)
HLineari
h
(1)
Concat
HLineary
(++1
HLinear2
hug
Figure 3: Ilusstaaionn of the I-th block in the simple hyperbolic neural network.
In our test, take d = 64. We sample input and output data from two wrapped normal distributions with different mcans (input: origin output: E2H(164) and variances (input: diag(1es), output: 3 x diag (la)). Taking the input as (0)), we fit. (4) to the oulput data. We record the average gradient norm of the three hyperbolic linear layers each block. The results for L = 64 blocks and = 128 blocks are shown Figure g Clearly, for the first 20 blocks, the Lorentz tangent concatenation leads significantly larger gradient norms. This difference in norms is clearer when the network is deeper. The gradients from the Lorentz direct concatenation much more stable.

---

Published in Transactions on Machine Learning Research (09/2024)
4 Hyperbolic Auto-Encoder Generative Adversarial Networks
4.1 Architecture of HAEGAN
(a)
Encoders
Zembdd
Decoders
Although recent proposals of hyperbolic generative models have transferred VAE and GAN to the hy-
Training Data
Reconstruc
tion
perbolic domain (Nggano et al. 2019 Mathieu et al.) 2019 Dai et al. 2021b Lazcano et al. 2021), they are known to suffer from numerical instability, which hinders building large-scale hyperbolic generative models. To this end, we design HAEGAN model to contain both hyperbolic AE and hyperbolic GAN. First, we train a hyperbolic AE and use the encoder to embed the dataset into latent hyper- bolic space. Then, we use our hyperbolic GAN to learn the latent distribution of the embedded data. Finally, we sample hyperbolic embeddings using the generator and use the decoder get samples in the original space. An illustration ion of HAEGAN is shown in Figure
Encoders
Freal
Training Data
Wrapped Normal Distribution
Generator
fake
Critic
(c)
Decoders
Generation
The overall structure of HAEGAN distributes learn- ing into two architectures and thus reduces the scale of either standalone network. Nevertheloss, both the GAN and the AE require carefully chosen structures in order to guarnntee numerical stability. We discuss the hyperbolic GAN and hyperbolic AE in the next two subsections respectively.
Figure 4: Overview of HAEGAN. (a) The hyperbolic AE. (b) The hyperbolic GAN for generating the latent embeddings. The encoders in (b) are identical to (a). (c) The process for sampling molecules. The generator (c) is identical to (b) and the decoders in (c) are identical to (a).
4.2
Hyperbolic GAN
4.2.1
Architecture of HGAN
We could use the hyperbolic linear layers in $2.2 to define a hyperbolic GAN whose generator and critic are in the hyperbolic space. The generator pushes forward a wrapped normal distribution G(o, I) to a hyperbolic distribution cascading of Igen hyperbolic linear layers. Specifically, we sample (0) G(o, I) from the wrapped normal distribution (Nagano al. 2019), and produce Ziake following
HL.inar d. (2(1-)), 1 1=1,'. Agen=
(8)
Ziake 2(22n))
The critic aims to distinguish between fake data generated from the generator and real data. It contains cascading of lais hyperbolic linear layers, and centroid distance layer whose output is score in R. Specifically,
HLineara, I.d (z(1-1)), 1= 1, 1,... laiss
(9)
HCDista am . (Haie)).
4.2.2 Hyperbolic Wasserstein GAN and Gradient Penalty
We adopt the framework of Wasserstein GAN (Aroovkyy et al. 2017 which aims to minimize the Wasserstein-1 (W1) distance between the distribution pushed forward by the generator and the data distri- bution. Since de is a valid metric, the W1 distance between two hyperbolic distribution P,, defined on the Lorentz space is
inf E(a.v)~ (dc W1 yE)()..P.)
(10)

---

Published in Transactions on Machine Learning Research (09/2024)
Tosa dats
Samples
Ioput date
Samples
Input
Samples
Figure Input data and generated samples from the hyperbolic GAN. The hyperbolic data points are transformed to the tangent space of the origin by the logarithmic map.
where (P,,,,P)) is the set all joint distributions whose marginals are Pr and Fg. respectively. By Selikrrovicc-Rubinstee duality (Villani 2009), we have the following more tractable form of W1 distance
Wi(Pr,Ps)
sup Ex-p, [D(x)] - EE-P, [D(x)].
(11)
where the supremum is over all 1-Lipschitz functions D L + R, represented by the critic. To enforce the 1- 1-Lipschitz constraint, we adopt a penalty term on the gradient following Gulrajani el al. 2017). The loss function thus
LWGAN
E [D( R-P3
E [D(x)]+ a-
+ E [(IVD(0)lc- 1)²]. d.-P;
(12)
where D is the critic, (D()) is the Riemannian gradient of D(x) at at I3 is the generator distribution and P, is the data distribution. Crucially P& samples uniformly along the geodesic between pairs of points sampled from Pg and Pr, instead of linear interpolation. This manner of sampling validated in the following proposition,
Proposition 4.1. Let P. and Pg be two distributions in Li and f be an optimal solution of max fl Ey=2,(f()) - [f(x) where I|-1e is the Lipschitz norm. Let a be the optimal coupling between P,, and Pg that minimizes WGP,,, inf inf Fxcy=-ldc(a, y)}, where ITPP,, PP)) is the set of joint distributions <x. y) whose marginals are P, and P.g, respectively. Let xt 7(), S S be the geodesic belween and y, such that 7()) r()) (1) v. TILk, Mluelllellc dc y). If S is differentiable and (xx y) 0, then it holds that
Vi P(o.y)< Vs (xx) = de y)
1.
(13)
The proof of this proposition can be found in Appendix D.I
4.2.3 Toy Distribution Generation
As proof-of-concept, we use a set of challenging toy 2D distributions explored by Rozen et al. (2021) to test the effectiveness the hyperbolic WGAN-GP formulation. For our experiment, the training data are prepared the following manner. We first sample 5,000 points from the toy 2D distributions and scale the coordinates to []., Then, we use the E2H operation (4 map the points to the hyperbolic space. These points are treated the input data of the hyperbolic GAN. Next, we use the hyperbolic GAN to learn the hyperbolic toy distributions. The generator and the critic both contain layers of hyperbolic linear layers and 64 hidden dimensions at each layer. The input dimension for the generator is 128.
After we Irain the hyperbolic GAN, we sample from it and compare with the input data. Note that the input data and the generated samples are both in the hyperbolic space. To illustrate them, we map both the input data and the generated samples to the tangent space of the origin by applying the logarithmic map. We present the mapped input data and generated samples in Figure 5 Clearly, the hyperbolic GAN can faithfully represent the challenging toy distributions in the hyperbolic space.
8

---

Published in Transactions on Machine Learning Research (09/2024)
4.3 Hyperbolic AE
HAEGAN enjoys expressivity and flexibility in choosing the AE for embedding the hyperbolic distribution. For many datascts with hierarchical structures that are not readily presented in the hyperbolic space, the hyperbolic AE can also learn to find their appropriate hyperbolic representations. In this case, the first layer of the hyperbolic AE is constructed by the hyperbolic embedding layer (Nagano et al. 2019 from Euclidean to the hyperbolic space.
The design of AE structure varies according to the type of data, which will be described in detail for each experimental task. Nevertheless, one common operation for building hyperbolic decoders for tree-structured data is message passing, which requires concatenating messages from different parts of a tree. We use our novel concatenation operation ($3) here to enforce stable training.
5 Experiments
In this section, we present two experiments utilizing HAEGAN: Random Tree Generation ($5.1) and de novo Molecular Generation ($..2) Additionally, a proof-of-concept MNIST generation experiment included in the Appendix E
5.1 Random Tree Generation
Recent studies (Boguná et al., 2010 Krioukov et al. 2010 Sala et al. 2018 Sonthalia & Gilbert 2020) have found that hyperbolic spaces are suitable for tree-like graphs. Thus, we perform an experiment which we use HEAGAN to generate random trees. We compare the performance of HAEGAN with other hyperbolic models. In this experiment, the AE consists of tree encoder and tree decoder. The details of the model architecture is described in Appendix F.2
5.1.1 Experimental Settings
Dataset Our dataset consists of 500 randomly generated trees. Each tree is created by converting a uniformly random Prüfer sequence (Prüfer 1918). The number of nodes in each tree uniformly sampled from [20, 50]. The dataset is randomly split into 400 for training and 100 for testing.
Basclines and Ablations We compare HAEGAN with the following bascline hyperbolic generation meth- ods. HGAN, where we only use hyperbolic Wasserstein GAN without the AE structure, where the tree decoder as the generator and a hyperbolic MLP as the critic. HVAE-w and HVAE-1, where we use the same AE but follow the ELBO loss function used by Mathieu et al. (2019) instead having a GAN ("w" and "r" refer to using wrapped and Riemannian normal distributions, respectively). Although we mainly focus on hyperbolic methods, we also compare with the following Fuclidean generation methods: GraphRNN (You et al. 2018b) and AEGAN, The latter has the same architecture as HAEGAN but all layers and operations are Euclidean.
As discussed in previous sections, our default choice in HAEGAN is to use Lorentz direct Concatenation and fully hyperbolic lincar layers (Chen et al. 2021). We also consider the following ablations HAEGAN: HAEGAN-H, where the fully hyperbolic linear layers are replaced with the tangent linear layers defined by Ganea al. (2018); HAEGAN-B, where the concatenation in HAEGAN is replaced by 3-concatenation (Shimizu et al. 2021 HAEGAN-T, where the concatenation is replaced by the Lorentz tangent concatena- tion.
Metrics We use the following metrics from You et al. 2018b) | Lo evaluate the models: The Maximum Mean Discrepancy (MMD) of degree distribution (Degree), MMD of the orbit counts statistics distribution (Orbit); Average difference of orbit counts statisties (Orbit), Betweenness Centrality (Betweenness). Close- ness Centrality (Closeness). All metries are calculated between the test dataset and 100 samples generated from the models.
9

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Results of the tree generation experiments. "NaN" indicates NaN loss during training.
MMD (x10-))
Average Difference
Time
Model
Concat
HNN
Degree
Orbis
Orbit
Belweenneess
Closencns
(s/step)
HGAN
Direct
Fully
2.0600000100
0.0560:2000ZZ
0.1315=0.008
0.0271
0.0229:30.00 00088
1.517220.0200
HVAE-W
Direct
Fully
NaN
N&N
NaN
NaN
1.6712+000257
HVAE.
Dimet
Fully
NaN
NaN
NaN
NaN
NaN
1.702410.0220
GraphRNN
N/A
N/A
2.6810+0.03522
00000+0.0044
0.1439=0.0033
0.0256+0.0001
0.0221++0000
0.0033=0.0012
AEGAN
N/A
N/A
1.3130+0.0001
0.0500 0,000
0.1 0.12035=00003
0. 0.025770.000
0.0219:40.00M 0007
1187/1000188
HAEGAN-H
Direct
Tangeat
0.743010.01.2
0.010011.00006
0.1382
0. 0.0245.10.00
0.0220110.000
1.85131000234
HABGAN-3
Beta
Fully
0.4700=0:0088
0.0010=#0.0015
0.1299=0.0033
0.026120.0m 0010
0.022110.000 0000
1.7529=0.0192
HAEGAN-T
Tangent
Fully
1.31400+000077
0.0620+0.0023
0.1316+0.0000
0. 0.0242+000012
0.0219++00000
1.6385=0.0181
HABGAN
Direct
Fully
0.1560=t0.0022
0.0050+0.0001
0.1233=00.0000
0.0237=000010
0.0217::0.0008
1.3116=0.0157
5.1.2 Results
Table presents results and runtime of all models. For all metrics, a smaller number implies a better result. Our default choice of HAEGAN (with direct concatenation and fully hyperbolic lincar layers) performs the best across all metrics except the MMD of orbit counts statisties distribution, in which it just marginally particular, the Lorentz direct concatenation generally performs better
falls behind the $-concatenation. and more efficiently than Lorentz tangent concatenation and 8-concatenation. Also, the fully hyperbolic linear layer superior the tangent linear layer in both effectiveness and efficiency. Our results also show the advantage of the overall framework compared with either single GAN VAE. On hand, the performance of HAEGAN is much better than HGAN. On the other hand, note that the hyperbolic VAE-based methods suffer from numerical instability for this simple dataset even when using fully hyperbolic linear layers and direct concatenation, ion, possibly because of the complicated ELBO loss. Finally, we remark that it is clear from the results that the hyperbolic models are better at generating trees than the Euclidean ones.
5.2 De Novo Molecular Generation
It crucial task in machine learning to learn the structure of molecules, which has important appli- cations in the discovery of drugs and protcins (El- ton et al. 2019). Since molecules naturally show graph structure, many recent works use graph neural networks to extract their information and accord- ingly train molecular generators (Simonovsky & Ko- modakis 2018 De Cao Kipl, 20188 Jin el al. 2018)
zG
Molccular Graph
Graph Encoder
Graph Decoder
2019). In particular, .Jin et al. (2018 2019) pro- posed a bi-level representation of molocules where both a junction-tree skeleton and a molecular graph are used to represent the original molecular data. In this way, a molecule is represented in a hierarchical manner with a tree-structured scaffold. Given that hyperbolic spaces can well-embed such hierarchical and troc-structured data (Peng et al., (021), we ex- pect that HAEGAN can leverage the structural in- formation. To validate its effectiveness, in this sec- tion, we test HAEGAN using molecular generative tasks, where the latent distribution is embedded in a hyperbolic manifold.
Junction Tree
Tree
Hyperholic
Tree Decoder
Encoder
Embedding
Figure 6: Illustration of the auto-encoder used in the HAEGAN for molecular generation. The input molec- ular graph is firstly coarsened into the junction tree. Then both of them are encoded using graph and tree encoders to their respective hyperbolic embeddings zT and zç. To reconstruct the molecule, we first decode the junction tree from 27, and then reconstruct the molecular graph using the junction tree and 2g.
In our experiments, we design both a hyperbolic tree AE and a hyperbolic graph AE in our HAEGAN embed the structural information of the atoms in each molecule, illustrated in Figure 6 Specifically, our model takes molecular graph as the input, passes the original graph to the graph encoder and feeds
10

---

Published in Transactions on Machine Learning Research (09/2024)
the corresponding junction tree to the tree encoder, acquiring hyperbolic latent representations 2g of the graph, as well as 27 for the junction tree. Then, the junction tree decoder constructs a tree from 2T autoregressively. Finally, the graph decoder recovers the molecular graph using the generated junction tree and zg- The hyperbolic representation is supposed to better leverage the hicrarchical structure from the junction-tree than the graph neural networks (Jin et al. 2018 2019). We describe the detailed model architecture in Appendix
5.2.1 Experimental Settings
Dataset We train and test our model on the MOSES benchmarking platform (Polykovskiy et al. 2020 which is refined from the ZINC dataset (Sterling & Trwin 2015) and contains about 1.58M training, 176k test, and 176k scaffold lest molecules. The molecules in the scaffold test set have different Bemii-Murcko scaffolds [Bemis & Murcko 1996), which represent the core structures of compounds, than both the training and test sets. They are used to determine whether a model can generate novel molecular scaffolds.
Baselines We compare our model with the following baselines: non-neural models including the Hidden Markov Model (HMM), the N-Gram generative model (NGram) and the combinatorial generator; neu- ral methods including CharRNN (Segler al. 2018); AAE (Kadurin et al.) 2017a b Polykovskiy et al. 2018), VAF (.6mez-Bombarel et al.) 2018 Blaschke et al. 2018), JTVAE (Jin et al.
2018), LatentGAN
Prykhodko et al. 2019), GraphINVENT (Mercado et al.) 2021), DiGress (Vignac et al. 2023), cMolGPPT
Wang et al. 2023). Group SELFIES [Cheng et al. (023). The benchmark results from HMM to LatentGAN
are taken from Polykovskiy et al.
(2020) the results from GraphINVENT to Group SELFIES are taken
from respective papers.
Ablations On one hand, we consider a Euclidean counterpart of HAEGAN, named as AEGAN, to examine whether the hyperbolic setting indeed contributes. The architecture of AEGAN the same as HAEGAN, except that the hyperbolic layers are replaced with Euclidean ones. On the other hand, we also report the following alternative hyperbolic methods: HVAE-w and HVAE-I, where we use the same tree and graph AE but follow the ELBO loss function used by Mathieu et al. (2019) instead of having GAN ("w" and "r" refer to using wrapped and Riemannian normal distributions, respectively); HGAN, where we train end-to-end hyperbolic WGAN with the graph and tree decoder as the generator, and the graph and tree encoder as the critic; HAEGAN-H, HAEGAN-3, and HAEGAN-T as introduced in $5.1
Metrics We briefly describe how the models are evaluated. Detailed descriptions of the following metrics can be found in the MOSES benchmarking platform (Polykovskiy et al.) 2020). We generate a set of 30,000 molecules, which we call the " "generated set". We first report the Validily metrics, which indicates the percentage of valid molecules in the generated set. Then, we also evaluate the following structure-related metrics by comparing the generated set with the test set and the scaffold set: Fréchet ChemNet Distance (FCD), Similarity to Nearest Neighbor (SNN) and the Scaffold similarity (Scaf). FCD is the difference in the distribution of the last layer of ChemNet. SNN is the average Tanimoto similarly (Tanimoto) 1958) between the generated molecule and its nearest neighbor in the reference set. Scaf is cosine distances between the scaffold frequency vectors [Bemis & Murcko 1996) of the generated and reference sets. In particular, SNN compares the detailed structures while Scaf compares the skeleton structures. By considering them with both the test and the scaffold test sets, we measure both the structural similarity to training data and the capability of searching for novel structures.
5.2.2 Results
We report in Table 2 the performance of HAEGAN and the basclines. For each metric described above, we take the mean and standard deviation from three independent samples. We use bold font to highlight the best performing model in each criterion.
First of all, HAEGAN achieves perfect validity scores, which implics the hyperbolic embodding adopted by HAEGAN does not break the rule of molecule structures and does not induce mode collapse. Moreover, our model significantly outperforms the baseline models in the SNN metric. This means that the molecules
11

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: Performance in Validity, FCD, SNN, and Scaf metrics. Reported (mean + std) over three indepen- dent samples. HVAE-W, HVAE-r, HGAN, HAEGAN-H, HAEGAN-B, HAEGAN-T are not included in the table as they all produce "NaN".
FCD (1)
SNN (1)
Seal (t)
Model
Validity (1)
Test
TestSF
Test
TestSF
Test
TestSF
Trase
0.008
0.476
0. 0.6419
0.5859
0.9907
HMM
0.076++00322
24.466-2.5522
25.431 4:2.5599
0.388++00011
0:380++0.011
0.204
0.049-+0.018
NGram
0.238:10.003
0.307
6.231:10.0900
0.521=40.001
0.499-20.001
0.530=#0.00 0.01
0.098::0.014
Combinatorial
1.0:00.0
4.238=0.037
4.511+0.0274
0.451=000001
0.439++0.001
0.445=0.000
0.087:+0.003
CharRNN
0.975:1.002 026
0.073=0.022
0.520=10.0379
0. 0.60240.001
0.565+1.014
0.924=00000 0.006
0,110:10.008 008
AAE
0.937 034
0.2563002003
1.05740. 2375
0.608+10.004
0.055
0.902 0.038
0.079:10.009
VAE
0.977+00001
0.099-:0.0125
0.567
0.626:10.001
0.008
0.939=#0.000 0.002
0.059+000010
JTVAE
1.0.10.0
0:297:1000887
0.828::0.0117
0.548:100008
0.519-10.007
0.896:20.00 004
0,101=000011
LatemGAN
0.897=0.003
0.395+0.02334
0.938+0005531
0.537++0.001
0.513++00001
0.887=0.001
0.107
GraphINVENT
0.964
0.682
1.223
0.569
0.539
0.885
0.127
DiGress
0.857
NA
1.19
NA
0.52
NA
0.148
eMolGPT
0.988
NA
NA
0.619
0.578
NA
NA
Group SELPIES
1.0:0.00
0.0999
0.734:10.0109
0.605:20.004
0.539:10.003
0.96510.002
0.061=:00055
HARGAN (Ours)
1.0::0.0
0.216+000004
0.751+400001
0.658+00.00%
0.613:10.0003
0.897
O. 0.005
ABGAN (Ouns)
1.0.1..0
0.381440.0082
0.869+50.0074
0.450-10.006
0.452-10.006
0.203:0.0004
0.058+10.008
generated by our model have a closer similarity to the reference set. implies that our model better captures the underlying structure of the molecules and our hyperbolic latent space is more suitable for embedding molecules than its Euclidean counterparts. Our model also achieves competitive performance in the Scaf metric when the reference set is the scaffold test set. Given the small FCD, this shows that our model is better at searching on the manifold of scaffolds and can generate examples with novel core structures.
Next, although AEGAN can also achieve very good performance in validity, uniqueness, and novelty, we notice the big margin HAEGAN has over AFGAN in the strueture-related metrics. This suggests that working with the hyperbolic space is necessary in our approach and the hyperbolic space better represents structural information.
Lastly, the alternative hyperbolic models all suffer from numerical instability and training reports NaN. This is not surprising since hyperbolic neural operations are known to casily make training unstable, especially in deep and complex networks. The result reveals the stronger numerical stability of HAEGAN, which highlights the importance of (1) the overall framework of HAEGAN (v.s. HVAE and HGAN); (2) the fully hyperbolic layers (v.s. HAEGAN-H); (3) the Lorentz direct concatenation (v.s. HAEGAN-B,, HAEGAN-T).
6 Conclusion and Limitations
We proposed HAEGAN, hybrid generative framework capable of generating both synthetic hyperbolic data and real molecular data with comparable performance in structure-related metrics. It is an effective hyperbolic generative model that accommodates numerically stable deep architectures. This is attributed to the following: first, the overall hybrid framework; second, the fully hyperbolic operations performed in the Lorentz space; third, the direct concatenation. We expect that HAEGAN can be applied to broader scenarios due to the flexibility designing the hyperbolic AE and the possibility of building deep models.
Despite the promising results, we point out two possible limitations of the current model. First, not all complex modules are directly compatible with HAEGAN. Indeed, if the gated recurrent units (GRU) were used in our molecular generation task, the complex structure of GRU would cause unstable training (we observed NaN in experiments) and that is why a hyperbolic lincar layer is used instead. Nevertheless, we expect defining more efficient hyperbolic operations that incorporate recurrent operations may alleviate the problem and leave it to future work. Second, although the hyperbolic operations in HAEGAN do not require going back and forth between the hyperbolic and the tangent spaces, we neod use exponential maps when sampling from the wrapped normal distribution. We will also work on more efficient ways of sampling from the hyperbolic Gaussian.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Acknowledgement
D.Z. is supported in part by National Natural Science Foundation of China (Grant No. 12301117). E.Q. is supported in part by the Office of Naval Research (ONR) under grant N00014-23-1-2887. We thank Marcus Werner from Duke Kunshan University for helpful discussions.
References
James W Anderson. Hyperbolic geometry. Springer Science & Business Mcdia, 2006.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-223. PMLR, 2017.
Gregor Bachmann, Gary Bécigneul, and Octavian Ganea. Constant curvature graph convolutional networks. In International Conference on Machine Learning, pp. 486 496. PMLR, 2020.
Guy W Bemis and Mark A Murcko. The propertics of known drugs. 1. molocular frameworks. Journal of medicinal chemistry, (9(15) 2887-2893, 1996.
Thomas Blaschke, Marcus Olivecrona, Ola Engkvist, Jürgen Bajorath, and Hongming Chen. Application of generative autoencoder in de novo molecular design. Molecular informatics, 37(1-2):1700123, 2018.
Marián Boguná, Fragkiskos Papadopoulos, and Dmitri Krioukov. Sustaining the internet with hyperbolic
mapping. Nature communications, 1(1):1 8, 2010.
Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton. Latent variable modelling with hyperbolic normalizing flows. In International Conference Machine Learning, pp- 1045-1055, PMLR, 2020.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint orXiv:2104.14/4 2021.
James W Cannon, William J Floyd, Richard Kenyon, Walter R Parry, et al. Hyperbolic geometry. Flavors of geometry, (1(59-115):2, 1997.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovee. Hyperbolic graph convolutional neural net- works. Advances in neural information processing systems, 32:4868 4879, 2019.
Ines Chami, Albert Gu, Dat P Nguyen, and Christopher Ré. Horopca: Hyperbolic dimensionality reduction via horospherical projections. In International Conference on Machine Learning, pp. 1419- 1429.. PMLRR, 2021.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobscn. Residual flows for invertible generative modeling. Advances in Newral Information Processing Systems, 32, 2019.
Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Fully hyperbolic neural networks. arXie preprint arXiv:2105.14686, 2021.
Austin H Cheng, Andy Cai, Santiago Miret, Gustavo Malkomes, Mariano Phielipp, and Alán Aspuru-Guzik. Group selfies: robust fragment-based molecular string representation. Digital Discovery, 2(3):748 758, 2023.
Seunghyuk Cho, Juyong Lee, and Dongwoo Kim. Hyperbolic vae via latent gaussian distributions. Advances in Neural Information Processing Systems, 36, 2024.
Jindou Dai, Yuwei Wu, Zhi Gao, and Yunde Jia. A hyperbolic-to--hperbbii graph convolutional network. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 154-163, 2021a.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Shuyang Dai, Zhe Gan, Yu Cheng, Chenyang Tao, Lawrence Carin, and Jingjing Liu. Apo-vae: Text generation in hyperbolic space. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 416 431, 2021b.
Nicola De Cao and Thomas Kipf. MolGAN: An Implicit Generative Model for Small Molccular Graphs. ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generalive Models, 2018
Daniel C Elton, Zois Boukouvalas, Mark D Fuge, and Peter W Chung, Deep learning for molecular design a review of the state of the art. Molecular Systems Design & Engineering, 4(4):828 849, 2019.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020.
Pengfei Fang, Mehrtash Harandi, and Lars Petersson. Kernel methods in hyperbolic spaces. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10665-10674, 2021.
Xingcheng Fu, Yisen Gao, Yuecen Wei, Qingyun Sun, Hao Peng, Jianxin Li, and Xianxian Li. Hyperbolic geometric latent diffusion model for graph generation. orXiv preprint arXiv:2405.031 2024.
Octavian Ganca, Gary Béeigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural
information processing systems, 31:5345-53555 2018.
Rafael BombarBo,barelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato. Benjamin Sánchez- Laanchez-eeneling, Dennis Sheberla, Jorge Aguilera-Iparrae Iparragurre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous represontation of molecules. ACS central science, 4(2):268 276, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Albert Gu, Frederic Sala, Beliz Gunel, and Christopher Ré. Learning mixed-curvature representations in product spaces. In International Conference on Learning Represenlations, 2019.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (ods.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International Conference on Machine Learning, pp. 2323-2332 PMLR, 2018.
Wengong Jin, Kevin Yang, Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- to-bbth translation for molecule optimization. In International Conference on Learning Representations, 2019.
Artur Kadurin, Alexander Aliper, Andrey Kazemnov, Polina Mamoshina, Quentin Vanhaelen, Kuzma Khrabrov, and Alex Zhavoronkov. The cornucopia of meaningful leads: Applying deep adversarial au- tocncoders for new molecule development in oncology. Oncotarget, 8(7):10883, 2017a.
Artur Kadurin, Sergey Nikolenko, Kuzma Khrabrov, Alex Aliper, and Alex Zhavoronkov. drugan: an advanced generative adversarial autoencoder model for de novo generation of new molecules with desired molecular properties in silico. Molecular pharmaceutics, 14(9):3098 3104, 2017b.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks.
International Conference on Learning Representations (ICT.R), 2017.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Dmitri Krioukov, Fragkiskos Papadopoulos, Maksim Kitsak, Amin Vahdat, and Marián Boguná. Hyperbolic geometry of complex networks. Physical Review E, 82(3):036106, 2010.
Marc Law, Renjie Liao, Jake Snell, and Richard Zemel. Lorentzian distance learning for hyperbolic repre- sentations. In International Conference on Machine Learning, pp. 3672 3681. PMLR, 2019.
Diego Lazcano, Nicolás Fredes Franco, and Werner Creixell. HGAN: Hyperbolic generative adversarial network. IEEE Access, 9:96309 96320, 2021. doi: 10.1109/ACCESS 2021.304472..
Yann LeCun, Corinna Cortes, and CJ Burges. Muist handwritlen digit database. ATT Labs [Online]. Available: http://yane http://yannllecnnccoo coann.cco//eee cenne//////eee //mdb//mist,,..mmmm 2, 2010.
Xuechen Li, Ting-Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp- 3870- 3870- PMLR, 2020.
Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. Advances in Newral Information Processing Systems, 32:8230 8241, 2019.
Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. Differ- entiating through the fréchet mean. In International Conference on Machine Learning, pp. 6393 6403. PMLR, 2020.
Emile Mathieu, Charline Le Lan, Chris J. Maddison, Ryota Tomioka, and Yee Whye Teh. Continuous hier- archical representations with poincaré variational auto-encoders. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d' d'AlchéBBue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Rocío Mercado, Tobias Rastemo, Edvard Lindelöf, Günter Klambauer, Ola Engkvist, Hongming Chen, and Esben Jannik Bjerrum. Graph networks for molecular design. Machine Learning: Science and Technology, 2(2):025,23, 2021.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped normal distri- bution hyperbolic space for gradient- based learning. In International Conference Machine Leorning, pp. 4693-4702 PMLR, 2019.
Maximilian Nickel and Douwe Kiela. Learning contimuous hierarchies in the lorentz model of hyperbolic geometry. In International Conference on Machine Learning, pp. 3779-3788. PMLR, 2018.
W. Peng, T. Varanka, A. Mostafa, H. Shi, and G. Zhao. Hyperbolic deep neural networks: A survey. IEEE Transactions on Pallern Analysis & Machine Intelligence, December 2021. doi: 10.1109/ 202M 3136921.
Wei Peng, Jingang Shi, Zhaoqiang Xia, and Guoying Zhao Mix dimension in poincaré geometry for 3d skeleton-based action recoguition. In Proceedings of the 28th ACM International Conference on Multime- dia, pp. 1432- 1440, 2020.
Daniil Polykovskiy, Alexander Zhebrak, Dmitry Vetrov, Yan Ivanenkov, Vladimir Aladinskiy, Polina Mamoshina, Marine Bozdaganyan, Alexander Aliper, Alex Zhavoronkov, and Artur Kadurin. Entan- gled conditional adversarial autoencoder for de novo drug discovery. Molecular pharmaceuties, 15(10):
4398-4405, 2018.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengelnn Sergey Golovanov, Oktai Tatanov, Stanislav Belyacv, Rauf Kurbanov, Alekscy Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Moloc- ular sets (moses): a benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:1931, 2020.
Heinz Prüfer. Neuer beweis eines satzes üiber permutationen. Arch. Math. Phys, 27(1918):742 744, 1918.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Oleksii Prykhodko, Simon Viet Johansson, Panagiotis-Chrie Chriisoss Kotsias, Josep Arús-Pous, Esben Jannik Bjerrum, Ola Engkvist, and Hongming Chen. A de novo molecular generation method using latent vector based generative adversarial network. Journal of Cheminformatics, 11(1):1-11) 13, 2019.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Aduances in neural information processing systems, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esscr, and Björn Ommer. High-resolution image synthesis with latent diffussion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognation, pp. 10684-10695, 2022.
Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman. Moser flow:
Divergence-based genera-
tive modeling on manifolds. Advances in Neural Information Processing Systems, 34, 2021.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time scries. Advances in neural information processing systems, 32, 2019.
Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. Representation tradcoffs for hyperbolic embed- dings. In International conference on machine learning, pp. 4460 4469. PMLR, 2018
Marwin H. S. Segler, Thierry Kogej, Christian Tyrchan, and Mark P. Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Central Science, 4(1):120- 131, 2018. doi: (0.1021/acscentsc) 7b00001 7600512.
Chence Shi*, Minkai Xu*, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. GraphAF: flow- based autoregrossive model for molecular graph generation. In International Conference Learning Represenlations (ICLR), 2020.
Ryohei Shimizu, YUSUKE Mukuta, and Tatsuya Harada. Hyperbolic neural networks +. In International Conference on Learning Representations, 2021.
Martin Simonovsky and Nikos Komodakis. GraphVAE: Towards generation of small graphs using variational autoencoders. In International conference on artificial neural networks, pp. Springer, 2018.
Rishi Southalia and Anna Gilbert. Tree! am no tree! i am low dimensional hyperbolic embedding. Advances in Newral Information Processing Systems, 33:845-856, 2020.
Teague Sterling and John Irwin. Zinc 15 - ligand discovery for everyone. Journal of Chemical Information and Modeling, 55(1)):2324 2337, 2015. doi:
5b00559. PMID: 26479676.
Li Sun, Zhongbao Zhang, Jiawei Zhang, Feiyang Wang, Hao Peng, Sen Su, and Philip S Yu. Hyperbolic variational graph neural network for modeling dynamic graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 0175, 2021.
Taffee T Tanimoto, Elementary mathematical theory of classification and prediction. 1958.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Clément Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Di- gress: Discrete denoising diffusion for graph generation. In Proceedings of the 11th International Conference on Learning Representations, 2023.
Códric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 5337-5344, 2019. Xiao Wang, Yiding Zhang, and Chuan Shi. Hyperbolic heterogencous information network embedding. In
Ye Wang, Honggang Zhao, Simone Sciabola, and Wenlu Wang. cmolgpt: A conditional generative pre-trained transformer for target-speeific de novo molecular generation. Molecules, 28(11 (8(11),4430, 2023.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Lingfeng Wen, Xuan Tang, Mingjie Ouyang, Xiangxiang Shen, Jian Yang, Daxin Zhu, Mingsong Chen, and Xian Wei. Hyperbolic graph diffusion model. In Proceedings of the AAAT Conference on Arlificial Intelligence, volume 38, pp. 15823 15831, 2024.
Zhenxing Wu, Dejun Jiang, Chang-Yu Hsieh, Guangyong Chen, Ben Liao, Dongsheng Cao, and Tingjun Hou. Hyperbolic relational graph convolution networks plus: simple but highly efficient qsar-modeling method. Briesings in Bioinformalics, 22(5):bbab112, 2021.
Menglin Yang, Min Zhou, Zhihao Li, Jiahong Liu, Lujia Pan, Hui Xiong, and Irwin King. Hyperbolic graph neural networks: A review of methods and applications. arXiv preprinl arXie:2202 20n:222221385,, 2022.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018a.
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. In International conference on machine learning, pp. 5708 5717. PMLR, 2018b.
Ke Yu, Shyam Visweswaran, and Kayhan Batmanghelich. Semi-supervised hierarchical drug embedding in hyperbolic space. Journal of chemical information and modeling, (0(12):5647-5657 2020.
Tao Yu and Christopher M De Sa. Numerically accurate hyperbolic embeddings using tiling- based models. Advances in Newral Information Processing Systems, 32, 2019.
Tao Yu and Christopher M De Sa. Representing hyperbolic space accurately using multi-component floats. Advances in Neural Information Processing Systems, 34:15570 15581, 2021.
Yiding Zhang, Xiao Wang, Chuan Shi, Xunqiang Jiang, and Yanfang Fanny Ye. Hyperbolic graph attention network. IEEE Transactions on Big Data, 2021.
17

---

Published in Transactions on Machine Learning Research (09/2024)
A Preliminaries
A.1 Hyperbolic Geometry
We describe some fundamental concepts in hyperbolic geometry related to this work.
The Lorentz Model The Lorentz model L (C, g) of an n dimensional hyperbolic space with constant
negative curvature K an n--dimensional manifold C embedded in the (n+ -iddeeniion Minkowski space,
together with the Riemannian metric tensor g = (i where 1, denotes the n-dimensional vector
whose entries all I's. Every point in Lx is represented by [is)
It
2t 0, R and satisfies
(x 1/K, where ) is the Lorentz inner product induced by g:
ye == x gy = - Jtit + t., Vo: 2, y E LR
(14)
Geodesics and Distances Geodesics are shortest paths in manifold, which generalize the notion of "straight lines" in Euclidean geometry. In particular, the length of a geodesic in L (the "distance") between x, E Lik is given by
1 cosh -1 "(K(x, (K) V-K
dc(x,
(15)
Tangent Space For each point æ E LK the tangent space at x is TaL = {y R (g. x) 0}. It is a first order approximation of the hyperbolic manifold around a point and is subspace of R=+1, We denote ||a||c V (v. 2)2 as the norm of v E TaLk
Exponential and Logarithmic Maps The exponential and logarithmic maps are maps between hy- perbolic spaces and their tangent spaces. For æ, y Li and v TeLk, the exponential map exp (v) La I maps tangent vectors to hyperbolic spaces by assigning to the point exp K (v) ((1), where
is the geodesic satislying 7(0) x and 7((0) Specifically,
exp (v) = cosh(() sinh (0) _ -. b= V-Kllollc.
(16)
The logarithmic map log (y) Lb
Tallikk is the inverse map that satisfies loy (ex) v. Specifically,
cosh (1b K 1y d calle 1 ((x,y)c
the
log (3)
(17)
Parallel Transport For two points v, Lix the parallel transport from to y defines map PTK which "transports" a vector from Tallik to TyT.ik along the geodesic from æ to y. Parallel transport preserves the metric, ie. Vu, U E TaLi, (PT (v), PT (m)) (v. u) In particular, the parallel transport in T.K is given by
(y,v) v)c
P P (v) -1/K
(ar y). (x, y)
(18)
A.2
Additional Hyperbolic Neural Operations
A.2.1 Hyperbolic Aggregation Operations
In hyperbolic space, the 'average' operation not well-defined, i.c. the Euclidean mean does not guarantee the result still in hyperbolic space. There are two generalizations of Euclidean mean in hyperbolic space, the Fréchet Mean and the Hyperbolic Centroid.
18

---

Published in Transactions on Machine Learning Research (09/2024)
Fréchet Mean Fréchet Mean is defined as the direct generalization of Euclidean mean for any metric space. In hyperbolic space, it can be defined as:
Ho = arg min
Σ uid (a,, 1 HELK
(19)
However, the Fréchet mean does not have closed form solution in hyperbolic space. There are works for
calculating the hyperbolic Fréchet mean iteratively (Lou et al. 2020 Gu et al. 2019). but they were quite inefficient and hard to pass gradients.
Hyperbolic Centroid The notion of a centroid is extended to L by Law et al. (2019), defined to be the point fx that minimizes a weighted sum of the squared Lorentzian distance:
arg min μ
Σ
(20)
MELK
where (xx, J/N is the set of points to aggregate, v is the weight vector whose entries satisfy 14 2 0, E, 0, = 1, N. The squared Lorentzian distance is defined as
(x y) = o a112 = 2/K 2/xx, y)
(21)
It satisfies all the axioms of distance metric except the triangle inequality. Thus, it is a Fréchet mean of pscudo-hyperbolic space. Fortunately, this centroid has closed form solution given by
( 10"
Di=1 K4
(22)
V-K
Σ D 12
A.2.2 Hyperbolic Graph Neural Network
Given the hyperbolic linear layers and hyperbolic aggregation, it is straightforward to generalize the Euclidean
Graph Convolutional wetworks (Kipl & Welling 2017) to hyperbolic space (Chami al. 2019 Chen et al. 2021):
_ (IGCNN(((-1))
(ICent((HLineee)) a.d (w!! 1)
(23)
where x is the feature of node v in layer I, di denotes the dimensionality of layer I, and N (v) is the set of neighbor points of node v.
Since hyperbolic space is better at embedding hierarchal data, graphs with Gromov hyperbolicity å (Sonthalia & Gilbert 2020) similar that of hyperboloid can we naturally embedded in hyperbolic space.
A.2.3 HyperbolicEEucliden Conversion Layers
Sometimes, it is necessary to find a learnable way to convert data between hyperbolic and Euclidean space. In this section, we introduce some learnable layers to convert between them.
Euclidean to Hyperbolic It is possible that dataset is originally represented as Euclidean, albeit having hicrarchical structure. In this case, the most obvious way of processing is to view the data in the tangent space and use the exponential or logarithmic maps to transform it to hyperbolic space. In order to map E _ to the hyperbolic space LØ, Nagano et al. (2019) add zero padding to the front of t to make it a vector in ToLk, and then apply the exponential map. This Euclidean to Hypertolic (E2H) operation was originally used for sampling, but can also be generally used to map from the Euclidean to the hyperbolic spaces. Specifically,
y E2H1 (t) expo ((P)),
(24)
where o V -1/K, K, O,...,,. 00 is the hyperbolic origin.
19

---

Published in Transactions on Machine Learning Research (09/2024)
For better expressivity, especially when the imput R" is one-hot, can first map the imput to hidden embedding h Wn E R with trainable embedding matrix W E [R"xm, Then, it is mapped to hyperbolic space by the E2H operation defined as above. That is,
y
(x)) = E2H (Wx)
(25)
This layer previously used by Nagano et al. 2019) for word embedding.
Hyperbolic to Euclidean Some machine learning tasks requires Euclidean output (such as classification) thus necessary to construct a layer to transform Hyperbolic features to Euclidean. It is possible to directly use logarithmic map to convert hyperbolic features to the tangent space and view that as Euclidean features. However, is hard pass gradients through logarithmic map. Another way extract Euclidean features is by calculating distances, since distances intrinsically Euclidean.
The Hyperbolic Centroid Distance Layer (Liu et al. 2019 is numerically stable way of converting Hyper- bolic features to Euclidean. It maps points from L to R". Given an input E LK it first initializes m trainable centroids {cr Li, then produces a vector of distances
HCDist, (z) = [d ci) ...d (a, cm))
(26)
20

---

Published in Transactions on Machine Learning Research (09/2024)
B
Related Works
Machine Learning in Hyperbolic Spaces A central topic in machine learning is to find methods and architectures that incorporate the geometric structure of data (Bronstein et al. 2021). Due to the data representation capacity of the hyperbolic space, many machine learning methods have been designed for hyperbolic data. Such methods include hyperbolic dimensionality reduction (Chami et al. 2021 and kernel hyperbolic methods (Fang et al. 2021). Besides these works, deep neural networks have also been proposod in the hyperbolic domain. One of the earliest such models was the Hyperbolic Neural Network (Ganea et al., 2018) which works with the Poincaré ball model of the hyperbolic space. This was recently refined in the Hyperbolic Neural Network ++ (Shimizu et al., 2021). Another popular choice is to use the Lorentz model of the hyperbolic space (Chen et al. 2021 Yang et al. 20220. Our model also uses Lorentz space for mumerical stability.
Hyperbolic Graph Neural Networks Graph neural networks (GNNs) are successful models for learning
representations of graph data. Recent studies (Boguná et al.] 2010 Krioukov et al., 2010 Sala et al.
2018
Sonthalia & Gilbert 2020) have found that hyperbolic spaces are suitable for tree-
like graphs and variety
of hyperbolic GNNs [Chami et al.] 2019 Liu et al.. 2019
Bachmann et al. 2020
Dai et al.] 2021a Chen
et al. 2021 have been proposed. In particular, Chami et al.
(219 Lou et al.
(011 Bachmann et a.. 2020
all performed message passing, the fundamental operation in GNNs, in the tangent space of the hyperbolic
space. On the other hand, Dai et al. (2021a); Chen et al. p2021) designed fully hyperbolic operations so that message passing can be done completely in the hyperbolic space. Some recent works address special GNNs, For instance, (Sun et al. (2021) applied hyperbolic time embedding to temporal GNN, while (Zhang et al. (2021) designed hyperbolic graph attention network. We also notice the recent survey on hyperbolic GNNs by Vang et. al. (2022)
Hyperbolic Generative Models Generative neural networks in the Euclidean domain cannot embed information of the hyperbolic geometry. To address that, Nagano et al. (2019) designs a wrapped normal distribution in the hyperbolic space which enables taking gradient and uses it as the latent distribution of a variational autoencoder (VAE). Mathicu et al. (2019) considers both the wrapped normal distribution and maximum entropy normal distribution and uses them to construct an VAE on the Poincaré space. [Dai et al. (2021b) also builds a VAE in the Poincaré space, which uses the primal-dual formulation of the Kull- backLeibler (KL) divergence. Cho et al. (2024) proposes a mpeudo-Gaussinn manifold normal distribution for the latent space of VAE. Other than using VAE, other generation frameworks are also adapted to the hyperbolic space. For instance, Bose et al. (200)) lifts normalizing flows on the tangent plane of the hyperbolic space for generation; while Wen et a.. (002);; Fu et al. (002)) adapts diffusion models to the hyperbolic space. Lazcano el al. (2021) uses hyperbolic linear layers in GAN for image generation. Despite the importance of the GAN framework, we are unaware of other hyperbolic GAN models. Our proposed model contains a hyperbolic encoder-dccoder to learn the graph-to-graph mapping, as well as hyperbolic GAN for generating latent embeddings, whose generalor uses the wrapped normal distribution as input.
Molecular Generation Stato-of-tho-art methods for molecular generation usually treat molocules as ab- stract graphs whose nodes represent atoms and edges represent chemical bonds. Early methods for molecular graph generations usually generate adjacency matrices via simple multilayer perceptrons (Simonovsky & Ko-
modakis 2018 De Cao & Kipf 2018). Jiin et al. (2018 2019) proposed to treat a molecule as a multiresolution representation, with a junction tree scaffold, whose nodes represent valid molccular substructures. Other
molecular graph generation methods include You et al. (2018a); [hii* et al. (2020); Mercado et al.] (2021]
Vignac et al. (2023); Cheng et al. (002)). Methods that work with SMILES (Simplified Molecular Input
Line Entry System) notations instead of graphs include include[Segler et al.
(2018):
:
et al. 2018
Blaschke et al. (2018):
; Kadurin et al. (2017a) b): Polykovskiy et al. 2018
; Prykhodko et al. (2019) : Wang
et al. (2023). In particular, Prykhodko et al. (2019) proposed an AE-GAN
architecture which is similar to
ours, though they did not use the graph structure of the molecules. Since the hyperbolic space is promising
for tree-like structures, hyperbolic GNNs have also been recently used for molecular generation LLu et al.
2019 Dai et al. 2021a).
21

---

Published in Transactions on Machine Learning Research (09/2024)
C Additional Analysis of Concatenation
C.1 Proof of Theorem 3.1
Proof. 1. First, consider the case where y* is one the spatial components of y. According to (6) y" is
a copy an entry in æ;, for some E .1,,..,,N, Therefore, if i j. dyy /0x is a zero vector; = J. Oy" / a one-hot vector. In both cases, |J0y" /0m;, 1.
Next, consider the case where y¹ Fj
1/8 is the time component of y. According ( Oy" /0x,
and thus 8yy
|
S 1 since the curvature K < 0.
Liml | 1| R
s R
We conclude that |10y* /0x, || s 1 for any entry y" in y.
2. Write == [2,,0 2T_
N. According to the definition of the Lorentz tangent concatenation as well
as formulas [6] and 17 for E {1,-.. .N,, the p-th entry of the vector 21, can be written in the following concrete form:
Sinh(A) cosh -1
K V
| 11
Tl.p
2,
V-K K
sinh(A) A sinh -1
(v
K22 || |²_ "-K"
31.P
(27)
V-K
|0
where
Σ sinh -1 (V K32 || 11?
N
1/2
A V-K K
K -
(28)
Let j + Disserentiating 2I with respect to Kj the k-th entry of æjj. yields
0ZL.P Dz..k
cooh((A)
sinh(A)
(VX²) [ 11 K -
Ci
sinh-'
A
A
1
Tj,k
(29)
V /1,
V | 11?
1+K K2
K
where
G = sinh (VK2 |
[2221,0,,11 11- -1
F1.p | |I
(30)
does not depend on Ty,,
Arbitrarily take fixed Hs, for all with particularly 2j.6 > O. Also arbitrarily take fixed ni,9 for q # p.
We claim that 02, Dzi,.
as Fl.g 00.
To prove the claim, first note that
8C1 Dzi.p
| 112
V ||,,,,
K V
1+K R*
sinh
K23
(K21112,1 112 K- -
(31)
|
[
Since 22. < ||,,,11 and sinh ((V'" K2l wi, 2 PP---- K-1
O, the second term in (31) is positive. Con- sequently, 8C1/0 1m and thus Ct is positive term that increases with Ft Moreover,
le,,
22

---

Published in Transactions on Machine Learning Research (09/2024)
sinh-1 - | 112 1/K
Ji.k
in (29) is fixed positive term that does
V1zy, 2-2
K²
cosh(A)
(A)
(inh(A)
not depend on Fi. Hence, we only need to show that
Ji, + Since A -
A
3
Tt, oo, this reduces to proving f(t) cosh(() once we write explicitly that
s(nh(()
as 00, which is an immediate result
12
f
e
e-
(t
(t-1)) 1)ee
((t)
+
+
(32)
13
12
13
We conclude that for any 0, there exist {a, |i and an entry =² of 2, i.e. 21, above, for which all entries of 0z /0x, are no less than M. Thus it also holds that ||0>* 10x le xxl ll M.
C.2 Effect on Hyperbolic Distances
In this section, we perform additional analysis of Lorentz direct concatenation and Lorentz tangent concate- nation, particularly their effect on hyperbolic distances.
Distances to Origin First, we study the hyperbolic distances to the hyperbolic origin for both con- catenation methods. Suppose we have x E Lg and y E Lg. Let x = HCat(æ, y) E Lg x+-m- and z' (x, y) E T 1- be their hyperbolic direct concatenation and hyperbolic tangent concate- nation, respectively. We compare the difference between dz(z, o) and dg(2', o) as follows. Note that the distance between an arbitrary point a Li and the origin only depend on the time component:
dc(2,
V-K cosh '(K (1,(x,))) 0)
(33)
cosh -1 (-KKxt)
(34)
K
Hence, the distance information is completely contained the time component. After the concatenation, the time component is V23 + vi + 1/K. Consequently, for Lorentz direct concatenation, the distance is
1 V-K cosh
dc(z,0)=
1/K)
(35)
For Lorentz tangent. concatenation, since both the logarithmic and exponential maps reserve the distances, one has
de(z',0) Vdelæ, o)2 ddcy. o)2
(36)
(cosh 2 + cosh ( ((-Ky:))
Although the hyperbolic distance dz(z,, o) not the squared sum of dz o) and dc(y, de(2, o) is larger than each of dz(æ o) and dz(y, o). On the other hand, after concatenation, dz o) (z, d o). This relation agrees with the Euclidean concatenation. However, norm- corm-preservation is not why concatenation works in the Euclidean domain. Therefore, we don't consider this concatenation. The Lorentz direct concatenation is more efficient and stable, and no information is lost during concatenation. Therefore, it is still preferred as a neural layer.
advantage of the Lorentz langent
Relative Distances More importantly, we study how concatenation changes the relative distances, which is closely related to stability. Specifically, we perform the following experiments. Given æ, y. E L let x, = HCat(x,c) be the direct-coneatenatee version of x and c, while ye = HCat(y, e) be the direct- concatenated version of y and c. Similarly we denote x' HTCat(a e) and y' HTCat(y, c). Since the same vector e is attached æ and y, we naturally hope dz (xc, ye) and de(x's v.) do not deviate much from de (c(ar))) y)
23

---

Published in Transactions on Machine Learning Research (09/2024)
Direct Tangent
0.7
Direct
0.8
Tangent
Frequency
Frequeney
0.5
0.4
0.4
0.3
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0
4
Distance
Distance
Figure 7: Difference between concatenated distances and original distances with n = 3. Left: spatial normal. Right: wrapped normal.
1.75
Dinect Tangent
0.8
Direct
0.7-
Tangent
1.50
Frequency
1.25
Frequenes
1.00
0.75
0.3
0.50
0,2
0.25
0.1
0.00
0,0
0.5
1.0
1.5
2.0
2,5
0.0
0
2
3
6
Distance
Distance
Figure 8: Difference between concatenated distances and original distances with n 16. Left: spatial normal. Right: wrapped normal.
We describe our experiments as follows. Take K 1. We randomly sample three points independently from Lg as æ, y and e respectively. We have two scenarios for sampling the points: (1) "spatial normal": the points are sampled so that their spatial components follow the standard normal distribution; (2) "wrapped normal": the points are sampled from the wrapped normal distribution with unit variance. In each scenario, for E {3, 16, 64), we do the experiments for 10,000 times. We report the distances [dc( dz(æ, y)
and [dc(=',, v/.) dz( y) in Figures 7 as well as their differences in Figure 10
Our experiments clearly show that, especially for large dimensions, the distance between de (Tes Yc) and dz(æ, y) is smaller than the distance between de(3', v') and de(æ,y). In particular, in many cases, [dc( yo) dz y)1 is around zero. On the other hand, (dc( v') dc(æ, y) tend to be large when = 16, 64, especially when samples follow the wrapped normal distribution. From this result, the Lorentz Direct Concatenation should be preferred to the Lorentz Tangent Concatenation. In particular, the signifi- cant expansion of distance when concatenating with the same vector, in the case n = 64, may be one cause of numerical instability.
24

---

Published in Transactions on Machine Learning Research (09/2024)
3.5
Direct Tangcot
Direct
Tangent
3.0
0,7
Frequeney
2.5
Fregueney
0.6
0.5
0.4
1.5
1.0
0.5-
0.1
0.0
0.5
1.0
1.5
2.0
8
Distance
Distance
Figure Difference between concatenated distances and original distances with n 64. Left: spatial normal, Right: wrapped normal.
2.5
Dim-64
Dim-64
Dim-16
0,8
Dim-16
Frequency
Frequeney
1.0
0.4
0.5
0.2
0.0
-1.5
-1.0
-0.5
0.0
1.0
0.0
Difference
Difference
Figure Mustration of Idc(xee ye) - dc(x,, (2, Idc(x's y/) dc(a,, 2))- Left: spatial normal. Right: wrapped normal.
25

---

Published in Transactions on Machine Learning Research (09/2024)
D Additional Details in Hyperbolic GAN
D.1 Proof of Proposition 4.1
Proof.
For the optimal solution f", we have
P(a (M(v) - (x)) = de(y, 20) = 1.
(37)
Let (t) = f (xt) - fr (z), 0 S t, < 1. Following Guirajani et al. (2017). it is clear that is (x, y)- Lipschitz, and (x)) - (x) = (0))= = idc(æ, y), I (x.) (x) ldc( y) = (x) + llwellc..
Let u, و.(..y) T be the unit speed directional vector the geodesic point æs. Let a 1- L be a different curve with a(0) 2y and a a'(0) u4. Note that 71(1) dc(æ, y)a (0). Therefore,
(++ dc(x, )
lim (h) = lim R-0
lim D h-+
(38)
h
The directional derivative can be thus calculated as
Vu, (x:)
f (a(h)) 5f (a(0))
dr l (a(())
lim
===
h
h
f" Eet aztens
f (xx))
lim
(39)
h-+0
f (x)) + (t + (c(ma)) )d (x, 3) f (x) -tdc(x,)))
lim
h
lim h-+0 h
1.
Since f* 1-Lipschitz, we have | [x:))l 1. This implics
1 12 [+)(
(uz, Vf" (zx)) IV (x;) (u, Vf" (xx) urll
(40)
= VV, (z:) + V (x:) - -u f (x:)
=1+ (x:) 411
(x1) 2
Therefore, we have 1 = 1+ [V (xx) uell 11. vf (z:) uc This yields Vf" (xx) dm(.))))
26

---

Published in Transactions on Machine Learning Research (09/2024)
100
Generalor Loss
75
Critic Loss
-50
-100
Loss
Loss
0-
150
-25
200
250
Generstor Loss
-300
Critic Loss
100
-350
20
Epochs
Epochs
Figure 11: The training loss of generator and critic in the MNIST generation task. Left: hyperbolic Wasser-
stein GAN. Right: Euclidean Wasserstein GAN.
4
7
0
9
2
8
5
9
6
/
3
9
8
5
0
7
3
4
0
7
0
0
8
6
7
0
9
8
4
Figure 12: MNIST samples generated from HAEGAN.
E MNIST Generation Experiment
We train a HAEGAN with the MNIST dataset (LeCun et al., 2010 This serves as an important sanity check given the promise of using hyperbolic models for vision tasks (Mathieu et al., 2019 Nagano et al. 2019
Bose et al. 2020 We obverse faithfully generated examples.
E.1
Experimental Settings
Architecture In the HAEGAN for generating MNIST, the encoder of the AE consists of three convo- lutional layers, followed by an E2H layer and three hyperbolic linear layers, while the decoder consists of three hyperbolic linear layers, a logarithmic map to the Fuclidean space, and three deconvolutional layers. We remark that in our implementation, we use the LipSwish activation function (Chen et al. 2019 in the HLinear layers for its smooth second order gradients and better Lipschitz property.
Dataset The MNIST (Leunn et al. 2010 (Modified National Institute of Standards and Technology) dataset is widely used and well-known database of handwritten digits. It contains 60,000 training images and 10,000 test images of handwritten digits, 0 through 9. The images in the MNIST database are 28x28 pixels in size and are grayscale images.
Training We describe the training procedures as follows. Firstly, we normalize the MNIST datasct and train the AE by minimizing the reconstruction loss. Secondly, we use the encoder to embed the MNIST in hyperbolic space and train the hyperbolic GAN with the hyperbolic embedding. Finally, we sample a hyperbolic embedding using the generator and use it to produce an image by applying the decoder.
The training curves of the hyperbolic GAN of HAEGAN in the MNIST generation task are shown in Figure which compare with an Fuclidean Wasserstein GAN. The critic loss includes the gradient penalty
27

---

Published in Transactions on Machine Learning Research (09/2024)
Table 3: Quantitative comparison between HAEGAN and other methods in the MNIST generation task, Log Likelihood (+sttd) for different embedding dimensions is reported. 5000 samples to estimate the log- likelihood across all models. indicates the results are replicated with ours setting with the original source code. NaN indicates numerically unstable settings.
Dimensionnlltt
Model
5
20
N- VAE (Mathieu et al. P-VAE (Wrapped) (Mathieu al.
2019
14.450=:0.40
-114.70=:.100
-100.20:10.10
-97.60=
2019
-143.80:0.60
-114.70=0.10
-100.00=0.100
-97.10=t0100
P-VAE (Riemannian) (Mathieu et 8.
2019
142.50=+0.40
-114.10=0.20
-99.70=0.10
-97.00=
Vanilla VAE (Naaao et al.
2019
-10..12+0.53
-114.03+0.47
-99.52+0.36
-97.54++028
Hyperbolic VAE E. NNaaan et 2019
-1.9.05+0..57
-114.25+00.6
-99.42+0.35
-97.32-+0.29
N-VAE
Bose et al.
2020
-139.42::0.0
-115.6310.24
-100.02+0..7
-97.53+0.36
Bose al.
2020
NaN
-113.72+0.91
-99.8220.23
-97.25
WC Bose et al.
2020
-144
-115.244+.66
-98.73+0.38
-97.05=+0.27
TC
Bose et al.
2020
NaN
-112.55+0.29
-99.35+0.28
-97.12+0.22
WHIC
Bose
2020
-137.23+1.97
-112.83+0.54
-90.45+0.26
-97.03++0.17
AEGAN (Ours)
-140.13:0..11
-110.62::0.41
-97.93
-04.19+L0.53
IIAEGAN (Ours)
0..8.5440.25
-110.1740.52
-98.03+0.35
-96.62+:0.11
Table 4: Quantitative comparison between HAEGAN and HGAN Lazcano et al. 2021 in the MNIST
generation task. Fréchet inception distance (+sld) reported. Results for (Lazcano el al. directly from the paper.
2021 are taken
Model
FID
HCAN Lazcano
54.95
HWGAN
Lazcano et a.
2021
12.50
HCGAN
Lazcano et
12.43
HARGAN (Our)
8.01 8.01++0.22
term. We observe that the trend of loss in the hyperbolic GAN is similar to the Euclidean one (both the generator and critic) and no instability from the hyperbolic model.
E.2 Results
We present some generated samples in Figure 12 In Table and Table we report the quantitative results
for the MNIST generation task. First,, we compare the nogative log-likeling lig-likll (NLL) results between our
method and hyperbolic VAEs (Mathieu et al. 2019 Nagano et al. 2019 Bose et al. 2020 NLL estimated
with 5000 samples across all baselines and our model. The results of Mathieu al. (2019) directly taken
from the paper, and the results of Nagano et al. (2019) ; Bose al. (2020) is reproduced by us to ensure fair
comparison. Then, also calculate the FID and compare it with HGAN (Lazcano et al. 2021 Our NLL
results are comparable with the hyperbolic VA while FID is slightly better than HGAN.
28

---

Published in Transactions on Machine Learning Research (09/2024)
F Additional Details on Experiments
F.1
Experiment Details for MNIST Generation
We describe the detailed architecture for the MNIST Generation experiment.
F.1.1
Architecture Details of auto-Encoder
Encoder
Input: MNIST image with dimension (28 28)
Convolutional Neural Network Encoder
Convolutional layer
Input channel:
Output channel: 8
Kernel Size: 3
Stride: 2
Padding:
LipSwish
Convolutional layer
Input channel: 8
Output channel: 16
Kernel Size:
Stride: 2
Padding:
Batch normalization layer (16)
LipSwish
Convolutional layer
Input channel: 16
Output channel: 32
Kernel Size: 3
Stride: 2
Padding:
Batch normalization layer (32)
LipSwish
Convolutional layer
Input channel: 32
Output channel: 64
Kernel Size: 3
Stride: 2
Padding:
Map to hyperbolic space: R&4 + 64
Hyperbolic linear layers:
Input dimension: 64
Hidden dimension: 64
Depth:
Output dimension: 64
Output: hyperbolic embeddings in T
29

---

Published in Transactions on Machine Learning Research (09/2024)
Decoder
Input: hyperbolic embeddings in _
Hyperbolic linear layers:
Input dimension: 64
Hidden dimension: 64
Depth:
Output dimension: 64
Map to Euclidean space: Lg + R64
Transposed Convolutional Neural Network Decoder
Transposed Convolutional layer
Input channel: 64
Output channel: 64
Kernel Size: 3
Stride:
Padding:
Output Padding: 0
Batch normalization layer (64)
LipSwish
Transposed Convolutional layer
Input channel: 64
Output channel: 32
Kernel Size:
Stride: 2
Padding: 0
Output Padding: 0
Batch normalization layer (32)
LipSwish
Transposed Convolutional layer
Input channel: 32
Output channel: 16
Kernel Size: 3
Stride: 2
Padding:
Output Padding: 1
Batch normalization layer (16)
LipSwish
Transposed Convolutional layer
Input channel: 16
Output channel:
Kernel Size:
Stride: 2
Padding:
Output Padding: 1
Output: Reconstructed MNIST image with dimension (28 28)
30

---

Published in Transactions on Machine Learning Research (09/2024)
Hyperparameters
Manifold curvature: K 1.0
For all hyperbolic linear layers:
Dropout: 0.0
Use bias: True
Optimizer: Riemannian Adam (31 0.9, 32 0.9)
Learning Rate: le-4
Batch size: 32
Number of epochs: 20
F.1.2 Architecture Details of Hyperbolic Generative Adversarial Network
Generator
Input: points _ sampled from Ç(o, diag((12)) (1128))
Hyperbolic lincar layers:
Input dimension: 128
Hidden dimension: 64
Depth:
Output dimension: 64
Output: points in IL
Critic
Input: points in L
Hyperbolic linear layers:
Input dimension: 3
Hidden dimension: 64
Depth: 3
Output dimension:
Hyperbolic centroid distance layer: L + IR
Output: score in R
Hyperparameters
Manifold curvature: K = -1.0
Gradient penalty coefficient: À = 10
For all hyperbolic linear layers:
Dropout: 0.0
Use bias: True
Optimizer: Riemannian Adam (01 0, 0.9)
31

---

Published in Transactions on Machine Learning Research (09/2024)
Learning Rate: le-4
Batch size: 64
Number of epochs: 20
Gradient penalty A: 10
F.2
Experiment Details for Random Tree Generation
We describe the detailed architecture and settings for the tree generation experiments.
F.2.1
Model Architecture
Tree Encoder The tree encoder for the random tree T contains the following layers. First, each node feature Ey (which is the zero vector in the random trees) is mapped to the hyperbolic space via
a
E2Har, (Ir.).
(41)
Next, the hyperbolic feature is passed to hyperbolic GCN with ly layers
x ( = HGCN(a<(-1)), 1= 11,....m
(42)
Finally, we take the centroid of the embeddings of all vertices to get the hyperbolic embedding zg of the entire tree,
2y HC)nt(z(()))
(43)
Tree Decoder We adopted the tree decoder from Jin et al. 2018 2019 The tree T = (Vr, ET) is generated using a tree recurrent neural network in a top-down and node-by-node fashion. The generation process resembles a depth-first. traversal over the tree Staring from the rool, at each time step 4, the model makes a decision whether to continue generating a child node or backtracking its parent node. If it decides to generate a node, it will further predict the cluster label of the child node. It makes these decision based the messages passed from the neighboring node. We remark that do not the gated recurrent unit (GRU) for message passing. The complex structure of GRU would make the training process numerically unstable for our hyperbolic neural network (we observed NaN in our experiments). We simply replace it with a hyperbolic linear layer.
Message Passing Let É = ((11,1))....... im)) denote the collection of the edges visited in depth-firs first traversal over T, where m 2 Erl- We store hyperbolic message hi..j. for cach edge in E. Let E, be the set of the first / edges in E. Suppose at time step I, the model visits node i, and it visils node js al the next time step. The message hi,.j. is updated using the node feature I;, and inward messages hg We first use hyperbolic centroid to gather the inward messages to produce
Zuei HCent (HLinearrryy (fhr,d Je..a.o
(44)
and then map the tree node features to the hyperbolic space to produce
Zeur HEmbedan, dr (x:.).
(45)
Finally, we combine them using the Lorentz Direct Concatenation and pass them through a hyperbolic linear layer to get the message
hi..j HLincargxdy.dr (HCat Zuci )))
(46)
Topological Prediction At each time step 1, the model makes a binary decision on whether to generate a child node, using tree embedding zr, node feature Di, and inward messages hg using the following layers successively:
Zue HCent HCenteHLineara,, di ((he (thnseemaaee))) ((ns)))))
Zeur HEmbedyn, dr (2x,))
(47)
Zall HLineargxdy.6 (HCat({Zoun Znei, z7 2T))),
P: = 2
32

---

Published in Transactions on Machine Learning Research (09/2024)
Label Prediction If a child node ji is generated, use the tree embedding øy and the outward message hinse to predict its label. We apply the following two layers successively:
Zall HLinearzxdr (HCat (s =r))),
(48)
q: = Softmax(HCDistaa (Zall)).
The output a distribution over the label vocabulary. When ji is a root node, its parent in is dummy and the message is padded with the origin of the hyperbolic space hi,.. = o.
Training The topological and label prediction have induced losses. Suppose p., år are the the ground truth topological and label values, obtained by doing depth-first traversal on the real junction tree. The decoder minimizes the following cross-entropy loss:
m Liopa >
11 Σ
Lecons (Pe, Pt); Llabel
Leross 9t), Ltotal Lropo + BLabbll,
(49)
where Leross is the cross-entropy loss. During the training phase, use the teacher forcing strategy: after the predictions at cach time step, we replace them with the ground truth. This allows the model to learn from the correct history information.
F.2.2 Architecture Details
Tree Encoder
Input: tree
Hyperbolic GCN layers:
Input dimension: 1
Hidden dimension: 32
Depth:
Output dimension: 32
Hyperbolic centroid on all vertices
Output: tree embedding in L
Tree Decoder
Input: tree embedding in T.
Message passing RNN:
Input: node feature of current tree node, inward messages
Hyperbolic linear layer on inward messages: T + 12
Hyperbolic centroid inward messages
Hyperbolic lincar layer: L + L
Output dimension: 32
Topological Prediction:
Input: embedding, inward messages
Hyperbolic linear layer on inward messages: T + LL
Hyperbolic centroid inward messages
Lorentz Direct concatenation on inward message and tree embedding: L - L
Hyperbolic linear layer: T,64 L
33

---

Published in Transactions on Machine Learning Research (09/2024)
Hyperbolic centroid distance layer: L R
Softmax on output
Output dimension:
Output: tree
Hyperparameters
Manifold curvature: K 1.0
For all hyperbolic linear layers:
Dropout: 0.0
Use bias: True
Optimizer: Riemannian Adam (31 = 0.0, B2 0.999)
Learning rate: 5e-3
Learning rate scheduler: StepLR (step 20000, 7 = 0.5)
Batch size: 32
Number of epochs: 20
F.2.3
Architecture Details of Hyperbolic Generative Adversarial Network
Gencrator
Imput: points sampled from wrapped normal distribution G(o, diag(116)) in L
Hyperbolic linear layers for embedding:
Input dimension: 16
Hidden dimension: 32
Depth: 2
Output dimension: 32
Output: tree embedding in IL
Critic
Input: tree embedding in L
Hyperbolic lincar layers for tree embedding:
Input dimension: 32
Hidden dimension: 32
Depth:
Output dimcnsion: 32
Hyperbolic centroid distance layer: L - IR
Output: score in R
34

---

Published in Transactions on Machine Learning Research (09/2024)
Hyperparameters
Manifold curvature: K = -1.0
Gradient penalty coefficiet:: À = 10
For all hyperbolic linear layers:
Dropout: 0.1
Use bias: True
Optimizer: Riemannian Adam (51 = 0, B2 0.9)
Learning Rate: le-4
Batch size: 64
Number of epochs: 20
Gradient penalty A: 10
F.3
Experiment Details for Molecular Generation
We describe the detailed architecture and settings for the molccular generation experiments.
F.3.1 Model Architecture
Notation We denote a molecular graph as G (Va; FG), where Vo is the set nodes (atoms) and Fo is the set of edges (bonds). Each node (atom) VG has node feature B, describing its atom type and properties. The molecular graph is decomposed into junction tree T (Vr, Er) where Vr is the set of atom clusters. We use v, to represent graph nodes and i, to represent tree nodes, respectively, The dimensions of the node features of the graph a and the tree are denoted by dG, and dr,, respectively. The hidden dimensions of graph and tree embeddings are da, dr, respectively.
Graph and Tree Encoder The design of the graph encoder is the samne the encoder in the random tree generation experiment ( ($F.2.1 The graph encoder maps a graph to a latent hyperbolic feature zg:
x E2Háo, (xx);
x (x
1 1=...... ,c.
(50)
2G HCent (ao)..
The tree encoder similar with the graph encoder, it encodes the junction tree to hyperbolic embedding žy with a hyperbolic GCN of depth ly- The only difference that its input feature a, are one-hot vectors representing the atom clusters the cluster vocabulary. We need use a hyperbolic embedding layer as the first layer of the network accordingly.
x HEmbeder, dr (x:), 1= 1, Ir,
x (l-1))
(51)
27 (C)nt(w((r))).
we generate junction tree T (Vr, ET) using a tree recurrent neural network in top-down and node-by by-node fashion. Its architecture is the same the tree decoder in the random tree generation experiment F.2.1 and we will not elaborate here.
Tree Decoder Similar to Jin et al. 20088 2019
35

---

Published in Transactions on Machine Learning Research (09/2024)
Graph Decoder The graph decoder assembles molecular graph given junction tree T (V, E) and graph embedding 2c. Tet 9; be the set of possible candidate subgraphs around tree node i.e. the different ways of attaching neighboring clusters to cluster i. We want to design a scoring function for each candidate subgraph G.)) Ge. To this end, we first use the hyperbolic GCN and hyperbolic centroid to acquire the hyperbolic embedding & of each subgraph G.y) Specifically,
! E2Hddo (ar));
(1) (GCCN(x(--1)),
1 == 1-1,,.,, .l.
(52)
5 H
Then, the embedding of the subgraph is combined with the embedding of the molecular graph zg by the Lorentz Direct Concalenation to produce Lal and gathered by hyperbolic centroid to acquire score
Hlinear2xdo da (HCat((> (HCa =G])),
(53)
s) (CDista,.. E R.
Training We define the loss for the graph decoder to be the sum of the cross-entropy losses in each G. Specifically, suppose the correct subgraph is G
Lasam =
ΣE (1) log Σ
exp(s,0)
(54)
G EG
Similar to the tree decoder, we also use teacher forcing when training the graph decoder. We also utilize the LipSwish activation function (Chen et al. 2019) throughout the hyperbolic linear layers for stable and smooth second-order gradients.
F.3.2
Architecture Details of Hyperbolic Junction Tree Encoder-Decoder
Graph Encoder
Input: graph node features in 35
Map features hyperbolic space: 1235 + 35
Hyperbolic GCN layers:
Input dimension:
Hidden dimension: 256
Depth:
Oulput dimension: 256
Hyperbolic centroid on all vertices
Output: graph embedding in 2
Tree Encoder
Input: junction tree fcatures in R822
Hyperbolic embedding layer: 1828 - L
Hyperbolic GCN layers:
36

---

Published in Transactions on Machine Learning Research (09/2024)
Input dimension: 256
Hidden dimension: 256
Depth: 4
Output dimension: 256
Hyperbolic centroid on all vertices
Output: tree embedding in L
Tree Decoder
Input: tree embodding 256
Message passing RNN:
Input: node feature of current tree node, inward messages
Hyperbolic linear layer inward messages: L
L
Hyperbolic centroid on inward mossages
Hyperbolic embedding layer node feature: R28
25256
Lorentz Direct concatenation on node feature and inward message: 256 + L
Hyperbolic linear layer: L5122 + Lx
Output dimension: 256
Topological Prediction:
Input: tree embedding, node feature of current tree node, inward messages
Hyperbolic linear layer on inward messages: Lx + L
256
256
Hyperbolic centroid inward messages
Hyperbolic embedding layer on tree feature: 8828 +
L
256
Lorentz Direct concatenation node feature, inward message, and tree embedding: L
TGs
Hyperbolic linear layer: L + 256
Hyperbolic centroid distance layer: 25 - R²
Softmax on output
Output dimension:
Label Prediction:
Input: embedding, outward messages
Lorentz Direct concatcnation outward message, and tree feature: 2 + L
Hyperbolic linear layer: I_512 -> L
Hyperbolic centroid distance layer: 25 - R2288
Softmax on output
Output dimension: 828
Output: junction tree
Graph Decoder
Input: junction tree, tree message, and graph embedding
Construction candidate subgraphs
Hyperbolic graph convolution layers all subgraphs:
37

---

Published in Transactions on Machine Learning Research (09/2024)
Input dimension: 256
Hidden dimension: 256
Depth: 4
Output dimension: 256
Hyperbolic centroid on vertices of all subgraphs
Lorentz Direct concatenation on subgraph embedding and graph embedding: L 2 L2
Hyperbolic linear layer: 2 256
Hyperbolic centroid distance layer: L IR
Use subgraph score to construct molecular graph
Output: molecular graph
Hyperparameters
Manifold curvature: K 1.0
For all hyperbolic linear layers:
Dropout: 0.0
Use bias: True
Optimizer:
Riemannian Adam (31 = 0.0, = 0.999)
Learning rate: 5e-4
Learning rate scheduler: StepLR (step 20000, 7 0.5)
Batch size: 32
Number of epochs: 20
F.3.3
Architecture Details of Hyperbolic Generative Adversarial Network
Generator
Input: points sampled from wrapped normal distribution G(o, diag(1 (21)) in LL 128
Hyperbolic linear layers for graph embedding:
Input dimension: 128
Hidden dimonsion: 256
Depth:
Output dimension: 256
Hyperbolic linear layers for tree embedding:
Input dimension: 128
Hidden dimension: 256
Depth: 3
Output dimension: 256
Output: graph embedding and tree embedding in 1,128
38

---

Published in Transactions on Machine Learning Research (09/2024)
Critic
Inpul: graph embedding and tree embedding in LK
Hyperbolic lincar layers for graph embodding:
Input dimension: 256
Hidden dimension: 256
Depth:
Output dimension: 256
Hyperbolic linear layers for tree embedding:
Input dimension: 256
Hidden dimension: 256
Depth:
Output dimension: 256
Lorentz Direct concatenation on graph embodding and tree embedding: L 1
Hyperbolic linear layer: L + 2
Hyperbolic centroid distance layer: 1.250 lR
Output: score in R
Hyperparameters
Manifold curvature: K -1.0
Gradient penalty coefficien:: λ = 10
For all hyperbolic linear layers:
Dropout: 0.1
Use bias: True
Optimizer:
Riemannian Adam (3) 0, 0, 0.9)
Learning Rate: le-4
Batch size: 64
Number of epochs: 20
Gradient penalty A: 10
39


Reviews:
[{"review_id": "ZqKEISSskd", "reviewer": "Reviewer_sZTL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow. The structure and flow of the paper make for an easy reading experience, provided one is somewhat familiar with the topic. The proposed layers and models architectures are described in detail, and supplementary material regarding hyperbolic networks are particularly useful for understanding the underlying concepts. This additional information helped me as someone who is not  deeply knowledgeable in the topic of hyperbolic networks grasp the details of the model and contextualize the contributions made by the authors. \n- According to the results presented in this work, the model certainly seems competitive with the current state-of-the-art. The ablation studies included in the synthetic tree generation task also help disentangle how much the various design decisions of the authors contributed to the model's performance. There is however a caveat to this, which will be discussed as the first weakness.\n\nWeaknesses:\n- A major weakness of this manuscript is the lack of comparison with newer models, especially on the MOSES data, which is widely used as benchmark for graph generative models. The most recent model included in the benchmark comparison was published in 2019, the field has not stopped developing since then. More recent models such as DiGress (Vignac et al. 2023) and cMolGTP (Wang et al, 2023) have been successfully applied on the same dataset and performance metrics, and I am confident there are also other such recent models that I have missed. The absence of these comparisons makes it difficult to evaluate how HAEGAN stands relative to the latest advancements in the field, therefore without including these comparisons, the conclusion that HAEGAN provides state-of-the-art performance cannot be conclusively made.\n- Another glaring weakness is the outdated state of the references, with almost no citations from post-2021 research. This undermines the relevance of the work, as it does not acknowledge most recent contributions and developments in hyperbolic generative models or molecular generation. Updating the references and especially the relevant work section to include recent publications in the field is crucial to provide a more current perspective and underscore the novelty of the contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a novel hyperbolic generative network called Hyperbolic AE-GAN (HAEGAN), designed to improve the stability and performance of hyperbolic generative models.\n\nThe model is a hybrid combining a hyperbolic autoencoder (AE) and a hyperbolic generative adversarial network (GAN). This combination enables to model learn a encoder and decoder mapping to and from a hyperbolic latent space without having to optimize the numerically unstable ELBO required for hyperbolic VAEs, and to also leverage the effective discriminator mechanism of the GAN to learn a mapping from wrapped normal noise to the latent space. \n\nThe authors also include a novel Lorenz Direct Concatenation Layer, that uses insights from Fully Hyperbolic Linear Layers to enable a simple and numerically stable hyperbolic concatenation operation that avoids the need for exponential and logarithmic maps.They elso introduce a novel Wasserstein GAN formulation in the hyperbolic space and provide theoretical validation of the stability and effectiveness the introduced concatenation layer and Wasserstein loss function.\n\nFinally, the authors perform demonstrate the capabilities of the HAEGAN architecture using a synthetic tree generation experiment, as well as by applying in on popular molecular data generation dataset, where the model achieves state-of-the-art performance. The authors also evaluate a series of variants of their model to perform an ablation study of their architecture design decisions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors should include comparisons with more recent models, which is critical for demonstrating the claim that HAEGAN has state-of-the-art performance. This is crucial to securing my recommendation for acceptance.\n- The literature review, particularly the relevant work section needs be updated to reflect research in the relevant fields also post 2021. This is also crucial for securing my recommendation.\n- In Section 5.1.1, when describing the topological and label losses, it is unclear whether and how this two losses are combined when training the model. The authors should explain this more clearly.\n- The statement on page 6, section 4.3: \"Since most interesting datasets are not readily presented in the hyperbolic domain, the hyperbolic AE can also learn to represent them in the hyperbolic domain\" is unclear and should be rewritten for better understanding.\n- I found small small typos and grammatical that should be corrected:\n    Page 7: \"the model visit node\"\n    Page 10: \"The design of the graph encoder the same with the encoder in the random tree generation experiment\".", "broader_impact_concerns": "I have no broader impact concerns regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EkiIlbcsSd", "reviewer": "Reviewer_G1Qy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* Many contributions, fitting well together to make a whole model.\n* Encouraging results of the HAEGAN model on molecular generation.\n\n# Weaknesses\n* The first part of the paper (Sect. 2 and 3) are basically inaccessible to readers that are not familiar with hyperbolic geometry and provide little to no intuition.\n* The baselines for the main experiment on molecular generation (Sect. 5.2) are 5 years or older, even though the field is very active and many newer models exist. (See e.g. https://scholar.google.com/scholar?as_ylo=2020&q=snn&cites=2745109896754049393&scipsc=1 for a rough list of papers citing MOSES and reporting some related metrics since 2020)\n* While based on the MOSES benchmark, this submission drops half of the metrics proposed by MOSES (internal diversity, FCD, and the property distributions) - exactly those most consulted by practicioners in the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents new concatenation/split operations in the Lorentz model of hyperbolic geometry that are numerically more stable than alternatives. These operations are used to construct a hyperbolic auto-encoder/GAN model on top of a hyperbolic GCN architecture. Experiments on (molecular) graph tasks compare different model ablations and some baseline models, showing empirically that the model is functional.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Critical: Extend the evaluation by the missing MOSES metrics, especially the internal diversity and FCD values (which are scalar values easy to add to a table).\n* Critical: At least report results of more modern baselines on molecular generation\n* Recommendation: Drop the validity, uniqueness and novelty results from Table 2, as these are largely meaningless. (Validity is 1 by construction for JT-VAE-derived models, uniqueness just indicates that there is no mode collapse).\n\nMinor notes:\n* p2: hyperlolic -> hyperbolic\n* p11: \"For all the metrics, a larger number implies a better result.\" - this is not quite the case: SNN/Scaff values of 1 on the Test dataset would indicate full overfitting, for example. More generally, these numbers are hard to interpret as models need to strike a balance between staying close to the training data, while generating new structures that could be of interest, but are _not_ present in the training data.\n* p12: \" This shows that our model is better at searching on the manifold of scaffolds and can generate examples with novel core structures\" - the lack of FCD (or QED) scores here means that this can be trivially achieved by valid but unstable or unlikely structures that you should not want to generate in the first place.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4QiYol138r", "reviewer": "Reviewer_Bnmm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-\tPaper is clearly written, and the motivation for individual components is logical.\n-\tExtension of WGAN-GP to the hyperbolic space by sampling from the geodesic between the embedded data distribution and generator distribution is elegant. (Proposition 4.1)\n-\tProposal of Lorentz direct concatenation and the subsequent theoretical and empirical analysis of the gradient bounds, stable optimization and the effect on distances is interesting. (Theorem 3.1)\n-\tHAEGAN shows strong performance in several molecular generation metrics especially in structure related metrics, showing the effectiveness of using Hyperbolic space over Euclidean space for modeling molecular generation. (Tab. 2)\n-\tDetailed network architecture descriptions are provided which adds value for reproducibility. (Appendix F)\n\n\nWeaknesses:\n\n-\tThe idea of combining AE and modeling the latent distribution with GAN for molecular generation has been previously studied in LatentGAN [1]. While the authors compared with  LatentGAN in the experiments, the claim for novelty of the hybrid AE-GAN seems like an overstatement. While the molecular representation, modeling space, and the network architectures are different, but the overall construct is quite similar.\n-\tThe random tree generation results are not very informative without any measure of variance in the scores. Also, this makes it difficult to understand the ablations, since none of the other hyperbolic networks seem to work in the molecular generation task. (Tab. 1)\n-\tThe instability of every other hyperbolic method in the molecular generation task is quite strange, especially HGAN, since it also uses the proposed Lorentz direct concatenation and the hyperbolic WGAN-GP.  (Tab. 2)\n-\tDue to weakness 2 and 3, it is difficult to conclude on the stability and advantage of the proposed method over other hyperbolic tree generation methods.\n-\tThe authors claim that because its complex nature, GRU would lead to instability, however, no evidence has been provided to sunstantiate that claim. (Sec. 5.1.1). \n-\tThe MNIST experimental results in Table 3 are questionable since other methods’ scores are taken directly from the respective papers and were produced with different settings. (Tab. 3, 4)\n-\tThe interesting alayses of the stability of the the Lorentz direct split is relegated to the appendix. (Appendix C)\n\n[1] Prykhodko et al., 2019, A de novo molecular generation method using latent vector based generative adversarial network.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces HAEGAN, a hyperbolic version of the auto-encoding generative adversarial network. It utilizes an auto-encoder to map data into a hyperbolic latent space, followed by training a GAN on this space. The authors extend WGAN-GP to the hyperbolic space by sampling along the geodesic between the generator and data distributions. They introduce Lorentz Direct Concatenation and Split for tree-structured data, crucial for constructing hyperbolic tree decoders. With these advancements, they successfully generate new molecular structures.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical for acceptance:\n\n-\tInclude a discussion on positoning HAEGAN with respect to LatentGAN and clarify the novelty claim of the AE-GAN construct.\n-\tMove some experimental validation of WGAN-GP extension and Lorentz direct concatenation from appendix to main body, as they are the most important contibutions.\n-\tInclude variance measures for random tree generation task results.\n-\tInclude an explanation/analysis/discussion on the instability of all other hyperbolic methods in the molecular generation task. \n-\tProvide a fair comparison of metrics in the MNIST experiment.\n\n\nGood to have:\n\n-\tExperiments on applicability to a different dataset (either molecular, or image, or any other domain) to emphasize the broader applicability of the proposed concatenation/split operations and the hyperbolic extension of WGAN-GP.\n-\tSince random tree generation and molecular generation shares some of the layers, baselines and other information, probably the architecture and baseline descriptions could be condensed.\n-\tSome minor typographical errors need to be corrected.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have addressed all major issues and implemented most requested changes in the latest revision.
Decision: Accept (poster)