Autoencoding Hyperbolic Representation for Adversarial Generation
Authors: Eric Qu
Dongmian Zou
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning
hyperbolic neural network
numerical stability
generative models

Abstract: With the recent advance of geometric deep learning, neural networks have been extensively used for data in non-Euclidean domains. In particular, hyperbolic neural networks have proved successful in processing hierarchical information of data. However, many hyperbolic neural networks are numerically unstable during training, which precludes using complex architectures. This crucial problem makes it difficult to build hyperbolic generative models for real and complex data. In this work, we propose a hyperbolic generative network in which we design novel architecture and layers to guarantee stable training. Our proposed network contains three parts: first, a hyperbolic autoencoder (AE) that produces hyperbolic embedding for input data; second, a hyperbolic generative adversarial network (GAN) for generating the hyperbolic latent embedding of the AE from simple noise; third, a generator that inherits the decoder from the AE and the generator from the GAN. We call this network the hyperbolic AE-GAN, or HAEGAN for short. The architecture of HAEGAN fosters expressive representation in the hyperbolic space, and the specific design of layers ensures numerical stability. Experiments show that HAEGAN is able to generate complex data with state-of-the-art structure-related performance.

Under review as a conference paper at ICLR 2023
AUTOENCODING HYPERBOLIC REPRESENTATION FOR ADVERSARIAL GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
With the recent advance of geometric deep learning, neural networks have been extensively used for data in non non-Euclidean domains. In particular, hyperbolic neural networks have proved successful in processing hierarchical information of data. However, many hyperbolic neural networks are numerically unstable dur- ing training, which precludes using complex architectures. This crucial problem makes it difficult to build hyperbolic generative models for real and complex data, In this work, propose hyperbolic generative network in which design novel architecture and layers to improve stability in training. Our proposed net- work contains three parts: first, hyperbolic autoencoder (AE) that produces hy- perbolic embedding for input data; second, a hyperbolic generative adversarial network (GAN) for generating the hyperbolic latent embedding of the AE from simple noise: third, a generator that inherits the decoder from the AE and the generator from the GAN. We call this network the hyperbolic AE-GAN, or HAE- GAN for short. The architecture of HAEGAN fosters expressive representation in the hyperbolic space, and the specific design of layers ensures numerical sta- bility. Experiments show that HAEGAN is able to generate complex data with state-of-the-ard structure-related performance.
INTRODUCTION
digh-ddmenssonal data often show an underlying geometric structure, which cannot be easily cap- tured by neural networks designed for Euclidean spaces. Recently, there is intense interest in learn- ing good representation for hierarchical data, for which the most natural underlying geometry hyperbolic. A hyperbolic space is Riemannian manifold with constant negative curvature (An- derson 2006). The exponential growth of the radius the hyperbolic space provides high capacity, which makes it particularly suitable for modeling tree-like hierarchical structures. Hyperbolic rep- resentation has been successfully applied to, for instance, social network data in product recommen- Wu et al. 2021). and
dation (Wang et al.) 2019), molecular data in drug discovery Yu et al.) 2020
skeletal data in action recognition (Peng et al. 2020).
Many recent works (Ganea al. 2018 Shimizu al. 2021 Chen et al. 2021 have success- fully designed hyperbolic neural operations. These operations have been used in generative models for generating samples in the hyperbolic space. For instance, several recent works (Nagano al. 2019 Mathieu et al. 2019; Dai et al.] (2021b) have built hyperbolic variational autoencoders (VAE)
(Kingma & Welling 2014). On the other hand, [azcano al. (202T) have generalized generative
adversarial networks (GAN) Gooofflloo et al. | Arjovsky et al., 2017) to the hyperbolic space. However, the above hyperbolic generative models are known to suffer from gradient explosion when the networks are deep. In order to build hyperbolic networks that can generate real data, it is desired to have framework that has both representation power and numerical stability.
To this end, we design novel hybrid model which learns complex structures and hyperbolic em- beddings from data, and then generates examples by sampling from random noises in the hyperbolic space. Altogether, our model contains three parts: first, we use hyperbolic autoencoder (AE) to learn the embedding of training data in the latent hyperbolic space; second, we use a hyperbolic GAN to lear generating the latent hyperbolic distribution by passing wrapped normal noise through the generator; third, we generate samples by applying sequentially the generator of the GAN and the decoder of the AE. We name our model as Hyperbolic AE-GAN, or HAEGAN for short. The advan-

---

Under review as a conference paper at ICLR 2023
tage of this architecture is twofold: first, it enjoys expressivity since the noise goes through both the layers of the generator and the decoder; second, it allows flexible design of the AE according to the type of input data, which does not affect the sampling power of GAN. In addition. HAEGAN avoids the complicated form of ELBO in hyperbolic VAE, which is one source of numerical instability. We highlight the main contributions of this paper as follows:
HAEGAN is a novel hybrid AE-GAN framework for learning hyperbolic distributions that aims for both expressivity and numerical stability.
We validate the Wasserstein GAN formulation in HAEGAN, especially the way of sampling from
the geodesic connecting a real sample and a generated sample..
We design a novel concatenation layer in the hyperbolic space. We extensively investigate its
numerical stability via theoretical and experimental comparisons.
In the experiments part, we illustrate that HAEGAN is not only able to faithfully generate synthetic hyperbolic data, but also able generate real data with sound quality. particular, we consider the molecular generation task and show that HAEGAN achieves state-of-the-art performance, especially in metrics related to structural properties.
2
BACKGROUND IN HYPERBOLIC NEURAL NETWORKS
2.1 HYPERBOLIC GEOMETRY
Hyperbolic geometry special kind of Riemannian geometry with constant negative curva- ture (Cannon et al. 1997; Anderson, 2006). To extract hyperbolic representations, it is necessary to choose a "model" or coordinate system, for the hyperbolic space. Popular choices include the Poincaré ball model and the Lorentz model, where the latter is found to be numerically more stable (Nickel Kiela 2018 We work with the Lorentz model L3 = (C, g) with a constant nega- tive curvature K, which is an n-dimensional manifold L embedded in the (n + 1)-dimensional Minkowski space, together with the Riemannian metric tensor = diag([- [1 1),1), where 1, de- notes the n-dimensional vector whose entries all I's. Every point in L is represented by 1/K, where (--)ac is the Lorentz inner
[ 1 Ft > O,æ, E R", and satisfies (x, x) product induced by ( y) == x gy = -2,1/ x, Vs. æ. y E LL In the rest of the paper, we will refer to Ia as the "time component" and x, as the "spatial component" In the following, describe some notations. Extensive details are provided in Appendix
Notation We use de(æ, y) to denote the length of a geodesic ("distance" along the manifold) connecting x, y E T.k- For each point æ E T. the tangent space i is denoted by TaT-- The norm ||- = ((.)) For x, y E Li and v Ta we use exp K (v) to denote the exponential LL - TaLk to denote the logarithmic map For two points y. E LR use PT to denote the parallel transport map which "transports" a vector from TzL to TyLk along the geodesic from æ to y.
map of v at æ;; on the other hand, we use log K such that log K (exp (v))
2.2 FULLY HYPERBOLIC LAYERS
One way to define hyperbolic neural operations is to use the tangent space, which is Euclidean. However, working with the tangent space requires taking exponential and logarithmic maps, which cause numerical instability. Moreover, tangent spaces are only local estimates of the hyperbolic spaces, but neural network operations are usually not local. Since generative networks have complex structures, we want to avoid using the tangent space whenever possible. The following hyperbolic layers take a "fully hyperbolic" approach and perform operations directly in hyperbolic spaces.
The most fundamental hyperbolic neural layer, the hyperbolic linear layer (Chen al., 2021) is a trainable "linear transformation" that maps from L to LØ- We remark that "linear" is used to analogize the Euclidean counterpart, which contains activation, bias and normalization. In general, for an input æ E LR the hyperbolic linear layer outputs
VIn(Waa x. w) 1/K h(Wx,, x.
HLinear, n.m (x)
(1)
2

---

Under review as a conference paper at ICLR 2023
Here h(Wæ, v) >a(v" x+b")
[ (Wr(x) + b), where E g"+1 and W E ["=((++1) are trainable weights, b and D' are trainable biases, a is the sigmoid function, T is the activation function, and the trainable parameter À > 0 scales the range.
While a map between hyperbolic spaces such as HLinear is used most frequently in hyperbolic networks, it is also often necessary to output Euclidean features. A numerically stable way of obtaining Euclidean oulput features via the hyperbolic centroid distance layer (Liu et al.) 2019). which maps points from L to R.". Given input æ E L it first initializes trainable centroids {or 1 C k then produces a vector distances
y HCDistn, (æ) [dc ci) de(x, cm)]
We introduce more hyperbolic neural layers used in our model in Appendix A.2
3 HYPERBOLIC AUTO-ENCODER GENERATIVE ADVERSARIAL NETWORKS
3.1 HYPERBOLIC GAN
With the hyperbolic neural layers defined in 22.2 is not difficult to define a hyperbolic GAN whose generator and critic contain hyperbolic linear layers. The critic also contains a centroid distance layer so that its output a one-dimensional score. To produce generated samples, we sample z (0) from G(o, I), the wrapped normal distribution (Nagano et al.) 2019) and then pass through the generator. We follow the Wasserstein gradient penalty (WGAN-GP) framework (Gulrajani et al. 2017) to foster easy and stable training. The loss function, adapted from the Euclidean WGAN-GP,
LWGAN = E [D()]] - E [D(x)] $-P,
+ E [(IVD())] - 1)]]. dorP:
(3)
m-7,
where D is the critic, D(()) is the Riemannian gradient of D(x)) at a. P. is the generator distri- bution and PP, the data distribution. Most importantly, Pa samples uniformly along the geodesic between pairs of points sampled from PG and IP, instead of linear interpolation. This manner of sampling is validated in the following proposition, the proof of which can be found in B
Proposition 3.1. Let P., and Pg be two distributions in L and f be an optimal solution of |fll [f))) - En-F, [f(r) where I1-1ly is the Lipschitz norm. Let x be the optimal coupling between P,, and F. that minimizes W (Pr,, = infxen(B ..) E Ensyaaldel
y)l. where )) is the set of joint distributions x (xx y) whose marginals are PP, and Pg, respectively. Let 2r 7l), S 1 be the geodesic between and y, such that (0) x1) y, g
[IPP,, P.)
vt E TL ll = de(a, y). If f is differentiable and ((x = y) 0. then it holds that
P P(a.a))~ ( (xx) (xx) = de dc(x,v)
V2
=
(4)
The WGAN-GP formulation of the hyperbolic GAN is capable of sampling distributions in low- dimensional hyperbolic spaces. We illustrate learning 2D distributions in (
3.2 ARCHITECTURE OF HABGAN
Although a hyperbolic GAN defined above can faithfully generate hyperbolic distributions its train- ing is difficult. Specifically, numerical instability will be observed when we incorporate complex network architectures into the generator and the critic. To this end, we design our HAEGAN model to contain both a hyperbolic AE and hyperbolic GAN. First, we train a hyperbolic AE and use the encoder to embed the dataset into a latent hyperbolic space. Then, we use hyperbolic GAN to learn the latent distribution of the embedded data. Finally, we sample hyperbolic embeddings using the generator and use the decoder to get samples in the original space. An illustration of HAEGAN is shown in Figure [
In addition to expressivity, HAEGAN enjoys flexibility in choosing the AE for embedding the hy- perbolic distribution. If the original dataset is not readily presented in the hyperbolic domain, the hyperbolic AE can also learn the hyperholic representation. In this case, adopt the hyperholic
3

---

Under review as a conference paper at ICLR 2023
(6)
Training Data
Training
Wapppdd
Wiapped
Einenders
Encoders
Generator
Generator
gm
Decoders
Critsc
Decaders
Reconstruction
Generation
Figure I: Overview of HAEGAN. (a) The hyperbolic AE. (b) The hyperbolic GAN for generat-
ing the latent embeddings. The encoders in (b) identical to (a). (c) The process for sampling molecules. The generator in (c) is identical to (b) and the decoders in (c) are identical to (a).
embedding operation (Nagano et al. 2019) from Euclidean to the hyperbolic space. We review the details of this embedding operation in Appendix A.3
As sanity check, train HAEGAN with the MNIST dataset (LeCun et al. 2010 and present some generated samples in Figure 2 It clear that HAEGAN can faithfully generate synthetic examples. We describe the details and perform quantitative comparison with other hyperholic models regarding this task in Appendix C
4
7
2
0
9
2
8
5
9
6
/
3
9
8
5
0
7
3
4
0
7
0
8
8
7
0
9
8
4
Figure 2: Samples generated from the HAEGAN trained MNIST.
LORENTZ CONCATENATION
4.1 MOTIVATION AND DEFINITION
Concatenation and split are essential operations neural networks for feature combination, parallel computation, etc. For data with complex structures, the decoder of the hyperbolic AE in HAEGAN will need to produce parts of features and then combine them, where we need to perform concate- nation. However, there is obvious way of doing concatenation in the hyperbolic space. Shimizu et al. (202T) proposed Poincaré /--oonctenattoon and f9-plit in the Poincaré model. Specifically, they first the logarithmic map to lift hyperbolic points to the tangent plane of the origin, then perform Euclidean concatenation and split in this tangent space, and finally apply 8 regularization and apply the exponential map to bring back the Poincaré ball.
Since we use the Lorentz model, the above operations are not useful and we need to define concate- nation and split in the Lorentz space. One could define operations in the tangent space similarly to the Poincaré B-concatenation and f-split. More specifically, if we want to concatenate the in- put vectors { (x, N where each æ; E T.k. we could follow a "Lorentz tangent concatenation",
first lift each I; to v; = log (x,) = =[" E Rn and then perform the Euclidean concate-
nation to get = |0, vT.,..., uN Finally, we would get = exp (v) as a concatenated vector in the hyperiolic space. We denote = HTCCat((z;)) ( Similarly, we could perform
the "Lorentz tangent split" input æ; L with split uub-dimensions ΣN n4 to get
= log K (x) [0, wT_ IR ...., NN E [RaN] U; [ ToL and the split vectors
y: expo (w:) successively.

---

Under review as a conference paper at ICLR 2023
Unfortunately, there are two problems with both the Lorentz tangent concatenation and the Lorentz tangent split. First, they are not "regularized", which means that the norm of the spatial component will increase after concatenation, and decrease after split. This will make the hidden embeddings numerically unstable. This problem could be partially solved by adding a hyperbolic linear layer after each concatenation and split, similarly to Ganea al. (2018), that we have trainable scaling factor À to regularize the norm of the output. The second and more important problem is that if we use the Lorentz tangent concatenation and split in deep neural network, there would be too many exponential and logarithmic maps. It on one hand suffers from severe precision issue due to the inaccurate float representation (Yu De Sa 2019 202T), and on the other hand easily suffers from gradient explosion. Moreover, the tangent space is chosen at If the points concatenate are not close to o, their hyperbolic relation may not be captured very well. Therefore, we abandon the use of the tangent space and propose more direct and numerically stable operations, which we call the "Lorentz direct concatenation and split", defined as follows.
Given the input vectors {=:)==1 where each I; E L and M = E ni the Lorentz direct concatenation of (a:; ) is defined to be a vector E T given by
y HCaat(
27 + -N 1)/
N,
(5)
Note that each æi is the spatial component of 2;, If we consider æ; E L as a point in Ra++1, the projection of æ; onto the Euclidean subspace {0} x R.,, or the closest point there, is a; The Lorentz direct concatenation can thus be considered as Euclidean concatenation of projections, where the Euclidean concatenated point is mapped back to L by the inverse map of the projection. We remark that this concatenation directly inherits from the Lorentz model.
We also define the Lorentz split for completeness, though our main focus is on concatenation: Given an input E k the Lorentz direct split of æ, with sub-dimensions 11::"" .. nn where > n,
n, will be {y: J. where each y: E I is given by first splitting and then calculating the corresponding time dimension as y; le., -1/K
[It, V1,
yN.
4.2 ADVANTAGE OF LORENTZ DIRECT CONCATENATION
We show the advantage of direct concatenation in this section. Firstly, We state the following theo- retical result regarding the exploding gradient of the Lorentz tangent concatenation.
Let {x where x; E Li denote the input features. Let y = HCat((x,))) | 1) denote the output of the Lorentz direct concatenation and = HTCat((#;) denote the output of the Lorentz tangent concatenation. Fix j E {1, .N. The following results hold:
Theorem 4.1.
1. For any
{ (x)) N and any entry 3² of ||0y" f0x,,,..... balla....... æx 1. 1.
For any M >> O, there exist ) and an entry 2" of z for which ||0=*/ /0r, len.... .v 2 M.
This theorem shows that while the Lorentz direct concatenation has bounded gradients, there is no control on the gradients of Lorentz tangent concatenation. The proof can be founn in Appendix [.] and we give simple numerical validation in Appendix [.2]
We remark that in addition to ensuring bounded gradients, discarding exponential and logarithmic maps also makes sure that the concatenation operation does not suffer from inaccurate float represen- tations YYu & De Sa 2019 2021). There are two additional benefits: first, the direct concatenation less complex and thus more efficient than the tangent concatenation; second, the operation does not contain exponential functions and is thus GPU scalable (Choudhary & Reddy 2022).
Next, we design the following simple experiment to show the advantage of our Lorentz direct con- catenation over the Lorentz tangent concatenation when they are used in simple neural networks. The hyperbolic neural network in this simple experiment consists of a cascading of L blocks A d-dimensional input is fed into two different hyperbolic linear layers, whose outputs then catenated by the Lorentz direct concatenation and the Lorentz tangent concatenation, respectively. Then, the concatenated output further goes through another hyperbolic linear layer whose output
5

---

Under review as a conference paper at ICLR 2023
again d-dimensional. Specifically, for 0,... 1.
h (1) = Hlinearda (1) HCat(h) (1) 1/2 );
h2 = Hlineard, ( (z
(6)
x (1+1) Hlinearza.a(t"))
In our test, we take d = 64. We sample input and output data from two wrapped normal distributions with different means (input: origin o, output: E2H(161)) and variances (input: (iag(1es) output:
3 x (1e))) Taking the input as 201), we fit x(4) to the output data. We record the average gradient norm of the three hyperbolic linear layers in each block. The results for = 64 blocks and L 128 blocks are shown in Figure
Clearly, for the first 20 blocks, the Lorentz tangent concatenation leads to significantly larger gradient norms. This difference in norms is clearer when the network is deeper. The gradients from the Lorentz direct concatenation much more stable.
(0)
(b)
Direct Concatetation
Gradient Norm
140
Lorentz Tangest Concstenation
Gradient Norm
800
Lorentz Tangent Concstaration
600
100
80
40
200
6
20
30
40
50
60
0
20
40
60
80
100
120
Blocks
Blocks
Figure 3; Average gradient norm of the each block in training, (a) 64 blocks. (b) 128 blocks.
Finally, we remark that Lorentz direct concatenation is numerically stable in practice. In the gener- ation experiments which we introduce in the next section, if we used the Lorentz tangent concate- nation, the gradients would explode very rapidly (produce NaN), even during the first epoch.
More analysis about hyperbolic concatenations, particularly regarding their impact on hyperbolic distances and their stability, can be found in Appendix D.3
EXPERIMENTS
We perform the following two experiments: random tree generation and molecular generation.
5.1
RANDOM TREE GENERATION
Recent studies (Boguná al. 2010 Krioukov al., 2010 Sala al. 2018 Sonthalia & Gilbert 2020) have found that hyperbolic spaces are suitable for tree-like graphs. We first perform an ex- periment in which generate random trees. We compare the performance of HABGAN with other hyperbolic models. In this experiment, the AE consists of a tree encoder and tree decoder. The structure of the AE is explained in detail in Appendix E We remark that both the tree encoder and the tree decoder are also used in the molecular generation experiment.
Dataset Our dataset consists of 500 randomly generated trees. Each tree is created by converting uniformly random Prüfer sequence (Prüfer 1918). The number of nodes in each tree is uniformly sampled from [20,50]. The dataset is randomly split into 400 for training and 100 for testing.
Baselines and Ablations We compare HAEGAN with the following baseline hyperbolic genera- tion methods. HGAN, where we only use hyperbolic Wasserstein GAN without the AE structure, where the tree decoder as the generator and the tree decoder as the critic. HVAE-w w and HVAE-1, where we use the same AE but follow the ELBO loss function used by Mathieu et al. (019) instead of having a GAN ("w" and "Y" refer to using wrapped and Riemannian normal distributions, re- spectively). Although mainly focus hyperbolic methods, also compare with the following

---

Under review as a conference paper at ICLR 2023
Euclidean generation methods: GraphRNN YYuu et al. 2018b) and AEGAN. The latter has the same architecture as HAEGAN but all layers and operations are in the Euclidean space.
As discussed in previous sections, our default choice in HAEGAN is use Lorentz direct Con- catenation and fully hyperbolic linear layers (Chen et al., 202T). We also consider the following ablations of HAEGAN: HAEGAN-H, where the fully hyperbolic linear layers are replaced with the tangent linear layers defined by Ganea et al. (2018): HAEGAN- where the concatenation in HAE-
GAN is replaced by 8-concatenation (Shimizu et al. 2021); HABGAN-T, where the concatenation
is replaced by the Lorentz tangent concatenation discussed in $4
Metrics We use the following metrics from You et al. (018b) to evaluate the models: The Maxi- mum Mean Diserepancy (MMD) of degree distribution (Degree), MMD of the orbit counts statistics distribution (Orbit); Average difference of orbit counts statisties (Orbit), Betweenness Centrality (Betweenness), Closeness Centrality (Closeness). All metries are calculated between the test dataset and 100 samples generated from the models.
Results The results and runtime of all models are shown in Table 1 For all metrics, a smaller number implies a better result.
Our default choice of HAEGAN (with direct concatenation and fully hyperbolic linear layers) per- forms the best across all metrics except the MMD of orbit counts statistics distribution, in which it just marginally falls behind the 3-concatenation. In particular, the Lorentz direct concatenation gen- erally performs better and more efficiently than Lorentz tangent concatenation and 8-concatenation. Also, the fully hyperbolic linear layer is superior than the tangent linear layer in both effectiveness and efficiency. Our results also show the advantage of the overall framework compared with either a single GAN or VAE. On one hand, the performance of HAEGAN is much better than HGAN. On the other hand, we note that the hyperbolic VAE-based methods suffer from numerical instability for this simple dataset even when using fully hyperbolic linear layers and direct concatenation, possibly because of the complicated ELBO loss. Finally, we remark that it is clear from the results that the hyperbolic models are better at generating trees than the Euclidean ones.
Table 1: Results of the tree generation experiments. "NaN" indicates NaN reported during training.
MMD
Average Difference
Time
Concat
HNN
Degree
Orbit
Orbit
Betweenness
Closeness
(s/step)
HGAN
Direct
Fully
0,000566
0.000056
0.131509 NaN
0.027145 NaN
0.022921
1.5472
HVAE-W
Direct
Fully
NaN
NaN
NaN
1.6712
HVAE-1
Direct
Fully
NaN
NaN
NaN
NaN
NaN
1.7394
GraphRNN
N/A
N/A
0.002681
0. 0.000106
0. 0.143869
0,025565
0,022051
0.0933
AEGAN
N/A
N/A
0,001343
0.000050
0. 0.140482
0,025666
0,021855
1.1874
HAEGAN-H
Direct
Tangent
0.000743
0.000010
0,138211
0.024512
0.022037
1,8513
HAEGAN-3
Beta
Fully
0.000470
0.000001
0.129896
0.026102
0.022375
1.7529
HAEGAN-T
Tangent
Fully
0.000314
0.000052
0.131563
0.024171
0.021858
1.6385
HAEGAN
Direct
Fully
0.000156
0.000005
0.123286
0.023706
0.021740
1.3146
5.2 DE Novo MOLECULAR GENERATION WITH HAEGAN
It is a crucial task in machine learning to learn structure of molecules, which has important applica- tion discovery of drugs and proteins (Elton et al. 2019). Since molecules naturally show graph structure, many recent works use graph neural networks to extract their information and accordingly train molecular generators (Simonovsky & Komodakis 2018 De Cao Kipf 2018 Jin et al.) 2018) 2019). In particular, Jín et al. (2018) 2019) proposed level representation of molecules where both a junction-tree skeleton and molecular graph are used to represent the original molecular data. In this way, molecule is represented in a hierarchical manner with tree-structured scaffold. Given that hyperbolic spaces can well-embed such hierarchical and tree-structured data (Peng et al. [02T], we expect that HAEGAN can leverage the structural information. To validate its effectiveness, in this section, we test HAEGAN using molecular generative tasks, where the latent distribution is embedded in hyperbolic manifold.
7

---

Under review as a conference paper at ICLR 2023
In our experiments, we design both hyperbolic tree AE and a hyperbolic graph AE in HAE- GAN to embed the structural information of the atoms in each molecule. Specifically, our model takes molecular graph as the input, passes the original graph to the graph encoder and feeds the corresponding junction tree to the tree encoder, acquiring hyperholic latent representations za of the graph, well as 2T for the junction tree. Then, the junction tree decoder constructs tree from 2T autoregressively. Finally, the graph decoder recovers the molecular graph using the gener- ated junction tree and zg. Within the HAEGAN framework, this network contains fully hyperbolic layers and embedding layers, as well as the concatenation layers described in $ We describe the detailed structure of each component in HAEGAN and include a figure illustration in Appendix
The hyperbolic representation is supposed to better leverage the hierarchical structure from the junction-tree than the graph neural networks (Jin et al. 2018 2019).
Dataset We train and test our model on the MOSES benchmarking platform (Polykovskiy et al. 2020), which is refined from the ZINC dataset (Sterling & Irwin 2015) and contains about 1.58M training. 176k test, and 176k scaffold test molecules. The molecules in the scaffold test set have different Bemis-Murcko scaffolds (Bemis & Murcko, [996). which represent the core structures of compounds, than both the training and the test set. They are used to determine whether model can generate novel scaffolds absent in the training set.
We compare our model with the following baselines: non-neural models including the Hidden Markov Model (HMM), the N-Gram generative model (NGram) and the combinatorial gen- erator; and neural methods including CharRNN (Segler et al.
Baselines
2018), AAE (Kadurin et al. 2017a
Polykovskiy et al. 2018). VAE BombazlBombarlli al.
2018: Blaschke et al., 2018). JTVAE
Uin et al. 2018).
.AtentGAN (Prykhodko et al.] 2011). The benchmark results are taken from
PPolykoskkiy al.
2020
Ablations On one hand, we consider an Euclidean counterpart of HAEGAN, named as AEGAN, to examine whether the hyperbolic setting indeed contributes. The architecture of AEGAN is the same as HAEGAN, except that the hyperholic layers are replaced with Fuclidean ones.
On the other hand, we also report the following alternative hyperbolic methods: HVAE-w and HVAE-r, where we use the same tree and graph AE but follow the ELBO loss function used by Mathieu et al. (2019) instead of having GAN ("w"" and "Y" refer to using wrapped and Riemannian normal distributions, respectively); HGAN, where we train an end-to-end hyperbolic WGAN with the graph and tree decoder as the generator, and the graph and tree encoder as the critic; HAEGAN- H. HAEGAN-B and HAEGAN-T T as introduced in $.1
Metrics We briefly describe how the models are evaluated. Detailed descriptions of the following metries can be found in the MOSES benchmarking platform (Polykovskiy et al.) (020). We generate set of 30,000 molecules, which we call the "generated set". On one hand, report the Validity, Unique(ness) and Novelty scores, which are the percentage of valid, unique and novel molecules in the generated set, respectively. These are standard metrics widely used to represent the quality of the generation. On the other hand, we evaluate the following structure-related metrics by comparing Nearest Neighbor (SNN) and
the generated set with the test set and the scaffold set: Similarity the Scaffold similarity (Scaf). SNN is the average Tanimoto similarly (Tanimoto 1958) between generated molecule and its nearest neighbor in the reference set. Scaf is cosine distances between the scaffold frequency vectors (Bemis & Murcko, 1996) of the generated and reference sets. In particular, SNN compares the detailed structures while Scaf compares the skeleton structures. By considering them with both the test and the scaffold test sets, we measure both the structural simi- larity to training data and the capability of searching for novel structures.
Results We describe the detailed settings and architectures of HAEGAN in Appendix F.5 and present some generated examples in Appendix G We We report in Table performance of HAEGAN and the baselines. For each metric described above, we take the mean and standard deviation from three independent samples. For all the metrics, a larger number implies a better result. We use bold font to highlight the best performing model in each criteria.
We take the most updated results available http:://gttbb..cmmmm/// www.cuulassettoooooooo
8

---

Under review as a conference paper at ICLR 2023
Table 2: Performance in Validity, Unique(ness), Novelty, SNN, and Scaf metrics. Reported (mean - std) over three independent samples. HVAE-w, HVAE-r, HGAN, HAEGAN-H. HAEGAN-S, HAEGAN-T are not included in the table as they all produce "NaN", implying instability.
Model
Validity (?)
Unique (1)
Novelty (1)
SNN (1)
Scal (
Test
TestSP
Test
TeaSF
Truin
0.6419
0.5859
0.9907
0
HMM
0.07620.032
0.567=0.142
0.999=0.001
0.388=0,011
0.380=0.011
0.207+0.048
0.04920.015
NGram
0,238+0.003
0.922+0.002
0.969=0.001
0.521=0.001
0.49940.001
0.53040.016
0.09840.014
Combinatorial
1.000.0
0,991
0.968=0.001
0.451=0.001
0.439=0.001
0.445+0.006
0.087+0.003
CharRNN
0.975200.02
0.99920.001
0.842=0.051
0.602=00022
0.56520.014
0.924
0.1 1.110a0.1 O0S
AAE
0,937=0.034
0.997=0.002
0.793=0.022
0.608=0.004
0.568:0.005
0.902+0.038
0.079+0.009
VAE
0,977+0.001
0.998+0 001
0.695=0.000 007
0.626=0.001
0.578+00000
0.939=0.002
0.059+0.011
JTVAE
1.0:0.0
0,999=0.000 001
0.914=0.009
0.548=0.008
0.519=0.007
0.896400.00
0.101+0.00 011
LateniGAN
0.897=00003
0.997=0.001
0.950=0.001
0.537=0.001
0.51320.001
0.88740.001
0.107:0.010
HAEGAN (Ours)
1.0
1.0
0.905=0.006
0.631=0.004
0.593:0.002
0.874+0.002
0.113=0.0007
ABGAN
1.020.0
0.968=0.001
0.99520.009
0.45920.006
0.45240.006
0.20340.004
0.05820.008
First of all, HAEGAN achieves perfect validity and uniqueness scores, which implies the hyperbolic embedding adopted by HAEGAN does not break the rule of molecule structures and does not in- duce mode collapse. Moreover, our model significantly outperforms the baseline models in the SNN metric. This means that the molecules generated by our model have a closer similarity the refer- ence set. It implies that our model captures better the underlying structure of the molecules and our hyperbolic latent space more suitable for embedding molecules than its Euclidean counterparts. Our model also achieves competitive performance in the Scaf metric when the reference set is the scaffold test set. This shows that our model is better in searching on the manifold of scaffolds and can generate examples with novel core structures.
Next, although AEGAN can also achieve very good performance in validity, uniqueness and novelty, we notice the big margin HAEGAN has over AEGAN in the structure-related metrics. This suggests that working with the hyperbolic space is necessary in our approach and the hyperholic space better represents structural information.
Lastly, the alternative hyperbolic models all suffer from numerical instability and training reports NaN. This is not surprising since hyperbolic neural operations are known easily make training unstable, especially deep and complex networks. The result reveals the stronger numerical stabil- ity of HAEGAN, which highlights the importance of (1) the overall framework of HAEGAN (v.s. HVAE); (2) the fully hyperbolic layers (v.s. HAEGAN-H); (3) the Lorentz direct concatenation (v.s. HAEGAN-3, HAEGAN-T).
6 CONCLUSION AND FUTURE WORK
In this paper, proposed HAEGAN, a hybrid AE-GAN framework and showed its capability of generating faithful hyperbolic examples. We showed that HAEGAN is able to generate both synthetic hyperbolic data and real molecular data. In particular, HAEGAN delivers state-of-the- art results for molecular data in structure-related metrics. It is not only the first hyperbolic GAN model that achieves such effectiveness, but also a deep hyperbolic model that does not suffer from numerical instability. We expect that HAEGAN can be applied to broader scenarios due to the flexibility in designing the hyperbolic AE and the possibility of building deep models.
Despite the promising results, we point out two possible limitations of the current model. First, not all complex modules are directly compatible with HAEGAN. Indeed, if the gated recurrent units (GRU) were used in our molecular generation task, the complex structure of GRU would cause unstable training and that is why a hyperbolic linear layer is used instead. Nevertheless, we expect defining more efficient hyperbolic operations that incorporate recurrent operations may alleviate the problem and leave it to future work. Second, although the hyperbolic operations in HAEGAN do not require going back and forth between the hyperbolic and the tangent spaces, need to use exponential maps when sampling from the wrapped normal distribution. We will also work on more efficient ways of sampling from the hyperbolic Gaussian.

---

Under review as a conference paper at ICLR 2023
REFERENCES
James W Anderson. Hyperbolic geometry. Springer Science & Business Media, 2006.
Martin Arjovsky, Soumith Chintala, and Léon Botto.. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-223. PMLR, 2017.
Gregor Bachmann, Gary Bécigneul, and Octavian Ganea. Constant curvature graph convolutional networks. In International Conference on Machine Learning, pp. 486-496. PMI.,, 2020.
Guy W Bemis and Mark A Murcko. The properties of known drugs. 1. molecular frameworks. Journal of medicinal chemistry, 39(15):8887-289, 1996.
Thomas Blaschke, Marcus Olivecrona, Ola Engkvist, Jürgen Bajorath, and Hongming Chen. Appli- cation of generative autoencoder de novo molecular design. Molecular informatics, 37(1-2):
1700123, 2018.
Marián Boguná, Fragkiskos Papadopoulos, and Dmitri Krioukov. Sustaining the internet with hy- perbolic mapping. Nature communicutions, 1(1):1-8, 2010.
Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton. Latent variable modelling with hyperbolic normalizing flows. In International Conference on Machine Learning, pp. 1045-1055 PMIR, 2020.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arX4v:2104.13478, 2021.
James W Cannon, William J Floyd, Richard Kenyon, Walter R Parry, et al. Hyperbolic geometry. Flavors of geometry, 31(59-115);, 1997.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. Advances in neural information processing systems, 32:4868 4879, 2019.
Ines Chami, Albert Gu, Dat P Nguyen, and Christopher Ré. Horopca: Hyperbolic dimensionality reduction via horospherical projections. In International Conference on Machine Learning, pp- 1419-1429. PMLR, 2021.
Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Fully hyperbolic neural networks. arXiv preprint arXiv:2105. 14686, 2021.
Nurendra Choudhary and Chandan K Reddy. Towards scalable hyperbolic neural networks using taylor series approximations. arXiv preprint arXiv:2206.036100 2022.
Jindou Dai, Yuwei Wu, Zhi Gao, and Yunde Jia. A hyperboli--to-hyyeroooii graph convolutional network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 154-163, 202
Shuyang Dai, Zhe Gan, Yu Cheng, Chenyang Tao, Lawrence Carin, and Jingjing Liu. Apo-vae: Text generation in hyperbolic space. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp- 416 4316-31,270 2021b.
Nicola De Cao and Thomas Kipf. MolGAN: An Implicit Generative Model for Small Molecular Graphs. ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018.
Daniel C Elton, Zois Boukouvalas, Mark D Fuge, and Peter W Chung. Deep learning for molecular design a review of the state of the art. Molecular Systems Design & Engineering, 4(4):828-849, 2019.
Pengfei Fang, Mehrtash Harandi, and Lars Petersson. Kernl methods in hyperbolic spaces. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10665-10674, 2021.
10

---

Under review as a conference paper at ICLR 2023
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems, 31:5345- 5355, 2018.
Rafael Gomez-Bombareelle Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato Timothy Hirzel,
Benjamín Sánchez- Lengeling. Dennis Sheberla, Jorge Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven contin- uous representation of molecules. ACS centrul science, 4(2):268 276, 2018
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International Conference on Machine Learning, pp. 2323-2332. PMLR, 2018.
Wengong Jin, Kevin Yang, Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- graph translation for molecule optimization. In International Conference on Learning Represen- tations. 2019.
Artur Kadurin, Alexander Aliper, Andrey Kazennov, Polina Mamoshina, Quentin Vanhaelen, Kuzma Khrabrov, and Alex Zhavoronkov. The coruucopia of meaningful leads: Applying deep adversarial autoencoders for new molecule development in oncology. Oncotarget, 8(7):10883, 2017a.
Artur Kadurin, Sergey Nikolenko, Kuzma Khrabrov, Alex Aliper, and Alex Zhavoronkov. drugan: an advanced generative adversarial autoencoder model for de novo generation of new molecules with desired molecular properties in silico. Molecular pharmaceutics, 14(9):3098-1144, 2017b.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in PyTorch, 2020.
Dmitri Krioukov, Fragkiskos Papadopoulos, Maksim Kitsak, Amin Vahdat, and Marián Boguná. Hyperbolic geometry of complex networks. Physical Review E, 82(3):036106, 2010.
Mare Law, Renjie Liao, Jake Snell, and Richard Zemel. Lorentzian distance learning for hyperbolic representations. In International Conference on Machine Learning, pp. 3672-3681. PMLR, 2019.
Diego Lazcano, Nicolás Fredes Franco, and Werner Creixell. HGAN: Hyperbolic generative adver- sarial network. IEEE Access, 9:96309-96322 2021. doi: 10. 1 10/9/ACCCSSS.22221.4477
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. 2010.
Available; http:///wwm.ccc...co////////////
Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. Advances in Neural Information Processing Systems, -248112--244,,, 2019.
Emile Mathieu, Charline Le Lan, Chris J. Maddison, Ryota Tomioka, and Yee Whye Teh. Con-
tinuous hierarchical representations with poincaré variational auto-encoders. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped nor- mal distribution on hyperbolic space for gradient- gradient-based learning. In International Conference on Machine Learning, pp. 4693-4702. PMLR, 2019.

---

Under review as a conference paper at ICLR 2023
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representa- tions. Advances in neural information processing systems, 30, 2017.
Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In International Conference on Machine Learning, pp. 3779-3788, PMLR, 2018.
W. Peng, T. Varanka, A. Mostafa, H. Shi, and G. Zhao. Hyperbolic deep neural networks: A survey, IEEE Transactions on Pattern Analysis Machine Intelligence, December 2021. doi: 10.119//PAMM222111333222
Wei Peng, Jingang Shi, Zhaoqiang Xia, and Guoying Zhao. Mix dimension in poincaré geometry for 3d skeleton-based action recognition. Proceedings of the 28th ACM International Conference on Multimedia, pp. 1432-1440, 2020.
Daniil Polykovskiy, Alexander Zhebrak, Dmitry Vetrov, Yan Ivanenkov, Vladimir Aladinskiy, Polina Mamoshina, Marine Bozdaganyan, Alexander Aliper, Alex Zhavoronkov, and Artur Kadurin. En- tangled conditional adversarial autoencoder for de novo drug discovery. Molecular pharmaceu- tics, 15(10):4398 4405, 2018.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Molecular sets (moses): a benchmarking platform for molecular generation models. Frontiers pharmacology, 11:1931, 2020.
Heinz Prüfer. Neuer beweis eines satzes über permutationen. Arch. Math. Phys, 27(1918))742-744 1918.
Oleksii Prykhodko, Simon Viet Johansson, Panagiotis-Christos Christos Kotsias, Josep Arús-Pous, Es- ben Jannik Bjerrum, Ola Engkvist, and Hongming Chen. A de novo molecular generation method using latent vector based generative adversarial network. Journal of Cheminformatics, 11(1):1- 13, 2019.
Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman. Moser flow: Divergence- based generative modeling on manifolds. Advances in Neural Information Processing Systems, 34, 2021.
Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. Representation tradeoff's for hyperbolic embeddings. In International conference on machine learning, pp. 4460-4469. PMLR, 2018.
Marwin H. S. Segler, Thierry Kogej, Christian Tyrchan, and Mark P. Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Central Science, 4(1): 120-131, 2018. doi: 10. /t02///csccenssc.700055222
Chence Shi Minkai Xu*, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. GraphAF: flow-based autoregressive model for molecular graph generation. In International Conference on Learning Representations (ICLR), 2020.
Ryohei Shimizu, YUSUKE Mukuta, and Tatsuya Harada. Hyperbolic neural networks+++ In Inter- national Conference on Learning Representations, 2021.
Martin Simonovsky and Nikos Komodakis. GraphVAE: Towards generation of small graphs using variational autoencoders. In International conference on artificial neural networks, pp. 412-422. Springer, 2018
Rishi Sonthalia and Anna Gilbert. Tree! i am no tree! am a low dimensional hyperbolic embedding. Advances in Neural Information Processing Systems, 33:845-856, 2020.
Teague Sterling and John J. Irwin. Zine 15 ligand discovery for everyone. Journal of Chemical Information and Modeling, 55(11):2324-255) 2337, 2015. doi: 10. 10. 10221//as.ccm.550005333. PMID: 26479676.
12

---

Under review as a conference paper at ICLR 2023
Li Sun, Zhongbao Zhang, Jiawei Zhang, Feiyang Wang, Hao Peng. Sen Su, and Philip S Yu. Hyper- bolic variational graph neural network for modeling dynamic graphs. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 4375-4383, 2021.
Taffee T Tanimoto. Elementary mathematical theory of classification and prediction. 1958.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Xiao Wang, Yiding Zhang, and Chuan Shi. Hyperbolic heterogeneous information network embed- ding. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 5337-5344, 2019.
Zhenxing Wu, Dejun Jiang, Chang-Yi Yu Hsieh, Guangyong Chen, Ben Liao, Dongsheng Cao, and Tingjun Hou. Hyperbolic relational graph convolution networks plus: simple but highly efficient qsar-modeling method. Briefings in Bioinformatics, 22(5):bbabl 12, 2021.
Menglin Yang, Min Zhou, Zhihao Li, Jiahong Liu, Lujia Pan, Hui Xiong, and Irwin King. Hyperbolic graph neural networks: A review of methods and applications. arXiv preprint arXiv:2202. 13852, 2022.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018a.
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generat- ing realistic graphs with deep auto-regressive models. In International conference on machine learning, pp. 5708-5717. PMLR, 2018b.
Ke Yu, Shyam Visweswaran, and Kayhan Batmanghelich. Semi-supervised hierarchical drug em- bedding in hyperbolic space. Journal of chemical information and modeling, 60(12):5647-5557, 2020.
Tao Yu and Christopher M De Sa. Numerically accurate hyperbolic embeddings using tiling-based models. Advances in Neural Information Processing Systems, 32, 2019.
Tao Yu and Christopher M De Sa. Representing hyperbolic space accurately using molti-compooent floats. Advances in Neural Information Processing Systems, 34:15570-15581, 2021.
Yiding Zhang. Xiao Wang, Chuan Shi, Xunqiang Jiang, and Yanfang Fanny Ye. Hyperbolic graph attention network. IEEE Transactions on Big Data, 2021.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
The Appendix is organized as follows. In | we describe the preliminaries on hyperbolic geometry. In BB we describe the details of hyperbolic GAN and demonstrate generating toy distributions. In $C we show detailed results from MNIST generation using HAEGAN. In $D we present more analysis for the Lorentz direct concatenation. In
we provide a detailed description of the AE used in we carefully describe all the experimental details and neural network architectures that did not cover in the main text for reproducing our work. In G we illustrate a subset of examples of molecules generated by HAEGAN.
S
HAEGAN for molecular generation. In F
A PRELIMINARIES
A.1 1
HYPERBOLIC GEOMETRY
We describe some fundamental concepts in hyperbolic geometry related to this work.
The Lorentz Model The Lorentz model L (C. of dimensional hyperbolic space with constant negative curvature is an vi-dimensional manifold L embedded in the ( 1)-dimensional Minkowski space, together with the Riemannian metric tensor diag([ [[11, 1, 1, where 1, de-
notes the n-dimensional vector whose entries are all I's. Every point in L is represented by
x
2t > 0, Es E R" and satisfies (æ, x = 1/K, where (; e is the Lorentz inner
product induced by g:
x )) x gy Fife + x Mo. H, y E L
Geodesics and Distances Geodesics are shortest paths in manifold, which generalize the no- tion of "straight lines" in Euclidean geometry. In particular, the length of a geodesic in Lg (the "distance") between æ, y L is given by
1 cosh - 'K v).
dc(x, 3) V-K
(8)
Tangent Space For each point a E Lg the tangent space at is Ta = {y E Ra+1 (y, c = 0}. It first order approximation of the hyperbolic manifold around point and is subspace of R=+1, We denote |0ا|c = (v, v) as the norm of E Ta
Exponential and Logarithmic Maps The exponential and logarithmic maps are maps between hyperbolic spaces and their tangent spaces. For x, E IL and v E T the exponential map exp K (v) TaLk +
L' maps tangent vectors to hyperbolic spaces by assigning v to the point exp (v) = (1), where y is the geodesic satisfying 7(0) = x and /((0) = v. Specifically,
exp K (v) = cosh(o) + sinh () V
-. ø = V-Kl|
The logarithmic map log (v) I. - TaILik is the inverse map that satisfies log (exp K (v:) Specifically,
v.
log (y) cosh ((5) y whe
3 = K ((x, y-
(10)
V-K K ly wal
Parallel Transport For two points ar, y E T. the parallel transport from i to y defines map PT which "transports" vector from TaL to TyL along the geodesic from to y. Parallel
transport preserves the metric, i.e. Vu, E Talk (PT" particular, the parallel transport in L'Kk is given by
(v),
PT PT (u)) = (v, u) In
(y. 2)2
P x (v)
(x + y)
(1I)
-1/K
(x, y)
14

---

Under review as a conference paper at ICLR 2023
A.2 MORE HYPERBOLIC LAYERS
We reviewed the Lorentz linear layer and the centroid distance layer in the main text. In this section, we review more hyperbolic layers.
The notion of the "centroid" of set of points is important in formulating attention mechanism and feature aggregation. In the Lorentz model, with the squared Lorentzian distance defined as
de (az, y) 2/K 2(z, y) y E L'k, the hyperbolic centroid (Law et al. 2019) is defined to be the minimizer that solves minuelg E pidl (2), (e) subject to æ, i == 1, .N. A closed form of the centroid is given by
R. 24 2
Σ, 14 0,
E V:
= HCent( (
(12)
V-K
where is the matrix whose i-th row is x Extracting the hyperbolic centroid following hyperbolic linear layers produces hyperbolic graph convolutional network (GCN) layer (Chen et al. 2021):
HGHH (1-)),) = HCent( (HLiieeHHiaaaaa... (z([-1) u (o))).1
(13)
where x. is the feature of node in layer I, di denotes the dimensionality of layer I, and ((v is the set of neighbor points of node v.
A.3 EMBEDDING FROM EUCLIDEAN TO HYPERBOLIC SPACES
It is possible that a dataset is originally represented Euclidean, albeit having hierarchical struc- ture. In this case, the most obvious way of processing is to use the exponential or logarithmic maps that
represent data in the hyperbolic space. In order to map Rn the hyperbolic space - (Ngano et al. (2019) add zero padding to the front of to make a vector in Tally and then apply the exponential map. This Euclidean to Hyperbolic (E2H) operation was originally used for sampling, but can also be generally used to map from the Euclidean to the hyperbolic spaces. Specifically.
y = E2H (t) = expo (IP)),
(14)
where o [V-1/K, ..... is the hyperbolic origin.
For better expressivity, especially when the input E RR is one-hot, one can first map the input to a hidden embedding = W E Rm with a trainable embedding matrix W Raxm, Then, is mapped to hyperbolic space by the E2H operation defined above. That is,
y HEmbed,,, (w) E2H (Wæ).
(15)
This layer was previously used by Nagano et al. (2099) for word embedding.
We remark that besides using the E2H operation, other embedding methods (Nickel & Kiela 2017] 2018 Sala et al.) 2018: (onthalia & Gilbert 2020) exist. We use E2H our experiments since is simple to incorporate it in HAEGAN.
A.4 MORE RELATED WORKS
Machine Learning in Hyperbolic Spaces A central topic in machine learning is find methods and architectures that incorporate the geometric structure of data (Bronstein et al.] 2021). l). Due to the data representation capacity of the hyperbolic space, many machine learning methods have been designed for hyperbolic data. Such methods include hyperbolic dimensionality reduction (Chami jet al. 202I) and kernel hyperbolic methods (Fang et al. 2021). Besides these works, deep neural networks have also been proposed in the hyperpolic domain. One of the earliest such model was the Hyperbolic Neural Network (Ganea et al. 2018) which works with the Poincaré ball model of the hyperbolic space. This recently refined in the Hyperbolic Neural Network ++ (Shimizu et al. 202I). Another popular choice is to use the Lorentz model of the hyperbolic space (Chen et al. 2021
Yang et al. 2022). Our model also uses Lorentz space for numerical stability.
15

---

Under review as a conference paper at ICLR 2023
Hyperbolic Graph Neural Networks Graph neural networks (GNNs) successful models for
learning representations of graph data. Recent studies (Boguná et al. 2010 Krioukov et al. 2010:
Sala et al. 2018 Sonthalia & Gilbert 2020) have found that hyperbolic spaces are suitable for
tree-like graphs and a variety of hyperbolic GNNs (Chami et al. 2019 Liu et al. 2019 Bachmann
al. 2020 Dai et al.] 202Ta Chen et al.
202T have been proposed. In particular, Chami et al.
[2019) Liu et al. (2019): Bachmann et al. (2020) all performed message passing, the fundamental
operation in GNNs, in the tangent space of the hyperbolic space. On the other hand. Dai et al.
(021a); Chen et al. (202T) designed fully hyperbolic operations so that message passing can be done completely in the hyperbolic space. Some recent works address special GNNs. For instance, Sun et al. (2021) applied hyperbolic time embedding to temporal GNN, while Zhang et al. (2021) designed a hyperbolic graph attention network. We also notice the recent survey on hyperbolic GNNs by Yang et al. (2022).
Molecular Generation State-of-the-arr methods for molecular generation usually treat molecules as abstract graphs whose nodes represent atoms and edges represent chemical bonds. Early methods for molecular graph generations usually generate adjacency matrices via simple multilayer percep- trons (Simonovsky & Komodakis 2018 De Cao & Kipf 2018). Recently, [ et al. (2018) 2019) proposed to treat a molecule as multiresouuiioaal representation, with a junction-tres trect scaffold, whose nodes represent valid molecular substructures. Other molecular graph generation methods in- clude (You et al. 201 Sa Shi* et al. 2020). Methods that work with SMILES (Simplified Molecular Input Line Entry System) notations instead of graphs include (Segler et al. 2018, Bomber-Bo et al. 2018 Blaschke et al. 2018 Kadurin et al. [017a ( PPolykoskky et al.][]0018 ( et al.]
2019). Since the hyperbolic space promising for tree-like like structures, hyperbolic GNNs have also
been recently used for molecular generation (Liu et al. 2019 Dai et al. 2021a).
B HYPERBOLIC GENERATIVE ADVERSARIAL NETWORKS
B. DETAILS THE HYPERBOLIC GAN
Wrapped normal distribution The wrapped normal distribution is a hyperbolic distribution whose density can be evaluated analytically and is differentiable with respect to the parameters Nagano et al. 2019). Given J L and Σ E RTx", to sample normal distribution G(u, Σ), we first sample a vector from the Euclidean normal distribution
L from the wrapped
[0]
(0 Σ). then identify e as an element v E ToL so that v
We parallel transport this v to
i
u= = PT and then finally map u to x = exp exp E Lix-
Generator and critic The generator pushes forward a wrapped normal distribution ((o. I) to a hyperbolic distribution via a cascading of hyperbolic linear layers. The critic aims to distinguish between fake data generated from the generator and real data. It contains a cascading of hyperbolic linear layers, and a centroid distance layer whose output is score in R.
Training We adopt the framework of Wasserstein GAN (Arjovsky et al. 201 7), which aims to minimize the Wasserstein-1 (W1) distance between the distribution pushed forward by the generator and the data distribution. Since de a valid metric, the W, distance between two hyperbolic distribution P., P,, defined on the Lorentz space is (IP, P.y yEI(FF.P..)
inf
((x,))~~ldc<0,, y)],
(16)
where [1(P,,, IP)) is the set of all joint distributions whose marginals are IP, and PP,, respectively. By Centtorovic--Rubisseee duality (Villani 2009), we have the following more tractable form of Wi distance
WI,,,, = sup Eg-P, [D(x) - Exw,, E |DI
(17)
where the supremum is over all I-Lipschitz functions LL IR represented by the critic. To enforce the 1-Lipschitz constraint, we adopt penalty term on the gradient following Gulrajani et al. 2017). The loss function is thus
LWGAN = E [D(E)] 3
E waF
[D(x]]
E + A =
[[IVD())]C - 1)²].
(18)
16

---

Under review as a conference paper at ICLR 2023
where (z))) is the Riemannian gradient of (z)) at 2ł, 1 is the generator distribution and P, is the data distribution, Ps samples uniformly along the geodesic between pairs of points sampled from Fg and P. Next, prove Proposition 3.1 to validate this sampling regime. The proof is obtained by carefully transferring the Fuclidean case (Gulrajani et al. 2017) to the hyperbolic space.
B.2 PROOF OF PROPOSITION
Proof. For the optimal solution f*, we have
P P(a.y))
- (2)) (())
de(v,
(19)
Let (t) = (x:) (x)), 0 t,t' < 1. Following Gulrajani et al. 2017 it is clear that ub is
dz(a, y) y)-Lipschitz, and (x - f ( = (t) = (dc( (= = f (x) + tdc( y) f=(x) c.
Let u, = [ε(يN)) E TLK be the unit speed directional vector of the geodesic at point xt. Let [- + Li be a differentiable curve with a<D) 2, and a' a'(0) ue Note that 7(()) dz ()a'(0) Therefore.
lim a(h) = lim (++ dz wi k-0
lim a x+ 3g k-0
(20)
h-00
The directional derivative be thus calculated
d
S" (a(li) 5 (a(0))
V uu fr (xt)
f*((TT)
lim
dr
h
==0
f" De+ (ettgttno)) (x:s)
lim h-10
(21)
f (xx) + (t dclan)) )d((x, y) f (x) - =dc((, y)
lim h-+0
h
lim =1
h- h
Since f" is I-Lipschitz, we have (x $ 1. This implies
1 12 (v))2
= (uz. Vf (zx) + (x,) (u, Vf" (2,)) u, 4.11
(22)
Vu, (x:) + v (z:) u, u. f (x:)
= 1+ | (xt) - -ul1 2 1.
Therefore, we have 1 = 1 + V (x:) ue 122 vf" (æx) ue This yields V (xx) "c(m.y))
B.3 TOY DISTRIBUTIONS
We use set of challenging toy 2D distributions explored by Rozen et al. (2021) to test the effec- tiveness of the hyperbolic GAN. We create the dataset the same way using their code For our experiment, the training data are prepared in the following manner. We first sample 5,000 points from the toy 2D distributions and scale the coordinates [-1, Then, we use the F2H operation 14 to map the points the hyperbolic space. These points treated as the input data of the hyperbolic GAN. Next, we use the hyperbolic GAN to learn the hyperbolic toy distributions. The generator and the critic both contain 3 layers of hyperbolic linear layers and 64 hidden dimensions at
17

---

Under review as a conference paper at ICLR 2023
Input
Samples
Inpai data
Samples
Input
Samples
Figure 4: Input data and generated samples from the hyperbolic GAN. The hyperbolic data points are transformed to the tangent space of the origin by the logarithmic map.
each layer. The input dimension for the generator is 128. We present the training details in Appendix
After we train the hyperbolic GAN, we sample from it and compare with the input data. Note that the input data and the generated samples are both in the hyperbolic space. To illustrate them, we map both the input data and the generated samples to the tangent space of the origin by applying the logarithmic map. We present mapped input data and generated samples in Figure Clearly, hyperbolic GAN can faithfully represent the challenging toy distributions in the hyperbolic space.
C MORE RESULTS FROM MNIST GENERATION WITH HAEGAN
In the HAEGAN for generating MNIST, the encoder of the AE consists of three convolutional layers, followed by an E2H layer and three hyperbolic linear layers, while the decoder consists of three hyperbolic linear layers, logarithmic map the Euclidean space, and three deconvolutional layers. We describe the training procedures as follows. Firstly, we normalize the MNIST dataset and train the AE by minimizing the reconstruction loss. Secondly, we use the encoder to embed the MNIST in hyperbolic space and train the hyperbolic GAN with the hyperbolic embedding. Finally, we sample hyperbolic embedding using the generator and use to produce an image by applying the decoder. We describe the detailed architecture and settings in Appendix F4
The training curves of the hyperbolic GAN of HAEGAN in the MNIST generation task are shown in Figure 5 which we compare with an Euclidean Wasserstein GAN. The critic loss includes the gradient penalty term. We observe that the trend loss in the hyperbolic GAN similar to the Euclidean (both the generator and critic) and no instability from the hyperbolic model.
100
75
Generator Loss
Critic Loas
50
-50
25.
100
150
-25
-200
250
-55
-30
Critic Loss
100
3350
10
20
40
50
70
10
20
40 Epuchs
50
60
70
Epochs
Figure 5: The training loss of generator and critic in the MNIST generation task. Left: hyperbolic Wasserstein GAN. Right: Euclidean Wasserstein GAN.
In Table 3 and Table 4
report the quantitative results for the MNIST generation task. First, we compare the negative log-likelihood (NLL) results between our method and hyperbolic VAEs
Mathieu al. 2019 Nagano et al. 2019 Bose et al. 2020). The results directly taken from
18

---

Under review as a conference paper at ICLR 2023
the respective papers, which were produced from different numbers of samples: in Nagano et al. (2019), the NLL is calculated with 500 samples, while Mathieu et al. (2019) used 3,000 samples. We generate 5,000 samples and compare the NLL with them. Then, also calculate the FID and compare it with HGAN (Lazcano et al., 2021). Our NL. results are comparable with the hyperbolic VAEs while FID is slightly better than HGAN.
Table 3: Quantitative comparison between HAEGAN and other methods in the MNIST generation task. Log Likelihood (1sttd) for different embedding dimensions is reported. We use 5000 samples to estimate the log-likelihood. Results for (Mathieu al. 2019; Nagano et al. 2019 Bose et al. 2020 are taken directly from the respective paper. * indicates numerically unstable settings.
Model
Dimensionality
10
20
N-VAF Mathieu et
2019
1144.=004
.11.7=0.11
100.2=0.1
99.640.1
P-VAE (Wrappdd) (Mathicn et a 2019
-143.8=0.6
-114.7=0.1
-100.0=0.1
-97. -97.1
P--AE (Riemannian)
-142.520.4
-114.1=0.2
997.020.1
Vanilla VAE (Naciino et mt
2019
140.4520.47
-10578820.51
-86.25e0.55
-77.89.0.66
Hyperholic VAT Nagano
79.2340.20
AEGAN (Oun)
141.1320.49
-111.41=0.48
-97.8300.37
19011840.1
HARGAN (Ours)
140.24:0.55
1111.29=0.59
98.15+0.41
91.37+0.39
2
V-VAE Bose all
2020
1130
-11.6600.2
-100.0e0.02
BL-VAE [Sose
2024
-113.720,9
-99 99.810.2
NC
Bose
2040
-139.2:0.4
115.220..
98.70.3
TC Hose et
MTA
-112.5=0.2
-99.3+0.2
WEC
2020
-136.5#2.1
-112.8:0.5
-99.4
Table 4: Quantitative comparison between HAEGAN and HGAN Lazcano et al. 2021) in the
MNIST generation task. Fréchet inception distance (#std) is reported. Results for Lazcano et al., 202T are taken directly from the paper.
Model
FID
HGAAN
Lazcano et al
2021
$4.95
HCGAN
azcano al
2021
12.43
HAEGAN
(Ous)
8.05=0.37
D MORE ANALYSIS OF CONCATENATION
D.I PROOF OF THEBREM 4.1
Proof. 1. First, consider the case where y* is one of the spatial components of According to (5).
y" copy of an entry x;, for some E {1. ,N). Therefore, if + j. Oy" /0x; zero vector; if = Oy" /0æ, one-hot vector. In both cases, ||0y" /0x < 1.
Next, consider the case where y² Yt the time component of y. According to Oy" / =
Bj.
and thus Oy"
|
< since the curvature K 0.
2 2
0x,
We conclude that |yy" /0xx || < 1 for any entry 3 in y.
2. Write = [ _I_
According the definition of the Lorentz tangent concatenation well as formulas (9) and 110) for I {1, N). the p-th entry of the vector 21, be written
N.
19

---

Under review as a conference paper at ICLR 2023
in the following concrete form:
cosh -1
K V (-K
|00, 10.112-0
sinh(4 (A)
A
V-K
| |I
sinh (v K'2 ||21,
112
K -1
sinh(4) A)
J1,
(23)
V-K
|| ||
where
N Σ
1/2
A= V-K K
sinh-
K2 || 11?
K
(24)
Let j # I. Differentiating Zi,p with respect to 2j,ks the k-tth entry of æ,,, yields
0ZL.p
Ci
(cosh(A)
sinh(A)
sinh -1
(V
|| (V
K -1)
j
A
A3
(25)
V ||
V ||aj 11
1+K
K
where
Cj = =ishin--1 (V [0,, 11-11 -1)
F1,
(26)
|
does not depend on ; Arbitrarily take fixed x., for all * / with particularly zj,4 O. Also arbitrarily take fixed 2t,q for f p. We claim that 02i,p + oo as F1,0 + oc.
jz
To prove the claim, first note that
Oc
27,p
+
| 11
||a; 11
||m 112 11
K
sinh
K
||,,
12- K -
1
x Fl,p
(27)
x
Since 2²,p I || and sinh (VKK3 1 Consequently, 0C1/0mz > and thus Ci a positive term that increases with F1.p- More- over, sinh-! (v R2 x K
0, the second term in (27) is positive.
|
Ij.h
in 25 a fixed posi-
V || 11' 1/K
|
1+K
cosh A
sinh A)
tive term that does not depend on F1,y- Hence, we only need to show that
+ 00
A
A
cosh (t)
sinh(t)
as Fi, p + co. Since A - oo as Ti,p - xc, this reduces to proving f(t) as + oo, which an immediate result once we write explicitly that
+ 0o
t2
t
e
+ (t 1)e*
((t) = 2 13
(28)
12
We conclude that for any M 0, there exist {x and an entry 8" of z, i.c. 2t,p above, for which all entries of 0=" /0n, are less than M. Thus it also holds that ||0=" /dar lmy ax > M. n
20

---

Under review as a conference paper at ICLR 2023
(a5)
(62)
(b3)
-10
Figure 6: Illustration of the gradients of both concatenation methods: (a) Lorentz tangent concate- 0z, izs0, (a3) & (b3) dr,
0z.1
nation (b) Lorentz direct concatenation. (al) & (bl) dr, (a2) & (b2) dr,
D.2 GRADIENT CONCATENATION
In this section, we numerically validate Theorem 4.1 by showing the gradients of the Lorentz tan- gent concatenation and the Lorentz direct concatenation under the following simple setting. We concatenate two 1-dimensional Lorentz. vectors to obtain a 2-dimensional Lorentz vector.
Denote 2 1
EL_, to be the two input vectors under the Lorentz model with K 1. For both the Lorentz tangent concatenation and the Lorentz direct concatenation, let x = [zt, 200, 2
E _ and y
e L22 denote the concatenated vector of x and y. Nu- merically, consider the range of x. Vs [[1000, 100) and caleulate the gradients of each entry of
θz
and Oz Or.
with respect to Zs. that is, Ox.
Oz.
E(-00,0000
.. 1l-1000100)
1..88E
E....EI-1000 -11.8.E-10001000 100
We plot them in Figure 6 We remark that due symmetry, the graphs are the same for Dy.
From Figure 6, we observe unbounded gradients in each component of the gradient of the Lorentz tangent concatenation. On the other hand, all the components of the gradient of the Lorentz direct concatenation has an absolute value bounded by 1. The results of this numerical experiment have validated the conclusion in Theorem
D.3 ANALYSIS OF EFFECT HYPERBOLIC DISTANCES
In this section, we perform additional analysis Lorentz direct concatenation and Lorentz tangent concatenation, particularly their effect on hyperbolic distances.
First, we study the hyperbolic distances to the hyperbolic origin for both concatenation meth- ods. Suppose have E LL and y E LL Let z' HTCat(z, y) pp+m-1 be their hyperbolic direct concatenation and hyperbolic tangent concatenation, respectively. We compare the difference between dz(z,, o) and dz(2',, .)) as follows. Note that the distance between arbitrary point L and the origin only depend the time
HCat(æ, y) L and
21

---

Under review as a conference paper at ICLR 2023
Direct Tangent
Direct Tingent
0.8
0.6
Frequency
Frequeney
0.5
0.6
0.4
0.4
0.2
0.0
0.0
0.0
0.5
1.5
2.0
2.5
3.0
3.5
4
Distance
Distance
Figure 7: Difference between concatenated distances and original distances with n = 3. Left: spatial normal. Right: wrapped normal.
Direct
L75
Tang
0.7
Tingent
1.50
0.6
Frequency
Frequeney
1.25
0.5
1.00
0.4
0.75
0.50
0.2
0.25
0.1
0.00
0.0
0.0
0.5
1.0
1,5
2.0
2.5
Distance
Distance
Figure 8: Difference between concatenated distances and original distances with n 16. spatial normal. Right: wrapped normal.
Left:
component:
1
dc(x.o)=
V-K cosh - (k,
V-K cosh (- Krz).
(29)
Hence, the distance information is completely contained the time component. After the concatena- tion, the time component /x² + vi + 1/K. K. Consequently, for Lorentz direct concatenation, the distance is
dc(2, 0) 1 cosh -1 -K v2 1/K
(30)
V-K
For Lorentz tangent concatenation, since both the logarithmic and exponential maps reserve the distances, one has
de(z',0) Vdc(x, + +d
(31)
-V-K (-osh) + cosh (-sh-2(-KKy))
Although the hyperbolic distance dz(2,0) is not the squared sum of dz(æ, o) and de(y, o), dc(2, o) is larger than each of de(xr, o) and dc(y, o). On the other hand, after concatenation, (z',,', dx o) + de (y, o). This relation agrees with the Euclidean concatenation. However, norm-preservation is not why concatenation works in the Euclidean domain. Therefore, we don't consider this tion more efficient and stable, and no information is lost during concatenation. Therefore, is still preferred as neural layer.
advantage of the Lorentz tangent concatenation. The Lorentz direct concatena-
More importantly, we study how concatenation changes the relative distances, which is closely related to stability. Specifically,, perform the following experiments. Given x, y, E L.K.
22

---

Under review as a conference paper at ICLR 2023
3.5
Direct Tingent
0.8
Direct
Tingent
3.0
0.7
Frequen
2.5
Frequeney
-0.6
0.5
0.4
1.5
0.3
1.0
0.5
0.1
0.0
0.0
0.0
0.5
1.0
2.0
6
Distance
Distance
Figure 9: Difference between concatenated distances and original distances with 64. Left:
spatial normal. Right: wrapped normal.
Dim=64
Dim=64
Dim=16
Dim- 16
2.0
Dim-3
Dim-3
Frequency
Frequency
0.6
1.0
0.4
0.2
2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
0.0
Difference
Difference
Figure/ 10:
|d<(x Mc) - dc(x, v)1 - Idc(x'c. vc) dc(x, Left: spatial normal.
Right:
wrapped normal.
let Øc HCat(æ c) be the direct-concatenated version æ and c, while ye = HCat c) be the direct-concatenater version of y and Similarly denote xy HTCat(æ, c) and V. = -HCCal (y, c). Since the same vector e is attached to æ and y, we naturally hope del (Re, ye) and de(w', y') do not deviate much from de(a, (æx y).
We describe our experiments as follows. Take K = -1 We randomly sample three points inde- pendently from Lix as æ, y and respectively. We have scenarios for sampling the points: (1) "spatial normal": the points are sampled so that their spatial components follow the standard normal distribution; (2) "wrapped normal": the points are sampled from the wrapped normal distribution with unit variance. In each scenario, for E {3. 16, 64), we do the experiments for 10,000 times. We report the distances (x He dz(x, y) and |dc(''s v²) dz y) Figures as well as their differences in Figure 10
Our experiments clearly show that, especially for large dimensions, the distance between de (mey Ye) and (x, y) is smaller than the distance between de(='' v²)) and de (æx, y). In particular, in many cases, (dg ye) dg(2x (x, y) around zero. On the other hand, |dc(x' y²)) - de (x, 3) tend to 16, 64, especially when samples follow the wrapped normal distribution. From
be large when this result, the Lorentz Direct Concatenation should be preferred to the Lorentz Tangent Concatena- tion. In particular, the significant expansion of distance when concatenating with the same vector, in the case n = 64, may be one cause numerical instability (note that we use the same dimensionality for the experiment in $4.2 and in the molecular generation task).
23

---

Under review as a conference paper at ICLR 2023
E DETAILS OF AE IN MOLECULAR GENERATION
In this section, we carefully describe the encoders and decoders, which compose the AE used in the HAEGAN for molecular generation. The basic structure of the AE in the molecular generation task illustrated in Figure
Notation We denote a molecular graph as G = (Vc; EG), where Va is the set of nodes (atoms) and EG the set of edges (bonds). Each node (atom) E VG has a node feature 2, describing its atom type and properties. The molecular graph is decomposed into a junction tree T = (Vr, Er) where Vr is the set of atom clusters. We use u, tt w to represent graph nodes and i, tree nodes, respectively. The dimensions of the node features of the graph De and the tree æ; are denoted by dg. and dr,, respectively. The hidden dimensions of graph and tree embeddings are do, dr, respectively.
represent
ZG
Molecular Graph
Graph Encoder
Graph Decoder
Junction Tree
Tree
Hyperbolic
Tree
Encoder
Embedding
Decoder
Figure 11: Illustration of the autoencoder used in the HAEGAN for molecular generation. The input molecular graph is firstly coarsened into the junction tree. Then both of them are encoded using graph and tree encoders to their respective hyperbolic embeddings 2T and zc. To reconstruct the molecule, we first decode the junction tree from Z,, and then reconstruct the molecular graph using the junction tree and 2G-
E. GRAPH AND TREE ENCODER
The graph encoder for the molecular graph contains the following layers. First, each node feature x, is mapped to the hyperbolic space via
(
E2H doo
(x.).
(32)
Next, the hyperbolic feature passed to hyperbolic GCN with la layers
HGCN(x 1 1, ,6.
(33)
Finally, we take the centroid of the embeddings of all vertices to get the hyperbolic embedding of the entire graph.
2G = (c)))
(34)
The tree encoder similar with the graph encoder, encodes the junction tree to hyperbolic em- bedding žy with hyperbolic GCN of depth ly. The only difference is that its input feature x; 's one-hot vectors representing the atom clusters in the cluster vocabulary. We need to use a hyperbolic embedding layer as the first layer of the network accordingly. Altogether, the tree encoder contains
24

---

Under review as a conference paper at ICLR 2023
the following layers in a sequential manner:
z; (0) H dr (2:s;;
x HGCN(al--)) 1 1= 1,..... IT,
(35)
2T HC))t(x!!7)).
E.2 TREE DECODER
we generate a junction tree T = (Vr, Fr)) using tree recurrent neural network in top-down and node-by-node fashion. The generation process resembles a depth- first traversal over the tree T', Staring from the root, at each time step t, the model makes a decision whether to continue generating child node or backtracking to its parent node. If it decides to generate new node, it will further predict the cluster label of the child node. It makes these decision based on the messages passed from the neighboring node. We remark that we do not use the gated recurrent unit (GRU) for message passing. The complex structure of GRU would make the training process numerically unstable for our hyperbolic neural network. We simply replace it with hyperbolic linear layer.
Similar to Jin et al. 2018
2019
(Im im)} denote the collection of the edges visited in depth-first traversal over T, where m = Forl. We store a hyperbolic message hic5, for each edge in E. Let E, be the set of the first t edges in E. Suppose at time step f, the model visit node i, and it visits node Je at the next time step. The message h is updated using the node feature x;, and inward messages ha We first use hyperbolic centroid to gather the inward messages to produce
Message Passing Let E =
nei HCent (Ho (HLineard,, ddz (f J B.,,)
(36)
and then map the tree node features to the hyperbolic space to produce
Zeur HEmbeddr, dy (2:,)-
(37)
Finally, we combine them using the Lorentz Direct Concatenation and pass them through hyper- bolic linear layer to get the message
hi.. = HLinear2xdr.dr
(HCat( Zeur: )).
(38)
Topological Prediction each time step t, the model makes a binary decision on whether to generate a child node, using tree embedding zr, node feature Dip and inward messages ha,s, using the following layers successively:
nei
HCent HCent(HLinearaay.d dr ((he, ((sDER))
Zeur
HEmbeda,, dr (zi,),
(39)
Zall HLinearax dr (HCat( (Zeur: Zuei, 2r
P: Softmax(HCDista, (Zall)).
Label Prediction If child node je is generated, we use the tree embedding 2y and the outward message higg predict its label. We apply the following two layers successively:
Zall HLinearzxdr,dr (HCat((ti.
(40)
q. Softmax(HCDDssaarr (zat)).
The output qe is distribution over the label vocabulary. When ji is root node, its parent in is dummy and the message is padded with the origin of the hyperbolic space h = o.
Training The topological and label prediction have two induced losses. Suppose p., å. are the the ground truth topological and label values, obtained by doing depth-first traversal on the real junction tree. The decoder minimizes the following cross-entropy loss:
7
I Tropo
Tecrose Pt), Laaeel = Σ
Tieross (g,, qc);
(41)
t=1
t=1
where Leross is the cross-entrope loss. During the training phase, use the teacher forcing strategy: after the predictions at each time step, we replace them with the ground truth. This allows the model to learn from the correct history information.
25

---

Under review as a conference paper at ICLR 2023
E.3 GRAPH DECODER
The graph decoder assembles molecular graph given a junction tree T = (V, E) and graph em- bedding zg- Let Gú be the set of possible candidate subgraphs around tree node i.e. the different ways of attaching neighboring clusters to cluster i. We want to design a scoring function for each candidate subgraph G E Gs.
To this end, we first use the hyperbolic GCN and hyperbolic centroid to acquire the hyperbolic embedding 2 of each subgraph G Specifically,
x (0) E2Hda, (x.), 0
x (1) (G 1= 1, lc.
(42)
2 0 HCent(a"la)).
Then, the embedding of the subgraph is combined with the embedding of the molecular graph 2g by the Lorentz Direct Concatenation to produce
all Hlinearzddo.da (HCatt(Zzg0) =G])),
(43)
which is then passed to the hyperbolic centroid distance layer get a score
HCDist (Zal)) E TR.
(44)
Training We define the loss for the graph decoder to be the sum of the cross-entropy losses in each G. Specifically, suppose the correct subgraph is G!
L Lassm = Σ
(1) -log Σ exp(s(0)
(45)
EG.
Similar to the tree decoder, we also use teacher forcing when training the graph decoder.
F DETAILED SETTINGS OF EXPERIMENTS
F.1 OPTIMIZATION
For all the experiments, we use the Geoopt package (Kochurov al. (0200) for Riemannian opti- mization. In particular, we use the Riemannian Adam function for gradient descent. We also use Geoopt for initializing the weights all hyperbolic linear layers our model with the wrapped normal distribution.
F.2 ENVIRONMENTS
All the experiments in this paper are conducted with the following environments.
GPU: RTX 3090
CUDA Version: 11.1
PyToorch Version: 1.9.0
RDKit Version: 2020.09.1.0
F.3 EXPERIMENT DETAILS FOR TOY DISTRIBUTION
We describe the details for the toy distribution experiment.
26

---

Under review as a conference paper at ICLR 2023
F.3.1
ARCHITECTURE DETAILS OF HYPERBOLIC GENERATIVE ADVERSARIAL NETWORK
Generator
Input: points in L sampled from ((o, diag(125h)
Hyperbolic linear layers:
Input dimension: 256
Hidden dimension: 128
Depth: 3
Output dimension: 3
Output: points in L
Critic
Input: points in L
Hyperbolic linear layers:
Input dimension: 3
Hidden dimension: 128
Depth: 3
Output dimension: 128
Hyperbolic centroid distance layer: LK -
Output: score IR
Hyperparameters
Manifold curvature: -1.
Gradient penalty coefficient:
For all hyperbolic linear layers:
Dropout: 0.0
Use bias: True
Optimizer: Riemannian Adam (31 0, = 0.9)
Learning Rate: 1e-4
Batch size: 128
Number of epochs: 20
Gradient penalty A: 10
F.4 EXPERIMENT DETAILS FOR MNIST GENERATION
We describe the detailed architecture for the MNIST Generation experiment.
F,4.1
ARCHITECTURE DETAILS OF AUTO-ENCODER
Encoder
Input: MNIST image with dimension (28 28)
Convolutional Neural Network Encoder
Convolutional layer
* Input channel:
* Output channel: 8
* Kernel Size: 3
* Stride:
Padding:
27

---

Under review as a conference paper at ICLR 2023
Leaky ReLU (0.2)
Convolutional layer
Input channel:
Output channel: 16
* Kernel Size: 3
* Stride: 2
* Padding:
Batch normalization layer (16)
Leaky ReLU (0.2)
Convolutional layer
Input channel: 16
Output channel: 32
Kernel Size: 3
* Stride: 2
* Padding:
Batch normalization layer (32)
Leaky ReLU (0.2)
Convolutional layer
Input channel: 32
Output channel: 64
Kernel Size: 3
* Stride: 2
* Padding:
Map to hyperbolic space: R64 + L
Hyperbolic linear layers:
Input dimension: 64
Hidden dimension: 64
Depth: 3
Output dimension: 64
Output: hyperbolic embeddings in L
Decoder
Input: hyperbolic embeddings in L
Hyperbolic linear layers:
Input dimension: 64
Hidden dimension: 64
Depth: 3
Output dimension: 64
Map to Euclidean space: IL + R61
Transposed Convolutional Neural Network Decoder
Transposed Convolutional layer
Input channel: 64
Output channel: 64
Kernel Size: 3
* Stride:
* Padding:
* Output Padding: O
Batch normalization layer (64)
Leaky ReLU (0.2)
Transposed Convolutional layer
28

---

Under review as a conference paper at ICLR 2023
Input channel: 64
Output channel: 32
Kernel Size: 3
Stride: 2
Padding:
Output Padding: 0
Batch normalization layer (32)
Leaky ReLU (0.2)
Transposed Convolutional layer
Input channel: 32
Output channel: 16
Kernel Size: 3
Stride:
Padding:
Output Padding:
Batch normalization layer (16)
Leaky ReLU (0.2)
Transposed Convolutional layer
Input channel: 16
Output channel:
Kernel Size: 3
Stride:
Padding:
Output Padding:
Output: Reconstructed MNIST image with dimension (28 x 28)
Hyperparameters
Manifold curvature: K 1.0
For all hyperbolic linear layers:
Dropout: 0.0
Use bias: True
Optimizer: Riemannian Adam (81 0.9, 82 0.9)
Learning Rate: le-4
Batch size: 32
Number of epochs: 20
F.4.2 ARCHITECTURE DETAILS OF HYPERBOLIC GENERATIVE ADVERSARIAL NETWORK
Generator
Input: points L sampled from G(o, (iag(1)2s))
Hyperbolic linear layers:
Input dimension: 128
Hidden dimension: 64
Depth: 3
Output dimension: 64
Output: points in 61
29

---

Under review as a conference paper at ICLR 2023
Critic
Input: points in Lx
Hyperbolic linear layers:
Input dimension: 3
Hidden dimension: 64
Depth: 3
Output dimension: 64
Hyperbolic centroid distance layer: 64 -
Output: score in IR
Hyperparameters
Manifold curvature: K = 110
Gradient penalty coefficient: À 10
For all hyperbolic linear layers:
Dropout: 0.0
Use bias: True
Optimizer: Riemannian Adam (31 = -0,33 = 0.9)
Learning Rate: le-4
Batch size: 64
Number of epochs: 20
Gradient penalty A: 10
F.5
EXPERIMENT DETAILS FOR MOLECULAR GENERATION
We describe the detailed architecture and settings for the molecular generation experiments.
F.5.1
ARCHITECTURE DETAILS OF HYPERBOLIC JUNCTION TREE ENCODER DECDDDER
Graph Encoder
Input: graph node features in 35
Map features to hyperbolic space: R:5 + L
Hyperbolic GCN layers:
Input dimension: 35
Hidden dimension: 256
Depth: 4
Output dimension: 256
Hyperbolic centroid on all vertices
Output: graph embedding in L
Tree Encoder
Input: junction tree features in 828
Hyperbolic embedding layer: 828 25
Hyperbolic GCN layers:
Input dimension: 256
Hidden dimension: 256
Depth: 4
Output dimension: 256
Hyperbolic centroid on all vertices
Output: tree embedding in 2,256
30

---

Under review as a conference paper at ICLR 2023
Tree Decoder
Input: tree embedding in 1
Message passing RNN:
Input: node feature of current tree node, inward messages
Hyperbolic linear layer on inward messages: L + L
256
Hyperbolic centroid on inward messages
Hyperbolic embedding layer on node feature: 18828 + 1
Lorentz Direct concatenation node feature and inward message: L L Hyperbolic linear layer: I,512 - L
Output dimension: 256
Topological Prediction:
Input: embedding, node feature of current node, inward messages
Hyperbolic linear layer on inward messages: 256 + K
256
Hyperbolic centroid inward messages
Hyperbolic embedding layer on tree feature: 1888 + L
Lorentz Direct concatenation on node feature, inward message, and tree embedding:
L
+ L
Hyperbolic linear layer: - + L
Hyperbolic centroid distance layer: 15 R2
Softmax on output
Output dimension: 2
Label Prediction:
Input: tree embedding, outward messages
Lorentz Direct concatenation outward message, and tree feature: 1225 - 1.
Hyperbolic linear layer: L + LK
Hyperbolic centroid distance layer: L + 528
Softmax output
Output dimension: 828
Output: junction tree
Graph Decoder
Input: junction tree, tree message, and graph embedding
Construction candidate subgraphs
Hyperbolic graph convolution layers all subgraphs:
Input dimension: 256
Hidden dimension: 256
Depth: 4
Output dimension: 256
Hyperbolic centroid on vertices of all subgraphs
Lorentz Direct concatenation subgraph embedding and graph embedding: 2
L512
Hyperbolic linear layer: I
L
Hyperbolic centroid distance layer: 2 -
Use subgraph score to construct molecular graph
Output: molecular graph
31

---

Under review as a conference paper at ICLR 2023
Hyperparameters
Manifold curvature: K = -100
For all hyperbolic linear layers:
Dropout: 0.0
Use bias: True
Optimizer: Riemannian Adam (B1 0.0, .2 0.999)
Learning rate: 5e-4
Learning rate scheduler: StepLR (step = 20000, 7 0.5)
Batch size: 32
Number of epochs: 20
F.5.2
ARCHITECTURE DETAILS OF HYPERBOLIC GENERATIVE ADVERSARIAL NETWORK
Generator
Input: points sampled from wrapped normal distribution G(o, (128)(112)) in Lk
Hyperbolic linear layers for graph embedding:
Input dimension: 128
Hidden dimension: 256
Depth: 3
Output dimension: 256
Hyperbolic linear layers for tree embedding:
Input dimension: 128
Hidden dimension: 256
Depth: 3
Output dimension: 256
Output: graph embedding and tree embedding 2
Critic
Input: graph embedding and tree embedding in I_
Hyperbolic linear layers for graph embedding:
Input dimension: 256
Hidden dimension: 256
Depth: 2
Output dimension: 256
Hyperbolic linear layers for tree embedding:
Input dimension: 256
Hidden dimension; 256
Depth: 2
Output dimension: 256
Lorentz Direct concatenation on graph embedding and tree embedding: L L512
Hyperbolic linear layer: 1 + 256
K
Hyperbolic centroid distance layer: 2 - IR
Output: score in IR
32

---

Under review as a conference paper at ICLR 2023
Hyperparameters
Manifold curvature: K = 1.0
Gradient penalty coefficient: À = 10
For all hyperbolic linear layers:
Dropout: 0.1
Use bias: True
Optimizer: Riemannian Adam (81 0, 0.9)
Learning Rate: le-4
Batch size: 64
Number of epochs: 20
Gradient penalty A: 10
F.6
EXPERIMENT DETAILS FOR TREE GENERATION
We describe the detailed architecture and settings for the tree generation experiments.
F.6.1
ARCHITECTURE DETAILS OF HYPERBOLIC TREE REEDDER-DECODER
Tree Encoder
Input: tree
Hyperbolic GCN layers:
Input dimension:
Hidden dimension: 32
Depth: 2
Output dimension: 32
Hyperbolic centroid on all vertices
Output: tree embedding in L
Tree Decoder
Input: tree embedding in I
Message passing RNN:
Input: node feature of current tree node, inward messages
Hyperbolic linear layer on inward messages: L - 1
Hyperbolic centroid on inward messages
Hyperbolic linear layer: L L
Output dimension: 32
Topological Prediction:
Input: embedding, inward messages
Hyperbolic linear layer on inward messages: L - L
Hyperbolic centroid on inward messages
Lorentz Direct concatenation inward message and tree embedding: L + L64
Hyperbolic linear layer: 04 L
Hyperbolic centroid distance layer: IL +
Softmax on output
Output dimension: 2
Output: tree
33

---

Under review as a conference paper at ICLR 2023
Hyperparameters
Manifold curvature: K = -100
For all hyperbolic linear layers:
Dropout: 0.0
Use bias: True
Optimizer: Riemannian Adam (3) 0.0, 82 0.999)
Learning rate: 5e-3
Learning rate scheduler: StepLR (step 20000, 7 0.5)
Batch size: 32
Number of epochs: 20
F.6.2
ARCHITECTURE DETAILS OF HYPERBOLIC GENERATIVE ADVERSARTAL NETWORK
Generator
Input: points sampled from wrapped normal distribution G(o, diag in L
Hyperbolic linear layers for tree embedding:
Input dimension: 16
Hidden dimension: 32
Depth: 2
Output dimension: 32
Output: tree embedding in T.
Critic
Input: tree embedding in L
Hyperbolic linear layers for tree embedding:
Input dimension: 32
Hidden dimension: 32
Depth: 2
Output dimension: 32
Hyperbolic centroid distance layer: L + R
Output: score in IR
Hyperparameters
Manifold curvature: K = -1.0
Gradient penalty coefficient: À = 10
For all hyperbolic linear layers:
Dropout: 0.1
Use bias: True
Optimizer: Riemannian Adam (81 0, = 0.9)
Learning Rate: le-4
Batch size: 64
Number of epochs: 20
Gradient penalty X: 10
34

---

Under review as a conference paper at ICLR 2023
G MOLECULE EXAMPLES
We show a subset molecule examples generated by HAEGAN.
Figure 12: Molecule examples generated by HABGAN.
35


Reviews:
[{"review_id": "zEoyo71iQX", "reviewer": "Reviewer_6PpY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a new network that combines autoencoder and generative adversarial network to guarantee stable training of hyperbolic neural networks. Experiments show that the proposed method can generate complex data, and there are some interesting results in the experiments. Nevertheless, there are still some issues that need to be resolved.", "summary_of_the_review": "Overall, I still have some concerns about the organization and clarify the proposed technique. The experiments are also limited.", "strength_and_weaknesses": "Pros:\n1. This paper designs a model that can guarantee stable training of hyperbolic neural networks.\n2. In the experiments, different metrics have been utilized to evaluate the proposed method. There are some interesting experimental results in this paper.\n\nCons:\n\n1. It seems that this work simply combines the hyperbolic neural networks with autoencoder and generative adversarial networks. I think the authors have to clarify the novelty and contributions of this work more clearly.\n\n2. It is not clear how the proposed model can guarantee stable training. A further explanation/clarification is necessary.\n\n3. The organization of this paper is somewhat confusing. Lots of the contents in this paper are utilized to introduce the background (e.g., Section 2, Section 4.1, Section 6). I think the authors should put more effort on explaining the proposed model. Maybe some background introduction can be moved into Appendix. The organization and presentation in this paper should be modified.\n\n4. The scalability of the proposed technique should be discussed.\n\n5. The experiments are limited, the authors should carry out experiments on one or two more datasets.\n\n6. This work lacks complexity analysis, the authors should analyze the complexity (e.g., time complexity) of the proposed method and compare it to state-of-the-art.\n\n7. Many notations have been used in this paper, I suggest the authors could try to make a table to explain all the notations in the Appendix.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GF1DseZ0X8Z", "reviewer": "Reviewer_V3p6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new architecture for generative modeling that makes use of hyperbolic geometry. Specifically, the builds on Chen et al 2022 to propose a generative framework that combines both Autoencoders and GANs. Additionally, the paper also a presents a new method for splitting and concatenating vectors in the Lorentz manifold. The paper adapts the $W$-GAN framework and proves a similar proposition for the characterization of the Wasserstein Distance. \n\nThe paper then tests these pieces out individually - generating images from MNIST, testing their concatenation method and then finally generating molecules using their method. ", "summary_of_the_review": "In summary, I think this is a well written paper with interesting results. However, some parts could be better explained. ", "strength_and_weaknesses": "**Strengths**\n--\n\n1) I think Proposition 3.1 is a strength. WGANs have improved GAN stability and being able to use a similar formulation (beyond the architecture mentioned here) is interesting and significant. \n2) The method seems to perform well and produce valid and diverse molecules. \n3) The paper is very well written. It is very easy to read and follow. All details are present and it should be easily reproducible. \n4) The ablation study is well conducted. \n\n**Weakness**\n---\n\n1) The motivation for the architecture is not quite clear. The authors, mention that training Hyperbolic Neural Networks is unstable and hence GANs can't model complex distributions. While I tend to believe this for the older methods that use the Tangent space, I however, do not know if this is true for the newer method that uses Lorentz rotations and boosts. Hence evidence of this via experimentation would be appreciated. \n\nSecond, this issues rather being resolved is pushed to a different part and then not discussed. In a sense, the method still learns a map from wrapped gaussians to the data distribution (GAN generator + Autoencoder decoder). Hence the autoencoder must now figure out how to map from a complex distribution to a simple distribution and then back to the complex distribution. Maybe because the training of autoencoders is more stable than GANs this is okay. \n\nFinally, the representation learned by the autoencoder is supposed to ``simpler'' than the original representation. But lower dimensional representations of data need not be simpler than higher dimensional representations. More discussion on this is needed. \n\n2) While the paper in general does a good literature survey, the paper has a section 2.3 which is on Embedding form Euclidean to Hyperbolic Spaces, which misses crucial literature. The paper claims it is unavoidable to use the exponential or logarithmic maps to represent the data hyperbolic space. However, this is not true. There are many embedding techniques such as Nickel and Kiela NeurIPS 2017, Nickel and Kiela ICML 2018, Sala, De Sa, Gu and Re ICML 2018, Sonthalia and Gilbert NeurIPS 2020. In particular, Sala et al 2018 and Sonthalia and Gilbert 2020 do not use the exponential or logarithmic maps. (This is a more minor comment.)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-r36piVfoHQ", "reviewer": "Reviewer_PLXj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A proposal of hyperbolic generative models is provided in the paper based on some existing structures in the literature, a stable training/optimization is observed empirically. The authors adopt the framework to learning molecule structure with the goal to learn the structural information. ", "summary_of_the_review": "The paper proposes a new framework of hyperbolic GAN and adopt it in a new task, learning the structure of molecules. The proposed model outperforms Euclidean baselines, where hyperbolic baselines are not available for comparison. However, the paper is lack of enough novelty, as most used layers are proposed before. ", "strength_and_weaknesses": "Strength. -- The proposed framework follows clear logic and easy to follow, mostly used the layer definition and function in [Chen et al. 2021] fully hyperbolic neural network. The authors also propose a straightforward version for concatenating/split vectors in the Lorentz model, which empirically shows stable gradients during optimization. The comparison of Lorentz Direct Concatenation and Lorentz Tangent Concatenation looks interesting and helpful. \n\n-- The authors evaluate the proposed HAEGAN on learning molecules structure and compare against several baselines using structure metrics. \n\n-- The improvement on learning structure of molecules seems to be significant when compared with various baselines. \n\nWeakness. -- Novelty of the model. Apart from the proposed concatenating/split, all other layers were proposed in prior works, particularly from [Chen et al. 2021], the whole framework looks to me a combination of existing pieces into a hyperbolic version of common GAN.\n\n-- The authors claim that the stable training of the model is guaranteed. However, the numerical stable property is only evaluated for the concatenating/split operation empirically. The numerical stable hyperbolic centroid distance layer is proposed in early work. There is no theory to guarantee the stable training. Particularly, large gradients into exp/log map is one side of the numerical instability, the other side of the numerical instability comes from float representation of hyperbolic space, a line of work such as [Yu et al. 2019], which happens when the hyperbolic point becomes far from the origin. Please modify the claim in the paper in an appropriate way. \n\nSuggestion and Question:\n-- Instead of putting 5 rows of NaN in table 1, you can just illustrate them in plain text. Though still it looks surprising to me that all hyperbolic baselines produce NaN, can you find a hyperbolic baseline without NaN, i.e., with less epochs or small graph just for a comparison?\n\n* [Chen et al. 2021], Fully hyperbolic neural networks. \n* [Yu et al. 2019], Numerically Accurate Hyperbolic Embeddings Using Tiling-Based Models", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject