Learning Hyperbolic Representations of Topological Features
Authors: Panagiotis Kyriakis
Iordanis Fostiropoulos
Paul Bogdan
Venue: ICLR 2021 Poster (2021)
Keywords: representation learning
hyperbolic deep learning
persistent homology
persistence diagrams

Abstract: Learning task-specific representations of persistence diagrams is an important problem in topological data analysis and machine learning. However, current state of the art methods are restricted in terms of their expressivity as they are focused on Euclidean representations. Persistence diagrams often contain features of infinite persistence (i.e., essential features) and Euclidean spaces shrink their importance relative to non-essential features because they cannot assign infinite distance to finite points. To deal with this issue, we propose a method to learn representations of persistence diagrams on hyperbolic spaces, more specifically on the Poincare ball. By representing features of infinite persistence infinitesimally close to the boundary of the ball, their distance to non-essential features approaches infinity, thereby their relative importance is preserved. This is achieved without utilizing extremely high values for the learnable parameters, thus the representation can be fed into downstream optimization methods and trained efficiently in an end-to-end fashion. We present experimental results on graph and image classification tasks and show that the performance of our method is on par with or exceeds the performance of other state of the art methods.


Published as a conference paper at ICLR 2021
LEARNING HYPERBOLIC REPRESENTATIONS OF TOPO- LOGICAL FEATURES
Panagiotis Kyriakis
Iordanis Fostiropoulos
Paul Bogdan
University of Southern Califoriaa Los Angeles, USA
University of Southem California
University of Southenn California
Angeles, USA fost i inttrrrpeeee usc. edu
Angeles, USA pbogdan@usc. edu
pkyriake @usc. edu
ABSTRACT
Learning task-specific representations of persistence diagrams an important problem in topological data analysis and machine learning. However, current methods are restricted terms of their expressivity as they are focused on Euclidean representations. Persistence diagrams often contain features of infinite persistence (i.e., essential features) and Euclidean spaces shrink their importance relative non-essential features because they cannot assign infinite distance finite points. To deal with this issue, we propose a method to learn representations persistence diagrams hyperbolic spaces, By representing features of infinite persistence infinitesimally close to the boundary of the ball, their distance to non-essential features approaches infinity, thereby their relative importance is preserved. This is achieved without utilizing extremely high values for the learnable parameters, thus, the representation can be fed into downstream optimization methods and trained efficiently in an end-to-end fashion. We present experimental results on graph and image classification tasks and show that the performance of our method is on par with or exceeds the performance of other state of the art methods.
specifically the Poincare ball.
INTRODUCTION
Persistent homology is topological data analysis tool which tracks how topological features (c.g. connected components, cycles, cavities) appear and disappear as we analyze the data at different scales or in nested sequences of subspaces ①1 2). nested sequence of subspaces known as a filtration. As an informal example of a filtration consider an image of variable brightness. As the brightness is increased, certain features (edges, texture) may become less or more prevalent. The birth of a topological feature refers to the "time" (i.e., the brightness value) when it appears in the filtration and the death refers to the "time" when it disappears. The lifespan of the feature is called persistence. Persistent homology summarizes these topological characteristics in a form of multiset called persistence diagram, which is a highly robust and versatile descriptor of the data. Persistence diagrams enjoy the stability property, which ensures that the diagrams of two similar objects similar 3 Additionally, under some assumptions, one can approximately reconstruct the input space from a diagram (which is known solving the inverse problem) (4. However, despite their strengths, the space of persistence diagrams lacks structure as basic operations, such as addition and scalar multiplication, are well defined. The only imposed structure induced by the Bottleneck and Wasserstein metrics, which are notoriously hard to compute, thereby preventing us from leveraging them for machine learning tasks.
To address these issues, several vectorization methods have been proposed. Some of the earliest approaches are based on kernels, i.e., generalized products that turn persistence diagrams into elements of a Hilbert space. Kusano et al. (5) propose a persistence weighted Gaussian kernel which allows them to explicitly control the effect of persistence. Alternatively, Carrière et al. (6) leverage the sliced Wasserstein distance to define a kernel that mimics the distance between diagrams. The approaches by Bubenik 7 based on persistent landscapes, by Reininghaus et al. (8) based on scale space theory and by Le et al. (9) based on the Fisher information metric are along the same line of work. The major drawback in utilizing kernel methods is that they suffer from scalability issues as the training scales poorly with the number of samples.
Related Work.

---

Published as a conference paper at ICLR 2021
In another line of work, researchers have constructed piiite-dimessional embeddings, i... transfor- mations turning persistence diagrams into vectors in a Euclidean space. Adams et al. (I0) map the diagrams to persistence images and discretize them to obtain the embedding vector. Carrière et al. (1I) develop a stable vectorization method by computing pairwise distances between points in the persistence diagram. An approach based interpreting the points in the diagram roots of a complex polynomial is presented by Di Fabio (12). Adcock et al. (13) identify an algebra of polynomials on the diagram space that can be used as coordinates and the approach is extended by Kališnik in (14) to tropical functions which guarantee stability. The common drawback of these embeddings that the representation is pre-defined, i.e., there exist no learnable parameters, therefore, it is agnostic to the specific learning task. This is clearly sub-optimal as the eminent success of deep learning has demonstrated that it is preferable to learn the representation.
The more recent approaches aim at learning the representation of the persistence diagram in an end-to-end fashion. Hofer et al. (15) present the first input layer based on a parameterized family of Gaussian- like functionals, with the mean and variance learned during training. They extend their method in (T6) allowing for a broader class of parameterized function families to be considered. It is quite common to have topological features of infinite persistence (,, i.e., features that never die. Such features are called essential and in practice are usually assigned a death time equal to the maximum filtration value. This may restrict their expressivity because it shrinks their importance relative to non-essential features. While we may be able to increase the scale sufficiently high and end up having only one trivial essential feature (i.e.. the 0-th order persistent homology group that becomes a single connected component at a scale that is sufficiently large), the resulting persistence diagrams may not be the ones that best summarize the data in terms of performance on the underlying learning task. This is evident in the work by Hofer et al. (15) where the authors showed that essential features offer discriminative power. The work by Carrière et al. (T7), which introduces a network input layer the encompasses several vectorization methods, emphasizes the importance of essential features and is the first one to introduce a deep learning method incorporating extended persistence as way to deal with them.
In this paper, we approach the issue of essential features from the geometric viewpoint. We are motivated by the recent success of hyperbolic geometry and the interest in extending machine learning models to hyperbolic spaces or general manifolds. We refer the reader to the review paper by Bronstein et al. (18) for overview of geometric deep learning. Here, we review the most relevant and pivotal contributions in the field. Nickel et al. (19) 20) propose Poincaré and Lorentz embeddings for learning hierarchical representations of symbolic data and show that the representational capacity and generalization ability outperform Euclidean embeddings. Sala et al. (2D propose low-dimensional hyperbolic embeddings of hierarchical data and show competitive performance on WorldNet. Ganea et al. (22) generalize neural networks to the hyperbolic space and show that hyperbolic sentence embeddings outperform their Euclidean counterparts on a range of tasks. Gulcherhe et al. (23) introduce hyperbolic attention networks which show improvements in terms of generalization on machine translation and graph learning while keeping a compact representation. In the context of graph representation learning. hyperbolic graph neural networks (24) and hyperbolic graph convolutional neural networks (25) have been developed and shown to lead to improvements various benchmarks. However, despite this success of geometric deep learning, little work has been done in applying these methods to topological features, such as persistence diagrams.
The main contribution of this paper is to bridge the gap between topological data analysis and hyperbolic representation learning. We introduce method to represent persistence diagrams hyperbolic space, more specifically on the Poincare ball. We define a learnable parameterization of the Poincare ball and leverage the vectorial structure of the tangent space to combine (in manifold- preserving manner) the representations of individual points of the persistence diagram. Our method learns better task-specific representations than the state of the art because it does not shrink the relative importance of essential features. In fact, by allowing the representations of essential features to get infinitesimally close to the boundary of the Poincare ball, their distance to the representations of non-essential features approaches infinity, therefore preserving their relative importance. To the best of our knowledge, this is the first approach for learning representations of persistence diagrams in non-Euclidean spaces.
2

---

Published as a conference paper at ICLR 2021
2 BACKGROUND
In this section, we provide a brief overview of persistent homology leading up to the definition of persistence diagrams. We refer the interested reader to the papers by Edelsbrunner al. 1 for detailed overview of persistent homology. An overview of homology can be found in the Appendix.
Persistent Homology. Let K be simplicial complex. A filtration of K is a nested sequence of subcomplexes that starts with the empty complex and ends with K,
0 = Ko C Ki S ....
C Ka=
(I)
A typical way to construct a filtration is to consider sublevel sets of real valued function, f K + R. Let (1) < <... < Od be sorted sequence of the values of f(K). Then, we obtain filtration by setting
Ko= and K; = / a:l) for sisd d.
We can apply simplicial homology to each of the subcomplexes of the filtration. When 0 sis S the inclusion K; S Kj induces a homomorphism
fhi H. (K()
(3)
on the simplicial homology groups for each homology dimension n. We call the image of fi n-th persistent homology group and it consists of homology classes born before i that are still alive at j. A homology class a is born at K; if it is not in the image of the map induced by the inclusion K;-1 C K, Furthermore, if a is born K;, it dies entering Kj if the image of the map induced by Ki-1 C Kリ- does not contain the image of a but the image of the map induced by K;- S K2 does. The persistence of the homology class a is j - i. Since classes may be born at the same and die at the same j. we can use inclusion-exclusion to determine the multiplicity of each (i,j),
P - - - - 3 + 1/,1.,
(4)
where the n-th persistent Betti numbers 3:. are the ranks of the images of the 7-th persistent homology group, i.e., (hh = =rank(im )), and capture the number of n-dimensional topological features that persist from j. By setting /n) still persist at the end of the filtration (j = d), which are known as essential features.
1/h 31-1.
account for features that
Persistence Diagrams. Persistence diagrams are multisets supported by the upper diagonal part of the real plane and capture the birth/death of topological features (i.e., homology classes) across the filtration.
Definition 2.1 (Persistence Diagram). Let A= for RA mult(r) oc) } be the multiset of the diagonal A = {(1), x2) E R22 : 11 = x2). where mult(.)) denotes the multiplicity function and let [ 222) TR. U U Uo) 22 > 21). Also, let be a homology dimension and consider the sublevel set filtration induced by a function f: K + E over the complex K. Then, a persistence diagram, (f) is multiset of the form (x) = {2 : x E R.) A constructed by inserting each
point (a;, aj) for i < j with multiplicity n (or 14:0 if it is an essential feature). We denote the space of all persistence diagrams with D.
Definition 2.2 (Wasserstein distance and stability). Let Da),), Eg(g) be two persistence diagrams generated by the filtration induced by the functions f.g : K + IR, respectively. We define the Wasserstein distance
& = inf
Σ |k- m()))))) ZED
(5)
where p. q E N and the infimum is taken over all bijections 1) : (J) Eg).) The special case p 0o is known as Bottleneck distance. The persistence diagrams are stable with respect to the Wasserstein distance if and only if ())(), (g)) < || -gllx
Note that bijection 7) between persistence diagrams is guaranteed to exist because their cardinalities are equal, considering that, as per Def. 2.1 the points on the diagonal are added with infinite multiplicity. The strength of persistent homology stems from the above stability definition, which essentially states that the map taking a sublevel function to the persistence diagram is Lipschitz continuous. This implies that if two objects are similar then their persistence diagrams are close.
3

---

Published as a conference paper at ICLR 2021
y
death
22
1:0
T Tzy B
Complex K. Filratioo f
I
81
Dn(f)
oo p())
1/2
0
M(D. 03
3/1
+/1
birth
Figure I: Illustration of our method: Initially, the points are transferred via the auxiliary transformation and the parameterization é to the Poincare ball B. where learnable parameters are added. Then. the logarithmic map is used for transforming the points to the tangent space Tzo B. Finally, the resulting vectors are added and transformed back to the manifold the exponential map. Note that the persistence diagram is mapped to single point on the Poincare ball (i.c., (D, () E B).
PERSISTENT POINCARE REPRESENTATIONS
In this section, we introduce our method (Fig. D for learning representations of persistence diagrams on the Poincare ball. We refer the reader to the Appendix for some fundamental concepts of differential geometry.
The Poincare ball is m-dimensional manifold (B,g%) where far TM ||>||| < <1}
is the open unit ball. The space in which the ball is embedded is called ambient space and is assumed to be equal R" The Poincare ball is conformal (i.e., segee-presrrving the Euclidean space but does not preserve distances. The metric tensor and distance function are as follows
-.e = 229
X
2
dz(z,, v) arccos (1+ 2 (1 ||>||| )1 Ilvl))
I v/1
(6)
1
1- ||=|?
where gf = Im the Euclidean metric tensor. Eq. 6 highlights the benefit using the Poincare ball for representing persistence diagrams. Contrary to Euclidean spaces, distances in the Poincare ball can approach infinity for finite points. This space is ideal for representing essential features appearing in persistence diagrams without squashing their importance relative non-essential features. Informally, this is achieved by allowing the representations of the former ones to get infinitesimally close to the boundary, thereby their distances to the later ones approach infinity. Fig. 2 provides an illustration.
We gradually construct our representation through composition of individual transformations. The first step is to transfer the points to the ambient space (i.e., "2" of the Poincare ball. Let Z be persistence diagram. We introduce the following auxiliary transformation
p: R R
This auxiliary transformation is essentially a nigh-dimensionaa embedding and may contain Icarnable parameters. Nonetheless, our main focus is to learn a hyperbolic representation and, therefore, we assume that is not learnable. Later in this section, we analyze conditions on to guarantee the stability and expressiveness of the hyperbolic representation.
The second step is to transform the embedded points from the ambient space to the Poincare ball. When referring to points on a manifold,
important to define coordinate system. A homeomorphism 1/ B - Ih is called coordinate chari and gives the local coordinates on the manifold. The inverse map Φ : 1m + B, called a parameterization coordinates. The main idea is inject learnable parameters into this parameterization. The injected parameters could be any form of differentiable functional that preserves the homomorphic property. Differentiability is needed such that representation be fed to downstream optimization
B and gives the ambient
Thee sublevel set function and the homology dimension omitted.
4

---

Published as a conference paper at ICLR 2021
methods. In our construction, utilize variant of the generalized spherical coordinates. Let E I be a vector of m parameters. We define the learnable parameterization Φ Rm x R + as follows
2 arctan 0171 and y: 0: + arccos for 2, ..m,
U1 1+
(8)
π
7
=3+1 2 + The small positive constant e is added to ensure that the denominator in Eq. [ not zero. Intuitively, Eq. [ [corresponds to scaling the radius of the point by factor 02 and rotating it by 0g radians across the angular axes. The scaling and rotation parameters are learned during training. Note that the form of yı ensures that representation belongs in the unit ball for values of 0,. The coordinate chart is not explicitly used our representation; it is provided in the Appendix for the sake of completeness.
where r² 2
The third step is to combine the representations of each individual point of the persistence diagram into single point in the hyperbolic space. Typically, in Euclidean spaces, this is done by concatenat- ing or adding the corresponding representations. However, in non-Euce Eucliddann spaces such operations not sanifold-preservingg Therefore, transform the points from the manifold to the tangent space, combine the vectors via standard vectorial addition and transform the resulting vector back to the manifold. This approach is based the exponential and logarithmic maps
(9) The exponential map allows us transform vector from the tangent space the manifold and its inverse (i.e., the logarithmic map) from the manifold to the tangent space. For a general manifold, is hard to find these maps details). Fortunately, for the Poincare ball case, they have analytical expressions, given as follows
expx T.B + B and log, B T.B.
need to solve for the minimal geodesic curve (see Appendix for more
Au | exp2 (v) = x @ (anh (~e_ 2
v |
2
+ y
log.r
Lanh || + wll
(10)
1 + vll'
where @ denotes the Möbius addition, which is manifold-preservee preserving operator (i.e., for any y I B = b y B). The analytical expression is given in the Appendix. The transformations given by these maps are norm-pret worm-preerrvnng ie., for example, the geodesic distance from x to the transformed point (x) coincides with the metric norm ||v||, induced by the metric tensor g This is important property importance of topological features) to be preserved when transforming to and from the tangent space. We now combine the aforementioned transformations and define the Poincare hyperbolic representation followed by its stability theorem.
need the distance between points (and therefore the relative
Definition 3.1 (Poincare Representation), Let E D be the persistence diagram to be represented in m-dimensional Poincare ball (B, ..)) embedded in R and xo B be given point. The representation of 'D on the manifold B is defined as follows
Φ D x R. B,
+ (D,0) = exPa.
Σ logzo (o(((x)))) TED
(11)
where the exponential and logarithmic maps are given by Eq. 10 and the learnalee parameterization and the auxiliary transformation by Eq, g and Eq. 7 respectively.
Theorem (Stability of Hyperbolic Representation). Let D,E two persistence diagrams and consider an auxiliary transformation p R + Rm that is
Lipschitz continuous w.r.t the induced metric norm ||-Il,-
p(x) 0 for all x E RA.
Additionally, assume that 2g = 0. Then, the hyperbolic representation given by Eq. 11 iis stable w.r.t the Wasserstein distance when p = 1, i.e., there exists constant K >> 0 such that
d 0), 4EE 0)) < K (w( (D,E)
(12)
where dæ is the geodesic distance and f is the Wasserstein metric with the g-norm replaced by the induced norm Il. (i.e., the norm induced by the metric tensor g, Appendix A.2
5

---

Published as a conference paper at ICLR 2021
(B.
30
20
H:
binh
Figure 2: Left: Example graph from the AMDB-BINARY dataset. Middle: Persistence diagrams extracted using the fieoois-Riips filtration. The dashed line denotes features infimite persistence. which are represented by points maximal death value cqual to 90 (i.c., by points finite persistence). Right: Equivalent representation on the 2-dimensional Poincare ball. Features of infinite persistence are mapped infinitesimally close to the boundary. Therefore, their distance to finite persistence features approaches infinity (d ²²
The proof of Theorem [ (given in the Appendix) results from a general stability theorem (3) and is on par with similar results for other vectorizations (1O) or representations (15) of persistence diagrams. One subtle difference is that Theorem | Jsess the induced norm rather than the q-norm appearing in the Wasserstein distance. However, since the induced norm implicitly depends on the chosen point To, which, per requirements of Theorem T is assumed to be equal to the origin, there is no substantial difference. The fact that we require the auxiliary transformation important to theoretically guarantee stability. Intuitively, this can be understood by recalling (Def. 2.1 that all (infinite) points on the diagonal are included in the persistence diagram. By mapping the diagonal to zero and taking Zy = O, we ensure that the summation Eq. collapses to zero when summing over the diagonal.
be zero on the diagonal is
Finally, we note that the assumptions of Theorem are not restrictive. In fact, we can easily find Lipschitz continuous transformations that are zero the diagonal RA, such as the exponential and rational transformations proposed by Hofer et al. (15). Additionally, we note that the assumptions of Theorem do not prohibit from choosing an "under-powered" or degenerate p. For example, P satisfies the assumptions and therefore leads to stable representation. However, such representation obviously useful for learning tasks. An implicit requirement, that guarantees not only the stability but the expressiveness of the results representation, is that p does not result in any information loss. This requirement is satisfied by picking a p that is injective, which, given that it is higher dimensional embedding, is a condition easy to satisfy. In practice, we use variant of the exponential transformation by Hofer et al. (5)). The exact expression is given in the Appendix.
EXPERIMENTS
We present experiments on diverse datasets focusing on persistence diagrams extracted from graphs and grey-scale images. The learning task is classification. Our representation acts input to a neural network and the parameters are learned end-to-ens via standard gradient methods. The architecture well as other training details are discussed the Appendix. The code to reproduce our experiments is publicly available at tttttts://// /ttl:p:://iiii http:/gitttt tuu.. com/ httpppkyiikkk wttpls//eemmanniiie fold//
Ablation Study: To highlight what extent our results are driven by the hyperbolic embedding, we perform an ablation study. In more detail, we consider three variants of our method:
1. Persistent Poincare (P-Poinc): This is the original method presented in Sec.
2. Persistent Hybrid (P-Hybrid): Same as P-Poinc with the Poincare ball replaced by the Euclidean space. This implies that the exponential and logarithmic maps (Eq. 10 reduce to the identity maps, i.e., exp_(v) (v) = I + log, (y) = y z. The learnable parameterization is in Eq. 8
3. Persistent Euclidean (P-Eu)l): Same as P-Hybrid with Eq. replaced with simple addition of the learnable parameters, i.e., y x: + 0.

---

Published as a conference paper at ICLR 2021
Baseline- Essential Features Separation: To highlight the benefit of unified Poincare representa- tion, we design a natural baseline that treats essential and non-essential features separately. In more detail, for each point (b, d) E D, we calculate its persistence d - b and then compute the histogram of the resulting persistence values. For essential features, we compute the histogram of their birth times. Then, we concatenate those histograms and feed them as input the neural network (architecture described in the Appendix). We consider the case where the essential features are included (baseline w/ essential) and the case where they are discarded (baseline w/o essential).
Since our method essentially represents each per- sistence diagram on m mn-dimensional Poincare ball, it may introduce substantial information compression when the points in the diagrams are not of the same order as m. A trivial approach to counteract this issue to use high value for m. However, we experimentally observed that a high manifold dimension does not give the optimal classification performance and it adds a computational overhead in the construction of the computation graph. Empirically, the best approach is to keep m at moderate values (in the range m = 3 to concatenate the outputs. Each replica is called a projection base and for their number we explored values dependant on the number of points in the persistence diagram. Persistence diagrams obtained from images tend to have substantially fewer points than diagrams obtained from graphs. Therefore, for images, we explored moderate values for K, i.e., 5 10, whereas for graphs increased K in the range 200 - 500. Essentially, we treat both m and K as hyper--aaaamters, explore their space following the aforementioned empirical rules and pick the optimal via the validation dataset. As we increase m, it is usually prudent to decrease K to maintain similar model capacity.
Manifold Dimension and Projection Bases:
12), replicate the representation K times and
4.1 GRAPH CLASSIFICATION
In this experiment, we consider the problem of graph classification. We evaluate our approach using social graphs from (26). The REDDIT-BINARY dataset contains 1000 samples and the graphs correspond to online discussion threads. The task is to identify to which community given graph belongs. The REDDII-5K and the REDDIT-/2K are a larger variant of the former dataset and contain 5K and 12K graphs from 5 and 11 subreddits, respectively. The task to predict to which subreddit given discussion graph belongs. The IMDB-BINARY contains 1000 ego-networks of actors that have appeared together in any movie and the task is to identify the genre (action or romance) to which an ego-graph belongs. Finally, the IMDB-MULTI contains 1500 ego-networks belonging to 3 genres (action, romance, sci-fi). We train our model using 10-fold cross- cross-valideen validation with a 80/20 split.
Graphs are special cases of simplicial complexes, therefore, defining filtrations is straightforward. We use two methods: The first one captures global topological properties and is based on shortest paths. In this case, the graph 9 = (V, E) is lifted metrie space (V, d) using the shortest path distance d: V x V + Rzo between two vertices, which can be easily proved to be a valid metric. Then, we define the Vietoris-Rips complex of as the filtered complex V (G) that contains subset of V as a simplex if all pairwise distances in that subset are less than or equal to 8, or, formally,
(13) This approach essentially interprets the vertices of the graph as a point cloud in metric space, with the distance between points given by the shortest path between the corresponding vertices. In the case of unweighted graph, we assign unit weight across edges. In Fig. 2 we show a sample persistence diagram extracted with the Vietoris-Rips filtration and the corresponding representation of the features on the Poincare ball. The second method captures local topological properties and is based on vertex degree. Given graph G- = (V, E), a simplicial complex can be defined the union of the vertex and edge sets, i.e., K = Ko U K1, where Ku = {u : C E V} and K₁ = {(u, v) (u. v) E E). The sublevel function is defined as follows: f(v = deg(v) for e E K, and f(u, v) = max { f(u).(v)
VR, {(ro, "1,-.. .u))
d( (1; n2) S 8, Vi,
for (u, v) E K1, where deg(v) is the vertex degree Then, the filtration is given by Eq. 2
We compare our method against several state of the art methods for graph classification. In particular, we compare against: (1) the Weisfeiler-Lehman (WL) graph kernel (27). (2) the Deep Graph Kernel (DGK) (26). (3) the Patchy-SAN (PSCN) (28), the Weighted Persistence Image Kernel (WKPI) (2)), (5) the Anonymous Walk Embeddings (AWE) (30), (6) the Graph Isomorphism Network with Graph Filitraton Learning (GIN-GFL) (1). Additionally, we compare against the PersLay input layer by Carrière et al. (17) which utilizes extended persistence as alternative way to deal with essential features and the Learnable Representation based on Gaussian- like structure elements (GLR)
presented by Hofer et al. (15)
7

---

Published as a conference paper at ICLR 2021
Table 1: Classification accuracy (mean-istd or min-max range, if available).
IMDB-M
IMDB-B
REDDIT-B
REDDIT-SK
REDDIT-12K
baseline w/o ess.
38.434
65.78+1 25
67.33+1.55
39.30+1.12
28.35=1.11
baseline w/ ess.
38.78
66.56 1.1
69.33
38.45+0.98
30.43
WL
49.33
73. 73.40.+4.63
81.10+1.90
49.44+2.36
38. 38.1841.31
DGK
10.52
66.96
78.04 78.04
11.27
32.22
PSCN
45.23
71. 71.00.1..29
16.30
49.10
10.322
WKPI
49.540.10
75. 10
n/a
59.50+0.60
48.40
AWE
51.54+3.61
74.45+5.88
87.8942.53
54.74+1.91
39.20 39.20+2.0
GIN-GFL
49.70=2.9
74.50+44.00
90.3042.60 n/a
55.70+2.90
1/a
PersLay
48.8-52.2
71.2-72.6
55.6-56-5
47.7-49.1
GLR
n/a
n/a
n/a
54.5
44.5
P-Eucl
46.45
67.5413.54
71.45
43.1513.12
32.56.3.66
P-Hybrid
47.87+2.03
42.4844.57
72.87+1.75
46.85+2.17
46.42
P-Poinc
57.31+4.27
81.86+4.26 2.22
79.78+3.21
51.71+3.01
42.16
We run simulations for different manifold dimensions (ranging from m m 12) and pick the best one using the mean (across all 10 folds) cross-validation accuracy as criterion. We report the mean and standard deviation for the best m on Table We observe that the performance of the P-Eucl method is poor but the addition of the learnable parameterization (i.e., P-Hybrid method) leads to small but consistent improvement. The best performance achieved by the fully hyperbolic representation (P-Poinc) and is par or exceeds the performance of the state of the art. This suggests that the hyperbolic representation is vital for obtaining good performance. Additionally, the baselines have poor performance irrespectively of whether not essential features have been included via the histogram of their birth times. This indicates that treating essential features separately and including them as additional inputs is not sufficient for good performance. These results support our initial motivation for representing persistence diagrams on the Poincare ball. Also, observe that our method out- out-performs all the competitor methods for the IMDB datasets, which have a small number of nodes and edges (approx. 10-20 nodes and 60-90 edges) while the REDDIT ones are bigger networks (approx. 400-500 nodes and 500-600 edges). This may indicate that the Poincare representation is best suited for smaller, less complex graphs.
4.2 IMAGE CLASSIHCATION
Even though it is well known that convolutional neural networks have achieved unprecedented success as feature extractors for images, we present an image classification case-study using topological features as a proof-of- with the structure of simplicial complex. In theory, we could construct Vietoris-Rips complexes, as in Eq. 13 by interpreting pixels as a point cloud. However, this is not the most natural representation of an image, which has grid structure. Therefore, we exploit this structure by constructing cubical complexes. i..., unions of cubes aligned on 2D grid. As in the case of graphs, we use two methods. For the first method, called cubical filtration, we use the grey-scale image directly and represent each pixel as 2-cube. Then, all of its faces are added to the K-th complex. We obtain a sublevel function by extending the grey-scale value I(v) of pixel a v to all cubes in K as follows:
for our method. Contrary to graphs, images are not inherently equipped
f(a)
min I(T), K. face of
(14)
In the second experiment, we consider the problem of image classification. We utilize two standard- ized datasets: the MNIST, which contains images of handwritten digits, and the Fashion-MNiD MNIST, which contains shape images of different types of garment (e.g.. T-shirt, trouser, etc.). Each dataset contains total of 70K (60K train, 10K validation, 10 folds) grey-scale images of size 28 x 28. Both datasets are balanced and categorized into 10 classes. Finally, a grey-scale filtration is obtained using The second method is called height filtration and uses the binarized version of the original image. We define the height filtration by choosing a direction U E R² of unit norm and by assigning to each pixel p of value in the binarized image a new value equal to (p, U), i.e. the distance of pixel p to the plane defined by v. This creates new gcey- image which is then fed to the aforementioned cubical filtration. We note that the height filtration deserves special attention in persistent homology
Eq. 2
8

---

Published as a conference paper at ICLR 2021
0.7
Table 2: Classification accuracy
0.6
MNIST
F-MNIST
0.56
015
PWGK
75.31
28.13
0.4
PI 94.39
31.26
0.9
10.3
0.85
GLR
93.31
56.85
0.2
P-Eucl
92.27
30.21
0.1
0.8
P H-Hybrid
93.78
38.45
01
2
3
5 cpach
P-Poine
95.91
72.28
sprads
Figure 3: Plotting the train loss (left) and the validation accuracy (middle) over training epochs for the MNIST dataset using two different dimensions for the Poincare ball (m = 3 and m 9).
because complexes to which it applied can be approximately reconstructed from their persistence diagrams (4). In practice,
direction vectors that distributed uniformly across the unit cycle. We choose 30 and 50 directions for the MNIST and Fashion -MNIST, respectively.
We compare method against three baselines that encompass all methods for handling persistence diagrams: (1) the Persistence Weighted Gaussian Kernel (PWGK) approach proposed by Kusano
al. (5) (2) the Persistence Images (PI) embedding developed by Adams al. (TO), and (3) the Learnable Representation based on Gaussian-like structure elements (GLR) by Hofer et al. (15). We run our simulations for manifold dimensions equal to m 3 and m 9 and report the best results in Table Our method outperforms all other methods, in some cases by a considerable margin. We also study how the manifold dimension affects the train loss and validation accuracy. The results are shown in Fig. 3 Observe that for
9 the train loss decreases more rapidly and the validation accuracy increases more rapidly the first few epochs. This suggests that higher manifold dimension may be slightly better for representing persistence diagrams. Additionally, we observed that the Poincare representation tends to generalize better than its Euclidean counterpart. In fact, the validation accuracy of the P-Eucl method started decreasing after the first 2-3 epochs, whereas the P-Poinc method showed a saturation rather than decrease. This was observed without and is strong empirical finding that
modifying the dropout rate any other demonstrates the superiority of Poincare representations.
CONCLUSION
We presented the first, to the best of our knowledge, method for learning representations of per- sistence diagrams in the Poincare ball. Our main motivation for introducing such method that persistence diagrams often contain topological features of infinite persistence (i.e., essential features) the representational capacity of which may be bottlenecked when representing them in Euclidean spaces. This stems from the fact that Euclidean spaces cannot assign infinite distance to finite points. The main benefit of using the Poincare space is that by allowing the representations of essential features get infinitesimally close to the boundary of the ball their distance to con-essential features approaches infinity, therefore preserving their relative importance. Directions for future work include the learning of filtration and/or scale end-to-end well investigate whether not there exists some hyperbolic trend in distances appearing in persistence diagrams that justify the improved performance especially on small graphs.
ACKNOWLEDGEMENTS
The authors gratefully acknowledge the support by the National Science Foundation under the Career Award CPS/CNS- -NCCC/S--145660,,. the NSF award under Grant Numbers CCF-1837131 MCB-1936775, CNS-1932620, and CMMI 1936624 and the DARPA Young Faculty Award and DARPA Director's Fellowship Award, under Grant Number 166001-17-1-4044 and Northrop Grumman grant. The views, opinions, and/or findings contained in this article are those of the authors and should not be interpreted as representing the official views or policies, either expressed or implied by the Defense Advanced Research Projects Agency, the Department of Defense the National Science Foundation.

---

Published as a conference paper at ICLR 2021
REFERENCES
[1] Herbert Edelsbrunner and John Harer. Persistent homology a survey. Discrete Computational Geometry- DCG, 453, 01 2008.
[2] Herbert Edelsbrunner and Morozov. Persistent homology: theory and practice. European Congress of Mathematics, pages 3 31-50, 01 2014.
[3] David Cohen-Steiner, Herbert Edelsbrunner, and John Harer. Stability of persistence diagrams. Discrete & Computational Geometry, 37(1):103-122) 2007.
[4] K. Turner, S. MMkkherjee, and D. M. Boyer. Persistent homology transform for modeling shapes and surfaces. Information and Inference: A Journal of the IMA, 3(4):310-344, Dec 2014.
[5] Genki Kusano, Kenji Fukumizu, and Yasuaki Hiraoka. Persistence weighted gaussian kernel for topological data analysis. In Proceedings of the 33rd International Conference on International Conference Machine Learning Volume 48, ICMI;16, page 2004-2013. JMLR.org. 2016.
[6] Mathieu Carrière, Marco Cuturi, and Steve Oudot. Sliced wasserstein kernel for persistence diagrams. Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, page 664-673. JMLR.org, 2017.
[7] Peter Bubenik. Statistical topological data analysis using persistence landscapes. Journal of Machine Learning Research, 16(3):77-102, 2015.
[8] J. Reininghaus, S. Huber, U. Bauer, and R. Kwitt. A stable multi-scale kernel for topological machine learning. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4741-4748 2015.
[9] Tam Le and Makoto Yamada. Persistence fisher kernel: A riemannian manifold kernel for persistence diagrams. S. Bengio, Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnetr, editors, Advances in Neural Information Processing Systems 31, pages 10007- 10018. Curran Associates, Inc., 2018
[10] Henry Adams, Tegan Emerson, Michael Kirby, Rachel Neville, Chris Peterson, Patrick Shipman, Sofya Chepushtanova, Eric Hanson, Francis Motta, and Lori Ziegelmeier. Persistence images: stable vector representation of persistent homology. Journal of Machine Learning Research, 18(8):1-35, 2017.
[11] Frédéric Chazal, Brittany Terese Fasy, Fabrizio Lecci, Alessandro Rinaldo, and Larry Wasser- man. Stochastic convergence of persistence landscapes and silhouettes. In Proceedings the Thirtieth Annual Symposium on Computational Geometry, SOCG' 14, page 474-483, New York, NY, USA, 2014. Association for Computing Machinery.
[12] Barbara Di Fabio and Massimo Ferri. Comparing persistence diagrams through complex vectors. In Vittorio Murino and Enrico Puppo, editors, Image Analysis and Processing ICIAP 2015, pages 294-305, Cham, 2015. Springer International Publishing.
[13] Aaron Adcock, Erik Carlsson, and Gunnar Carlsson. The ring of algebraic functions on persistence bar codes. Homology, Homotopy and Applications, 18, 04 2013.
[14] Sara Kalisnik. Tropical coordinates the space of persistence barcodes. Foundations of Computational Mathematics, 1911::00-112, 2019.
[15] Christoph Hofer, Roland Kwitt, Mare Niethammer, and Andreas Uhl. Deep learning with topo- logical signatures. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page 1633-1643, Red Hook, NY, USA, 2017. Curran Associates Inc.
[16] Christoph D. Hofer, Roland Kwitt, and Mare Niethammer. Learning representations of persis- tence barcodes. Journal of Machine Learning Research, 20 (0(16)):1 145, 2019.
10

---

Published as a conference paper at ICLR 2021
[17] Mathieu Carrière, Frédéric Chazal, Yuichi Ike, Théo Lacombe, Martin Royer, and Yuhei Umeda. Perslay: A neural network layer for persistence diagrams and new graph topological signatures. In Silvia Chiappa and Roberto Calandra, editors. The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy), volume 108 of Proceedings of Machine Learning Research, pages 2786-2786 2796. PMLR, 2020.
[18] M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18 42, 2017.
[19] Maximillian Nickel and Douwe Kiela, Poincaré embeddings for learning hierarchical represen- tations. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6338-6347. Curran Associates, Inc., 2017.
[20] Maximilian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz. model of hyperbolic geometry. Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3776-3785. PMLR, 2018.
[21] Frederic Sala, Chris De Sa, Albert Gu, and Christopher Re. Representation tradeoffs for hyperbolic embeddings. In Jennifer Dy and Andreas Krause, editors. Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1460-4469, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
[22] Octavian Ganea, Gary Becigneul, and Thomas Hofmann. Hyperbolic neural networks. In
S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 5345-5355. Curran Associates, Inc., 2018
[23] Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, and Nando de Freitas. Hyperbolic attention networks. In International Conference Learning Representations, 2019.
[24] Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buus E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems 32, pages 8230-8241
Curran Associates, Inc., 2019.
[25] Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 4868-4879. Curran Associates, Inc., 2019.
[26] Pinar Yanardag and S. V.N. Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '15, page 1665,1374, New York, NY, USA, 2015, Association for Computing Machinery.
[27] Nino Shervashidze, Paseal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M. Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(77):2539-251, 2011.
[28] Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In Proceedings of the 33rd International Conference on International Conference Machine Learning Volume 48, ICML'16, page 2014-2022 JMLR.org. 2016.
[29] Qi Zhao and Yusu Wang. Learning metrics for persistence-based summaries and applications for graph classification. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bur Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 9855-98666 2019.
11

---

Published as a conference paper at ICLR 2021
[30] Sergey Ivanov and Evgeny Burnaev. Anonymous walk embeddings. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, vol- ume 80 of Proceedings of Machine Learning Research, pages 1195-2195 Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
[31] Christoph Hofer, Florian Graf, Bastian Rieck, Marc Niethammer, and Roland Kwitt. Graph filtration learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th Inter- national Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4323, Virtual, 13-18 Jul 2020. PMLR.
[32] Manfredo Perdigao do Carmo, Riemannian geometry. Birkhäuser, 1992.
12

---

Published as a conference paper at ICLR 2021
A HOMOLOGY AND DIFFERENTIAL GEOMETRY
A.1 HOMOLOGY
Homology is general method for associating chain of algebraic objects (e.g abelian groups) to other mathematical objects such as topological spaces. The construction begins with a chain group, Cn. whose elements are the n-chains, which for a given complex are formal sums of the n n--imennional cells. The boundary homomorphism, O, Cn + Cn+1, maps each r--hain to the sum of the (n -iddiennii faces of its n-cell, which is (n - 1) c)hchiin. Combining the chain groups and the boundary maps in the sequence, we get the chain complex:
But2, Cn+1 Butl, Cn B.,, 02, C1 0 Cu 0 O.
(15)
The kernels, Zn - (er(On), ). and the images, Bn im((n++)) of the boundary homomorphisms are called cycle and boundary groups, respectively. A fundamental property of the boundary homomor- phism is that its square is zero, 0, = 0, which implies that, for every n, the boundary forms a subgroup of the cycle, i.e., im(0a+1) S ker(0,). Therefore, we can create their quotient group
Hn == (0.( (imm))+)mm = Zn/ Bu,
(16)
which is called n-th homology group and its elements are called homology classes. The rank, 3n ran) of this group is known as the n-th Betti number.
Even though homology can be applied to any topological space, we focus on simplicial homology, i.e. homology groups generated by simplicial complexes. Let K be simplicial complex and Kn its n-skeleton. The chain group (K) is the free abelian group whose generators are the n-dimensional simplexes of K. For simplex = [mu, E Kn, we define the boundary homomorphism as
Ox Σ Fo ... I4-1, 1441 xp)
(17)
and linearly extend this to C. CK)K,, i.e., On =
A.2 RIEMMANIAN MANIFOLDS
An mn-dimensioall manifold can be seen a generalization of a 2D surface and is space that can be locally approximated by RT. The space in which the manifold is embedded is called ambient space. We assume that the ambient space is the standard Euclidean space of dimension equal to the dimension of the manifold, i.e, R For I E M, the tangent space T..M of M at x is the linear vector space spanned by the m linear independent vectors tangent to the curves passing through x. A Riemannian metric g = (gz)EEM on M is a collection of inner products ga T.M x T + IR varying smoothly across M and allows us define geometric notions such as angles and the length of curve. A Riemannian manifold is a smooth manifold equipped with Riemannian metric g. The Riemannian metric g gives rise to the metric norm I|-I1, and induces global distances by integrating the length of shortest path between two points x, y E MM
d(x, 4) = =inf L''' (5(),
(18)
where y M 1], H. is such that 7(0) = and 711 = y. The unique smooth path y of minimal length between two points æ and y is called a geodesic and the underlying shortest path length is called geodesic distance. Geodesics and can be seen as the generalization of straight-lines. For point x E M and a vector v E T,M, let yo be the geodesic starting from x with an initial tangent vector equal to v, i.e., 7o (0) the exponential map are guaranteed by the classical existence and uniqueness theorem of ordinary differential equations and Gauss's 's Lemma of Riemannian geometry (32).
and (0) v. The existence, uniqueness and differentiability of
A.3 POINCARE BALL
We define the Möbius addition for any x, on the Poincare ball:
(1 + 2(x, w) + |Iell )x + (1 ||x| 12) 10y
y
(19)
1 + 2(z,, w) + ||=0|1 ||v11
13

---

Published as a conference paper at ICLR 2021
The Möbius addition is manifold preserving operator that allows to add point belonging in the Poincare ball without leaving the space. Finally, we provide the coordinate chart 3b .M - I2m i... the inverse of the parameterization in Eq. g which is defined as follows:
I2 11 II sin (y1) cos (yi), for i ,,..., and Zm 21 II sin (v₂) sin (ym). j=2
(20)
Note that we have omitted the learnable parameters. The coordinate chart is not explicitly needed for constructing the representation. However, could be useful as an intermediate layer between our representation and standart deep neural network architecture. Its purpose would be to transfer the representation from the manifold to a Euclidean space which can be fed into a standart DNN without having to define its operations the manifold.
B THEORETICAL RESULTS
Proof of Theorem Let E two persistence diagrams and q & bijection that achieves the infinum in Eq
Consider the subset DD = 'D \ Ra. which is essentially the original diagram without the points in the diagonal. Then, we have the following sequence of incqualities
dss( (D,
(D, 0), ((E,))
(21)
ds
exPao
Σ
logzo (o(((z)))) exp2o
Σ SEC
logao (o(p(2))))
(22)
ZED
< K
10g (d(p(x)))
10% (d( (d(p(())))
(23)
RED
DEE
$
Kg
log.r, (o(p(x)))
Σ log (e((((x))))
(24)
mEDo
DEDo
< Kg Σ
||ogoro (p(p(x))) log (,((((()))))
(25)
aEDo
s K2Ki Σ [((p(()) - M((((()))) S KeKiKo
Σ [l((z) - p(nx)))l
(26)
rDDo
zEDo
< KKKKK
Σ 1 S Ku (D, ZEDo
(27)
where wi is the Wasserstein distance with the g-norm replaced with the norm induced by the metric tensor of the Poincare ball. Eq. [3 from the smoothness of the exponential map (assuming that it has Lipschitz constant equal to K2). Eq. 24 follows by substituting the bijection v and because, per requirements of Theorem D the auxiliary transformation is zero the diagonal RA and
To = O. Note that the requirements summation over E RA collapses to follows from the smoothness of the logarithmic map and the parameterization (assuming K2 and K₂ Lipschitz constants, respectively). Finally, Eq. 27 follows from the assumed, as per Theorem smoothness of and by using the definition of Wasserstein distance. This concludes the proof.
Theorem [
combined with Eq. 8 10 and [9] imply that the
Eq. 25 follows from the triangle inequality. Eq. 26
C IMPLEMENTATION DETAILS
Auxiliary Transformation: Even though several choices for the auxiliary transformation p R - R. defined in Eq. 7 are possible, we present the one used in our implementation. Our choice for p is based the exponential structure elements introduced by Hofer al. (15). Formaaly, for i = 1,2,... m, we define the following
p((x1, = e (n #1. 3- -lu (u) 12) -2,))
F1 + x2
(28)
21 F2-
14

---

Published as a conference paper at ICLR 2021
Then, the auxiliary transformation is p: (21,22) - (p:(21. ( x22)))²=, where (J1,& (2,x)= the means. In contrast to Hofer et al. (15), those means are not learned, but we rather fix them to pre-defined values and keep then constant during training. Additionally, above-defined satisfies the stability conditions given by Theorem i.e., zero on the diagonal and Lipschitz continuity. Finally, under the assumption of all means being unique, it is injective on R. and, therefore, introduces no information loss.
easily prove that the
Network Architecture: We implemented all algorithms in TensorFlow 2.2 using the TDA-Toolki and the Scikit-TDA for extracting persistence diagrams and run all experiments on the Google Cloud AI Platform. To show the versatility of our approach, we used the same network architecture across all of our simulations and our proposed representation acts as an input. In more detail, for each individual filtration that we extract from the data (images or graphs), we create different input layer. The input to this layer is the persistence diagrams of all homology dimension given by the corresponding filtration. Given the nature of the data (graphs and images), the homology dimensions with non-trivial persistence diagrams layer processes the persistence diagrams of each homology class independently and outputs the representations (i.e., Eq. ITp for each one of them. Following that, concatenate and flatten the previous outputs and fed the resulting vector to a batch normalization layer. Then, we use two dense layers of 256 and 128 neurons, respectively, with Rectified Linear (ReLu) activation function. This is followed by a dropout layer to prevent overfitting and a final dense layer as an output. We consider two different implementations of the the above architecture. The first one is based on the hyperbolic neural networks introduced by Ganea et al. in (22). It re-defines key neural network operations in the Poincare ball and, therefore, allows to directly utilize the output of our representation in the neural network. The second implementation uses the coordinate chart given by Eq. 20 to project the representation back Euclidean space and feed it to a standardized implementation of the aforementioned layers. This method is more convenient in practice as it maintains compatibility with existing backbone networks trained in Euclidean geometry.
only the first two (i.e., Ho and Ifj). Our input
Hyperparameters and Training: To train the neural network, we use the Adam optimizer (8) 0.9, B2 = 0.999) with an initial learning rate of 0.001 and batch size equal to 64. We use a random uniform initializer in the interval [-0.05, 0.05) for all learnable variables. For all graph datasets trained the network for 100 epochs and halved the learning rate every 25 epochs. For the MNIST and Fashion-MNIST datasets used 10 and 20 epochs, respectively, and no learning rate scheduler. We tune the dropout rate manually by starting from really low values and monitoring the validation set accuracy. In general, noticed that the network did not tend overfit, therefore, we kept the rate low values (0-0.2) for all experiments.
15


Reviews:
[{"review_id": "zCqrv0z_kLH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to learn a representation for the persistence diagram (PD) in the hyperbolic space to incorporate the essential features (i.e., infinite persistence). The authors show that the hyperbolic representation has stability. Empirically, the authors illustrate that the hyperbolic representation for PD compares favorably with other baselines on graph and image classification.\n\nThe motivation to use hyperbolic representation to include essential features for PD is interesting. The authors give some details about the background (e.g., persistence diagrams, Poincare ball). However, the main part of the framework and how to learn the parameters of the embedding are missing.\n\nAs I understand, the authors propose: \n\n(1) to use some auxiliary transformation to lift points of $R^2$ into $R^m$ (however, there are no properties or information about this auxiliary transformation, and how one can do it)\n\n(2) projects points in $R^m$ into the Poincare ball, parameterized by $\\theta$. Why ones need this parameterization, and why this parameterization is important in applications? (It seems the authors combine projection with some transformation here?). It is better in case the authors give the projection (w.r.t. what distance?) and then transform the projected points. \n\n+ There is no description of the space of $\\Theta$ for the parameterization?\n\n(3) combine the representations of each point in PD. It seems that the authors use the sum of all points w.r.t. hyperbolic manifold to represent PD.\n\n+ I also concern about the usage of exp and log map at point $x$ for this combination. Typically, the tangent space is just simply a \"flattened\" space of the hyperbolic space at point $x$, it can only preserve the geometry for some close neighbor points of $x$ (this \"flat\" approach has a large distortion for those points which are far to $x$. It is better in case the authors give more discussion about how to choose $x$, and how it affects the geometry of the hyperbolic?\n\nThe main framework to learn those parameters is not presented in the main manuscript, and it is also unclear how one uses the learned hyperbolic representation for the downstream task?\n+ It seems that the authors incorporate the procedure to learn those parameters inside the networks for classification and learn end-to-end?\n\nSomehow, it seems that the authors propose to use a hyperbolic embedding within a neural network as a classifier for PDs. (in my opinion, the novelty may be incremental in case the authors simply replace the Gaussian-like embedding of Hofer et al. into a hyperbolic embedding inside a neural network. However, the hyperbolic representation for PD may be still interesting by itself. )\n\nOverall, I think this work seems interesting and has good potential. The authors may need to describe more details about the neural framework use to learn the parameters of the hyperbolic representation.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "q4Pm8b_u4DP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Synopsis of the paper\n\nThis paper proposes a novel embedding or representation algorithm for\npersistence diagrams, i.e. topological descriptors. Existing methods are\nrestricted because they do not feature *learnable* or *trainable*\nparameters for their representations (with the exception of methods such\nas the one by Hofer et al. or PersLay). This method, by contrast,\npresents a trainable embedding to the Poincaré ball, thus representing\npersistence diagrams in a hyperbolic space.\n\nThis has the advantage of being more appropriate for representing\n*essential features*, i.e. features of infinite persistence. Thus,\nit is possible to perform end-to-end training of neural networks\nthat use persistence diagrams as their input.\n\nExperiments with graph classification and image classification tasks\ndemonstrate the utility of the proposed method.\n\n# Summary of the review\n\nThis is a very well-written paper, with a strong contribution to\ntopological data analysis and machine learning. The paper is technically\nsound, apart from some minor inconsistencies, which I shall discuss\nbelow. It is exciting for me to see how to obtain fully-trainable\nembeddings here, and I am happy to endorse this paper.\n\nThere are a few issue that I would like to see rectified in a revision\nof the paper, though:\n\n1. Some details on the method are missing. This concerns primarily the\n   learnable auxiliary transformation from Eq. 7. While the paper\n   subsequently discusses potential choices for this function, this is\n   not specified. In fact, the method tying everything together even\n   cites this equation again, without providing a definition for it.\n\n   This needs to be rectified---I am assuming that one could, for\n   example, use a transformation such as the one provided by Hofer et\n   al.; is this correct?\n\n2. The experiments on graph classification are lacking some comparison\n   partners. Specifically when discussing topology-based approaches, it\n   is useful to compare to other topology-based approaches. Here are\n   some suggestions:\n\n      - Hofer et al.: *Graph Filtration Learning*\n      - Rieck et al.: *A Persistent Weisfeiler–Lehman Procedure for Graph Classification* \n      - Zhao & Wang: *Learning metrics for persistence-based summaries and applications for graph classification*\n\n    There is an overlap of the data sets assessed using these methods,\n    so the appropriate results could be cited and used. I wanted to\n    raise this concern primarily because I am aware of other methods\n    obtaining somewhat better classification performance in some cases.\n    \n    It would strengthen the paper immensely if it could perform some\n    form of 'ablation study', i.e. highlighting to what extent the\n    improved results are driven by the embedding in hyperbolic space, or\n    the choice of filtration, etc.\n\n3. Adding to this, some details of the method need to be described\n   better; only the supplements briefly mention how $m$, the embedding\n   parameter, is chosen in the end, but I would prefer a more in-depth\n   analysis of the choice of this parameter. Is it sufficient to pick,\n   say, $m = 32$ for all practical purposes? Or does it make sense to\n   concatenate the representations afterwards, as discussed in the\n   supplements? Adding more details will aid in understanding the paper\n   and, ultimately, promote further adoption of the method.\n\n# Detailed comments\n\n- In the abstract, I would say that 'Existing methods are restricted in\n  terms of their expressivity'. I would use 'bottleneck' only to refer\n  to the eponymous distance, in order to prevent confusion.\n\n- The definition of 'filtration' in the introduction is incorrect;\n  a filtration is a *sequence* of subspaces. \n\n- I would not refer to diagrams being *injective*. I understand what is\n  mean here, viz. the fact that under some assumptions (!), one can\n  reconstruct the input space from a diagram (this is known as solving\n  the inverse problem), but I think this might be slightly confusion\n  here. Why not focus on the expressivity properties of diagrams and the\n  fact that one can *approximate* their inputs under some conditions?\n\n- The comment on 'extended persistence' on p. 2 is slightly imprecise;\n  'extended persistence' is well studied now, but PersLay is indeed the\n  first 'deep learning' method incorporating it. This should be\n  clarified.\n\n- When discussing filtrations and homology groups, I would stick with\n  $\\subseteq$ instead of $\\subset$. The former is more generic and would\n  make the write-up more consistent, as currently both forms are being\n  mixed.\n\n- As outlined above, I would suggest to be more verbose when it comes to\n  the description of the method in terms of the individual functions,\n  i.e. $\\psi$ and $\\rho$.\n\n- Am I mistaken or could $\\rho$ also be *any* universal approximator,\n  such as a set function or, more generically, a deep neural network?\n  Would it be possible to use the method by Hofer et al. (2017) to\n  obtain this map and *then* subsequently train a better hyperbolic\n  embedding?\n\n- When running the experiments, is $m$ fixed or is the best $m$ selected?\n  I see that $m \\in \\{2, \\dots, 12\\}$, but I do not understand how this\n  is actually used in the network. Figure 4 is somewhat helpful here,\n  but it would be interesting to see what happens for a range of\n  parameters.\n\n- It is my understanding that the Euclidean method P-Eucl is only driven\n  by $\\rho$. Is this correct? How critical is this choice in practice?\n  Moreover, how was it chosen for this paper? I am asking because it is\n  clear that the hyperbolic embedding helps improve predictive\n  performance, but I wonder what would happen if one uses an\n  'underpowered' $\\rho$ function.\n\nAll in all, I feel that this could make a very strong addition to the\ntopological machine learning literature!\n\n# Style & clarity\n\nThe paper is very well-written, the authors are to be commended for\nthat. I have a few minor suggestions, some of which are more personal\npet peeves, but which might help make this paper shine even more!\n\n- 'root of a complex polynomial' --> 'roots of a complex polynomial' (?)\n\n- I would prefer not to use citations as nouns, i.e. I would prefer\n  writing 'Kusano et al. (5)' instead of 'In (5), Kusano et al.'; the\n  former strikes me as more readable and also translates well to\n  different citation styles.\n\n- The image of $f_n^{i,j}$ is *the* $n$th persistent homology group\n\n- Whenever possible, I would use $\\operatorname{text}$ for operators or\n  functions, instead of 'raw' $text$. This concerns for example the rank\n  function but also the degree function.\n\n- I would write 'Poincaré' everywhere\n\n- 'persistent diagram' --> 'persistence diagram'\n\n- 'persistent image' --> 'persistence image'\n\n- 'Vietories' --> 'Vietoris'\n\n- 'gray-scale' --> 'grey-scale' (or vice versa, if American English is\n  the preferred spelling)", "extra_scores": {}, "extra_text": {}}, {"review_id": "NTpqIcJOSoT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a new representation of persistence diagrams that can include `''essential features''. Essential features correspond to the intrinsic topology of the underlying space that will not die during the filtration. To include the fact that the essential features are infinitely far from other normal features in the diagram, the authors proposed to use a Poincare ball representation, which maps the diagram into a disk whose boundary is infinitely far from inside. \n\nThe authors further proposed a classifier that learns the parameterization of the embedding of a diagram in the Poincare ball. The presentation learning procedure seems to be similar to (Hofer et al. 17), except for using a Poincare ball representation instead of the Euclidean square/triangle representation. Experiments are carried on graph classification tasks and image classification task.\n\nOn the positive side, I think the proposed representation well unified essential and non-essential topological structures. It is elegant and well-thought. A stability theorem is proven (in a similar manner as other known representations). The references are also reasonably complete.\n\nHowever, I am having doubts on the practical motivation of the representation in the learning context. To me, essential features can be considered by simply adding the histogram of their birth times as additional features (to the neural networks at an appropriate layer). I think this approach is a natural and necessary baseline to be compared with.\n\nGenerally, the empirical results are not particularly strong. On graph datasets, the proposed method is only winning 2 out of 5 times. Many important baselines are also missing. For graph classification methods, state-of-the-art classifiers such as GIN and GraphSAGE should be at least compared with. For topological classifiers, many kernel methods could be compared with: PWGK, PI (both of which were used in the other experiment), also Sliced-Wasserstein Kernel. I honestly think an ensemble of these methods and the histogram of birth times of essential diagrams can be easy to tune and perform better. \n\nOther questions/comments:\n\nThe references are not standard and need to be fixed. \n\nFor graph with Rips filtration, why are there 1-dim essential homology? Wouldn't all 1D homology features be killed eventually?\n\nDetails of the classifier (how the representation is learned) is still not clear even after reading the section in the supplemental material. \n\nExperimental details are missing. I understand this is 80% training 20% testing. But how many folds were used to evaluate? Some baseline numbers are very similar to the numbers in the GIN paper. However, the GIN paper was using a 10-fold cross-validation. So there is some discrepancy in the experiments. I would appreciate if the authors could kindly elaborate.\n\nOverall I think this is a nice mathematical formulation to incorporate essential features into the representation of persistent homology. But the practical usage in learning is not very convincing. Fundamentally, the essential features are completely different from non-essential ones. There is nothing in between them. Thus the benefit of a unified representation does not bring much more information than simply treating them separately.\n\n\n** After rebuttal: \nI am increasing the score to 6. I appreciate the authors' response to the reviews. They did the additional experiments I asked for. \n\nAs I stated in the original review, I really liked the unified approach. It is elegant and is nicely presented.\n\nAfter reading the authors' response to R4, it is clarified that the 1D essential homology is because the computation over all threshold is too expensive. I think there might be an opportunity to better justify this paper: we often have to stop the filtration early due to computational concern. This unified representation could potentially be a good solution for this: without computing the actually death time, the unified solution can still 'learn' the real death time of the 1D essential classes. The authors might want to discuss or ideally empirically verify this in the final version. For example, can you show that using the new approach, and stopping earlier during the filtration, the unified classifier can be as good as when we run the whole filtration and compute the real death  time for all 1D classes. Moreover, it will be ideal if the authors can manage to show that the unified approach can actually learn the real death time for these fake essential classes (I do not know how). This way the paper can potentially have a bigger impact. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LNuETgQ_0qh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to learn representations of topological persistence diagrams in hyperbolic spaces (Poincare balls). They provide a step-by-step methodology, and an algorithm for creating the representation. They also compare their approach with various graph classification and other ways of representing persistence diagrams.\n\nStrengths:\n=========\n1. This is the first work that learns hyperbolic representations of persistence diagrams.\n2. The methodology is sufficiently clear and the paper is reasonably well-written.\n3. Applications to graph data as well as image data show some promise.\n\nThings to improve:\n=================\n1. My biggest question in this paper is around the assumptions the authors have made and the motivation. They have mentioned that essential features have infinite persistence and truncating the persistence is not the best thing to do. I agree, however, there is really only one essential feature - the 0th order persistent homology group that becomes a single connected component at a scale that is sufficiently large. Since this \"essential\" group is one and the same for all point clouds that we compare,  they always match to one another, and it is safe to say that this truly essential group can be ignored in ideal cases. All other \"essential\" groups are only artifacts of our construction, which happen because we do not use a large enough scale for computational or some other reason.\n\n2. Even in an ideal setting (with large enough scales), it would be good to discuss why hyperbolic coordinates make sense. Why do we expect the distance between the persistence homology module to have some hyperbolic trend (see fig. 2 in https://dawn.cs.stanford.edu/2018/03/19/hyperbolics/). Some thought and discussion around this would be helpful to understand the motivation of the approach.\n\n3. Based on fig. 2 I was initially under the impression that individual points will be mappable between persistence diagrams and hyperbolic representations, but it took some more reading to understand that there is one single representation for the entire persistence diagram in the hyperbolic space. Perhaps the authors can make it clear somewhere (even in Fig. 2).\n\n4. How is x0 chosen in (11). Why is summing in tangent space meaningful? Why not some  other operation?\n\n5. What is the absolute state-of-the-art  for Table 1 and Table 2? I am not holding this as a negative, but it may be good for the readers to know where TDA methods stand in these applications.\n\n6. Along similar lines, are there any applications that are especially suited to TDA that the authors can demonstrate? \n\n7. The authors seem to have used multiple runs for results in Table 1, but the text in Sec. 4.1 does not indicate how it is done. Please elaborate.\n\n8. Why are some values missing for some methods in Table 1.\n\n9. Why does P-Eucl make sense as a baseline? Is this like a so-called ablation study for this procedure? How are the representations created for P-Eucl? Please feel free to add more details on this in supplement.\n\n10. In Sec. 4.2, do the  compared methods also use the same simplicial construction (cubical complexes constructed in a  specific way)? If so, would the compared baselines work better with some other construction?\n\nIn summary, this is an interesting direction, but clarity is needed regarding the motivation,  and presenting extra details along the way will also be appreciated.  Finally, it could make sense to think about  an appropriate application.\n\nUpdate post-rebuttal\n==================\n- I am happy with the thorough engagement by the authors and their clarifications. So  I am bumping the up the score a notch.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel method for representing persistence diagrams by embedding them to a Poincare ball.  The representation is learnable, and unifies essential and non-essential features.   The experimental comparisons with existing representation methods show significant improvements in performance.   
The flexible data-driven embedding to a suitable geometric space is a novel idea, which will certainly advance the usefulness of TDA.  The  experimental resutls demonstrate well the advantage of the proposed repesentation.  The authors have also addressed the review comments appropriately, with some extra experiments.  This is also a good addition.   
Decision: Accept (Poster)