Title: Neural Latent Geometry Search: Product Manifold Inference via Gromov-Hausdorff-Informed Bayesian Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Representation Learning

Product Manifolds

Bayesian Optimization

Gromov-Hausdorff Distance
Abstract: Recent research indicates that the performance of machine learning models can be improved by aligning the geometry of the latent space with the underlying data structure. Rather than relying solely on Euclidean space, researchers have proposed using hyperbolic and spherical spaces with constant curvature, or combinations thereof, to better model the latent space and enhance model performance. However, little attention has been given to the problem of automatically identifying the optimal latent geometry for the downstream task. We mathematically define this novel formulation and coin it as neural latent geometry search (NLGS). More specifically, we introduce an initial attempt to search for a latent geometry composed of a product of constant curvature model spaces with a small number of query evaluations, under some simplifying assumptions. To accomplish this, we propose a novel notion of distance between candidate latent geometries based on the Gromov-Hausdorff distance from metric geometry. In order to compute the Gromov-Hausdorff distance, we introduce a mapping function that enables the comparison of different manifolds by embedding them in a common high-dimensional ambient space. We then design a graph search space based on the notion of smoothness between latent geometries and employ the calculated distances as an additional inductive bias. Finally, we use Bayesian optimization to search for the optimal latent geometry in a query-efficient manner. This is a general method which can be applied to search for the optimal latent geometry for a variety of models and downstream tasks. We perform experiments on synthetic and real-world datasets to identify the optimal latent geometry for multiple machine learning problems.

Neural Latent Geometry Search: Product Manifold Inference via

Abstract

Recent research indicates that the performance of machine learning models can be improved by aligning the geometry of the latent space with the underlying data structure. Rather than relying solely on Euclidean space, researchers have proposed using hyperbolic and spherical spaces with constant curvature, or combinations thereof, to better model the latent space and enhance model performance. However, little attention has been given to the problem of automatically identifying the optimal latent geometry for the downstream task. We mathematically define this novel formulation and coin it as neural latent geometry search (NLGS). More specifically, we introduce an initial attempt to search for a latent geometry composed of a product of constant curvature model spaces with a small number of query evaluations, under some simplifying assumptions. To accomplish this, we propose a novel notion of distance between candidate latent geometries based on the Gromov-Hausdorff distance from metric geometry. In order to compute the Gromov-Hausdorff distance, we introduce a mapping function that enables the comparison of different manifolds by embedding them in a common highdimensional ambient space. We then design a graph search space based on the notion of smoothness between latent geometries, and employ the calculated distances as an additional inductive bias. Finally, we use Bayesian optimization to search for the optimal latent geometry in a query-efficient manner. This is a general method which can be applied to search for the optimal latent geometry for a variety of models and downstream tasks. We perform experiments on synthetic and real-world datasets to identify the optimal latent geometry for multiple machine learning problems.

1 Introduction

There has been a recent surge of research employing ideas from differential geometry and topology to improve the performance of learning algorithms [Bortoli et al., 2022, Hensel et al., 2021, Chamberlain et al., 2021, Huang et al., 2022, Barbero et al., 2022a,b]. Traditionally, Euclidean spaces have been the preferred choice to model the geometry of latent spaces in the ML community [Weber, 2019, Bronstein et al., 2021]. However, recent work has found that representing the latent space with a geometry that better matches the structure of the data can provide significant performance enhancements in

Álvaro Arroyo ˚

both reconstruction and other downstream tasks [Shukla et al., 2018]. In particular, most works have employed constant curvature model spaces such as the Poincaré ball model [Mathieu et al., 2019], the hyperboloid [Chami et al., 2019], or the hypersphere [Zhao et al., 2019], to encode latent representations of data in a relatively simple and computationally tractable way.

While individual model spaces have sometimes shown superior performance when compared to their Euclidean counterparts, more recent works [Gu et al., 2018, Skopek et al., 2019, Sáez de Ocáriz Borde et al., 2023b,a, Zhang et al., 2020, Fumero et al., 2021, Pfau et al., 2020] have leveraged the notion of product spaces (also known as product manifolds ) to model the latent space. This idea allows to generate more complex representations of the latent space for improved performance by taking Cartesian products of model spaces, while retaining the computational tractability of mathematical objects such as exponential maps or geodesic distances , see Sáez de Ocáriz Borde et al. [2023b]. Despite the success of this methodology, there exists no principled way of obtaining the product manifold signature ( i.e. , the choice and number of manifold components used to generate the product manifold and their respective dimensionalities) to optimally represent the data for downstream task performance. This procedure is typically performed heuristically, often involving a random search over the discrete combinatorial space of all possible combinations of product manifold signatures, which is an exceedingly large search space that hampers computational efficiency and practical applicability. Some other work related to latent space geometry modelling can be found in Lubold et al. [2023], Hauberg et al. [2012], Arvanitidis et al. [2017].

Contributions. 1) In this paper, we consider a novel problem setting where we aim to search for an optimal latent geometry that best suits the model and downstream task. As a particular instance of this setting, we consider searching for the optimal product manifold signature. Due to the conceptual similarity with neural architecture search (NAS) strategies [Elsken et al., 2018, Zoph and Le, 2016, Pham et al., 2018], we coin this problem as neural latent geometry search (NLGS) , which we hope will encourage additional work in the direction of optimal latent geometry inference. We test our framework on a variety of use cases, such as autoencoder reconstruction [Mathieu et al., 2019] and latent graph inference [Sáez de Ocáriz Borde et al., 2023b], for which we create a set of custom datasets.

2) To search for the product manifold signature, we must be able to compare product manifolds. This is traditionally done by computing the Hausdorff distance between manifolds [Taha and Hanbury, 2015], which however requires manifolds to reside in the same metric space. To address this limitation, in this work we develop to our knowledge the first computational method to mathematically compare product manifolds. Our approach generalizes classical algorithms and allows the comparison of manifolds existing in different spaces. This is achieved by defining an isometric embedding that maps the manifolds to a common high-dimensional ambient space, which enables the computation of their Gromov-Hausdorff distances.

3) Leveraging the Gromov-Hausdorff distances between candidate latent space manifolds, we design a principled and query-efficient framework to search for an optimal latent geometry, in the sense that it yields the best performance with respect to a given machine learning model and downstream task.

Our approach consists of constructing a geometry-informed graph search space where each node in the graph represents a unique candidate product manifold, associated with the model performance using this manifold as its embedding space. The strength of edges in the graph are based on the inverse of the Gromov-Hausdorff distance, thereby encoding a notion of 'closeness" between manifolds in the search space. We then perform efficient search over this space using Bayesian optimization (BO). We compare our proposed method with other search algorithms that lack the topological prior inherent in our model. Empirical results demonstrate that our method outperforms the baselines by a significant margin in finding the optimal product manifold.

Outline. In Section 2 we discuss manifold learning, and product manifolds of constant curvature model spaces such as the Euclidean plane, the hyperboloid, and the hypersphere. We also review relevant mathematical concepts, particularly the Hausdorff and Gromov-Hausdorff distances from metric geometry [Gopal et al., 2020]. Section 3 presents the problem formulation, the proposed methodology to compare product manifolds, as well as how the search space over which to perform geometry-informed Bayesian optimization is constructed. Finally, Section 4 explains how our custom

synthetic and real-world datasets were obtained, and the empirical results. Lastly, in Section 5 we conclude and discuss avenues for future work. 2

2 Background

Manifold Learning Manifold learning is a sub-field of machine learning that uses tools from differential geometry to model high-dimensional datasets by mapping them to a low-dimensional latent space. This allows researchers to analyze the underlying structure of data and improve machine learning models by capturing the geometry of the data more accurately. Manifold learning is based on the assumption that most observed data can be encoded within a low-dimensional manifold (see Figure 1) embedded in a high-dimensional space [Fefferman et al., 2013]. This has seen applications in dimensionality reduction [Roweis and Saul, 2000, Tenenbaum et al., 2000], generative models [Goodfellow et al., 2014, Du et al., 2021, Bortoli et al., 2022], and graph structure learning for graph neural networks (GNNs) [Topping et al., 2021]. In all these application, the key is to find a topological representation as an abstract encoding that describes the data optimally for the downstream task.

Product Manifolds. In this work, we model the geometry using model space Riemannian manifolds (Appendix A.1) and Cartesian products of such manifolds. The three so-called model spaces with constant curvature are the Euclidean plane, E n ' E d E K E ' R d E , where the curvature K E ' 0 ; the hyperboloid, H n ' H d H K H ' t x p P R d H ` 1 : x x p , x p y L ' 1 { K H u , where K H ă 0 and x¨ , ¨y L is the Lorentz inner product; and the hypersphere, S n ' S d S K S ' t x p P R d S ` 1 : x x p , x p y 2 ' 1 { K S u , where K S ą 0 and x¨ , ¨y 2 is the standard Euclidean inner product. These have associated exponential maps and distance functions with closed form solutions, which can be found in Appendix A.2. A product manifold can be constructed using the Cartesian product P ' Ś n P i ' 1 M d i K i of n P manifolds with curvature K i and dimensionality d i . Note that both n P and d i are hyperparameters that define the product manifold P and that must be set a priori. On the other hand, the curvature of each model space K i can be learned via

ϕ

i

ϕ

´

1

i

ϕ

´

1

j

ϕ

j

M

U

i

U

j

ϕ

i

p

U

i

q

R

m

ψ

ij

ϕ

j

p

U

j

q

R

m

gradient descent. One must note that the product manifold construction makes it possible to generate more complex embedding spaces than the original constant curvature model spaces, but it does not allow to generate any arbitrary manifold nor to control local curvature.

Hausdorff and Gromov-Hausdorff Distances for Comparing Manifolds. The Hausdorff distance between two subsets of a metric space refers to the greatest distance between any point on the first set and its closest point on the second set [Jungeblut et al., 2021]. Given a metric space X with metric d X , and two subsets A and B , we can define the Hausdorff distance between A and B in X by

Apriori this quantity may be infinite. Hence we will restrict to compact subsets A and B . In this case, we can equivalently define d H p A,B q as the smallest real number c ě 0 such that for every a P A and every b P B there exist a 1 P A and b 1 P B such that both d X p a, b 1 q and d X p a 1 , b q are at most c .

We note that the previous definition does not require any differentiable structures on X , A and B . They can be merely metric spaces. This generality allows us to distinguish the metric properties of Euclidean, hyperbolic and spherical geometries beyond analytic notions such as curvature. However, the definition in Equation 1 only allows to compare spaces A and B that are embedded in a certain metric space X . The notion of distance that we shall consider is the Gromov-Hausdorff distance, which we define below in Equation 3.

Now, given two metric spaces A and B , we denote by ES p A,B q (standing for 'embedding spaces of A and B ') as the triple p X,f,g q where X is a metric space and f : A Ñ X and g : B Ñ X are isometric embeddings. We define the Gromov-Hausdorff distance between A and B as:

We should note that, since we assume that both A and B are compact, there is a trivial upper bound for their Gromov-Hausdorff distance in terms of their diameters. The diameter of a metric space Y is defined to be diam p Y q ' sup y,y 1 P Y d Y p y, y 1 q . Given a 0 P A and b 0 P B , we can define the isometric embeddings f : A Ñ A ˆ B and g : B Ñ A ˆ B given by f p a q ' p a, b 0 q and g p b q ' p a 0 , b q . It is easy to see that d A ˆ B,f,g H p A,B q ď max p diam p A q , diam p B qq . Since the triple p A ˆ B,f,g q belongs to ES p A,B q , we can estimate d GH p A,B q ď max p diam p A q , diam p B qq . To compare E n , H n and S n , we propose taking closed balls of radius one in each space. Since balls of radius one in any of these spaces are homogeneous Riemannian manifolds, they are isometric to each other. By estimating or providing an upper bound for their Gromov-Hausdorff distance, we can compare the spaces. This notion of distance between candidate latent geometries will later be used to generate a search space for our framework. With exactly an analogous argument as before, we can notice that given two compact balls of radius one B and B 1 , of centres x 0 and x 1 0 , we can embed B into B ˆ B 1 by the mapping f : b ÞÑ p b, x 1 0 q . From here, it is obvious to see that d B ˆ B 1 ,f, id H p B,B ˆ B 1 q ' 1 . In particular, this gives us the bound d GH p B,B 1 q ď 1 . This is the estimation we will take as the Gromov-Hausdorff distances of product manifolds that simply differ in one coordinate (such as, say, E 2 and E 2 ˆ H 2 ).

3 Neural Latent Geometry Search: Latent Product Manifold Inference

In this section, we leverage ideas discussed in Section 2 to introduce a principled way to find the optimal latent product manifold. First, we introduce the problem formulation of NLGS. Next, we outline the strategy used to compute the Gromov-Hausdorff distance between product manifolds, and we discuss how this notion of similarity can be used in practice to construct a graph search space of latent geometries. Lastly, we explain how the Gromov-Hausdorff-informed graph search space can be used to perform NLGS via BO.

3.1 Problem Formulation

The problem of NLGS can be formulated as follows. Given a search space G denoting the set of all possible latent geometries, and the objective function L T,A p g q which evaluates the performance of a given geometry g on a downstream task T for a machine learning model architecture A , the objective is to find an optimal latent geometry g ˚ :

In our case we model the latent space geometry using product manifolds. Hence we effectively restrict Equation 4 to finding the optimal product manifold signature:

where n P is the number of model spaces composing the product manifold P , t d i u i P n P are the dimensions of each of the model spaces of constant curvature, and t K i u i P n P their respective curvatures. We further simplify the problem by setting d i ' 2 , @ i , and by restricting ourselves to K i P t´ 1 , 0 , 1 u , in order to limit the size of the hypothesis space.

3.2 Quantifying the Difference Between Product Manifolds

Motivation. From a computational perspective, we can think of the Hausdorff distance as a measure of dissimilarity between two point sets, each representing a discretized version of the two underlying continuous manifolds we wish to compare. Taha and Hanbury [2015] proposed an efficient algorithm to compute the exact Hausdorff distance between two point sets with nearly-linear complexity leveraging early breaking and random sampling in place of scanning. However, the original algorithm assumes that both point sets live in the same space and have the same dimensionality, which is a

limiting requirement. Gromov-Hausdorff distances, as opposed to usual Hausdorff distances, allow us to measure the distance between two metric spaces that a priori are not embedded in a common bigger ambient space. However, this has the caveat that they are not computable. For instance, for our application we must calculate the distance between each pair of the following three spaces: E n , S n and H n . However, S n does not isometrically embed in E n and hence in order to compare E n and S n , we must work in a higher dimensional ambient space such as E n ` 1 . In the case of H n , finding an embedding to a Euclidean space is more complicated and is described in Appendix B.2 ( H n will be embedded isometrically into E 6 n ´ 6 ). However, in this process there will be choices made about which embeddings to consider (in particular, we cannot exactly compute the infimum that appears in the definition of Gromov-Hausdorff distance in Equation 3). Likewise, using the original algorithm by Taha and Hanbury [2015] it is not possible to compute the Hausdorff distance between product manifolds based on an unequal number of model spaces, for instance, there is no way of computing the distance between E n and E n Ś H n . In this section we give an upper bound for d GH p E n , S n q , and then describe an algorithm to give an upper bound for d GH p E n , H n q and d GH p S n , H n q .

Strategy. The spaces E n and S n isometrically embed into E 6 n ´ 6 in many ways. This may seem redundant because both spaces already embed in E n ` 1 . However, the interest of considering this higher dimensional Euclidean space is that H n will also isometrically embed into it. Crucially, this will provide a common underlying space in which to compute Hausdorff distances between our geometries E n , S n and H n , which will lead to an estimation of their mutual Gromov-Hausdorff distance. The embedding of H n into E 6 n ´ 6 that we shall describe appears in Henke and Nettekoven [1987] and is made explicit in [Blanuša, 1955]. We also refer the reader to the exposition [Brander, 2003, Chapter 5], which puts this results in a broader context while also summarising related advances on the topic of isometrically embedding homogeneous spaces into higher dimensional ones. For n ' 2 , we name this embedding F : H 2 Ñ E 6 (Appendix B.2). For simplicity, in our experiments in Section 4, we will work with product manifolds generated based on constant curvature model spaces of dimension n ' 2 .

Now we can summarise our strategy to estimate d GH p B E 2 , B H 2 q as follows (for d GH p B H 2 , B S 2 q it will be entirely analogous). The first step consists of approximating our infinite smooth spaces by finite discrete ones. For this, we consider several collections of points t P i u i P I in E 2 that are sufficiently well distributed. The exponential map can be applied to the collection of points exp : T 0 H 2 -R 2 Ñ B H 2 to get several collections of points Q in B H 2 (again, well distributed by construction). In addition, we will consider several isometric embeddings f k : B E 2 Ñ R 6 . Hence, we take

In Appendix B, we gradually unravel the previous formula and give explicit examples of the involved elements. In particular, in Appendix B.1 we explain how to generate points in the balls of radius one, and in Appendix B.2 how to describe the isometric embedding F : B H 2 Ñ E 6 . The results obtained for the Gromov-Hausdorff distances between product manifolds are used to generate the graph search space introduced in the next section.

3.3 The Gromov-Hausdorff-Informed Graph Search Space

Gromov-Hausdorff Edge Weights. In Section 2, we used Cartesian products of constant curvature model spaces to generate candidate latent geometries. We now turn our attention to constructing a search space to find the optimal latent geometry. To do so, we first consider all possible combinations of product manifolds based on a given number of model spaces, represented by n s . Furthermore, we denote the total number of products (model spaces) used to form the product manifold P with n p . Conceptually, n s is the number of model space types used, while n p refers to the overall number, or quantity , of model spaces that form the resulting product space. For instance, if only the Euclidean plane, the hyperboloid, and the hypersphere

are taken into account, then n s ' 3 . If all product mani- fold combinations are considered, the number of elements in the search space increases to ř n p i ' 1 n i s . However, we assume commutativity for latent product manifolds, implying that the output of a trained neural network with a latent geometry M i Ś M j should be the same as that with M j Ś M i . We refer to this concept as the symmetry of parameterization , as neural networks can rearrange the weight matrices of their neurons to achieve optimal performance for two equivalent latent manifolds. This assumption reduces the search space from growing exponentially to O p n 2 p q , assuming three constant curvature model spaces are considered (see Appendix B.5 for a more complete explanation).

H

2

Ś

H

2

E

2

Ś

H

2

S

2

Ś

H

2

E

2

Ś

E

2

E

2

Ś

S

2

S

2

Ś

S

2

We model the search space as a graph over which we perform BO. More formally, we consider a graph G given by p V , E q , where V ' t v i u N i ' 1 are the nodes of the graph and E ' t e j u j M ' 1 is the set of edges, where edge e j ' p v i , v j q connects nodes v i and v j . The topology of the graph is encoded in its adjacency matrix , which we denote with A P R N ˆ N . In our case, we focus on a weighted and undirected graph, meaning that A i,j ' A j,i , and A i,j can take any positive real values. We consider a setting in which the function f p¨q to minimize is defined over the nodes of the graph, and the objective of using BO is to find the node associated to the minimum value v ˚ ' arg min v P V f p v q .

In our setting, each node in the graph represents a different latent space product manifold, and the value at that node is the validation set performance of a neural network architecture using this latent geometry. We use the inverse of the Gromov-Hausdorff distance between product manifolds to obtain edge weights.

We denote by w M 1 , M 2 (edge weights) the inverse Gromov-Hausdorff distance between model spaces M 1 and M 2 . The approximate values of these coefficients are presented in Table 1. In particular, the Gromov-Hausdorff distance d GH p E 2 , S 2 q used to compute w E 2 , S 2 is derived analytically in Appendix B.3, while the remaining coefficients are obtained through computational approximations of Equation 6, which are depicted in Appendix B.4. Further, the Gromov-Hausdorff distance between manifolds of different dimensions is one (see Section 2).

In order to impose additional structure on the search space, we only allow connections between nodes corresponding to product manifolds which differ by a single model space. For example, within the same dimension, S 2 ˆ H 2 and E 2 ˆ H 2 would be connected with edge weighting w E 2 , S 2 while S 2 ˆ H 2 and E 2 ˆ E 2 would have no connection in the graph. Furthermore, product manifolds in different dimensions follow the same rule. For instance, we would have a connection of strength one between E 2 ˆ H 2 and E 2 ˆ H 2 ˆ H 2 , but no connection between, for instance, E 2 and E 2 ˆ H 2 ˆ H 2 or S 2 and E 2 ˆ H 2 . This construction induces a sense of directionality into the graph and generates clusters of product manifolds of the same dimension. Finally, it should be mentioned that in practice there are only four edge weights. For example, the connectivity strength between E 2 ˆ H 2 and H 2 ˆ H 2 is w E 2 , H 2 since d GH p E 2 ˆ H 2 , H 2 ˆ H 2 q ' d GH p E 2 , H 2 q given that d GH p H 2 , H 2 q ' 0 . Visual representations of the graph search space can be found Figures 2 and 3. Note that both figures use the same colour scheme, and in Figure 3 there is an increase in the dimen-

corresponding to the three model spaces). We refer the reader to Appendix B.6 for additional sionality of the product manifolds from left to right (e.g. on the top left corner, one can see a triangle visualizations.

Bayesian Optimization over the Graph Search Space. We aim to find the minimum point within the graph search space through the use of BO (see Appendix C). While performing BO on graphs allows us to search for the minimum over a categorical search space, some of the key notions used in BO over Eucledian spaces do not translate directly when operating over a graph. For example, notions of similarity or 'closeness" which are trivially found in Eucledian space through the ℓ 1 or ℓ 2 norms require more careful consideration when using graphs.

In our setting, we employ a diffusion kernel [Smola and Kondor, 2003, Kondor and Vert, 2004] to compute the similarity between the nodes in the graph. The diffusion kernel is based on the eigendecomposition of the graph Laplacian L P R N ˆ N , defined as L ' D ´ A where D is the degree matrix of the graph. In particular, the eigendecomposition of the Laplacian is given by L ' UΛU T , where U ' r u 1 , . . . , u N s is a matrix containing eigenvectors as columns and Λ ' diag p λ 1 , . . . , λ N q is a diagonal matrix containing increasingly ordered eigenvalues. The covariance matrix used to define the GP over the graph is given by

where β is the lengthscale parameter. Our approach is therefore conceptually similar to that of [Oh et al., 2019], which employ a diffusion kernel to carry out a NAS procedure on a graph Cartesian product. We highlight, however, that the main contribution of this work is the construction of the graph search space, and we employ this search procedure to showcase the suitability of our method in finding the optimal latent product manifold. For reference, we note that other works employing Bayesian optimization in graph-related settings include Cui and Yang [2018], Como et al. [2020], Ma et al. [2019], Ru et al. [2020], Cui et al. [2020], Wan et al. [2021].

4 Experimental Setup and Results

In this section, a detailed outline of the experimental results is provided. It comprises of experiments performed on synthetic datasets, as well as experimental validation on custom-designed real-world datasets obtained by morphing the latent space of mixed-curvature autoencoders and latent graph inference. The results demonstrate that the Gromov-Hausdorff-informed graph search space can be leveraged to perform NLGS across a variety of tasks and datasets.

4.1 Synthetic Experiments on Product Manifold Inference

In order to evaluate the effectiveness of the proposed graph search space, we conduct a series of tests using synthetic data. We wish to present a setting for which the latent optimal product manifold is known by construction. To do so, we start by generating a random vector x and mapping it to a 'ground truth' product manifold P T , which we choose arbitrarily, using the corresponding exponential map. The resulting projected vector is then decoded using a neural network with frozen weights, f θ , to obtain a reference signal y P T , which we wish to recover using NLGS. To generate the rest of the dataset, we then consider the same random vector but map it to a number of other product manifolds t P i u i P n P with the same number of model spaces as P T but different signatures. Decoding the projected vector through the same neural network yields a set of signal, t y P i u i P n P . We use the aforementioned signals, to set the value associated with each node of the graph search space to be MSE p y P T , y P i q . In this way, our search algorithm should aim to find the node in the graph that minimizes the error and hence find the latent geometry that recovers the original reference signal.

Our method consists in using BO over our Gromov-Hausdorff-informed search space. We compare it against random search, and what we call 'Naive BO', which performs BO over a fully-connected graph which disregards the Gromov-Hausdorff distance between candidate latent geometries. The figures presented, namely Figures 4 and 5, display the results. Each plot illustrates the performance of the algorithms and baselines as they select different optimal latent product manifolds denoted as P T . Notably, the figures demonstrate that the algorithm utilizing the Gromov-Hausdorff-informed search space surpasses all other baselines under consideration and consistently achieves the true

function minimum. Figure 4 generally requires fewer iterations compared to Figure 5 to attain the global minimum, primarily due to the smaller size of the search space. It is important to note that our benchmark solely involves the same algorithm but with a change in the search space to a topologically uninformative one. This modification allows us to evaluate the impact of incorporating this information into the optimization process. We have not considered other benchmarks such as evolutionary algorithms since they always rely on a notion of distance between sampled points. Furthermore, as there are no existing algorithms to compute the distance between arbitrary product manifolds, we have not included these methods in our benchmark as they would simply converge to random search. All results are shown on a log scale, and we apply a small offset of ε ' 10 ´ 3 to the plots to avoid computing log p 0 q when the reference signal is found.

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

GH BO

Naive BO

Random

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

GH BO

Naive BO

Random

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

Minimum

4.2 Experiments on Real-World Datasets

To further validate our method, we conduct additional tests using image and graph datasets. Specifically, we focus on image reconstruction and node classification tasks.

Image Reconstruction with Autoencoder

We consider four well-known image datasets: MNIST [Deng, 2012], CIFAR-10 [Krizhevsky et al., 2014], Fashion MNIST [Xiao et al., 2017] and eMNIST [Cohen et al., 2017]. Some of these datasets have been shown to benefit from additional topological priors, see Khrulkov et al. [2020] and Moor et al. [2020]. We use an autoencoder, detailed more thoroughly in Appendix D, to encode the image in a low dimensional latent space and then reconstruct it based on its latent representation. To test the performance of different latent geometries, we project the latent vector onto the product manifold being evaluated and use the reconstruction loss at the end of training as the reference signal to use in the graph search space. We consider a search space size of n p ' 7 for MNIST and n p ' 8 for CIFAR-10, Fashion MNIST and eMNIST. The results are displayed in Figure 6. In line with previous experiments, the Gromov-Hausdorff-informed search graph enables us to find better solutions in a smaller amount of evaluations.

0

10

20

30

40

50

# Iterations

-6

-5

-4

-3

-2

Minimum

MNIST

GH BO

Naive BO

Random

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

CIFAR-10

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

fMNIST

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

eMNIST

Latent Graph Inference

Finally, we consider searching for the optimal product manifold for a graph neural network node classification task using latent graph inference. In particular, we build upon previous work by Sáez de Ocáriz Borde et al. [2023b] which used product manifolds to produce richer embeddings spaces for latent graph inference. We consider the Cora [Sen et al., 2008] and Citeseer [Giles et al., 1998] citation network datasets, and a search space consisting of product manifolds of up to seven model spaces n p ' 7 . The aim is to find the latent space which gives the best results for the node classification problem using minimal query evaluations. Performing BO alongside the Gromov-Hausdorff informed search space gives a clear competitive advantage over Naive BO and random search in the case of Cora. For Citeseer, the experiments do not give such a clear-cut improvement, which can be attributed to the lack of smoothness of the signal on the graph search space and its incompatibility with the intrinsic limitations of the diffusion kernel.

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

-2

-1

Minimum

GH BO

Naive BO

Random

0

10

20

30

40

50

# Iterations

-7

-6

-5

-4

-3

Minimum

CiteSeer

5 Conclusion

In this work, we have introduced neural latent geometry search (NLGS) , a novel problem formulation that consists in finding the optimal latent space geometry of machine learning algorithms using minimal query evaluations. In particular, we have modeled the latent space using product manifolds based on Cartesian products of constant curvature model spaces. To find the optimal product manifold, we propose using Bayesian Optimization over a graph search space. The graph is constructed based on a principled measure of similarity, utilizing the Gromov-Hausdorff distance from metric geometry. The effectiveness of the proposed method is demonstrated through our experiments conducted on a variety of tasks, based on custom-designed synthetic and real-world datasets.

Limitations and Future Work. While the NLGS framework is general, we have restricted ourselves to using product manifolds of constant curvature model spaces to model the geometry of the latent space. Furthermore, we have only considered curvatures t´ 1 , 0 , 1 u , and model spaces of dimension two in order for the optimization problem to be tractable. In future research, there is potential to explore alternative approaches for modeling the latent space manifold. Additionally, the field of Bayesian Optimization over graphs is still in its early stages. Incorporating recent advancements in the area of kernels on graphs [Borovitskiy et al., 2021, Zhi et al., 2023] could lead to improved performance. In our current research, our emphasis is on introducing NLGS and providing an initial solution under a set of simplifying assumptions related to the potential latent manifolds available and the optimization algorithm. The investigation of the impact of using different similarity measures to compare latent structures also remains a subject for future research.

Societal Impact Statement

This work is unlikely to result in any harmful societal repercussions. Its primary potential lies in its ability to enhance and advance existing data modelling and machine learning methods.

References

Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. arXiv preprint arXiv:1710.11379 , 2017.

Federico Barbero, Cristian Bodnar, Haitz Sáez de Ocáriz Borde, Michael Bronstein, Petar Veliˇ ckovi´ c, and Pietro Liò. Sheaf neural networks with connection laplacians. In Topological, Algebraic and Geometric Learning Workshops 2022 , pages 28-36. PMLR, 2022a.

Federico Barbero, Cristian Bodnar, Haitz Sáez de Ocáriz Borde, and Pietro Liò. Sheaf attention networks. In NeurIPS 2022 Workshop on Symmetry and Geometry in Neural Representations , 2022b.

Danilo Blanuša. Über die Einbettung hyperbolischer Räume in euklidische Räume. Monatsh. Math. , 59:217-229, 1955.

Viacheslav Borovitskiy, Iskander Azangulov, Alexander Terenin, Peter Mostowsky, Marc Deisenroth, and Nicolas Durrande. Matérn gaussian processes on graphs. In International Conference on Artificial Intelligence and Statistics , pages 2593-2601. PMLR, 2021.

Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and A. Doucet. Riemannian score-based generative modeling. ArXiv , abs/2202.02763, 2022.

David Brander. Isometric embeddings between space forms. 2003. Master thesis.

Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021.

Benjamin Chamberlain, James Rowbottom, Davide Eynard, Francesco Di Giovanni, Xiaowen Dong, and Michael Bronstein. Beltrami flow and neural diffusion on graphs. Advances in Neural Information Processing Systems , 34:1594-1609, 2021.

Ines Chami, Rex Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks, 2019.

Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN) , pages 2921-2926. IEEE, 2017.

Giacomo Como, Raffaele Damiano, and Fabio Fagnani. Discrete bayesian optimization algorithms and applications. 2020.

Jiaxu Cui and Bo Yang. Graph bayesian optimization: Algorithms, evaluations and applications. ArXiv , abs/1805.01157, 2018.

Jiaxu Cui, Bo Yang, Bingyi Sun, Xia Ben Hu, and Jiming Liu. Scalable and parallel deep bayesian optimization on attributed graphs. IEEE Transactions on Neural Networks and Learning Systems , 33:103-116, 2020.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Yilun Du, Katherine Collins, Joshua B. Tenenbaum, and Vincent Sitzmann. Learning signal-agnostic manifolds of neural fields. In Neural Information Processing Systems , 2021.

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. J. Mach. Learn. Res. , 20:55:1-55:21, 2018.

Charles Fefferman, Sanjoy K. Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. arXiv: Statistics Theory , 2013.

Marco Fumero, Luca Di Cosmo, Simone Melzi, and Emanuele Rodolà. Learning disentangled representations via product manifold projection. ArXiv , abs/2103.01638, 2021.

C Lee Giles, Kurt D Bollacker, and Steve Lawrence. Citeseer: An automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries , pages 89-98, 1998.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS , 2014.

Dhananjay Gopal, Aniruddha V. Deshmukh, Abhay S. Ranadive, and Shubham Yadav. An introduction to metric spaces. 2020.

Albert Gu, Frederic Sala, Beliz Gunel, and Christopher Ré. Learning mixed-curvature representations in product spaces. In International Conference on Learning Representations , 2018.

Søren Hauberg, Oren Freifeld, and Michael Black. A geometric take on metric learning. Advances in Neural Information Processing Systems , 25, 2012.

Wolfgang Henke and Wolfgang Nettekoven. The hyperbolic n -space as a graph in Euclidean p 6 n ´ 6 q -space. Manuscripta Math. , 59(1):13-20, 1987.

Felix Hensel, Michael Moor, and Bastian Alexander Rieck. A survey of topological machine learning methods. Frontiers in Artificial Intelligence , 4, 2021.

Chin-Wei Huang, Milad Aghajohari, A. Bose, P. Panangaden, and Aaron C. Courville. Riemannian diffusion models. ArXiv , abs/2208.07949, 2022.

Paul Jungeblut, Linda Kleist, and Tillmann Miltzow. The complexity of the hausdorff distance. In International Symposium on Computational Geometry , 2021.

Anees Kazi, Luca Cosmo, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael Bronstein. Differentiable graph module (DGM) for graph convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-1, 2022.

Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6418-6428, 2020.

Risi Kondor and Jean-Philippe Vert. Diffusion kernels. kernel methods in computational biology , pages 171-192, 2004.

Wouter Kool, Herke van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement, 2019.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar. html , 55(5), 2014.

Shane Lubold, Arun G Chandrasekhar, and Tyler H McCormick. Identifying the latent space geometry of network models through analysis of curvature. Journal of the Royal Statistical Society Series B: Statistical Methodology , 85(2):240-292, 2023.

Lizheng Ma, Jiaxu Cui, and Bo Yang. Deep neural architecture search with deep graph bayesian optimization. 2019 IEEE/WIC/ACM International Conference on Web Intelligence (WI) , pages 500-507, 2019.

Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh. Continuous hierarchical representations with poincaré variational auto-encoders. Advances in neural information processing systems , 32, 2019.

Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of bayesian methods for seeking the extremum. Towards global optimization , 2(117-129):2, 1978.

Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International conference on machine learning , pages 7045-7054. PMLR, 2020.

Changyong Oh, Jakub Tomczak, Efstratios Gavves, and Max Welling. Combinatorial bayesian optimization using the graph cartesian product. Advances in Neural Information Processing Systems , 32, 2019.

David Pfau, Irina Higgins, Aleksandar Botev, and Sébastien Racanière. Disentangling by subspace diffusion. ArXiv , abs/2006.12982, 2020.

Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. ArXiv , abs/1802.03268, 2018.

Sam T. Roweis and Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science , 2000.

Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborne. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels. arXiv preprint arXiv:2006.07556 , 2020.

Haitz Sáez de Ocáriz Borde, Alvaro Arroyo, and Ingmar Posner. Projections of model spaces for latent graph inference. In ICLR 2023 Workshop on Physics for Machine Learning , 2023a.

Haitz Sáez de Ocáriz Borde, Anees Kazi, Federico Barbero, and Pietro Lio. Latent graph inference using product manifolds. The Eleventh International Conference on Learning Representations , 2023b.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Ankita Shukla, Shagun Uppal, Sarthak Bhagat, Saket Anand, and Pavan K. Turaga. Geometry of deep generative models for disentangled representations. Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing , 2018.

Ondrej Skopek, Octavian-Eugen Ganea, and Gary Bécigneul. Mixed-curvature variational autoencoders. arXiv preprint arXiv:1911.08411 , 2019.

Alexander J Smola and Risi Kondor. Kernels and regularization on graphs. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings , pages 144-158. Springer, 2003.

Abdel Aziz Taha and Allan Hanbury. An efficient algorithm for calculating the exact hausdorff distance. IEEE Transactions on Pattern Analysis and Machine Intelligence , 37:2153-2163, 2015.

Joshua B. Tenenbaum, Vin De Silva, and John C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science , 2000.

Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. ArXiv , abs/2111.14522, 2021.

Wan, H. Kenlay, B. Ru, A. Blaas, M. A. Osborne, and X. Dong. Adversarial attacks on graph classifiers via bayesian optimisation. Conference on Neural Information Processing Systems , 2021.

Melanie Weber. Curvature and representation learning: Identifying embedding spaces for relational data. 2019.

Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning . MIT press Cambridge, MA, 2006.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Sharon Zhang, Amit Moscovich, and Amit Singer. Product manifold learning. In International Conference on Artificial Intelligence and Statistics , 2020.

Deli Zhao, Jiapeng Zhu, and Bo Zhang. Latent variables on spheres for autoencoders in high dimensions. arXiv: Learning , 2019.

Yin-Cong Zhi, Yin Cheng Ng, and Xiaowen Dong. Gaussian processes on graphs via spectral kernel learning. IEEE Transactions on Signal and Information Processing over Networks , 2023.

Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. ArXiv , abs/1611.01578, 2016.

A Additional Background

A.1 Differential Geometry, Riemmanian Manifolds, and Product Manifolds

Differential geometry is a mathematical discipline that employs rigorous mathematical techniques, including calculus and other mathematical tools, to investigate the properties and structure of smooth manifolds. These manifolds, despite bearing a local resemblance to Euclidean space, may exhibit global topological features that distinguish them from Euclidean space. The fundamental objective of differential geometry is to elucidate the geometric properties of these spaces, such as curvature, volume, and length, through rigorous mathematical analysis and inquiry.

A Riemannian manifold generalizes Euclidean space to more curved spaces, and it plays a critical role in differential geometry and general relativity. A Riemannian manifold is a smooth manifold M equipped with a Riemannian metric g , which assigns to each point p in M an inner product on the tangent space T p M . This inner product is a smoothly varying positive definite bilinear form on the tangent space given by g p : T p M ˆ T p M Ñ R , which satisfies the following properties: symmetry, g p p X,Y q ' g p p Y, X q for all X,Y P T p M ; linearity, g p p aX ` bY, Z q ' ag p p X,Z q ` bg p p Y, Z q for all X,Y,Z P T p M and scalars a, b P R ; positive definiteness: g p p X,X q ą 0 for all non-zero X P T p M . The metric g p induces a norm || . || p on T p M given by || X || p ' a g p p X,X q . This norm allows us to define the length of curves on the manifold, and hence a notion of distance between points. Specifically, given a curve c : r 0 , 1 s Ñ M and a partition 0 ' t 0 ă t 1 ă ... ă t n ' 1 , the length of the curve c over the partition is given by: L p c q ' ř || c 1 p t i q|| c p t i q , where c 1 p t i q denotes the tangent vector to c at time t i , and || ¨ || c p t i q denotes the norm induced by the metric at the point c p t i q . The total length of the curve is obtained by taking the limit of the sum as the partition becomes finer.

On the other hand, a product manifold results from taking the Cartesian product of two or more manifolds, resulting in a manifold with a natural product structure. This enables researchers to examine the local and global geometry of the product manifold by studying the geometry of its individual factors. Riemannian manifolds and product manifolds are essential mathematical concepts that have far-reaching applications in diverse fields such as physics, engineering, and computer science. In the appendix of this paper, we provide a more in-depth look at these concepts, including their properties and significance.

A.2 Constant Curvature Model Spaces

Amanifold M is a topological space that can be locally identified with Euclidean space using smooth maps. In Riemannian geometry, a Riemannian manifold or Riemannian space p M , g q is a real and differentiable manifold M where each tangent space has an associated inner product g known as the Riemannian metric. This metric varies smoothly from point to point on the manifold and allows for the definition of geometric properties such as lengths of curves, curvature, and angles. The Riemannian manifold M Ď R N is a collection of real vectors that is locally similar to a linear space and exists within the larger ambient space R N .

Curvature is effectively a measure of geodesic dispersion. When there is no curvature geodesics stay parallel, with negative curvature they diverge, and with positive curvature they converge. Constant curvature model spaces are Riemannian manifolds with a constant sectional curvature, which means that the curvature is constant in all directions in a 2D plane (this can be generalized to higher dimensions). These spaces include the Euclidean space, the hyperbolic space, and the sphere. The Euclidean space has zero curvature, while the hyperbolic space has negative curvature and the sphere has positive curvature. Constant curvature model spaces are often used as reference spaces for comparison in geometric analysis and machine learning on non-Euclidean data.

Euclidean space,

is a flat space with curvature K E ' 0 . We note that in this context, the notation d E is used to represent the dimensionality. In contrast, hyperbolic and spherical spaces possess negative and positive curvature, respectively. We define hyperboloids H d H K H as

and hyperspheres S d S K S as

where K H ă 0 and x¨ , ¨y L is the Lorentz inner product

where K S ą 0 and x¨ , ¨y 2 is the standard Euclidean inner product

Table 2 summarizes the key operators in Euclidean, hyperbolic, and spherical spaces with arbitrary curvatures. It is worth noting that the three model spaces discussed above can cover any curvature value within the range of p´8 , 8q . However, there is a potential issue with the hyperboloid and hypersphere becoming divergent as their respective curvatures approach zero. This results in the manifolds becoming flat, and their distance and metric tensors do not become Euclidean at zerocurvature points, which could hinder curvature learning. Therefore, stereographic projections are considered to be suitable alternatives to these spaces as they maintain a non-Euclidean structure and inherit many of the properties of the hyperboloid and hypersphere.

A.3 Constructing Product Manifolds

In this appendix, we provide additional details about the generation of product manifolds. In mathematics, a product manifold is a type of manifold obtained by taking the Cartesian product of two or more manifolds. Informally, a product manifold can be thought of as a space formed by taking multiple smaller spaces and merging them in a way that maintains their individual structures. For instance, the surface of a sphere can be seen as a product manifold consisting of two real lines that intersect at each point on the sphere. Product manifolds play a crucial role in various fields of mathematics, including physics, topology, and geometry.

A product manifold can be defined as the Cartesian product:

Here, K i and d i represent the curvature and dimensionality of the space, respectively. Points x p P P are expressed using their coordinates:

Also, the metric of the product manifold decomposes into the sum of the constituent metrics

hence, p P , g P q is also a Riemannian manifold. It should be noted that the signature of the product space is parametrized with several degrees of freedom. These include the number of components used in the product space, the type of model spaces employed, their dimensionality, and curvature. If we restrict P to be composed of the Euclidean plane E d E K E , hyperboloids H d H j K H j , and hyperspheres S d S k K S k of constant curvature, we can rewrite Equation 13 as

where K E ' 0 , K H j ă 0 , and K S k ą 0 . Hence, P would have a total of 1 ` n H ` n S component spaces, and total dimension d E ` Σ n H j ' 1 d H j ` Σ n S k ' 1 d S k . Distances between two points in the product manifold can be computed aggregating the distance contributions d i from each manifold composing the product manifold:

The overline denotes that the points x p 1 and x p 2 have been adequately projected on to the product manifold using the exponential map before computing the distances.

A.4 The Rationale Behind using Product Manifolds of Model Spaces

Most arbitrary Riemannian manifolds do not have closed-form solutions for their exponential maps and geodesic distances between points, as well as for other relevant mathematical notions. The exponential map takes a point on the manifold and exponentiates a tangent vector at that point to produce another point on the manifold. The geodesic distance between two points on a manifold is the shortest path along the manifold connecting those points.

For simple, well-studied manifolds like Euclidean space, hyperbolic space, and spherical space, closed-form solutions for exponential maps and geodesic distances are available. This is because these manifolds have constant curvature, which allows for more straightforward calculations. However, for most other manifolds, the exponential map and geodesic distances must typically be computed numerically or approximated using specialized algorithms. These computations often involve solving differential equations, and depending on the manifold's curvature and geometry, these solutions can be quite complex and may not have closed-form expressions. For certain types of manifolds, like product manifolds where each factor is a well-understood manifold, the computation of exponential maps and geodesic distances can sometimes be simplified due to the separability of the metric. However, scaling these methods to more general, complex manifolds can be challenging and computationally intensive.

As mentioned, closed-form solutions for exponential maps and geodesic distances are not common for arbitrary Riemannian manifolds, and computational methods often rely on numerical techniques and algorithms due to the complexity of these calculations. These properties have made product manifolds the most attractive tool to endow the latent space with richer structure while retaining computational traceability, see [Gu et al., 2018, Skopek et al., 2019, Sáez de Ocáriz Borde et al., 2023b,a, Zhang et al., 2020, Fumero et al., 2021, Pfau et al., 2020]. This motivates us to use product manifolds in our setup, which keeps our work in line with the state-of-the-art approaches in the literature. We highlight that one of the central contributions of our paper is to shed light on the problem of NLGS, and provide an initial solution to the problem in the context of the current state-of-the-art in geometric

latent space modelling. Future research avenues could explore generalizations of this which account for new techniques that emerge in the literature, potentially comparing a more general set of latent manifolds.

A.5 Geometry of Negative Curvature: Further Discussion

Lastly, in this appendix, we aim to provide an intuition regarding closed balls in spaces of curvature smaller than zero. The hyperbolic spaces we consider in this article are hyperbolic manifolds (that is, hyperbolic in the differentiable sense). However, there are other generalisations of the notion of hyperbolicity that carry over to the setting of geodesic metric spaces (such as Gromov hyperbolicity). This theory unifies the two extremes of spaces of negative curvature, the real hyperbolic spaces H n and trees. Recall that a tree is a simple graph that does not have closed paths. Trees provide a great intuition for the metric phenomena that we should expect in spaces of negative curvature. For example, let us denote by G an infinite planar grid (observe that this graph is far from being a tree), and let us denote by T a regular tree of valency 4 (this means that every vertex in T is connected to exactly other four vertices). The 2-dimensional Euclidean space E 2 is quasi-isometric to G and the 2-dimensional real hyperbolic space H 2 is quasi-isometric (but not isometric) to T . This observation is due to Švarc and Milnor. Recall that the graphs G and T are endowed with the natural path metric. A ball of radius n ą 0 in G has between n 2 and 4 n 2 points (polynomial in n ), while a ball of radius n in T has exactly p 4 n ´ 1 q{ 3 points (exponential in n ). Now let us come back to H 3 , where the 'continuous' analogue of this principle is also true; that is, that there exists a constant c ą 1 such that for every n ą 0 , one needs at least c n balls of radius one to cover a ball of radius n in H 3 . If we only want to understand the local geometry of H 3 , we can re-scale this principle to say the following. Let B H 3 1 denote the ball of radius one in H 3 . For all positive integers n ą 0 , one needs at least c n balls of radius 1 { n to cover B H 3 1 . This tree-like behaviour should be compared with the opposite principle that holds in E 3 , being that for all n ą 0 , one needs at most 8 n 3 (a polynomial bound) of balls of radius one to cover B E 3 1 . Essentially, we should think of B H 3 1 as a manifold that is isomorphic to B E 3 1 which is difficult to cover by balls of smaller radius, in fact, it will be coarsely approximated by a finite tree of balls of small radius (the underlying tree being bigger as the chosen radius is smaller). Furthermore, the philosophy that travelling around B H 3 1 becomes more and more expensive as we leave the origin is made accurate with the consideration of divergence functions: geodesics in H 3 witness exponential divergence. More precisely, there exists a constant c ą 1 such that for every r ą 0 and for every geodesic γ : R ÝÑ H 3 passing through the origin at time t ' 0 it is true that if α is a path that connects γ p´ r q and γ p r q and that lies entirely outside of the ball of radius d p 0 , γ p r qq , then α must have length at least c r . Again, this is better appreciated when put in comparison with the Euclidean space E 3 . Any geodesic (any line) γ : R ÝÑ E 3 passing through the origin at time t ' 0 will have the property that for all r ą 0 there is a path of length πr ` ε , with ε ą 0 arbitrarily small, that lies entirely outside of the ball of radius r with centre at the origin. The length πr ` ε is linear in r , as opposed to the exponential length that is required in hyperbolic spaces.

B Further Details on the Gromov-Hausdorff Algorithm for Neural Latent Geometry Search

As discussed in Section 3, there are limitations to the Hausdorff distance as a metric for comparing point sets that represent discretized versions of continuous manifolds. The original algorithm by Taha and Hanbury [2015] for computing the Hausdorff distance assumes that the point sets reside in the same space with the same dimensionality, which is a limiting requirement. However, the GromovHausdorff distance proposed in this work allows us to compare metric spaces that are not embedded in a common ambient space, but it is not exactly computable. We will focus on providing an upper bound for d GH p E n , S n q and describing an algorithm for obtaining upper bounds for d GH p E n , H n q and d GH p S n , H n q too. As previously mentioned, E n and S n can be isometrically embedded into E 6 n ´ 6 in many ways, which provides a common underlying space for computing Hausdorff distances between E n , S n , and H n . The embeddings of different spaces require making choices: in particular, there is no way to exactly compute the infimum in the definition of Gromov-Hausdorff distance given in Section 2:

and

it will be easy to generate collections of points in B E 2 and B S 2 . As we anticipated above in the outline of our strategy to estimate d GH p B E 2 , B H 2 q , in order to give explicit well-distributed collection of points in B H 2 , it is enough to give a well-distributed collection of points in B E 2 and consider its image under the exponential map exp 0 : B E 2 Ñ B H 2 , where we have identified B E 2 with the ball of radius one of R 2 -T 0 H 2 , i.e. the tangent space of H 2 at the point 0 . However, we should remark that our description of H n will not be any of the three most standard ones: namely the hyperboloid model , the the Poincaré ball model , or the upper half-plane model . We describe H n as the differentiable manifold R n (of Cartesian coordinates x, y 1 , . . . , y n ´ 1 ) equipped with the Riemannian metric g ´ 1 ' d x 2 ` e 2 x p d y 2 1 `¨¨¨ d y 2 n ´ 1 q . This metric is complete and has constant sectional curvature of -1, which implies that p R n , g ´ 1 q is isometric to H n . In the hyperboloid model of H 2 , we view this space as a submanifold of R 3 (although with a distorted metric, not the one induced from the ambient R 3 ). In this model of H 2 , one can explicitly describe by the following assignment exp 0 : R 2 Ñ H 2 by

We use this explicit formula of the exponential map with coordinates in the hyperboloid model of H 2 , to give a explicit formula for the exponential map with coordinates in the model of H 2 that we introduced above, denoted by p R 2 , g ´ 1 q . In order to change coordinates from the hyperboloid model to the Poincaré ball model, we use the following isometry (which is well-known and can be thought of as a hyperbolic version of the stereographic projection from S 2 to R 2 ):

To change coordinates from the Poincaré ball model to the upper half-plane model, one can use the following isometry:

so we resort to numerical approximations which are later described in this appendix.

To estimate the mutual Gromov-Hausdorff distance between the infinite smooth spaces E 2 and H 2 , the first step is to approximate them using finite discrete versions. This is done by considering well-distributed collections of points in E 2 and H 2 , obtained by applying the exponential map to a collection of points in R 2 . Multiple isometric embeddings of E 2 into R 6 are also considered. The estimation is obtained by computing the minimum of the Hausdorff distance between the point sets obtained from the collections in R 6 and the isometric embedding of H 2 into R 6 . The same applies for spherical space.

B.1 Generating Points in the Corresponding Balls of Radius One

All our computations will be done in dimension two, so we will simply precise how to generate points in B E 2 , B S 2 and B H 2 . For B E 2 and B S 2 , we will use very elementary trigonometry. For B H 2 , we will need an explicit description of the exponential map of H 2 . Using the descriptions

The previous assignment comes from the standard Möbius transformation z ÞÑ z ` i iz ` 1 that identifies the Euclidean ball of radius one of C and the upper half plane of C as Riemann surfaces (which we give explicitly in this case, although it is known to exist and to be unique by the classical Riemann mapping theorem). Finally, to go from coordinates in the upper half-plane model of H 2 to our model p R 2 , g ´ 1 q , we use the isometry

B.2 Embedding of H n into E 6 n ´ 6

We want to define an isometric embedding of B 1 H n into E 6 n ´ 6 ( F from Section 3.2). This higher dimensional space is our candidate to fit in the three geometries E n , H n and S n to compute their Hausdorff dimensions as subspaces of E 6 n ´ 6 and hence estimate their Gromov-Hausdorff distances. Before describing such embedding, we introduce several preliminary auxiliary functions.

Let χ p t q ' sin p πt q ¨ e ´ sin ´ 2 p πt q for non-integer values of t . A priori, the inverse of sin p 0 q ' 0 does not make sense but since lim t Ñ 0 ` χ p t q ' lim t Ñ 0 ´ χ p t q ' 0 , we can set χ p 0 q ' 0 so it is still continuous. In fact, it is smooth and, in particular, integrable. We can say the same at all points when t is an integer, so we set χ p t q ' 0 for all integers t and we obtain an smooth function χ defined on R .

We also define and

We set c to be the constant defined in terms of the following

We also define f 0 p x, y q ' ´ ş sinh ´ 1 p ye x q 0 a 1 ´ ε p t q 2 d t, log p a e ´ 2 x ` y 2 q , h p ψ p x, y qq ¯ , with ε being

This way, we can set

Recall that in our case, we use the function F p¨q ' f p n ' 2 , ¨q to map the set of points Q in H 2 to R 6 , and for computing

B.3 Estimation of the Gromov-Hausdorff distance between the Euclidean and Spherical Spaces

In this section we give the analytical derivation of the Gromov-Hausdorff distance between the Euclidean space E n and the Spherical space S n . We first explain the case n ' 1 , where we can visually understand the situation in a better way because all distances will be measured in E 2 . Afterwards, we replicate the same argument for arbitrary n . Recall that the Spherical model S n can be described as the metric subspace of E n ` 1 corresponding to the Euclidean sphere of radius one. Hence, as a subspace of R n ` 1 , it corresponds to tp x 0 , . . . , x n q : ř n i ' 0 x 2 i ' 1 u .

-2

-1

1

2

0

-1

1

The ball of radius one of S 1 , denoted by B S 1 , is highlighted in red. Our estimation of the GromovHausdorff distance between E 1 and S 1 is motivated by the following observation. In the y -axis, the red arc ranges from 1 ` cos p 1 q 2 to 1. If we consider the following blue segment, representing the ball of

-1

-0.5

0.5

1

1.5

0

0.5

1

radius one of E 1 (denoted by B E 1 ), we get

However, it can be seen that sup x P B E 1 d E 2 p x, B S 1 q ' 0 . 279 . More generally, if the blue line is chosen to be at the height 1 ´ x , for some x lying in the closed interval r 0 , 1 ´ cos p 1 qs , it is not hard to see that, for such embedding f : B E 1 Ñ E 2 ,

This can be seen in the following picture. The two pink lines represent the biggest lengths between

-1

-0.5

0.5

1

1.5

0

0.5

1

points in B E 1 and B S 1 , whose length is approximately equal to 0.257. Since we cannot compute exactly the Gromov-Hausdorff distance d GH p S 1 , E 1 q , choices have to be made. We have discussed why we would expect it to be somewhere in between 0 . 23 and 0 . 257 . Since we are considering a very simple embedding for these estimations, it is reasonable to expect d GH p S 1 , E 1 q to get closer to the lowest number when considering embeddings of these spaces in more complicated higher dimensional spaces, as the definition of the Gromov-Hausdorff distance allows.

For an arbitrary n , we reduce the estimation of d GH p B E n , B S n q to the one-dimensional case as follows. Analogously as we did for n ' 1 , given any value of x , we can consider B E n isometrically embedded in E n ` 1 by the map f x : B E n Ñ R n ` 1 defined by f pp x 0 , x 1 , . . . , x n qq ' p x 0 , x 1 , . . . , x n , x q . We view B S n isometrically embedded into R n ` 1 as the ball of radius one of S n Ă R n ` 1 with centre in the north pole of the sphere, i.e. the point with coordinates p 0 , 0 , . . . , 0 , 1 q . Given any unit vector u in R n ` 1 orthogonal to ⃗ n ' p 0 , 0 , . . . , 0 , 1 q , we define π u to be the two-dimensional plane linearly spanned by u and ⃗ n . It is clear that, as u ranges over the orthogonal space of p 0 , 0 , . . . , 0 , 1 q , the intersections B S n X π u cover the whole B S n and the intersections f x p B E n q X π u cover the whole f x p B E n q . Hence, in order to compute d E n ` 1 H p f x p B E n q , B S n q , it suffices to compute d E n ` 1 H p f x p B E n q X π u , B S n X π u q for all unit vectors u orthogonal to ⃗ n . Moreover, for two such unit vectors u and v , there is a rigid motion (a rotation) of the whole ambient space R n ` 1 , that fixes ⃗ n and that maps isometrically f x p B E n q X π u to f x p B E n q X π v and B S n X π u to B S n X π v . In particular,

So we can do the previous computation for the specific value of u ' p 0 , 0 , . . . , 1 , 0 q (for clarity, where we mean the unit vector where the n -th coordinate is equal to 1 and the rest are zero). Crucially, the projection of R n ` 1 unto R 2 (by projecting onto the last two coordinates) restrict to isometries on B S n X π u and B E n X π u , from where it is deduced, analogously as in 36, that, as long as x P r 0 , 1 ´ cos p 1 qs , we have the following:

B.4 Computational Implementation of the Gromov-Hausdorff Distance between the Remaining Model Spaces of Constant Curvature

In this appendix, we provide further details regarding how the Gromov-Hausdorff distances between the remaining manifolds (between B E 2 and B H 2 , and B S 2 and B H 2 ) were approximated computationally. Note that as discussed in Section 3.3, these distances will be leveraged to compare not only model spaces but product manifolds as well.

B.4.1 Discretizing the Original Continuous Manifolds

This section discuss how the the points in the corresponding balls of radius one described in Appendix B.1 are generated from a practical perspective. The Euclidean, hyperbolic, and spherical

spaces are continuous manifolds. To proximate the Gromov-Hausdorff distance between them we must discretize the spaces. A more fine-grained discretization results in a better approximation. As previously mentioned in Section 2, to compare E n , S n , and H n , we can take closed balls of radius one in each space. Since these spaces are homogeneous Riemannian manifolds, every ball of radius one is isometric. We can estimate or provide an upper bound for their Gromov-Hausdorff distance by using this method.

In the case of the Euclidean plane, we sample points from the ball, B E 2 ' tp r cos p t q , r sin p t qq : r P r 0 , 1 s , t P r 0 , 2 π qu , with a discretization of 10 , 000 points in both r and t . To obtain points in B H 2 we will use the points in B E 2 as a reference, and apply the exponential map, and the change of coordinates described in Appendix B.1 to convert points in B E 2 to points in B H 2 . However, due to numerical instabilities instead of sampling from B E 2 , we will restrict ourselves to B 1 E 2 ' tp r cos p t q , r sin p t qq : r P r 0 . 00000001 , 0 . 97 s , t P r 0 , 2 π qu and use a discretization of 10 , 000 as before. Lastly to sample points for the spherical space, B S 2 ' tp sin p β q cos p α q , sin p β q sin p α q , cos p β qq : α P r 0 , 2 π q , β P r 0 , 1 su , we use a discretization of 100 for both α and β . The granualirity of the discretization was chosen as a trade-off between resolution and computational time. We observed that the GromovHausdorff distance stabilized for our discretization. However, given the nature of the GromovHausdorff distance, it is difficult to conclude whether some unexpected behaviour could be observed with greater discretization.

B.4.2 Calculating Constants for the Embedding Function

The next step is to obtain a computational approximation of the mapping function defined in Appendix B.2. The embedding of H n into E 6 n ´ 6 requires computing several constants. Using a standard integral solvers A « 0 . 141328 , (Equation 25) can be approximated. To calculate the constant c « 10 . 255014502464228 , we discretize the input x P r´ 2 , 2 s using a step size of 10 ´ 8 to compute G 1 and G 2 and use a for loop to calculate the max in Equation 28. Note that this requires to constantly reevaluate the integrals for ψ 1 p x q and ψ 2 p x q . Likewise, G 1 and G 2 are also used to obtain ε in Equation 33, which is in turn used to calculate the mapping function that maps points in H 2 to the higher dimensional embedding Euclidean space E 6 . We use F to map B H 2 to E 6 , and we keep those points fixed in space.

B.4.3 Optimizing the Embedding Functions for the Euclidean and Spherical Spaces

and,

we must optimize f k and g k . These are the functions used to embed B E 2 and B S 2 in E 6 n ´ 6 . Note that in practice, B E 2 and B S 2 are discretized into P i H and P S i , respectively.

To optimize for f k we consider all possible permutations of the basis vectors of E 6 : t e 1 , e 2 , e 3 , e 4 , e 5 , e 6 u for each f k we consider two elements t e i , e j u and use those dimensions to embed E 2 in E 6 . In principle, we should also consider a small offset given by F p 0 q ' 0 , but it is zero regardless. Additionally, during the optimization we also add a small vector (with all entries but a single dimension between zero) to the mapping function to translate the plane in different directions by an offset of between ´ 0 . 5 and 0 . 5 , with a total of a 100 steps between these two quantities.

To optimize for g k we follow a similar procedure in which we consider all permutations of the basis vectors. Note however, that in this case we would have three basis vectors instead of two, given how B S 2 ' tp sin p β q cos p α q , sin p β q sin p α q , cos p β qq : α P r 0 , 2 π q , β P r 0 , 1 su is sampled. For each permutation family, P S i , we also consider its negative counterpart, ´ P S i , to compute the Gromov-Hausdorff distance as well as experimenting with offsetting the mapping function.

B.5 Derivation of Number of Product Manifold Combinations in the Search Space

Here, we derive the exact number of nodes in the graph search space in our setting. In particular, we consider the case with three model spaces (the Eucledian plane, the hyperboloid and the hypershere). The growth of the search space can be modelled with growth of a tree, as depicted in Figure 11.

E

n

H

n

S

n

E

n

Ś

E

n

E

n

Ś

H

n

E

n

Ś

S

n

H

n

Ś

H

n

H

n

Ś

S

n

S

n

Ś

S

n

To calculate the number of elements in the search space, we define the total number of products at level h of the tree as N . We have that N p h q ' N E p h q ` N H p h q ` N S p h q , where N E p h q is the number of Euclidean spaces added to the product manifolds at depth h of the tree, and N H p h q and N S p h q represent the same for the hyperboloid and the hypersphere respectively. By recursion, we can write

To be consistent with the previous notation, we write the above in terms of the number of products n p

The total number of nodes in the graph search space for a number of product n p is then

B.6 Visualizing the Graph Search Space

In this appendix, we give a visual depiction of the graph search space we construct for neural latent geometry search. Figure 12 and Figure 13 provide plots of the graph search space as the size of the product manifolds considered increases. For product manifolds composed of a high number of model spaces, visualizing the search space becomes difficult. We omit the strengths of the connections for visual clarity in this plots.

hence we have that

If we focus on Figure 12, we use e , h , and s to refer to the model spaces of constant curvature E 2 , H 2 , and S 2 . In this work, we have considered model spaces of dimensionality two, but the graph search space would be analogous if we were to change the dimensionality of the model spaces. Nodes that include more than one letter, such as hh , ss , ee , etc, refer to product manifolds. For example, hh would correspond to the product manifold H 2 ˆ H 2 . As discussed in Section 3.3, we can see that connections are only allowed between product manifolds that differ by a single model space. To try to clarify this further, we can see that e , h and s are all interconnected since they only differ by a single model space (one deletion and one addition). Likewise, e is connected to ee , eh , and ee since they only differ by a single model space (one addition). However, e is not connected to hs (one deletion and two additions) nor is ee connected to ss (two deletions and two additions).

h

e

s

hh

eh

ss

ee

es

hs

s

e

h

es

eh

hs

ss

ee

hh

hhs

ehs

ehh

hss

eeh

ees

ess

sss

hhh

eee

e

h

s

ee

eh

hh

hs

es

ss

ehs

ees

ess

hhs

ehh

eeh

sss

hss

eee

hhh

ehss

ehhs

hhhs

eehs

hhss

hsss

eees

eeeh

eess

ehhh

esss

eeee

ssss

hhhh

s

h

e

hs

es

ee

eh

hh

ss

eeh

hhs

ees

ehh

hss

ehs

ess

sss

hhh

eee

eehs

ehhs

eeeh

hhhh

eess

ehss

ssss

hhhs

hhss

eees

esss

ehhh

hsss

eeee

ehhss

eeess

eehss

hhsss

hhhhs

eeees

ehsss

ehhhh

ehhhs

hssss essss

hhhss

hhhhh

eesss

eehhh

eeeeh

eeehs

eeehh

sssss

eeeee

B.7 Motivating Gromov-Hausdorff Distances for Comparing Latent Geometries

The use of Gromov-Hausdorff distances can be motivated from a theoretical and practical perspective. From a theoretical perspective, the Gromov-Hausdorff distance offers a way to gauge the similarity between metric spaces. This is achieved by casting their representation within a shared space, all the while maintaining the original pairwise distances. In the context of machine learning models, particularly those involving generative models (such as V AEs or GANs), the latent space assumes a pivotal role in shaping the characteristics of the generated data. This also holds true in the case of reconstruction tasks, such as those carried out by autoencoders. Through the application of a metric that factors in their inherent geometries, the aim is to capture the concept of resemblance in data generation, reconstruction, and other downstream tasks which will be heavily affected by the choice of latent geometry.

While traditional metrics like Euclidean or cosine distances might suffice for some cases, they do not always capture the complex and nonlinear relationships between data points in high-dimensional spaces. The Gromov-Hausdorff distance considers the overall structure and shape of these spaces, rather than just individual distances, which can lead to better generalization and robustness. This is especially relevant when dealing with complex data distributions or high-dimensional latent spaces. Moreover, another appealing aspect of Gromov-Hausdorff distance is its invariance to isometric transformations, such as rotations, translations, and reflections. This is a desirable property when comparing latent spaces, as the metric used should reflect the similarity in shape and structure, rather than being influenced by trivial transformations.

While the direct relationship between Gromov-Hausdorff distance and model performance might not be immediately obvious, we argue that if two models have similar latent space geometries, this suggests that they might capture similar underlying data structures, which could lead to similar performance on downstream tasks. This is known as an assumption of smoothness in the NAS literature, see Oh et al. [2019]. However, it is important to note that this relationship is not guaranteed, and the effectiveness of Gromov-Hausdorff distance as a proxy measure depends on the specific application and the nature of the models being compared.

B.8 Method Scalability

The proposed approach sidesteps scalability concerns by preventing the need to recalibrate GH coefficients, as long as one adheres to latent spaces derived from products of model spaces of dimension two. If a higher-dimensional latent space was needed, this could be solved by adding additional 'graph slices" with augmented dimensions (as depicted in Figure 2). These slices can be integrated with the lower-dimensional counterpart of the graph search space following the procedure described in the paper. The number of coefficients remains constant at 3, rendering the GromovHausdorff distances free from extra computation overhead when increasing the dimension.

Moreover, in alignment with the manifold hypothesis, data is expected to exist within a lowdimensional manifold. This discourages the exploration of higher dimensions for latent space modeling. It is important to note, nonetheless, that the paper has already delved into relatively large search spaces, which include the exploration of high-dimensional latent spaces.

C Background on Bayesian Optimization

Bayesian optimization (BO) is a query-based optimization framework for black-box functions that are expensive to evaluate. It builds a probabilistic model of the objective function using past queries to automate the selection of meta-parameters and minimize computational costs. BO seeks to find the global optimum of a black-box function f : X Ñ R by querying a point x n P X at each iteration n and obtaining the value y n ' f p x n q ` ε , where ε ' N p 0 , σ 2 q is the noise in the observation. To do so, BO employs a surrogate to model the function f p¨q being minimized given the input and output pairs D t ' tp x i , y i ' f p x i qu N i ' 1 , and selects the next point to query by maximizing an acquisition function . In our work, we use the Gaussian Process (GP)[Williams and Rasmussen, 2006] as the surrogate model and expected improvement (EI)[Mockus et al., 1978] as the acquisition function.

Preliminaries. Bayesian optimization (BO) is particularly beneficial in problems for which evaluation is costly, behave as a black box, and for which it is impossible to compute gradients with respect to the loss function. This is the case when tuning the hyperparameters of machine learning models. In our

case, we will use it to find the optimal product manifold signature. Effectively, Bayesian optimization allows us to automate the selection of critical meta-parameters while trying to minimize computational cost. This is done building a probabilistic proxy model for the objective using outcomes recorded in past experiments as training data. More formally, BO tries to find the global optimum of a black-box function f : X Ñ R . To do this, a point x n P X is queried at every iteration n and yields the value y n ' f p x n q ` ε , where ε ' N p 0 , σ 2 q , is a noisy observation of the function evaluation at that point. BO can be phrased using decision theory

where N is the maximum number of evaluations, c is the cost incurred by each evaluation, and the loss L is minimized by finding a lower f p x q . In general, Equation 50 is intractable and the closed-form optimum cannot be found, so heuristic approaches must be used. Instead of directly minimizing the loss function, BO employs a surrogate model to model the function f p¨q being minimized given the input and output pairs D t ' tp x i , y i ' f p x i qu N i ' 1 . Furthermore, in order to select the next point to query, an acquisition function is maximized. In this work, the surrogate model is chosen to be a Gaussian Process (GP) [Williams and Rasmussen, 2006] and expected improvement (EI) [Mockus et al., 1978] is used as the acquisition function.

Gaussian Processes. A GP is a collection of random variables such that every finite collection of those random variables has a multivariate normal distribution. A GP is fully specified by a mean function , µ p x q , and covariance function (or kernel ), denoted as k p x , x 1 q . The prior over the mean function is typically set to zero, and most of the complexity of the model hence stems from the kernel function. Given t iterations of the optimization algorithm, with an input x t ' r x 1 , . . . , x t s T and output y t ' r y 1 , . . . , y t s T , the posterior mean and variance are given by

and where we define r K 1: t s i,j ' k p x i , x j q is the p i, j q -th element of the Gram matrix.

Expected Improvement. EI is a widely used acquisition function for Bayesian optimization. EI improves the objective function through a greedy heuristic which chooses the point which provides the greatest expected improvement over the current best sample point. EI is calculated as

where and Φ p¨q denotes the CDF of a standard normal distribution.

D Experimental Setup

In this section we provide additional details for the implementation of the experimental setup, including how the datasets are generated, relevant theoretical background, and the hyperparameters used for BO over our discrete graph search space.

D.1 Synthetic Experiments

To assess the effectiveness of the proposed graph search space, we conduct a series of tests using synthetic data. Our objective is to establish a scenario in which we have control over the latent space and manually construct the optimal latent space product manifold. To achieve this, we initiated the process by generating a random vector, denoted as x , and mapping it to a predetermined "ground truth" product manifold P T . This mapping is performed using the exponential map for the product manifold P T , which can be derived based on the model spaces of constant curvature that constitute it. Subsequently, the resulting projected vector is decoded via a neural network with fixed weights, denoted as f θ , to yield a reference signal y P T . The rationale behind employing a frozen network is

to introduce a non-linear mapping from the latent space to the signal space. This approach aims to mimic the behavior of a trained network in a downstream task, while disregarding the specific task or model weights involved. By utilizing a frozen network, we can capture the essence of a non-linear mapping without relying on the exact details of the task or specific model weights. The primary goal is to recover this reference signal using the neural latent geometry search (NLGS) approach. To generate the remaining dataset, we employ the same random vector and mapped it to several other product manifolds, denoted as t P i u i P n P , comprising an equivalent number of model spaces as P T but with distinct signatures. Decoding the projected vector using the same neural network produces a set of signals, denoted as t y P i u i P n P . To populate the nodes of the graph search space, we assign the corresponding value of mean squared error (MSE) between y P T and y P i for each pair. In this manner, our search algorithm aims to identify the node within the graph that minimizes the error, effectively determining the latent geometry capable of recovering the original reference signal. A schematic of the proposed method and specifics on the construction of the network used to decode the signal are shown in Figure 18 and Table 3.

P

exp

x

p

p

x

q

x

f

p¨q

y

D.2 Autoencoders

Autoencoders are a type of neural network architecture that can be used for unsupervised learning tasks. They are composed of two main parts: an encoder and a decoder. The encoder takes an input and compresses it into a low-dimensional representation, while the decoder takes that representation and generates an output that tries to match the original input. The goal of an autoencoder is to learn a compressed representation of the input data that captures the most important features, and can be used for tasks such as image denoising, dimensionality reduction, and anomaly detection. They have been used in a wide range of applications, including natural language processing, computer vision, and audio analysis.

Component

Layers

Encoder

Conv2d (1, 20, 3) - BatchNorm2d - SiLU

Conv2d (20, 20, 3) - BatchNorm2d - SiLU

.

.

.

(9 repetitions)

.

.

.

Conv2d (20, 2, 3) - BatchNorm2d - SiLU

Flatten - Linear - SiLU

Decoder

Linear - SiLU - Unflatten

ConvTranspose2d (2, 20, 3) - BatchNorm2d - SiLU

ConvTranspose2d (20, 20, 3) - BatchNorm2d - SiLU

.

.

.

(9 repetitions)

.

.

.

ConvTranspose2d (20, 1, 3) - BatchNorm2d - Sigmoid

The autoencoder's objective is to learn a compressed representation (latent space) of the input data and use it to reconstruct the original input as accurately as possible. In our experiments, the encoder takes an input image and applies a series of convolutional layers with batch normalization and SiLU activation functions, reducing the image's dimensions while increasing the number of filters or feature maps. The final output of the encoder is a tensor of shape p batchsize , latentdim , 6 , 6 q or p batchsize , latentdim , 10 , 10 q , where latentdim is the desired size of the latent space. After flattening this tensor, a fully connected layer is used to map it to the desired latent space size. The encoder's output is a tensor of shape p batchsize , latentdim q , which represents the compressed representation of the input image. The decoder takes the latent space representation as input and applies a series of transpose convolutional layers with batch normalization and SiLU activation functions, gradually increasing the image's dimensions while decreasing the number of feature maps. The final output of the decoder is an image tensor of the same shape as the input image. The loss functions used for training are the MSELoss (mean squared error) and BCELoss (binary cross-entropy) from the PyTorch library. A summary of the autoencoder is provided in Table 4.

D.3 Latent Graph Inference

Graph Neural Networks (GNNs) leverage the connectivity structure of graph data to achieve state-ofthe-art performance in various applications. Most current GNN architectures assume a fixed topology of the input graph during training. Research has focused on improving diffusion using different types of GNN layers, but discovering an optimal graph topology that can help diffusion has only recently gained attention. In many real-world applications, data may only be accessible as a point cloud of data, making it challenging to access the underlying but unknown graph structure. The majority of Geometric Deep Learning research has relied on human annotators or simplistic pre-processing algorithms to generate the graph structure, and the correct graph may often be suboptimal for the task at hand, which may benefit from rewiring. Latent graph inference refers to the process of inferring the underlying graph structure of data when it is not explicitly available. In many real-world applications, data may only be represented as a point cloud, without any knowledge of the graph structure. However, this does not mean that the data is not intrinsically related, and its connectivity can be utilized to make more accurate predictions.

For these experiments we reproduce the architectures described in Sáez de Ocáriz Borde et al. [2023b], see Table 5 and Table 6 for the original architectures. In our case, we use the GCN-dDGM model leveraging the original input graph inductive bias.

D.3.1 Differentiable Graph Module

In their work, Kazi et al. [2022] presented a general method for learning the latent graph by leveraging the output features of each layer. They also introduced a technique to optimize the parameters responsible for generating the latent graph. The key concept is to use a similarity metric between the latent node features to generate optimal latent graphs for each layer l . In this context, X p 0 q and A p 0 q represent the original input node feature matrix and adjacency matrix, respectively. So that

and A p 0 q ' A if the adjacency matrix from the dataset, or A p 0 q ' I if G ' p V , Hq . The proposed architecture in Kazi et al. [2022] consists of two primary components: the Differentiable Graph Module (DGM) and the Diffusion Module. The Diffusion Module, X p l ` 1 q ' g ϕ p X p l q , A p l q q , may be one (or multiple) standard GNN layers. The first DGM module takes the original node features and connectivity information and produces an updated adjacency matrix

In principle, the DGM module utilizes information from previous layers to generate adjacency matrices at each layer

To do so, a measure of similarity is used

In summary, the proposed approach utilizes a parameterized function f Θ p l q with learnable parameters to transform node features and a similarity measure φ to compare them. The function can be an MLP or composed of GNN layers if connectivity information is available. The output of the similarity measure is used to create a fully-connected weighted adjacency matrix for the continuous

differentiable graph module (cDGM) approach or a sparse and discrete adjacency matrix for the discrete Differentiable Graph Module (dDGM) approach, with the latter being more computationally efficient and recommended by the authors of the DGM paper [Kazi et al., 2022]. To improve the similarity measure φ and construct better latent graphs, the approach employs product spaces and Riemannian geometry. Additionally, an extra loss term is used to update the learnable parameters of the dDGM module.

Lastly, we will examine the dDGM module, which utilizes the Gumbel Top-k [Kool et al., 2019] technique to generate a sparse k -degree graph by stochastically sampling edges from the probability matrix P p l q p X p l q ; Θ p l q , T q , which is a stochastic relaxation of the kNN rule, where each entry corresponds to

T being a learnable parameter. The primary similarity measure utilized in Kazi et al. [2022] involved computing the distance between the features of two nodes in the graph embedding space

where ∆ p¨ , ¨q denotes a generic measure of distance between two points. They assumed that the latent features laid in an Euclidean plane of constant curvature K E ' 0 , so that

where d E is the distance in Euclidean space. Based on

where q P R N is uniform i.i.d in the interval r 0 , 1 s , we can sample the edges

k being the number of sampled connections using the Gumbel Top-k trick. The Gumbel Top-k approach utilizes the categorical distribution p p l q ij Σ r p p l q ir for sampling, and the resulting unweighted adjacency matrix A p l q p X p l q ; Θ p l q , T, k q is used to represent E p X p l q ; Θ p l q , T, k q . It is worth noting that including noise in the edge sampling process can generate random edges in the latent graphs, which can serve as a form of regularization.

Finally, we can summarize a multi-layer GNN using the dDGM module as

Equations 64 and 65 belong to the dDGM module, while Equation 66 is associated with the Diffusion Module. In our study, we extend Equation 59 to measure distances without relying on the assumption utilized in Kazi et al. [2022], which restricts the analysis to fixed-curvature spaces, specifically to Euclidean space where K E ' 0 .

D.4 Naive Bayesian Optimization

Naive BO in the main text considers performing BO over a fully-connected unweighted graph. Such a graph is latent geometry agnostic, that is, it does not include any metric geometry inductive bias to provide the optimization algorithm with a sense of closeness between the candidate latent geometries.

E Ablation of the Gromov-Hausdorff Coefficients

We evaluate the impact of Gromov-Hausdorff coefficients on the graph search space by conducting an additional set of experiments. We compare the use of an unweighted, pruned graph search space with the introduction of Gromov-Hausdorff coefficients in the datasets considered in this paper. The results are show in Figures 19 to 22 below.

0

10

20

30

40

50

# Iterations

8

7

6

5

4

3

2

1

Minimum

GH BO

0

10

20

30

40

50

# Iterations

8

7

6

5

4

3

2

1

0

Minimum

0

10

20

30

40

50

# Iterations

8

7

6

5

4

3

2

1

Minimum

0

10

20

30

40

50

# Iterations

8

7

6

5

4

3

2

1

Minimum

0

10

20

30

40

50

# Iterations

6

5

4

3

2

1

Minimum

GH BO

0

10

20

30

40

50

# Iterations

7

6

5

4

3

2

1

0

Minimum

GH BO

0

10

20

30

40

50

# Iterations

7

6

5

4

3

2

1

Minimum

GH BO

0

10

20

30

40

50

# Iterations

7

6

5

4

3

2

1

Minimum

GH BO

0

10

20

30

40

50

# Iterations

7.0

6.5

6.0

5.5

5.0

4.5

4.0

3.5

3.0

Minimum

MNIST

GH BO

0

10

20

30

40

50

# Iterations

6

5

4

3

2

Minimum

CIFAR-10

GH BO

0

10

20

30

40

50

# Iterations

6

5

4

3

2

Minimum

fMNIST

GH BO

0

10

20

30

40

50

# Iterations

7

6

5

4

3

2

1

Minimum

eMNIST

GH BO

0

10

20

30

40

50

# Iterations

6

5

4

3

2

1

0

Minimum

GH BO

0

10

20

30

40

50

# Iterations

6

5

4

3

2

1

Minimum

CiteSeer

GH BO

The plots above show that in the majority of cases, the GH weights appear to either match or enhance the performance of the search algorithm. This observation is particularly pronounced

in synthetic tasks, which can be intuitively justified due to the significant impact of changes in the latent geometry on the network's output in such a setup. However, we highlight that a large amount of the performance gains in the algorithm seem to come from the inductive bias endowed to the search algorithm through the graph search space. This could be due to the fact that in image reconstruction tasks, searching for the optimal dimension is more important for reconstruction than finding the optimal latent geometry within a particular dimension. Since the GH coefficients are unity between dimensions, this makes the performance of the two search spaces similar in this context.

It should be noted that setting the inverse of the Gromov-Hausdorff distance as the edges of a search graph is only one potential way to add a geometric inductive bias into the search space. Future work could use similar mechanisms as those developed in this paper but in a different context in order to improve optimization performance. Finally, we would like to highlight that a more thorough analysis of the links between the performance of neural networks and the similarity of their latent geometries is a relevant question that merits further study, which is left for future work.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of automatically identifying the optimal latent geometry for machine learning models. The neural latent geometry search (NLGS) searches for a latent geometry composed of a product of constant curvature model spaces with minimal query evaluations.The proposed method utilizes the Gromov-Hausdorff distance as a measure to compare and evaluate the distance between different metric spaces. By constructing a graph search space and leveraging Bayesian optimization, the method efficiently searches for the optimal latent geometry.Experimental results on synthetic and real-world datasets demonstrate the effectiveness of the proposed approach in identifying the optimal latent geometry for various machine learning problems.\n\nSTRENGTHS:\n- The authors propose a modeling approach where the latent geometry is represented using the Cartesian product of Riemannian manifolds with constant curvature.\n- This paper constructs a graph search space where each node represents a latent space product manifold. The inverse GH distance is used as the weight on the edges, and Bayesian optimization is employed to find the minimum within this space.\n- Appropriate figures are included in the paper, effectively illustrating the concepts and results.\n\nWEAKNESSES:\n- There is a concern regarding Section 3.1 regarding the potential for overfitting due to the similarity between the objective function $L_{T,A}(g)$ used to determine the latent geometry $g$ and the loss function in the downstream task.\n- The paper lacks adequate illustration and explanation for the rationale behind utilizing the Cartesian product of Riemannian manifolds with constant curvature, such as Euclidean space, hyperbolic space, and sphere. It would be beneficial for the authors to provide a more comprehensive insight into the reasons for this choice, including potential advantages like efficient computation or other relevant considerations.\n- Although I am not an expert in image reconstruction and latent graph inference, it could be beneficial to compare the GH-BO method with existing benchmarks for these tasks in order to evaluate its performance and assess its effectiveness and competitiveness.\n\n**Other comments:**\n\nIt seems that the pdf submitted includes the appendix.\n\nQUESTIONS:\n1. Is it possible to utilize the Gromov-Wasserstein (GW) distance, which is a reformulation of the Gromov-Hausdorff (GH) distance [A], as an alternative for practical computation, given the known computational challenges associated with the GH distance?\n  \n2. Why was the diffusion kernel chosen to compute similarity in lines 284-287 instead of using the inverse of the Gromov-Hausdorff distance directly?\n  \n3. In the experiment, what does the \"Naive BO\" method use to be the weight on the fully-connected graph?\n  \n\n[A] Mémoli, Facundo. \"Gromov–Wasserstein distances and the metric approach to object matching.\" *Foundations of computational mathematics* 11 (2011): 417-487.\n\nLIMITATIONS:\nThe authors address the limitations of their work. This work does not have a negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe goal of this paper is to learn the geometry of underlying given data points after mapping them into latent space. Rather than learning a general (nonparametric) Riemannian manifold from the embedding data, the paper models this manifold as a product of certain prototypical manifolds with constant curvature (Euclidean, spherical, hyperbolic). Perhaps the idea is that a general nonlinear manifold can be stitched together using patches from these three types of manifolds. The paper describes a way to map points on these prototypical manifolds to their tangent spaces and back, i.e., the inverse exponential and the exponential map. This way one can express an embedded data point as elements of manifolds using their Euclidean coordinates. Patching together of prototype manifolds is done using a graph representation. \n\nWith this geometry, the paper derives a framework to fit a latent manifold to the training data. The loss function is built using the Gromov-Hausdorff distance. This requires isometrically embedding the prototypical manifolds into larger Euclidean spaces, and then utilize Euclidean distances to specific Gromov-Hausdorff distances. The search for optimal graph representation of latent space manifold is conducted through Bayesian optimization.\n\nSeveral experiments on synthetic and real-world datasets are presented in support of this theory.\n\nSTRENGTHS:\n-- Tackles a difficult problem of manifold learning by doing to in the latent space. \n\n-- The idea of representing arbitrary manifolds by a patching of some constant curvature pieces is interesting. (However, one has to show mathematically that this construction covers all or most of the interesting scenarios.)\n\n-- The use of graph representation to patch together the pieces is also interesting and feasible. \n\n-- The experimental results are reasonably demonstrative of the ideas although I have not evaluated them too carefully.\n\nWEAKNESSES:\n-- Why is the manifold learning or fitting performed in the latent space and not the original data space. This involves a choice of embedding which could be arbitrary. Perhaps the isometric nature of the embedding allows this arbitrariness to be introduced. \n\n-- Is it not possible to provided a simple minded sketch of the overall framework first using basic mathematics? There are too many symbols introduced right away and that distracts from a quick, preliminary understanding of the paper.\n\nQUESTIONS:\nPlease see my comment above.\n\nLIMITATIONS:\nThe paper does not discuss any limitation specifically.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an approach to estimate the best latent geometry structure for a machine learning model, trained on a data distribution and for a specific task. The approach assumes a product manifold structure of the latent space of subspace of constant curvature, being able to model product spaces of euclidean, spherical and hyperbolic subspaces. The search over candidate geometries is performed by constructing a graph search space where each node in the graph represents a candidate product manifold, associated with the model performance. Edges represent the inverse of the Gromov- Hausdorff distance between candidate spaces. The search is then conducted using Bayesian optimization algorithms. Experiments on synthetic and real scenarios demonstrate that assuming a graph space which models the topology and geometry of the spaces (with the right distances on the edges) entail finding the correct minima and being more efficient in the optimization process.\n\nSTRENGTHS:\n- The paper experimental results highlight the importance of correctly modeling topology and geometry of the latent space in the search for the product latent space that better suits the data and the task at hand, outlining the importance of modeling the geometry in embedding spaces.\n\n- The paper is well written and clear.\n\n- The ideas developed and explained in the paper are relevant to the representation learning and geometric deep learning communities: they could be used for new modeling strategies of latent spaces of NNs, developing constraints, and posing the product space search itself as a learning problem to find the geometry that better represents the data for a given task.\n\nWEAKNESSES:\n- The approach seems not very efficient to scale up to real case scenarios due to the requirement of embedding everything  isometrically in a space of 6n-n dimensions when hyperbolic components are considered. Also it is not entirely evident how the approach would scale up in real scenarios, when growing the number of components (subspaces)  and their dimensions considered. \n\n- The experiment primarily shows that modeling the topology and geometry is important to search for the right geometry. While this is a good contribution by itself, in my opinion another great potential outcome of the pape would be interesting in showing  how the resulting product manifold approximates the latent space, to understand if this tells us something about the data, especially for real dataset (e.g. does the MNIST latent space is better modeled by an euclidean product manifold or different ones, how many components better approximate it, etc..)\n\n- It would be good to add some additional works modeling the latent space of NNs as a product manifold, e.g. : \n\n     - *Fumero, Marco, et al. \"Learning disentangled representations via product manifold projection.\" International conference on machine learning. PMLR, 2021.*\n\n     - *Pfau, David, et al. \"Disentangling by subspace diffusion.\" Advances in Neural Information Processing Systems 33 (2020): 17403-17415.*\n\n     - *Zhang, Sharon, Amit Moscovich, and Amit Singer. \"Product manifold learning.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.*\n\nQUESTIONS:\n- Does the method scales well with growing dimensions of the latent space or the requirement to embed everything isometrically for hyperbolic latent space is a bottleneck in scaling up everything?\n\n- Can the isometric embedding requirement be relaxed and allow for some bounded distortion in the embedding space? Could this help in reducing the dimensionality constraints on the embedding space?\n\n- Is the latent space search always made post hoc (after having trained a model)? Do you think that the model training would benefit for enforcing a given geometry during training as well?\n\n- The graph pruning strategy (considering models which differ by a single type of latent space and with similar number of subspaces) also loses something in the latent geometry selection strategy? I.e. would it more beneficial in some cases to have two models that differs of more than one component connected in the graph?\n\n- How is the neural network employed to decode the signals in experiment in section 4.1 trained?\n\n- Are the experiments performed always on a single vector projected on the different product manifold geometries? Especially for real case scenario would not be better to work with multiple samples or some averaged statistics of the latent space?\n\n- For real dataset  (e.g. MNIST autoencoder) it is correct to say that the result of the graph search provides the best geometry for the latent space among the one considered? Does this tell something specific about the geometry and topology of the data? What happens if the model is trained from scratch assuming the selected geometry, will the resulting model have better performances?\n\n- The diffusion kernels signature is nearly invariant to isometric transformation of the graph: does this bring any problem in the graph search (e.g. there are cases in which it would be useful to distinguish two nodes with isometrically similar signatures)? How many components (eigenvalues, eigenvectors pairs) are used to approximate the kernel?\n\nLIMITATIONS:\nThe limitations are correctly addressed in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA variety of recent advances have provided evidence that learning in the\nmanifold setting (i.e., assuming that high dimensional data lie along\nlow-dimensional latent manifolds, aka. the manifold hypothesis) is advantageous. However, determining the optimal geometry of the latent space often requires expertise or the use of expensive hyperparameter tuning.\nThe present work addresses the problem of learning the optimal latent space\ngeometry for a variety of latent variable models and downstream tasks as well as\nprovides a formal approach to the topic. To solve the problem, the authors\nintroduce the construction of a novel graph search space as a basis for applying\nBayesian optimization. The nodes carry information about different latent space\ngeometries and the edges carry weights from the calculation of the\nGromov-Hausdorff distance between them. As an example to highlight the details of the approach, product manifolds are used. Experiments with toy and real data sets complete the work and provide empirical evidence for the functionality of the approach.\n\nSTRENGTHS:\nI greatly appreciate any attempt to incorporate geometrically sensitive analysis\ninto machine learning. From my perspective it is advantageous, to assume that\ndata incorporates structural information and to give models the ability to\naddress them.\n\nI really like the careful, clear and independent introduction to the theoretical\ncontent; It makes it easy (I would assume even for non-experts) to easily follow the\nauthors' arguments.\n\nFurther, the content is well organized and, with very few exceptions, has no spelling or grammatical flaws. Empirical evidence is accessed through experiments on toy and real world data and the authors make effort to extend the small-scale experimental setting.\n\nWEAKNESSES:\n- W1: The theory is presented in a rather narrow setting of product\n  manifolds and the authors simplify the problem setting (e.g., see\n  Section 3.1) by choosing a rather low dimension (i.e., $d_i=2$) of each of the model spaces and restrict themselves\n  to a (very) limited selection of different possible curvatures $K_i \\in \\lbrace-1,\n  0, 1\\rbrace$. According to Section B in the appendix, estimation of the\n  Gromov-Hausdorff distance in those cases, i.e., in the class of\n  product manifolds parameterized by a limited number of signatures, already seems\n  laborious. The presented method is very model space specific and seems\n  difficult to generalize to arbitrary manifolds. Overall, this impression\n  contradicts my expectations of an automatically optimal latent geometry\n  recognition when learning any latent space models. However, will there always be the need of expert knowledge in the field of\n  differential geometry to first choose a class of manifolds, then derive a\n  class-specific Gromov-Hausdorff distance formula (if tractable or derive\n  sufficient upper bounds) and finally construct a graph search space?\n\n- W2: In fact, the authors are aware of, that the class of product manifolds are far from\n  being as rich as, e.g., the class of arbitrary Riemannian manifolds (cf. l.\n  112ff.). Is it possible to quantify how well this class of manifolds works as\n  a proxy for arbitrary latent space model classes? In other words, in the case\n  of product manifolds as a model class, how severe are the constraints imposed\n  when learning a latent space model, e.g., in the setting of variable Bayesian\n  inference using a variational auto-encoder model?\n\n- W3: Considering the points (i) I have rather a broad knowledge on the subject\n  and (ii) the authors admit that to their knowledge no computational method for\n  the mathematical comparison of product manifolds exists (see l. 58), it is\n  nevertheless questionable that no related work exists. I would cherish the\n  effort to address prior work even it is in the broader sense of latent space\n  geometry or addressed to single properties only, e.g., curvature (see Lubold\n  et al., Identifying the latent space geometry of network models through analysis of curvature).\n\n- W4: I'm interested in implementation details, in particular how the presented\n  geometry identification fits into a neural network pipeline of a latent space\n  model, for instance in a variation auto-encoder. A structured description of the\n  methodology in form of a code listing as well as code examples for the purpose\n  of reproducibility would be desirable.\n\nHowever, I would like to reiterate my appreciation for the authors' interest in\nexpanding the field of latent space models from a geometrical point of view. I'm\nsure that by working on the deficiencies mentioned, the work will gain in value\nand will deliver a valuable contribution to the community.\n\nQUESTIONS:\n- Q1: In Section 4.1, can you give more insight into the \"Ground Truth\" product manifold $\\mathcal{P}_{T}$? Even if this was chosen arbitrarily, what is its dimension?\n- Q2: What measure is used for evaluation in the experiments; i.e., what does\n  \"Minimum\" at the y-axis in all plots refer to? And are there any data\n  normalization's involved since I identify no changes in the scale of the plots.\n- Q3: I would say I got a good intuition for closed balls of radius one in spaces\n  of curvature zero or one, at least for dimensions lower 3. What's a good\n  intuition for closed balls in spaces of curvature smaller zero? Would be cool\n  to have a small section on this, e.g., as part of the appendix. Especially,\n  because those balls are used for comparison of model spaces (see l. 137ff.).\n  \n- Minor stuff:\n    - l. 129 check wording.\n    - l. 134 $b_0 \\in B$\n    - l. 145 $d_{GH}(B,B\\times B')$?\n    - l. 180 $\\mathbb{E}^{6n-6}$ ?\n    - l. 199 $d_{GH}(B_{\\mathbb{E}²},B_{\\mathbb{H²}})$ twice?\n    - l. 352 \"Figure 6._In\"\n\nLIMITATIONS:\nLimitations are addressed (see e.g. Chapter 5) and mainly concern the focus on a\nlimited class of models, namely product manifolds formed by spaces of constant\ncurvature. From my perspective there are important aspects that need to be\ndiscussed additionally, e.g., computational limitations in the case of high\ndimensional latent spaces, since here the estimation of the Gromov-Hausdorff\ndistance in combination with the application of Bayesian optimization usually\ncauses an enormous computational effort.  According to section B.4 in the\nappendix, restrictions related to estimating the Gromov-Hausdorff distance for\nthe (arguably rather simple) model class under consideration are already\nimplied. In the case of arbitrarily complex Riemannian manifolds, do these\ndifficulties even lead to intractability?\n\nNo evidence was found that the authors addressed potential negative societal impact. Please make the effort to include this mandatory information in your work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the past few years, there's been an interesting trend in the ML literature where people have proposed non-Euclidean latent space geometries to improve network performance. However, to my knowledge geometries have always been picked on an ad hoc basis or tuned naively as a hyperparameter - the most common being hyperbolic and hyperspherical spaces. This evokes a natural question: is there an efficient way to choose the best latent geometry? The authors formulate this choice as a task on its own: neural latent geometry search (NLGS).\n\nThe authors proceed to propose their own method for NLGS. The method chooses a latent geometry from the set of products of up to a fixed number $n_\\mathcal{P}$ of manifolds, where $n_\\mathcal{P}$ is a hyperparameter. At a high level, the authors metrize the set of product manifolds, use this metric to structure a graph over the set, and then use this graph as a search space for Bayesian optimization.\n- The metric is derived from the Gromov-Hausdorff distance between manifolds. The Gromov-Hausdorff distance isometrically embeds two manifolds into a common space ($6n - 6$ dimensional Euclidean space, for this work) and computes the maximum distance from a point on one manifold to the other manifold. Some effort is involved in actually estimating these distance values.\n- The authors then create a graph whose vertices are product manifolds and whose edges connect those product manifolds which differ by a single term in the product. The edges are weighted according to the inverse Gromov-Hausdorff distance.\n- The authors then proceed to use Bayesian optimization on the resulting graph to pick the latent geometry which minimizes the loss for the problem of interest.\n\nThis Gromov-Hausdorff Bayesian optimization (GH BO) method is tested against a naive baseline and Bayesian optimization on a fully connected graph. It outperforms the two methods on both synthetic and real-world experiments.\n\nSTRENGTHS:\n- NLGS is an interesting and well-motivated problem. Based on the broader literature's interest in latent geometries, NLGS could end up being a very impactful framework for model tuning.\n  - As a result, GH BO might also end up being a common tool in the hyperparameter search toolbox.\n- The theory and machinery used to develop their Gromov-Hausdorff search space is creative and interesting; this is a very original paper.\n- The whole paper is well-written and easy to follow. At a glance, even the appendices are a nice read.\n- Their approach very clearly empirically outperforms the baselines, which are reasonable in my opinion.\n\nWEAKNESSES:\n1. The choice of Gromov-Hausdorff distance as the metric strikes me as a bit arbitrary. Sure, it's *a* way to metrize the model spaces, but I do not think the paper very clearly justifies it as being the right way for the purpose of ML model performance. Why should I expect two models whose latent geometries are closer in Gromov-Hausdorff distance to perform more similarly? As a general principle of optimization, there needs to be some relationship between the objective function and the search space. My concern here is that this relationship is embodied by the structure of the graph (which is discussed for 1 paragraph), rather than the Gromov-Hausdorff weights themselves (which are developed over more than 2 pages + Appendix B).\n   - A natural ablation here would be to test BO with the same graph as is used in GH BO algorithm, except with uniform edge weights of 1 instead of the GH weights. This experiment would test whether the GH weights, and not just the choice of edges, are responsible for performance. Thus far the authors have only ablated against a complete graph.\n\n2. The experiments are not at all reproducible.\n   - The authors did not describe their network architectures. The appendix only contains a vague description of the autoencoder architecture without details on the number of layers, weights, filters, etc. Even fewer details are available for the other experiments.\n   - The authors did not provide their code. Since their method is somewhat involved, this will also hamper adoption by future work.\n\n3. Clearly GH BO performs better than other methods for the models given, but it's not clear whether these are actually *decent models*. As a result, it's unclear to me how much GH BO will help at scale in realistic settings. Clarifying the model architectures used in the experiments might help.\n\n4. On the theoretical side, several simplifications are made (mostly for computional purposes) which make the final algorithm much less interesting:\n   - The search space is restricted to products between manifolds of dimension 2.\n   - Only unitary (-1, 1) curvature values are used for the non-Euclidean model spaces due to the discrete setup, whereas other works tune the curvature through gradient descent (perhaps that can be done post hoc, but it would be nicer if it were integrated into NLGS).\n   - The authors pruned their search graph by removing edges wherein manifold products differ by more than one model space.\n   - As a result, they only needed to compute 3 distinct Gromov-Hausdorff distance values, resulting in only 3 distinct graph weights.\n\nI'm open to raising my score if the authors satisfactorily address concerns #1 and #2.\n\nQUESTIONS:\nWhat is your opinion on my concern #3? Do you think there's reason to expect GH BO will perform at scale?\n\nCan you please clarify this statement regarding other benchmarks: \"Furthermore, as there are no existing algorithms to compute the distance between arbitrary product manifolds, we have not included these methods in our benchmark as they would simply converge to random search.\"\n- Can you not metrize arbitrary product manifolds using distances within the graph structure you've designed?\n- Why would they converge to random search?\n\nLIMITATIONS:\nThe authors identify *constant curvature* model spaces as a limitation. They also point to limits in the field of Bayesian optimization on graphs.\n\nHowever, I believe some of the simplifications I pointed out in the weaknesses section are also pertinent. It would be helpful to either (1) explicitly raise these as limitations or (2) further justify them as good choices for the algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}