Neural Latent Geometry Search: Product Manifold Inference via Gromov-Hausdorff-Informed Bayesian Optimization
Authors: Haitz Sáez de Ocáriz Borde
Alvaro Arroyo
Ismael Morales López
Ingmar Posner
Xiaowen Dong
Venue: NeurIPS 2023 poster (2023)
Keywords: Representation Learning
Product Manifolds
Bayesian Optimization
Gromov-Hausdorff Distance
TL;DR: This paper aims at finding the optimal latent product manifold for machine learning models and tasks via Bayesian optimization.

Abstract: Recent research indicates that the performance of machine learning models can be improved by aligning the geometry of the latent space with the underlying data structure. Rather than relying solely on Euclidean space, researchers have proposed using hyperbolic and spherical spaces with constant curvature, or combinations thereof, to better model the latent space and enhance model performance. However, little attention has been given to the problem of automatically identifying the optimal latent geometry for the downstream task. We mathematically define this novel formulation and coin it as neural latent geometry search (NLGS). More specifically, we introduce an initial attempt to search for a latent geometry composed of a product of constant curvature model spaces with a small number of query evaluations, under some simplifying assumptions. To accomplish this, we propose a novel notion of distance between candidate latent geometries based on the Gromov-Hausdorff distance from metric geometry. In order to compute the Gromov-Hausdorff distance, we introduce a mapping function that enables the comparison of different manifolds by embedding them in a common high-dimensional ambient space. We then design a graph search space based on the notion of smoothness between latent geometries and employ the calculated distances as an additional inductive bias. Finally, we use Bayesian optimization to search for the optimal latent geometry in a query-efficient manner. This is a general method which can be applied to search for the optimal latent geometry for a variety of models and downstream tasks. We perform experiments on synthetic and real-world datasets to identify the optimal latent geometry for multiple machine learning problems.

Neural Latent Geometry Search: Product Manifold Inference via
Gromor-Hausdorff-lnooree Bayesian Optimization
Haitz Sáez de Ocáriz Borde Oxford Robotics Institute
Álvaro Arroyo* Oxford-Man Institute University of Oxford
University of Oxford
Ismael Morales López Mathematical Institute University of Oxford
Ingmar Posner
Xiaowen Dong
Oxford Robotics Institute University of Oxford
Machine Learning Research Group University of Oxford
Abstract
Recent research indicates that the performance of machine learning models can be improved by aligning the geometry of the latent space with the underlying data structure. Rather than relying solely on Euclidean space, researchers have proposed using hyperbolic and spherical spaces with constant curvature, or combi- nations thereof, to better model the latent space and enhance model performance. However, little attention has been given to the problem of automatically identi- fying the optimal latent geometry for the downstream task. We mathematically define this novel formulation and coin More specifically, we introduce an initial attempt to search for a latent geometry composed of product of constant curvature model spaces with small number of query evaluations, under some simplifying assumptions. To accomplish this, we propose a novel notion of distance between candidate latent geometries based distance from metric geometry. In order to compute
neural latent geometry search (NLGG)).
on the Gromov-Hausdee the Gromov-Hause Hausdorff distance, we introduce a mapping function that enables the comparison of different manifolds by embedding them in a common high- dimensional ambient space. We then design a graph search space based on the notion of smoothness between latent geometries, and employ the calculated dis- tances as an additional inductive bias. Finally, we use Bayesian optimization to search for the optimal latent geometry in query-efficient manner. This is gen- eral method which can be applied to search for the optimal latent geometry for a variety of models and downstream tasks. We perform experiments synthetic and real-world datasets to identify the optimal latent geometry for multiple machine learning problems.
Introduction
There has been recent surge of research employing ideas from differential geometry and topology to improve the performance of learning algorithms [Bortoli et al. 2022] Hensel et al. [021] Chamberlain et al. 202 Huang et al. 2022 Barbero et al. 2022a Traditionally. Euclidean spaces have been the preferred choice to model the geometry of latent spaces in the MI. community [Weber] 2019 Bronstein al. 2021). However, recent work has found that representing the latent space with geometry that better matches the structure of the data can provide significant performance enhancements in
*Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

both reconstruction and other downstream tasks (Shukla et al. 2018). In particular, most works have employed constant curvature model spaces such as the Poincaré ball model [Mathieu et al.) 2019), the hyperboloid [Chami et al. 2019), or the hypersphere (Zhao et al., 2019). to encode latent representations of data relatively simple and computationally tractable way.
While individual model spaces have sometimes shown superior performance when compared to their
Euclidean counterparts, more recent works [Gu al. 2018 Skopek et al. [011 Sáez de Ocáriz Borde Zhang et al. 2020. Fumero et al. 202 Plau et al. [0200] have leveraged the notion of product spaces (also known as product manifolds) to model the latent space. This idea allows to generate more complex representations of the latent space for improved performance by taking Cartesian products of model spaces, while retaining the computational tractability of mathematical objects such as exponential maps or geodesic distances, see (azzz de Ocariz Borde et al.) [2023b]. Despite the success of this methodology. there exists no principled way of obtaining the product manifold signature (i.e., the choice and number of manifold components used to generate the product manifold and their respective dimensionalities) to optimally represent the data for downstream task performance. This procedure is typically performed heuristically, often involving random search over the discrete combinatorial space of all possible combinations of product manifold signatures, which is an exceedingly large search space that hampers computational efficiency and practical applicability. Some other work related to latent space geometry modelling can be found in Lubold et al. (2023). Hauberg et al. (2012), Arvanitidis et al. [2017].
et al. 2023b a
Contributions. 1) In this paper, we consider novel problem setting where we aim to search for an optimal latent geometry that best suits the model and downstream task. As particular instance of this setting, we consider searching for the optimal product manifold signature. Due to the conceptual similarity with neural architecture search (NAS) strategies (Elsken et al., 2018 Zooph and Le] 2016 Pham et al. 2018). we coin this problem as neural latent geometry search (NLGS), which we hope will encourage additional work in the direction of optimal latent geometry inference. We test our framework on a variety of use cases, such as autoencoder reconstruction [Mathieu et al. 2019) and latent graph inference (Sáez de Ocáriz Borde et al. 2023b for which we create a set of custom datasets.
2) To search for the product manifold signature, we must be able to compare product manifolds. This traditionally done by computing the Hausdorff distance between manifolds (Taha and Hanbury 2015). which however requires manifolds to reside in the same metric space. To address this limitation, in this work we develop to our knowledge the first computational method to mathematically compare product manifolds. Our approach generalizes classical algorithms and allows the comparison of manifolds existing in different spaces. This is achieved by defining an isometric embedding that maps the manifolds to common high-dimensional ambient space, which enables the computation of their Gromov- Hausdorff distances.
3) Leveraging the Gromov-Haue Hausdorff distances between candidate latent space manifolds. we design a principled and query-efficient framework to search for an optimal latent geometry, in the sense that it yields the best performance with respect to a given machine learning model and downstream task.
Our approach consists of constructing a scometry-informee graph search space where each node in the graph represents a unique candidate product manifold, associated with the model performance using this manifold its embedding space. The strength of edges in the graph are based on the inverse of the Hrasdo-Hauudorff" distance, thereby encoding a notion of "closeness" between manifolds in the search space. We then perform efficient search over this space using Bayesian optimization (BO). We compare our proposed method with other search algorithms that lack the topological prior inherent in our model. Empirical results demonstrate that our method outperforms the baselines by a significant margin in finding the optimal product manifold.
Outline. In Section 2 we discuss manifold learning, and product manifolds of constant curvature model spaces such as the Euclidean plane, the hyperboloid, and the hypersphere. We also review relevant mathematical concepts, particularly the Hausdorff and
distances from metric geometry [Gopal et al., 2020). Section 3 presents the problem formulation, the proposed methodology compare product manifolds, as well how the search space over which to perform geometry-informed Bayesian optimization is constructed. Finally, Section I exxplans how our custom
2

---

distance, synthetic and real-worid datasets were obtained, and the empirical results. Lastly, in Section 5 conclude and discuss avenues for future work
we
2 Background
Manifold Learning Manifold learning is sub-field of machine learning that uses tools from differential geometry to model high-dimensional datasets by mapping them to a low-dimensional latent space. This allows researchers to analyze the underlying structure of data and improve machine learning models by capturing the geometry of the data more accurately. Manifold learning is based on the assumption that most observed data can be encoded within now-dimensional manifold
(see Figure I embedded applications in dimensionality reduction [Roweis and Saul 2000 Tenenbaum et al., 2000], generative models [Goodíellow et al. 2014 Du et al. al. |021] Bortoli et al. 2022]. and graph structure learning for graph neural networks (GNNs) [Topping et al. [2021] In all these application, the key is to find a topological representation as an abstract encoding that describes the data optimally for the downstream task.
a high-dimensional space [Fefferman et al. 2013]. This has seen
Product Manifolds. this work, we model the ge- ometry using model space Riemannian manifolds (Ap- pendix A.T) and Cartesian products of such manifolds. The three so-called model spaces with constant curva- ture are the Euclidean plane, E Ex Rd where the curvature Kg = O; the hyperboloid, H = - (xp E RRda+1 (xp: x = 1/Kn), where K₃ < 0 and <: is the Lorentz inner product; and the hypersphere, S" S
{xp E ds (Xp: Xp)2 = 1/Ks}, where Kg > 0 and (,,'2 is the standard Euclidean inner prod- uct. These have associated exponential maps and distance functions with closed form solutions, which can be found in Appendix A.2 A product manifold can be constructed using the Cartesian product P uP M K, of npp man- ifolds with curvature K; and dimensionality dz Note that both np and d, are hyperparameters that define the product manifold P and that must be set a priori. On the other hand, the curvature of each model space K₁ can be learned via gradient descent. One must note that the product manifold construction makes it possible to generate more complex embedding spaces than the original constant curvature model spaces, but it does not allow to generate any arbitrary manifold nor to control local curvature.
Figure I: Schematic of a manifold M and open subsets U, and Uj. An open chart is homeomorphism of open subset of the manifold onto an open sub- set of the Euclidean hyperplane. Here, Vay is a transition function.
Hausdorff and Haundofff Distances for Comparing Manifolds. The Hausdorff distance between two subsets a metric space refers to the greatest distance between any point on the first set and its closest point the second set (Jungeblut et al. 2021]. Given metric space X with metric dx, and two subsets and B, we can define the Hausdorff distance between and in X by
d (4, B) max suppdx (a, B), sup dx (b, A)
.EAA
B
priori this quantity may be infinite. Hence we will restrict compact subsets A and B. In this case, we can equivalently define du(AA A, B3 as the smallest real number e 2 0 such that for every a E A and every b E B there exist a' A and // B such that both dx 1/) and dx(a', b)
most
We note that the previous definition does not require any differentiable structures on X, A and B. They can be merely metric spaces. This generality allows to distinguish the metric properties of Euclidean, hyperbolic and spherical geometries beyond analytic notions such as curvature. However, the definition in Equation [ only allows to compare spaces A and B that are embedded in a certain metric space X. The notion of distance that we shall consider is the which we define below in Equation
Given a metric space X and two isometric embeddings f: X and g: B X, we define
dx (A, B) di ()(()),
(2)
For a high level description of the proposed framework, recommend skipping Section g
3

---

Now, given two metric spaces A and B. we denote by ES(A, B) (standing for "embedding spaces of A and B") as the triple fx, where X metric space and f A and X are isometric embeddings. We define the Grrmov-Hausdory distance between A and B as:
inf dGH B) (X.J.))ESS(A A,))
Gx...99 (M(((((())B)))))
We should note that, since we assume that both A and B are compact, there a trivial upper bound for their Hoomov-Hausdorff distance in terms of their diameters. The diameter of metric space Y is defined to be diam (Y) = sup y.yee dy (y. v/). Given dg E A and by E B, we can define the isometric embeddings f : A - A x B and B - x B given by f(a) = {a, bu) and g(b) = (av. b). It is easy to see that dæ 4Ax (A, B) & max (diam A), diam(B) B)). Since the triple (A x f.9) belongs to (S,, B), we can estimate (cH(A, B) & max (daam(A), diam( To compare IE", H" and S" we propose taking closed balls of radius one in each space. Since balls of radius one in any of these spaces are homogeneous Riemannian manifolds, they are isometric to each other. By upper bound for their Gromov-
estimating or providing spaces. This notion of distance between candidate latent geometries will later be used to generate a search space for our framework. With exactly an analogous argument as before, given two compact balls of radius one and B', of centres Zo and zys we can embed B into B x B' by the mapping f
distance, we can compare the
notice that
(b, x/). From here, it is obvious to that d B .f.id/ (B. B x B') 1. In particular, this gives us the bound dGH( B, B*) < 1. This is the estimation we will take as the Srrmov-HHasdore distances of product manifolds that simply differ in coordinate (such as, say, E² and 1 x IH²)
3 Neural Latent Geometry Search: Latent Product Manifold Inference
In this section, leverage ideas discussed in Section introduce principled way find the optimal latent product manifold. First, we introduce the problem formulation of NLGGS. Next, we outline the strategy used to compute the we discuss how this notion of similarity can be used in practice to construct a graph search space of latent geometries. Lastly, we explain how the Haanoo--audoorf-iinnnnoo graph search space can be used to perform NLGS via BO.
distance between product manifolds, and
3.1 Problem Formulation
The problem of NLGS can be formulated as follows. Given search space denoting the set of all possible latent geometries, and the objective function (g)4(g) which evaluates the performance of given geometry g on a downstream task for a machine learning model architecture A, the objective is to find an optimal latent geometry g*:
g arg min (g.)
g=0
In our case we model the latent space geometry using product manifolds. Hence we effectively restrict Equation 4 to finding the optimal product manifold signature:
n nip, [d.) .
[K3)* s
arg min (rp, (dilienp (Ki)ien)): d.
(5)
where np is the number of model spaces composing the product manifold P. (en are the dimen- sions of each of the model spaces of constant curvature, and [K,) jin) their respective curvatures. We further simplify the problem by setting d, = 2, Vi, and by restricting ourselves to K, E (-1,0, 13, in order to limit the size of the hypothesis space.
3.2 Quantifying the Difference Between Product Manifolds
Motivation. From computational perspective, we can think of the Hausdorff distance as measure of dissimilarity between two point sets, each representing a discretized version of the two underlying continuous manifolds we wish to compare. Taha and Hanbury 2015) proposed an efficient algorithm to compute the exact Hausdorff distance between two point sets with nearly-linear complexity leveraging early breaking and random sampling place of scanning. However, the original algorithm assumes that both point sets live in the same space and have the same dimensionality, which is

---

limiting requirement. Gromov-Hausdone distances, as opposed to usual Hausdorff distances, allow us measure the distance between two metric spaces that a priori are not embedded common bigger ambient space. However, this has the caveat that they are not computable. For instance, for our application we must calculate the distance between each pair of the following three spaces: TE", St and H However, 8" does not isometrically embed in E" and hence in order to compare EE and S", we must work in a higher dimensional ambient space such as E=* In the case of H"" finding an embedding to Euclidean space is more complicated and is described in Appendix B.2 (1H will be embedded isometrically into EE6n-9). However, in this process there will be choices made about which embeddings consider (in particular, we cannot exactly compute the infimum that appears in the definition of Haunoo-f distance in Equation 33. Likewise, using the original algorithm by Taha and Hanbury [015 it is not possible to compute the Hausdorff distance between product manifolds based on an unequal number of model spaces, for instance, there is no way of computing the distance between EE and E X THT In this section give an upper bound for daa (EE ,8"), and then describe an algorithm to give an upper bound for dc (E", IHT") and dc (
Strategy. The spaces En and S" isometrically embed into Æ in many ways. This may seem redundant because both spaces already embed in En+1_ However, the interest of considering this higher dimensional Euclidean space is that H" will also isometrically embed into it. Crucially, this will provide a common underlying space in which to compute Hausdorff distances between our geometries E", S" and IiI", which will lead to an estimation of their mutual distance. The embedding of H into g6n-6 that we shall describe appears in Henke and Nettekoven [987] and is made explicit in [Blanuša| [955]. We also refer the reader to the exposition [Brander 2003] Chapter 5], which puts this results in broader context while also summarising related advances on the topic of isometrically embedding homogeneous spaces into higher dimensional ones. For n 2, we name this embedding F : H2 - EE6 (Appendix [B.2). For simplicity, in our experiments in g we will work with product manifolds generated based on constant curvature model spaces of dimension n
Section
Now we can summarise our strategy to estimate do (g2, Bgy) as follows (for dGH (Bgz, Bga) it will be entirely analogous). The first step consists of approximating our infinite smooth spaces by finite discrete ones. For this, we consider several collections of points {P.) tez in E² that are sufficiently well distributed. The exponential map can be applied to the collection of points exp : TulH2 = R22 Bu
to get several collections of points Q in Bur (again, well distributed by construction). In addition, we will consider several isometric embeddings fO Bga R. Hence, we take
dGH (Bre, Bya) ve min d Ff..F (P. min di ((())) Gj.k
(6)
In Appendix B we gradually unravel the previous formula and give explicit examples of the involved elements. In particular, in Appendix [ we explain how to generate points in the balls of radius one, and in Appendix B.2 how to describe the isometric embedding F: Bg - IE, The results obtained for the Ctomovv-Haudoore distances between product manifolds are used to generate the graph search space introduced in the next section.
3.3 The Haudoor-Husddoff-Innnnoo Graph Search Space
Gromov-Hausdare Edge Weights. In Section 2, we used Cartesian products of constant curvature model spaces to generate candidate latent geometries. We now turn our attention to constructing search space find the optimal latent geometry. To do so, we first consider all pos- sible combinations of product manifolds based on given number of model spaces, represented by n., Furthermore, we denote the total number of products (model spaces) used to form the product manifold P with np. Conceptu- ally, ni, the number of model space types used, while np refers to the overall number, or quantity, of model spaces that form the resulting product space. For instance, if only the Euclidean plane, the hyperboloid, and the hypersphere
Table 1:
Estimated Gromov-Hausdorff distances (up to two decimal places) be- tween model spaces and corresponding edge weights in the graph search space.
Comparison Pair
doul
((()
(I22 $2
0.23
4.35
2 H
0.77
1.30
(S H
0.84
1.20
5

---

are taken into account, then ni = 3. If all product mani- fold combinations are considered, the number of elements in the search space increases to Σ n However, we assume commutativity for latent product manifolds, implying that the output of a trained neural network with a latent geometry M; X M, should be the same as that with M, X M, We refer to this concept the symmetry of parameterization, matrices of their neurons to achieve optimal performance for two equivalent latent manifolds. This assumption reduces the search space from growing exponentially to O(n)), assuming three constant curvature model spaces are considered (see Appendix [3.]] for a more complete explanation).
neural networks can rearrange the weight
We model the search space graph over which we perform BO. More formally, we consider graph G given by (V,E), where V = fur are the nodes of the graph and & = {e is the set of edges, where edge ej = (vi, U2) connects nodes v; and vg. The topology of the graph is encoded in its adjucency matrix, which we denote with A E RN*N, In our case, focus on a weighted and undirected graph. meaning that A.j = Aj... and Ai, can take any positive real values. We consider a setting in which the function f(-) to mínimize is defined over the nodes of the graph, and the objective of using BO to find the node associated to the minimum value 8 arg min,ey f(w).
S2
X S
E
X S
E
E
g2
2
B H
1I X 2
Figure 2: Slice of the graph search space for la- tent geometries of dimension 4: product manifolds obtained using? 2 models spaces of dimension 2. The graph edges are shown in different colours to depict a different degree connectivity (black: Wya, 52, red: W2a,F2, blue: uga,,z2 )²). this is deter- mined by the inverse of the distance between the different product manifolds.
In our setting, each node in the graph represents a different latent space product manifold, and the value at that node is the validation set performance of neural network architecture using this latent geometry. We use the inverse of the Gromov-HHasdone distance between product manifolds to obtain edge weights. We denote by MM,,M,, (edge weights) the inverse Gromov-Hausdorff distance between model spaces Mi and M2. The approximate values of these coefficients are presented in Table [ In particular, the Gromov-Hausdorff distance (E²2, S used to compute Wga,st is derived analytically in Appendix BB.3. while the remaining coefficients are obtained through computational approximations of [ which are depicted in Appendix B.4 Further, the Gromov-HHasdorf distance between manifolds different dimensions is one (see Section
In order to impose additional structure on the search space, we only allow connections between nodes cor- responding to product manifolds which differ by a single model space. For example, within the same dimension, $2 x 2 and E² x H² would be con- nected with edge weighting wga,g ga while $2 HP and E² E² would have no connection in the graph. Furthermore, product manifolds in different dimen- sions follow the same rule. For instance, we would have a connection of strength one between E² x Hi and E² ! H?2 but no connection between, for instance, E² and E² x H x H! or S² and E² H² This construction induces a sense of directionality into the graph and generates clusters of product man- ifolds of the same dimension. Finally, it should be mentioned that in practice there are only four edge weights. For example, the connectivity strength between E H and H x H2 is Wgg g² since dGH(EE x H2? H2 x H2) dcH(E' HH' given that de (H2 H²) = O. Visual representations of the graph search space
Figure 3: Example graph search space for prod- uct manifolds composed of up to seven model spaces. Manifolds of different dimensionality are connected with edges coloured grey. Node labels have been omitted for visual clarity.
be found Figures 2 and
Note that both figures use the same colour scheme, and in Figure| there is an increase in the dimen-

---

sionality of the product manifolds from left to right (e.g. on the top left corner, one can see a triangle corresponding to the three model spaces). We refer the reader to Appendix B.6 for additional visualizations.
Bayesian Optimization over the Graph Search Space. We aim to find the minimum point within the graph search space through the use of BO (see Appendix C). While performing BO on graphs allows us to search for the minimum over a categorical search space, some of the key notions used in BO over Eucledian spaces do not translate directly when operating over a graph. For example, notions of similarity "closeness" which are trivially found in Eucledian space through the & or lz norms require more careful consideration when using graphs.
In our setting, we employ a diffusion kernel (Smola and Kondor 2003 Kondor and Vert 2004 to compute the similarity between the nodes in the graph. The diffusion kernel is based on the cigendecomposition of the graph Laplacian L E ENN x. defined as L D A where D is the degree matrix of the graph. In particular, the eigendecoomooition of the Laplacian is given by L = UAU!, where U [u,,... ........ uN] is matrix containing eigenvectors as columns and A diag(A1...... Ax) is a diagonal matrix containing increasingly ordered eigenvalues. The covariance matrix used to define the GP over the graph is given by
K(V.V) Ue UU
where 3 is the lengthscale parameter. Our approach is therefore conceptually similar to that of Oh iet al. 20191, which employ a diffusion kernel to carry out a NAS procedure on a graph Cartesian product. We highlight, however, that the main contribution of this work is the construction of the graph search space, and we employ this search procedure to showcase the suitability of our method in finding the optimal latent product manifold. For reference, we note that other works employing Bayesian optimization in graph-related settings include Cui and Yang (2018), Como et al. [2020].
Ma et al. 2019], Ru al. (2020) Cui et al. (2020] Wan et al. [202]]
4 Experimental Setup and Results
In this section, detailed outline of the experimental results is provided. It comprises of experiments performed on synthetic datasets, as well as experimental validation on custom-designed real-world datasets obtained by morphing the latent space of mixed-curvature autoencoders and latent graph inference. The results demonstrate that the Hoomoo-aaudoorf-infoone graph search space can be leveraged to perform NLGS across a variety of tasks and datasets.
4.1 Synthetic Experiments on Product Manifold Inference
In order to evaluate the effectiveness of the proposed graph search space, we conduct a series of tests using synthetic data. We wish to present a setting for which the latent optimal product manifold is known by construction. To do so, we start by generating a random vector x and mapping it to a "ground truth" product manifold Pr, which we choose arbitrarily, using the corresponding exponential map. The resulting projected vector is then decoded using neural network with frozen weights, fe. obtain reference signal y²,, which we wish to recover using NLGS. To generate the rest of the dataset, we then consider the same random vector but map it to number of other product manifolds (Pa)ieny with the same number of model spaces as Pr but different signatures. Decoding the projected vector through the same neural network yields a set of signal, {y Jieup- We use the aforementioned signals, to set the value associated with each node of the graph search space to be MSE(y"T, y In this way, our search algorithm should aim to find the node in the graph that minimizes the error and hence find the latent geometry that recovers the original reference signal.
Our method consists in using BO over our Gromov-Hausdorif against random search, and what we call "Naive BO", which performs BO over a fully-connected graph which disregards the Haanoo-- distance between candidate latent geometries. The figures presented, namely Figures F and [5] display the results. Each plot illustrates the performance of the algorithms and baselines as they select different optimal latent product manifolds denoted as Pr. Notably, the figures demonstrate that the algorithm utilizing the Gromov- www.--Haaau-ooooonnnne search space surpasses all other baselines under consideration and consistently achieves the true
search space. We compare it
7

---

function minimum. Figure generally requires fewer iterations compared to Figure to attain the global minimum, primanly due to the smaller size of the search space. important note that our benchmark solely involves the same algorithm but with a change in the search space to topologically uninformative one. This modification allows us evaluate impact of incorporating this information into the optimization process. We have not considered other benchmarks such as evolutionary algorithms since they always rely on a notion of distance between sampled points. Furthermore, there are existing algorithms to compute the distance between arbitrary product manifolds, we have not included these methods in our benchmark as they would simply converge to random search. All results shown log scale, and apply small offset of 10-3 to the plots to avoid computing log(O) when the reference signal is found.
GHIB
Navve EO
Minimum
Minimum
Mtnimum
Figure 4: Results (mean and standard deviation over 10 runs) for candidate latent geometries involving product manifolds composed of 13 model spaces. For each plot a different ground truth product manifold Pr is used to generate the reference signal.
Rake 80
Minumum
Minimum
Minimmum
Tecaloons
Earations
llrrations
Minimu
Minimum
Misimum
Teraioos
Beraiion
Figure Results (mean and standard deviation over 10 runs) for candidate latent geometries involving product manifolds composed of 20 model spaces. Same setup as above.

---

4.2 Experiments on Real-World Datasets
To further validate our method, we conduct additional tests using image and graph datasets. Specifically, we focus on image reconstruction and node classification tasks.
Image Reconstruction with Autoencoder
We consider four well-known image datasets: MNIST Deng 2012
CIFAR-10 Krizhevsky et al.
2014], Fashion MNIST [iiao et al., 2017] and eMNIST Cohen al. 2017]. Some of these datasets have been shown to benefit from additional topological priors, see Khrulkov et al. 2020 and Moor al. 2020 We use an autoencoder, detailed more thoroughly in Appendix D to encode the image in low dimensional latent space and then reconstruct it based on its latent representation. To test the performance of different latent geometries, we project the latent vector onto the product manifold being evaluated and use the reconstruction loss at the end of training as the reference signal to use in the graph search space. We consider a search space size of Ωn = 7 for MNIST and np = 8 for C10, Fashion MNIST and eMNIST. The results are displayed in Figure 6 In line with previous experiments, the Gromov-Hauxdort Hausdorff informed search graph enables us to find better solutions in a smaller amount of evaluations.
MUST
DIAR-12
MAUST
#PRST
BD
Menmmmm
Danalies
Figure 6: Results (mean and standard deviation over 10 runs) for image reconstruction tasks.
Latent Graph Inference
Finally, we consider searching for the optimal product manifold for graph neural network node classification task using latent graph inference. particular, we build upon previous work by [Sáez de Ocáriz Borde et al. (223b) which used product manifolds to produce richer embeddings spaces for latent graph inference. We consider the Cora [Sen al. 2008) and Citeseer [Giles et al. 1998) citation network datasets, and search space consisting of product manifolds of up seven model spaces np = The aim is to find the latent space which gives the best results for the node classification problem using minimal query evaluations. Performing BO alongside the -oanno-- informed search space gives a clear competitive advantage over Naive BO and random search in the case of Cora. For Citeseer, the experiments do not give such clear-cut improvement, which can be attributed to the lack of smoothness of the signal on the graph search space and its incompatibility with the intrinsic limitations of the diffusion kernel.
Cara
CiteSeer
GH BD
Nalve Ranstam
Figure 7: Results (mean and standard deviation over 10 runs) for latent graph inference datasets.

---

5 Conclusion
In this work, we have introduced neural latent geometry search (NLGGS), novel problem formulation that consists in finding the optimal latent space geometry of machine learning algorithms using minimal query evaluations. In particular, based on Cartesian products of constant curvature model spaces. To find the optimal product manifold, we propose using Bayesian Optimization over a graph search space. The graph constructed based on a principled measure of similarity, utilizing the Gromov- Haandooff distance from metric geometry. The effectiveness of the proposed method is demonstrated through our experiments conducted on variety of tasks, based on deesigeed synthetic and real-world datasets.
have modeled the latent space using product manifolds
L Limitations and Future Work. While the NLGS framework is general, we have restricted ourselves to using product manifolds of constant curvature model spaces to model the geometry of the latent space. Furthermore, we have only considered curvatures {- 1, 0, 1], and model spaces of dimension two in order for the optimization problem to be tractable. In future research, there is potential to explore alternative approaches for modeling the latent space manifold. Additionally, the field of Bayesian Optimization over graphs is still in its early stages. Incorporating recent advancements in the area of kernels on graphs (Borovitskiy et al., 2021 Zhi al., 2023 could lead to improved performance. In our current research, our emphasis is on introducing NLGS and providing an initial solution under a set of simplifying assumptions related to the potential latent manifolds available and the optimization algorithm. The investigation of the impact of using different similarity measures to compare latent structures also remains subject for future research.
Socictal Impact Statement
This work is unlikely to result in any harmful societal repercussions. Its primary potential lies in its ability to enhance and advance existing data modelling and machine learning methods.
Acknowledgement
AA thanks the Rafael del Pino Foundation for financial support. AA and HSOB thank the Oxford-Mar Institute for computational resources. AA acknowledges the G-Research grant for travel assistance. XD acknowledges support from the Oxford-Man Institute and the EPSRC (FP/T023333/1).
References
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. arXiv preprint 1rXiv:1710.11 1379,, 2017.
Federico Barbero, Cristian Bodnar, Haitz Sáez de Ocáriz Borde, Michael Bronstein, Petar Veličković, and Pietro Liò. Sheaf neural networks with connection laplacians. In Topological, Algebraic and Geometric Learning Workshops 2022, pages 28-36. PMLR, 2022a.
Federico Barbero, Cristian Bodnar, Haitz Sáez. de Ocáriz Borde, and Pietro Liò. Sheaf attention networks. In NeurlPS 2022 Workshop on Symmetry and Geometry in Neural Representations, 2022b.
Danilo Blanuša. Über die Einbettung hyperbolischer Räume in euklidische Räume. Monatsh. Math., 59:217-229, 1955.
Viacheslav Borovitskiy, Iskander Azangulov, Alexander Terenin, Peter Mostowsky, Mare Deisenroth, and Nicolas Durrande. Matérn gaussian processes graphs. In International Conference Artificial Intelligence and Staristics, pages 2593 2593-2601. PMLR, 2021.
Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and A. Doucet. Riemannian score-based generative modeling. ArXiv, abs/2202.02763. 2022.
David Brander. Isometric embeddings between space forms. 2003. Master thesis.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesies, and gauges, 2021.
10

---

Benjamin Chamberlain, James Rowbottom, Davide Eynard, Francesco Di Giovanni, Xiaowen Dong. and Michael Bronstein. Beltrami flow and neural diffusion on graphs. Advances in Neural Information Processing Systems, 34: 14:1594-609, 2021.
Ines Chami, Rex Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks, 2019.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-2926. IEEE, 2017.
Giacomo Como, Raffaele Damiano, and Fabio Fagnani. Discrete bayesian optimization algorithms and applications. 2020.
Jiaxu Cui and Bo Yang. Graph bayesian optimization: Algorithms, evaluations and applications. ArXiv, abs/1805. 10ss/1005011,,, 2018.
Jiaxu Cui, Bo Yang, Bingyi Sun, Xia Ben Hu, and Jiming Liu. Scalable and parallel deep bayesian optimization on attributed graphs. IEEE Transactions on Neural Networks and Learning Systems, 33:103-116, 2020.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-14,, 2012.
Yilun Du, Katherine Collins, Joshua Tenenbaum, and Vincent Sitzmann. Learning manifolds of neural fields. In Neural Information Processing Systems, 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, J. Mach. Learn. Res., 20:55:1-55:: 2018.
Charles Fefferman, Sanjoy K. Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. arXiv: Statistics Theory, 2013.
Marco Fumero, Luca Di Cosmo, Simone Melzi, and Emanuele Rodolà. Learning disentangled representations via product manifold projection. ArXiv, abs/2103.01633 2021.
C Lee Giles, Kurt D Bollacker, and Steve Lawrence. Citeseer: An automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries, pages 89-98, 1998.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Dhananjay Gopal, Aniruddha V. Deshmukh, Abhay S. Ranadive, and Shubham Yadav. An introduc- tion to metric spaces. 2020.
Albert Gu, Frederic Sala, Beliz Gunel, and Christopher Ré. Learning mixed-curvature representations in product spaces. In International Conference on Learning Representations, 2015 88
Søren Hauberg, Oren Freifeld, and Michael Black. A geometric take on metric learning. Advances in Neural Information Processing Systems, 25, 2012.
Wolfgang Henke and Wolfgang Nettekoven. The hyperbolic n-space graph in Euclidean (6n 6)-space. Manuscripta Math., 19()),3320, 1987.
Felix Hensel, Michael Moor, and Bastian Alexander Rieck. A survey of topological machine learning methods. Frontiers in Artificial Intelligence, 2021.
Chin-Wei Huang, Milad Aghajohari, A. Bose, P. Panangaden, and Aaron C. Courville. Riemannian diffusion models. ArXiv, 0as/2208.009499 2022.
Paul Jungeblut, Linda Kleist, and Tillmann Miltzow. The complexity of the hausdorff distance. In International Symposium Computational Geometry, 2021.
Anees Kazi, Luca Cosmo, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael Bronstein. Differ- entiable graph module (DGM) for graph convolutional networks. IEEE Trunsactions on Puttern Analysis and Machine Intelligence, pages 1-1, 2022.

---

Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 6418-6428, 2020.
Risi Kondor and Jean- Phiilipppp Vert. Diffusion kernels. kernel methods in computational biology, pages 171-192, 2004.
Wouter Kool, Herke van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement, 2019.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar- dataset. online: http:://ww.. cs.
toronto. edu/kriz/cifrr html, 55(5), 2014.
Shane Lubold, Arun G Chandrasekhar, and Tyler H McCormick. Identifying the latent space geometry of network models through analysis of curvature. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(2):240-29) 292, 2023.
Lizheng Ma. Jiaxu Cui, and Bo Yang. Deep neural architecture search with deep graph bayesian optimization. 2019 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pages 500-507, 2019.
Emile Mathieu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee Whye Teh. Contin- uous hierarchical representations with poincaré variational auto-encoders. Advances in neural information processing systems, 32, 2019.
Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of bayesian methods for seeking the extremum. Towards global optimization, 2(117- 129)222 1978.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International conference machine learning, pages 7045-7054. PMLR, 2020.
Changyong Oh, Jakub Tomezak, Efstratios Gavves, and Max Welling, Combinatorial bayesian
optimization using the graph cartesian product. Advances in Neural Information Processing Systems, 32, 2019.
David Pfau, Irina Higgins, Aleksandar Botev, and Sébastien Racanière. Disentangling by subspace diffusion. ArXiv, abs/2006. 12982, 2020.
Hicu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. ArXiv, abs/ abs//180,.0326,, 2018
Sam T. Roweis and Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embed- ding. Science, 2000.
Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborne. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kemels. arXiv preprint arX1v:2006.07556, 2020.
Haitz Sáez de Ocáriz Borde, Alvaro Arroyo, and Ingmar Posner. Projections of model spaces for latent graph inference. In ICLR 2023 Workshop on Physics for Machine Learning, 2023a.
Haitz Sáez de Ocáriz Borde, Anees Kazi, Federico Barbero, and Pietro Lio. Latent graph inference using product manifolds. The Eleventh International Conference on Learning Representations, 2023b.
Prithviraj Sen, Galilco Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad Collective classification in network data. Al magazine, 29(3):93 90,,993,330 2008
Ankita Shukla. Shagun Uppal. Sarthak Bhagat, Saket Anand, and Pavan K. Turaga. Geometry deep generative models for disentangled representations. Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing, 2018.
Ondrej Skopek, Octavian-Fugen Ganea, and Gary Bécigneul. Mixed-curvature variational autoen- coders. arXiv preprint 4rX1v:1911.08411, 2019.
12

---

Alexander Smola and Risi Kondor. Kermels and regularization graphs. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, pages 144-158. Springer, 2003.
Abdel Aziz Taha and Allan Hanbury. An efficient algorithm for calculating the exact hausdorff distance. IEEE Transactions Pattern Analysis and Machine Intelligence, 37:2153-2211 2163, 2015.
Joshua B. Tenenbaum, Vin De Silva, and John C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 2000.
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. ArXiv, abs/21 00s/2111.14522,, 2021.
X. Wan, H. Kenlay, B. Ru, A. Blaas, M. A. Oshorne, and X. Dong. Adversarial attacks on graph classifiers via bayesian optimisation. Conference on Neural Information Processing Systems, 2021.
Melanie Weber. Curvature and representation learning: Identifying embedding spaces for relational data. 2019.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. MIT press Cambridge, MA, 2006.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint urXiv:1708.07744 2017.
Sharon Zhang, Amit Moscovich, and Amit Singer. Product manifold learning. International Conference on Artificial Intelligence and Statistics, 2020.
Deli Zhao, Jiapeng Zhu, and Bo Zhang. Latent variables spheres for autoencoders in high dimensions. arXiv: Learning, 2019.
Yin-Cong Zhi, Yin Cheng Ng, and Xiaowen Dong. Gaussian processes on graphs via spectral kemel learning. IEEE Transactions on Signal and Information Processing over Networks, 2023.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. ArXiv, Abs/1611.01178 2016.
13

---

A Additional Background
A.1 Differential Geometry, Riemmanian Manifolds, and Product Manifolds
Differential geometry is mathematical discipline that employs rigorous mathematical techniques, including calculus and other mathematical tools, to investigate the properties and structure of smooth manifolds. These manifolds, despite bearing local resemblance to Euclidean space, may exhibit global topological features that distinguish them from Euclidean space. The fundamental objective of differential geometry is to elucidate the geometric properties of these spaces, such curvature, volume, and length, through rigorous mathematical analysis and inquiry.
A Riemannian manifold generalizes Euclidean space to more curved spaces, and it plays a critical role differential geometry and general relativity. A Riemannian manifold a smooth manifold M equipped with Riemannian metric g. which assigns to each point p in M an inner product on the tangent space M. This inner product a smoothly varying positive definite bilinear form on the tangent space given by Ip T.M x T.M - R, which satisfies the following properties: symmetry, go(X, Y)= = gY, X) for all X,Y E T,M: linearity, 9p (aX + bY, 2) = (X, 2) + bgg 2) for all Y, Z E T,M and scalars a, b E I; positive definiteness: 9p (X, > 0 for all non-zero X M. The metric Op induces norm ||-Ilp on T,,M given by ||x||,
9p(p,, X). This norm allows us to define the length of curves on the manifold, and hence a notion of distance between points. Specifically, given a curve c: [0. 1] .M and a partition 0 = to where c'(t.) denotes the tangent vector e at time &g and || ||e(t,) denotes the norm induced by the metrie at the point c(t,). The total length of the curve is obtained by taking the limit of the sum as the partition becomes finer.
bn = 1, the
length of the curve с over the partition is given by: L(c) = [[c(((t))lccd
On the other hand, product manifold results from taking the Cartesian product of two more manifolds, resulting in manifold with a natural product structure. This enables researchers to examine the local and global geometry of the product manifold by studying the geometry of its individual factors. Riemannian manifolds and product manifolds are essential mathematical concepts that have far-reaching applications in diverse fields such physies, engineering, and computer science. In the appendix of this paper, we provide a more in-depth look at these concepts, including their properties and significance.
A.2 Constant Curvature Model Spaces
manifold M is topological space that can be locally identified with Euclidean space using smooth maps. In Riemannian geometry, a Riemannian manifold or Riemannian space (M.g) is real and differentiable manifold M where each tangent space has an associated inner product g known as the Riemannian metric. This metric varies smoothly from point to point on the manifold and allows for the definition of geometric properties such as lengths of curves, curvature, and angles. The Riemannian manifold M c N is collection of real vectors that is locally similar to linear space and exists within the larger ambient space RV
Curvature is effectively a measure of geodesic dispersion. When there is no curvature geodesics stay parallel, with negative curvature they diverge, and with positive curvature they converge. Constant curvature model spaces are Riemannian manifolds with a constant sectional curvature, which means that the curvature is constant in all directions in 2D plane (this can be generalized higher dimensions). These spaces include the Fuclidean space, the hyperbolic space, and the sphere. The Euclidean space has zero curvature, while the hyperbolic space has negative curvature and the sphere has positive curvature. Constant curvature model spaces are often used as reference spaces for comparison i geometric analysis and machine learning on oun-uuciidean data.
Euclidean space,
Eds Rd
(8)
flat space with curvature KE = O. We note that in this context, the notation de used to represent the dimensionality. In contrast, hyperbolic and spherical spaces possess negative and positive curvature, respectively. We define hyperboloids Hd
14

---

IHid KR
[xp E Rda+1 (Xp: xp>c = 1/Km):
(9)
where Kx < and (, e the Lorentz inner product
(x,y)c= -111
Σ
3zbl+ Vx R
(10)
j-2
and hyperspheres 8 as
S
[xp E Rd
(xp: Xp>2 = 1/Ka):
(11)
where Kş > 0 and (3,-2 is the standard Euclidean inner product
d Σ
(x,y)2
FJby Vx,y E
(12)
Table Sanee2summi summarizes the key operators Euclidean, hyperbolic, and spherical spaces with arbitrary curvatures. It is worth noting that the three model spaces discussed above can cover any curvature value within the range of (-oc, 20). However, there is potential issue with the hyperboloid and hypersphere becoming divergent as their respective curvatures approach zero. This results in the manifolds becoming flat, and their distance and metric tensors do become Euclidean at zero- curvature points, which could hinder curvature learning. Therefore, stereographic projections are considered be suitable alternatives these spaces they maintain oun-lucide structure and inherit many of the properties of the hyperboloid and hypersphere.
Table 2: Relevant operators (exponential maps and distances between two points) in Euclidean, hyperbolic, and spherical spaces with arbitrary constant curvatures.
Space Model
expx, (x)
d(x,y' y)
E. Euclidean
Xy
||x y112 arccosh ()(()))))
III hyperboloid
cosh
xp sinh (v (v-KEl v-Kulixi
6, hypersphere
(v Kg
xxp sin (</g|>>) /Ka//x||
arccos (Ks(x.y)2)
Kks
A.3 Constructing Product Manifolds
In this appendix, we provide additional details about the generation product manifolds. In mathematics, product manifold is type of manifold obtained by taking the Cartesian product of two or more manifolds. Informally, a product manifold can be thought of as a space formed by taking multiple smaller spaces and merging them in way that maintains their individual structures. For instance, the surface sphere can be seen as a product manifold consisting of two real lines that intersect at each point the sphere. Product manifolds play crucial role in various fields of mathematics, including physics, topology, and geometry.
A product manifold can be defined as the Cartesian product:
Pp P
X
M
(13)
Ki
1=1
Here, K; and represent the curvature and dimensionality of the space, respectively. Points x, E P expressed using their coordinates:
15

---

Xp - concat
(x(")
x
x
M
(14)
Also, the metric of the product manifold decomposes into the sum of the constituent metries
PyP
9P
Dh+
(15)
hence, (P. gp) is also Riemannian manifold. It should be noted that the signature of the product space parametrized with several degrees of freedom. These include the number of components used in the product space, the type of model spaces employed, their dimensionality, and curvature. If
df and hyperspheres &
restrict P to be composed of the Euclidean plane E hyperboloids H ka
of constant curvature, we can rewrite Equation 13
Yig
P
Edg
(16)
where K3 = 0, K < O, and K > 0. Hence, P would have a total of 1 + ny + ng component
spaces, and total dimension dg + == di + === d Distances between two points in the product manifold can be computed aggregating the distance contributions D from each manifold composing the product manifold:
Dp ( Xpa)
Σ
0
(17)
The overline denotes that the points Xp1 and Xp2 have been adequately projected on to the product manifold using the exponential map before computing the distances.
A.4
The Rationale Behind using Product Manifolds Model Spaces
Most arbitrary Riemannian manifolds do not have closed-form solutions for their exponential maps and geodesic distances between points, as well as for other relevant mathematical notions. The exponential map takes point on the manifold and exponentiates a tangent vector at that point to produce another point on the manifold. The geodesic distance between two points on a manifold the shortest path along the manifold connecting those points.
For simple, well-studied manifolds like Euclidean space, hyperbolic space, and spherical space, closed-form solutions for exponential maps and geodesic distances are available. This is because these manifolds have constant curvature, which allows for more straightforward calculations. However, for most other manifolds, the exponential map and geodesic distances must typically be computed numerically or approximated using specialized algorithms. These computations often involve solving differential equations, and depending on the manifold's 's curvature and geometry, these solutions can be quite complex and may not have closed-form expressions. For certain types of manifolds, like product manifolds where each factor is well-understood manifold, the computation of exponential maps and geodesic distances can sometimes be simplified due to the separability the metric. However, scaling these methods to more general, complex manifolds can be challenging and computationally intensive.
As mentioned, closed-form solutions for exponential maps and geodesic distances are not common for arbitrary Riemannian manifolds, and computational methods often rely on numerical techniques and algorithms due to the complexity of these calculations. These properties have made product manifolds the most attractive tool endow the latent space with richer structure while retaining computational traceability, see [Gu et al. 2018 Skopek et al., 2019 Sāez de Ocáriz Borde et al. 20233 Zhang jt al., 2020, Fumeer et al. 2021
Pfau et al.) 2020]. This motivates us to use product manifolds in our setup, which keeps our work in line with the state-of-the-art approaches in the literature. We highlight that one of the central contributions of our paper is to shed light on the problem of NI.GS, and provide initial solution to the problem in the context of the current of-t-oofthhearr in geometric
16

---

latent space modelling. Future research avenues could explore generalizations of this which account for new techniques that emerge in the literature, potentially comparing a more general set of latent manifolds.
A.5 Geometry of Negative Curvature: Further Discussion
Lastly, in this appendix, we aim to provide an intuition regarding closed balls in spaces of curvature smaller than zero. The hyperbolic spaces we consider in this article are hyperbolie manifolds (that is, hyperbolic in the differentiable sense). However, there are other generalisations of the notion of hyperbolicity that carry over the setting of geodesic metric spaces (such as Gromov hyperbolicity). This theory unifies the two extremes of spaces of negative curvature, the real hyperbolic spaces H" and trees. Recall that a tree is a simple graph that does not have closed paths. Trees provide great intuition for the metric phenomena that we should expect in spaces of negative curvature. For example, let denote by G' an infinite planar grid (observe that this graph is far from being tree), and let us denote by I' a regular tree of valency 4 (this means that every vertex in T is connected to exactly other four vertices). The 2-dimensional Euclidean space E²² is quasi-isometric G and the 2- 2-dimensional real hyperbolic space IH! is quasi-isometric (but not isometric) to T'. This observation is due to Svarc and Milnor. Recall that the graphs G and T are endowed with the natural path metric. A ball of radius n >> in G has between n² and 4n² points (polynomial in n1), while a ball of radius in T has exactly (4" 1)/3 points (exponential in n). Now let us come back to HP, where the "continuous" analogue of this principle also true; that is, that there exists a constant e >> such that for every n > O, one needs at least c" balls of radius one to cover a ball of radius n in H3 If we only want understand the local geometry of H3, we can re-scale this principle to say the following. Let BF" denote the ball of radius one in H For all positive integers n > O, one needs at least c" balls of radius 1 1/n to cover Bfia. This tree-like behaviour should be compared with the opposite principle that holds in E3, being that for all
0, one needs most 8n3 (a polynomial bound) of balls of radius one to cover BT Essentially, we should think of 3 as manifold that is isomorphic to BT which is difficult to cover by balls of smaller radius, in fact, it will be coarsely approximated by finite tree of balls of small radius (the underlying tree being bigger as the chosen radius smaller). Furthermore, the philosophy that travelling around B, becomes more and more expensive as we leave the origin is made accurate with the consideration of divergence functions: geodesics in IHI3 witness exponential divergence. More precisely, there exists a constant e > 1 such that for every > O and for every geodesic 7 R - H passing through the origin at time L = 0 it is true that if a is a path that connects 7 and y(r) and that lies entirely outside of the ball of radius d(0, 7r)),, then must have length at least c". Again, this is better appreciated when put in comparison with the Euclidean space IE3, Any geodesic (any line) y IR - EB' passing through the origin at time t = 0 will have the property that for all r > 0 there is path of length πर + E, with e > 0 arbitrarily small, that lies entirely outside of the ball of radius r with centre at the origin. The length xr + e is linear in r, as opposed to the exponential length that is required in hyperbolic spaces.
B Further Details on the Gromov-Hausdort Algorithm for Neural Latent Geometry Search
As discussed in Section 3 there are limitations to the Hausdorff distance metric for comparing point sets that represent discretized versions of continuous manifolds. The original algorithm by Taha and Hanbury [2015) for computing the Hausdorff distance assumes that the point sets reside in the same space with the same dimensionality, which is limiting requirement. However, the Gromov- Hausdorff distance proposed in this work allows to compare metric spaces that are not embedded in common ambient space, but it is not exactly computable. We will focus on providing an upper bound for deH (E", SS) and describing an algorithm for obtaining upper bounds for dcH (E", H") and dc HD) too. As previously mentioned, E" and S" can be isometrically embedded into R6x-6 in many ways, which provides a common underlying space for computing Hausdorff distances between E*, $", and " The embeddings of different spaces require making choices: in particular, distance given
there no way to exactly compute the infimum the definition of in Section 2
17

---

dG B)
inf (X.S.))ES((ABB)
H xx ((()))))((B))))
(18)
so we resort to numerical approximations which are later described in this appendix.
To estimate the mutual Gromov-Hausdorff distance between the infinite smooth spaces E² and III!, the first step is to approximate them using finite discrete versions. This is done by considering cell-distributed collections of points in E² and H², obtained by applying the exponential map to collection of points in IR?. Multiple isometric embeddings of E² into R are also considered. The estimation is obtained by computing the minimum of the Hausdorff distance between the point sets obtained from the collections TR and the isometric embedding of HH into R The same applies for spherical space.
B.1 Generating Points in the Corresponding Balls of Radius One
All our computations will be done in dimension two, so we will simply precise how to generate points in Bga, Bga and Bga. For Bga and Bsa, will use very elementary trigonometry. For Bga, we will need an explicit description of the exponential map of IH?. Using the descriptions
B2 ((cos((),rrsn(())) r [0, 11. ( [0,2x)]
(19)
and
Bga ([sin(3) cos(a),, (i) sin(a), cos(3)) [0, 2m), 8 [0, 1]},
(20)
it will be easy generate collections of points in Bg and Bga. As we anticipated above in the outline of our strategy to estimate dcH (cH(Bgz, Bgz), order to give explicit well-distributed collection of points in Bga, it is enough to give well-distributed collection of points in Bz= and consider its image under the exponential map expo Bga - Bga where we have identified Bga with the ball of radius one of R² = ToH2, i.. the tangent space of HH at the point O. However, should remark that our description of H will not be any of the three most standard ones: namely the hyperboloid model, the the Poincaré ball model, or the upper half half-plane model. We describe H as the diffcrentiable manifold Rn (of Cartesian coordinates x, y1..... y/ 1/-11) equipped with the Riemannian metric g-1 dx.2 e2x (dyf + +... ....dy?-1) 1)- This metric is complete and has constant sectional curvature -1, which implies that (R", 9-11) is isometric to H In the hyperboloid model of H2, we view this space
submanifold of R (although with distorted metric, not the one
induced from the ambient RR)). In this model of H one can explicitly describe by the following assignment expo IR2 1 by
x siihh(( / x² y²) ysinh( v x² + +2²) + y²
(x, -
cosh / x2 + y²)
(21)
V y²
We use this explicit formula the exponential map with coordinates in the hyperboloid model of HH, give explicit formula for the exponential map with coordinates in the model of H that introduced above, denoted by (IR2,g-1) --)).. In order to change coordinates from the hyperboloid model the following isometry (which is well-known and can be thought
to the Poincaré ball model, of as a hyperbolic version of the stereographic projection from S to RR:::
(1
y
(a, y, 2)
(1+2'
(22)
To change coordinates from the Poincaré ball model to the upper half-plane model, one can use the following isometry:
2y
1 x 322
(x, y) +
(23)
(x
1)2 + y²
(x - 1)2 + 32
18

---

The previous assignment comes from the standard Möbius transformation 2 s+1 that identifies the Euclidean ball of radius one of C and the upper half plane of C as Riemann surfaces (which we give explicitly in this case, although it is known exist and be unique by the classical Riemann mapping theorem). Finally, to go from coordinates in the upper half-plane model of H to our model (R*,g-1), we use the isometry
s=+1
(a,y) (- log y,, 2)).
(24)
B.2
Embedding of H" into E6n-6
We want to define an isometric embedding of B into E6n-6 (F' from Section 3.2). This higher dimensional space is our candidate to fit in the three geometries E²,, H and S to compute their Hausdorff dimensions as subspaces of 1g and hence estimate their Gromov-HHusdond distances. Before describing such embedding,
introduce several preliminary auxiliary functions.
Let x(t) sin(xt) e --sinn (xx)" for non-integer values of t. A priori, the inverse of sin does not make sense but since lim,- x(t = lim,-o- x(t) = 0, we can set x() = 0 so it still continuous. In fact, it is smooth and, in particular, integrable. We can say the same at all points when t is an integer, so we set x(t) O for all integers t and we obtain an smooth function x defined on I.
x(t)dl.
(25)
We also define
1 p++
v1(() = V A Jo x(t) dt,
(26)
and
v2((x) V
J. dt.
(27)
We set to be the constant
2 max (G1; G2)
(28)
defined in terms of the following
G
(sinh(z) - 1/1 (z()
(29)
dæ
L-2-2
d (sinh(x) x2((x)) [=[-22]
G2 da
(30)
Mx, y) sinh(x) 4/1 (x)( (x) cos(c- y), th (α) sin(c- v), viz(x) cos(c y), v2(xx) (2(())ss)nn sin(c.
sin(c-y)). v)).
(31)
((5,v) v) (sinh- log( 2e + y2)
(32)
We also define fo(x, 3) ( (iin "(ve2) v1 - =(t))ddt, /e 2x + 8²) h ((1(2,,,))) v))
with being
G + G2
(33)
22
19

---

This way. we can set
1
[a, (1.......... Vn-11
(Jo(2, vn. - 1,),,-..... fo(2, vn 1in-1).
Recall that our case, we use the function F(.) f(n 2,-) to map the points Q in H22 to Re and for computing
dGH( Bux) x min 8B .f..F (P,, Q5) min d (ft(P. 1.j.k
(34)
tj.k
B.3 Estimation of the Gromoy-Hausdorff distance between the Euclidean and Spherical Spaces
In this section we give the analytical derivation of the Gromov- distance between the Euclidean space EE and the Spherical space S We first explain the case 1, where can visually understand the situation in a better way because all distances will be measured in I.", Afterwards, replicate the same argument for arbitrary n. Recall that the Spherical model 3 can be described as the metric subspace of E=+1 corresponding to the Euclidean sphere of radius one. Hence, subspace of Rn+1 it corresponds to t ( Tn) EE= x,
Figure 8: The one-dimensional model of spherical geometry S1 isometrically embedded in R²
The ball of radius one of S1, denoted by Bg is highlighted in red. Our estimation of the Gromov- Hausdorff distance between E. and is motivated by the following observation. In the y-axis, the red are ranges from [+c0a((1) to 1. If we consider the following blue segment, representing the ball of
Figure 9: Comparison between Bg (in red) and Bg (in blue) inside E..
radius one of ! (denoted by BBg) we get
sup dge(x, Bg:) 1 - cos(1)
0.23.
(35)
ze Rg
2
However, it can be seen that supreBa dga (I, Bg:) 0.279. More generally, if the blue line is chosen to be at the height - x, for some lying the closed interval [0, - cos(1)) it not hard to see that, for such embedding f: Bg! E
=h (f(Ez)) BBi) = max {x. fr,
x, /(1 sin (1))² (1 cos(1) =)33
(36)
20

---

whose minimum is attained exactly when x v(1 sin [)]2 + (1- - 2 cos(1)) ( i.. when
sin (1-
0.257.
(37)
2osc)
This can be seen in the following picture. The two pink lines represent the biggest lengths between
Figure 10: Comparison between Bg (in red) and Bg: (in blue) inside E²2 and a schematic of the biggest lengths (in pink) between the two point sets.
points in Bg and Bgi, whose length is approximately equal to 0.257. Since we cannot compute exactly the why we would expect it to be somewhere in between 0.23 and 0.257. Since we are considering a very simple embedding for these estimations, it is reasonable expect (G' IR1) get closer to the lowest number when considering embeddings of these spaces in more complicated higher dimensional spaces, the definition of the Gromov-Hausdorff distance allows.
distance don(S', E)), choices have to be made. We have discussed
For an arbitrary n, we reduce the estimation of dgH [gg, Bgx) ) to the one-dimensional case as follows. Analogously
did for n 1, given any value of in E*+1 by the map f= Bg - Ra+1 defined by f((2g, .1..... We view B3 isometrically embedded into I " the ball of radius one of S" R with centre in the north pole of the sphere, i.c the point with coordinates (0, ...... 0, 1). Given any unit vector in m=++1 orthogonal plane linearly spanned by u and nT. It is clear that, as u ranges over the orthogonal space of O0, 1), the intersections B3* Th cover the whole Bg and the intersections fz (Bgz) o mT
consider Bga isometrically embedded
(xg, 21 In, x).
(0, -, 0, 1), define nu be the two-dimensional
cover the whole fz (Bgo). Hence, in order to compute dii d'=+1 (fa(Bez). Bg-). it suffices to compute 1 di (fz (Bg.) Tus Bg O Tu) for all unit vectors orthogonal to 7ī. Moreover, for two such unit vectors u and e. there a rigid motion (a rotation) of the whole ambient space 1=+11 that fixes ił and that maps isometrically (Bgg) ) Tu to fz
T5 and Bgn Tu to Bg n.. In particular,
d (B=( n Tu, Bg e π.) = di " (fa( n To. Bg 0 TT).
So we can do the previous computation for the specific value of u (0,0, ..... 1, 1,0) (for clarity, where we mean the unit vector where the 7ith coordinate is equal to and the rest are zero). Crucially, the projection of R unto R² (by projecting onto the last two coordinates) restrict to isometries on Bg O Tu. and Bg o Tu. from where it is deduced, analogously as in 36 that, as long as E [0, cos [)], have the following:
al (Bgx) o Tuy Bgn O nu) = max x, 2, (1- (i-
cos(1) x>3
B.4 Computational Implementation of the Gromov-Hausdorff Distance between the Remaining Model Spaces of Constant Curvature
In this appendix, we provide further details regarding how the Gromov-Hausdone distances between the remaining manifolds (between Bga and By,, and Bga and Bg²) approximated computa- tionally. Note that as discussed in Section 3.3, these distances will be leveraged to compare not only model spaces but product manifolds as well.
B.4.1 Discretizing the Original Continuous Manifolds
This section discuss how the the points the corresponding balls of radius one described Ap- pendix B.I are generated from practical perspective. The Euclidean, hyperbolic, and spherical
21

---

spaces are continuous manifolds. To proximate the Sromov-Hausdeee distance between them we must discretize the spaces. A more fine-grained discretization results better approximation. As previously mentioned in Section 2 to compare IR,, S", and IH", we can take closed balls of radius one in each space. Since these spaces are homogeneous Riemannian manifolds, every ball of radius one is isometric. We can estimate or provide an upper bound for their Gromov- distance by using this method.
In the case of the Euclidean plane, we sample points from the ball, Bga {(r cos(t), sin((t)) : E [0, [o, 2 with discretization of 10, 000 points in both r and t. To obtain points in Bya we will use the points in Bş² as a reference, and apply the exponential map, and the change of coordinates described in Appendix [BI to convert points in B to points in Bga However, due to numerical instabilities instead of sampling from Bga, we will restrict ourselves to By = {(r cos(t), r sin(t)) E [0. (0.00000001, 0.97], / [0, 2T)} and use a discretization of 10, 000 as before. Lastly to sample points for the spherical space, Bga {(sin(3) cos(a), sin(.) (.9) sin(a)) cos( 1]3, we use a discretization of 100 for both a and J3. The granualirity of the discretization was chosen as a trade-off between resolution and computational time. We observed that the Gromov- Hausdorff distance stabilized for our discretization. However, given the nature of the Gromov- Hausdorff distance, it is difficult to conclude whether some unexpected behaviour could be observed with greater discretization.
E [0, 2m), f
B.4.2 Calculating Constants for the Embedding Function
The next step is obtain computational approximation of the mapping function defined in Ap- pendix B.2 The embedding of H into gon-6 requires computing several constants. Using standard integral solvers A 0.141328, (Equation 25) can be approximated. To calculate the constant c x 20.2250144522444
we discretize the inpt E [- using step size of 10 to compute G1 and G2 and use a for loop to calculate the max in Equation 28 Note that this requires to constantly reevaluate the integrals for (xr) and (zr).. Likewise, G1 and G2 are also used to obtain
& in Equation 33,, which is turn used to calculate the mapping function that maps points in H² to
the higher dimensional embedding Euclidean space E We use F' to map Bga to E and we keep those points fixed in space.
B.4.3 Optimizing the Embedding Functions for the Euclidean and Spherical Spaces
Next to approximate the Fromov-Hausdorfe distances:
do (Bg, Bax) & min F (P!l = min di (F((P))
(38)
1.3.k H
and,
dcn( BBY) x min d .... (PS, Q2) = min di (P.),, i.j.
(39)
i,j.k
we must optimize far and gk. These are the functions used to embed Bga and Bga in Bon-6 Note that in practice, Bga and Bga are discretized into P. and P... respectively.
To optimize for fx we consider all possible permutations of the basis vectors of E (es,ez) ey) and use those dimensions should also consider a small offset given by F(O) 0, but it is
(e1, 62, e3; 64, es, [6] for each fa we consider two elements {ez,
to embed IE² in 156 In principle, zero regardless. Additionally, during the optimization we also add a small vector (with all entries but single dimension between zero) to the mapping function to translate the plane in different directions by an offset of between - 0.5 and 0.5, with a total of 100 steps between these two quantities.
To optimize for gle we follow a similar procedure in which we consider all permutations of the basis vectors. Note however, that this case we would have three basis vectors instead of two, given how Bga (3)) cos(a), (3) sin(a), sis(a) cos(3))
E [0, 2T,, 3 E [0. 1]} is sampled. For each permutation family, Pš. we also consider its negative counterpart, ps to compute the distance well experimenting with offsetting the mapping function.
22

---

B.5 Derivation of Number of Product Manifold Combinations in the Search Space
Here, we derive the exact number of nodes in the graph search space in our setting. In particular, we consider the case with three model spaces (the Eucledian plane, the hyperboloid and the hypershere). The growth of the search space can be modelled with growth of tree, as depicted in Figure
E
HI
S"
E" X E"
E" H"
E" X S" H
X H
H X S"
S" X S"
Figure 11: The growth of the graph search space as function of the number of model spaces used to generate product manifolds can be represented as a tree. Note that as discussed in Section 3.3 we assume commutativity: M, M, M2 x M,.
To calculate the number of elements in the search space, we define the total number of products at level of the number of Euclidean spaces added to the product manifolds at depth of the tree, and Nn) and (s)h) represent the same for the hyperboloid and the hypersphere respectively, By recursion, can write
N. We have that (h) ( + NNr Ns(h,, where NE)h) is the
NE(h) Ng 1)
(40)
(a NE(h 1) + Nu 1)
(41)
= =1th
(42)
Ns NE(h - 1) + Nu 1) + Ns(h - 1)
(43)
h(h+1) + 1) NE(1) + hl,(11 + Ns(1)
(44)
2
1(h+ 1)
+ h
(45)
2
hence we have that
h(h + 1)
N(h) 1 + (1 h) + (1
(46)
5
1
= 3 + h +
h
(47)
2 2
To be consistent with the previous notation, we write the above in terms of the number of products ny
N(n)) 3+
2 nD +
2 mp
(48)
The total number of nodes in the graph search space for number of product np is then
NT
N(1
(49)
B.6 Visualizing the Graph Search Space
In this appendix, we give visual depiction of the graph search space we construct for neural latent geometry search. Figure 12 and Figure 13 provide plots of the graph search space as size of product manifolds considered increases. For product manifolds composed of high number of model spaces, visualizing the search space becomes difficult. We omit the strengths of the connections for visual clarity in this plots.
23

---

If we focus on Figure 12 and S?, In this work, we have considered model spaces of dimensionality two, but the graph search space would be analogous if we were to change the dimensionality of the model spaces. Nodes that include more than one letter, such as hh, se, ee, etc, refer to product manifolds. For example, hh would correspond to the product manifold H HH2 As discussed in Section 3.3 we can see that connections are only allowed between product manifolds that differ by a single model space. To try to clarify this further, we a single model space (one deletion and one addition). Likewise, connected to ee, eh, and ee since they only differ by single model space (one addition). However, is not connected hs (one deletion and two additions) nor ee connected to ss (two deletions and two additions).
we use h, and s to refer to the model spaces of constant curvature E², IH?
that h and are all interconnected since they only differ by
Figure 12: Graph search spaces for np = 2 (left) and np = 3 (right). Strength connectivity is not depicted in the graph.
Figure 13: Graph search spaces for np 4 (left) and ny (right). Strength of connectivity is not depicted in the graph.

---

Figure 14: Graph search spaces for ny 8.
Figure 15: Graph search spaces for n, = 10.
25

---

Figure 16: Graph search spaces for np = 12.
Figure 17: Graph search spaces for n, = 14.
26

---

B.7 Motivating Gromov-Hauddorff Distances for Comparing Latent Geometries
The use of Fromov--Hasdooe distances can be motivated from theoretical and practical perspective. From a theoretical perspective, the Sromov-Hausdorf distance offers a way to gauge the similarity between metric spaces. This is achieved by casting their representation within a shared space, all the while maintaining the original pairwise distances. In the context of machine learning models, particularly those involving generative models (such as VAFs or GANs), the latent space assumes a pivotal role in shaping the characteristics of the generated data. This also holds true in the case of reconstruction tasks, such as those carried out by autoencoders. Through the application of a metric that factors in their inherent geometries, the aim is to capture the concept of resemblance in data generation, reconstruction, and other downstream tasks which will be heavily affected by the choice of latent geometry.
While traditional metrics like Euclidean or cosine distances might suffice for some cases, they do not always capture the complex and nonlinear relationships between data points in high- nigh-dimensional spaces. The Crromov-Hausdovv distance considers the overall structure and shape of these spaces, rather than just individual distances, which can lead to better generalization and robustness. This is especially relevant when dealing with complex data distributions or high-dimensional latent spaces. Moreover, another appealing aspect of Gromov- Hoanov-Hauddorff distance is its invariance to isometric transformations, such as rotations, translations, and reflections. This is a desirable property when comparing latent spaces, as the metric used should reflect the similarity in shape and structure, rather than being influenced by trivial transformations.
While the direct relationship between Haano--aaudorfff distance and model performance might not be immediately obvious, we argue that two models have similar latent space geometries, this suggests that they might capture similar underlying data structures, which could lead to similar performance on downstream tasks. This is known as an assumption of smoothness in the NAS literature, see Oh et al. [201)] However, it is important to note that this relationship is not guaranteed, and the effectiveness Gromov- Hausdoff distance as a proxy measure depends on the specific application and the nature of the models being compared.
B.8 Method Scalability
The proposed approach sidesteps scalability concerns by preventing the need recalibrate GH coefficients, as long as one adheres to latent spaces derived from products of model spaces of dimension two. If higher-dimensional latent space was needed, this could be solved by adding additional "graph slices" with augmented dimensions (as depicted in Figure 2}. These slices can be integrated with the lower-dimensional counterpart of the graph search space following the procedure described in the paper. The number of coefficients remains constant at 3, rendering the Gromov- Hausdorff distances free from extra computation overhead when increasing the dimension.
Moreover, in alignment with the manifold hypothesis, data is expected to exist within a low- dimensional manifold. This discourages the exploration of higher dimensions for latent space modeling. It is important to note, nonetheless, that the paper has already delved into relatively large search spaces, which include the exploration of high-dimensional latent spaces.
C Background on Bayesian Optimization
Bayesian optimization (BO) is a query-based optimization framework for black-box functions that are expensive to evaluate. It builds probabilistic model of the objective function using past queries to automate the selection of meta parameersssserss and minimize computational costs. BO seeks to find the global optimum of black-box function f X E by querying a point X,, X' at each iteration and obtaining the value yn = f(xx) + E, where = ~ (0, o²) is the noise in the observation. To do so, BO employs a surrogate to model the function f(-) being minimized given the input and output pairs D.- = {(x;, y = { N and selects the next point to query by maximizing an acquisition function. In our work, we use the Gaussian Process (GP) Williams and Rasmussen (2066] as the surrogate model and expected improvement (Mokkus et al.) 1978 as the acquisition function.
Preliminaries. Bayesian optimization (BO) is particularly beneficial in problems for which evaluation costly, behave as black box, and for which it impossible to compute gradients with respect to the loss function. This is the case when tuning the hyperparameters of machine learning models. In our
27

---

case, we will use it to find the optimal product manifold signature. Effectively. Bayesian optimization allows us to automate the selection of critical meta- parameters while trying to minimize computational cost. This is done building a probabilistic proxy model for the objective using outcomes recorded in past experiments as training data. More formally, BO tries to find the global optimum of black-box function f X' - R. To do this, point X, E X' is queried at every iteration and yields the value V/ = f(xa) + E, where e ~ [N0,, d²), is a noisy observation of the function evaluation at that point. BO can be phrased using decision theory
L(,, (xx) J N x c+ min f(x)) 1gng
(50)
where N is the maximum number of evaluations, e the cost incurred by each evaluation, and the loss L is minimized by finding a lower f(x). In general, Equation 50 intractable and the closed-form optimum cannot be found, so heuristic approaches must be used. Instead of directly minimizing the loss function, BO employs surrogate model to model the function f(-) being minimized given the input and output pairs D. = {(x,; Y/ = [[==))]== Furthermore, order to select the next point to query, an acquisition function is maximized. In this work, the surrogate model is chosen to be Gaussian Process (GP) [Willaams and Rasmussen 2006) and expected improvement (ED) [Mockus ct al. 1978 is used as the acquisition function.
Gaussian Processes. GP is collection of random variables such that every finite collection of those random variables has a multivariate normal distribution. A GP is fully specified by mean function, p(x), and covariance function (or kernel), denoted as k(x, x'). The prior over the mean function is typically set to zero, and most of the complexity of the model hence stems from the kernel function. Given / iterations of the optimization algorithm, with an input x, = [x1,..., 24 ] and output y; [yn,....uc]] the posterior mean and variance are given by
u( (Dr) k. + 1
(51)
and
( k(stt1, k(r++1, X1) [1. + I (144]="k(x,,tp 5(x1,
(52)
where we define [K:: = M(n, is the ji, element of the Gram matrix.
Expected Improvement. EI is widely used acquisition function for Bayesian optimization. EI improves the objective function through a greedy heuristic which chooses the point which provides the greatest expected improvement over the current best sample point. El is calculated as
(x) ((x) + (V(T())) 1)1.
(53)
where
T(x) = f(xbext) - μx())
(54)
(xx)
and ((-) denotes the CDF of a standard normal distribution.
D Experimental Setup
In this section we provide additional details for the implementation of the experimental setup, including how the datasets are generated, relevant theoretical background, and the hyperparameters used for BO over our discrete graph search space.
D.1 Synthetic Experiments
To assess the effectiveness of the proposed graph search space, we conduct a series of tests using synthetic data. Our objective is to establish a scenario in which we have control over the latent space and manually construct the optimal latent space product manifold. To achieve this, we initiated the process by generating a random vector, denoted as x, and mapping it to predetermined "ground truth" product manifold Pr. This mapping is performed using the exponential map for the product manifold Pr, which can be derived based on the model spaces of constant curvature that constitute it. Subsequently, the resulting projected vector is decoded via a neural network with fixed weights, denoted as fe, to yield a reference signal Pr. The rationale behind employing frozen network is
28

---

to introduce non-linear mapping from the latent space to the signal space. This approach aims to mimic the behavior of a trained network in a downstream task, while disregarding the specific task or model weights involved. By utilizing frozen network, we can capture the essence of non-linear mapping without relying on the exact details of the task or specific model weights. The primary goal is to recover this reference signal using the neural latent geometry search (NLGS) approach. To generate the remaining dataset, we employ the same random vector and mapped it to several other product manifolds, denoted (P; kiepp comprising but with distinct signatures. Decoding the projected vector using the same neural network produces hempp
equivalent number of model spaces as Pr
set of signals, denoted as {y h/mm.- To populate the nodes of the graph search space, assign
the corresponding value of mean squared error (MSE) between y and y for each pair. In this manner, our search algorithm aims identify the node within the graph that minimizes the error, effectively determining the latent geometry capable of recovering the original reference signal. A schematic of the proposed method and specifics on the construction of the network used to decode the signal are shown in Figure 18 and Table 3
P
expx
f()
X
Figure 18: Schematic of procedure used to generate synthetic datasets. We model f as an MLP.
Table 3: Summary of network to generate synthetic datasets.
Model
Linear (data dim, 100) ELU
Linear (100, 100) - ELU
Linear (100, 100) ELU
Linear (100, 5) ELU
D.2 Autoencoders
Autoencoders are a type of neural network architecture that can be used for unsupervised learning tasks. They are composed of two main parts: an encoder and decoder. The encoder takes an input and compresses into now-dimensional representation, while the decoder takes that representation and generates an output that tries to match the original input. The goal of an autoencoder is to learn compressed representation of the input data that captures the most important features, and can be used for tasks such as image denoising, dimensionality reduction, and anomaly detection. They have been used wide range of applications, including natural language processing, computer vision, and audio analysis.
29

---

Table 4: Autoencoder architecture summary.
Component
Layers
Conv2d (1, 20, 3) BatchNorm2d- SiLU
Conv2d (20, 20, 3) BatchNorm2d SiLU
Encoder
(9 repetitions)
Conv2d (20, 2, 3) BatchNorm22 SiLU
Flatten Linear SSLU
Linear SiLU Unflatten
Conv' Transpose2d (2. 20, 3) BatchNNorm2d SiLU
TonnTrraspposed (20, 20, 3) BatchNorm2d SiLUU
Decoder
(9 repetitions)
ConnTraaspose2d (20, 1, 3) BatchNorm22 Sigmoid
The autoencoder's objective is to learn a compressed representation (latent space) the input data and use it to reconstruct the original input as accurately possible. In our experiments, the encoder takes an input image and applies a series convolutional layers with batch normalization and SiLU activation functions, reducing the image's dimensions while increasing the number of filters or feature maps. The final output of the encoder is a tensor of shape (batchsize, lntmntdim, 6, 6) or (batchsize, latentdim, 10, 10), where latentdim is the desired size of the latent space. After flattening this tensor, fully connected layer used to map it to the desired latent space size. The encoder's output is tensor of shape (batchsize, latentdim), which represents the compressed representation of the input image. The decoder takes the latent space representation as input and applies series of transpose convolutional layers with batch normalization and SiLU activation functions, gradually increasing the image's 's dimensions while decreasing the number of feature maps. The final output of the decoder is image tensor of the same shape the input image. The loss functions used for training are the MSELoss (mean squared error) and BCELoss (binary cross-entropy) from the PyTorch library. A summary of the autoencoder is provided in Table
D.3 Latent Graph Inference
Graph Neural Networks (GNNs) leverage the connectivity structure of graph data to achieve state-of- the-art performance in various applications. Most current GNN architectures assume fixed topology of the input graph during training. Research has focused on improving diffusion using different types of GNN layers, but discovering an optimal graph topology that can help diffusion has only recently gained attention. many real-world applications, data may only be accessible as point cloud of data, making it challenging to access the underlying but unknown graph structure. The majority of Geometric Deep Learning research has relied on human annotators or simplistic pre- pre-processing algorithms to generate the graph structure, and the correct graph may often be suboptimal for the task at hand, which may benefit from rewiring. Latent graph inference refers to the process of inferring the underlying graph structure of data when it is not explicitly available. In many real-world applications, data may only be represented as a point cloud, without any knowledge of the graph structure. However, this does not mean that the data is not intrinsically related, and its connectivity can be utilized to make more accurate predictions.
For these experiments reproduce the architectures described in Sáez de Ocáriz Borde et al. 2023b
see Table 5 and Table for the original architectures. our case, we use the GCN-dDGM model leveraging the original input graph inductive bias.
30

---

Table 5: Summary of model architectures for experiments for Cora and CiteSeer.
Model
MLP
GCN
GCN-dDGM
Layer parameters
Activation
Layer type
NIA
N/A
dDGM
[No Taaturs 32)
ELD
Cincar
Graph Coiv""
Graph Cony
NIA
NA
dDGM
(32, 16)
ELU
Lineur
Graph Cunv
Graph Conv
dDGM
(16, 85
ELU
Laneur
Graph Conv""
Graph Conv
8)
Liner
(8, 8)
ELU
Lineur
Linear
Linear
Linear
Table 6: dDGM* and dDGM architectures for Cora and CiteSeer.
dDGM
dDGM
No. Layer parameters
Activation
Layer type
(No. features, 32)
ELU
Linear
Linear
(32, 16 per model space)
ELU Sigmoid
Linear
Graph Conv
(16 per model space, 4 per model space)
Linear
Graph Conv
D.3.1 Differentiable Graph Module
In their work, Kazi et al. [2022] presented a general method for learning the latent graph by leveraging the output features of each layer. They also introduced a technique to optimize the parameters responsible for generating the latent graph. The key concept is to use similarity metric between the latent node features to generate optimal latent graphs for each layer l. In this context, X and A (0) represent the original input node feature matrix and adjacency matrix, respectively. So that
(0)
-X2
(55)
(0)
and A (0) A if the adjacency matrix from the dataset, or (0) If (V, Ø). The proposed
architecture in Kazi et al. [2022] consists of two primary components: the Differentiable Graph
Module (DGM) and the Diffusion Module. The Diffusion Module, x(+1) = I g (!) A ()),, may
be one (or multiple) standard GNN layers. The first DGM module takes the original node features and connectivity information and produces an updated adjacency matrix
(1=) (1=1) DGM(X (0), A
(56)
In principle, the DGM module utilizes information from previous layers to generate adjacency matrices at each layer
X (I+1) A (I+1) DGMM(concat( (1()), x."')),A A
(57)
To do so, measure of similarity is used
((x1!
-(
(58)
summary, the proposed approach utilizes a parameterized function fee with learnable parameters to transform node features and a similarity measure &p compare them. The function can be an MILP or composed of GNN layers if connectivity information is available. The output of the similarity measure is used to create fully-connected weighted adjacency matrix for the continuous
31

---

differentiable graph module (cDGM) approach or a sparse and discrete adjacency matrix for the discrete Differentiable Graph Module (dDGM) approach, with the latter being more computationally efficient and recommended by the authors of the DGM paper (Kazi et al. 2022]. To improve the similarity measure y and construct better latent graphs, the approach employs product spaces and Riemannian geometry. Additionally, an extra loss term is used to update the learnable parameters of the dDGM module.
Lastly, we will examine the dDGM module, which utilizes the Gumbel Top-k [Kool et al. 2019 technique to generate sparse k-degree graph by stochastically sampling edges from the probability matrix (0) (X; ((),,) ,T), which is stochastic relaxation of the kNN rule, where each entry corresponds to
p = exp(->(T)(x") (1+1) (x(+))))))
(59)
T being learnable parameter. The primary similarity measure utilized in Kazi al. 2002 involved computing the distance between the features of two nodes in the graph embedding space
Pij exp(-TA(x, (1+1)
xx x(()))) exp(-
(x ())
(x)
(x))))
(60)
where - denotes a generic measure of distance between two points, They assumed that the latent features laid in an Euclidean plane of constant curvature KE = 0, so that
Pij
(x (x()), S 10 (x) (0)
1)),
(61)
where og is the distance in Euclidean space. Based on
(logo ()) (og(-))(()))
(62)
where q R.N is uniform i.i.d in the interval [0, 17. we can sample the edges
80 (x
() e k) = ((1, (1. (i, ja.) : N},
(63)
being the number of sampled connections using the Gumbel Top-k trick. The Gumbel Top-k approach utilizes the categorical distribution
Pu for sampling, and the resulting unweighted E.
adjacency matrix A (x @,,TT, is used to represent ((X(); ((), T, k). worth noting that including noise in the edge sampling process generate random edges in the latent graphs, which can serve as a form of regularization.
Finally, we can summarize multi-layer GNN using the dDGM module as
(1(+1)_
(!l) (concat(X* (1) xx))), A (1),
(64)
(1+1) p(!) (x(((x((+1))
(65)
((+1) Go g+ (x (1) A (1+1)).
(66)
Equations 64 and 65 belong to the dDGM module, while Equation 66]is associated with the Diffusion Module. In study, extend Equation 59 to measure distances without relying on the assumption utilized in Kazi et al. [2022), which restricts the analysis to fixed-curvature spaces, specifically to Fuclidean space where Kg O.
D.4 Naive Bayesian Optimization
Naive BO in the main text considers performing BO over a fully-connected unweighted graph. Such a graph is latent geometry agnostic, that is, it does not include any metric geometry inductive bias to provide the optimization algorithm with sense of closeness between the candidate latent geometries.
32

---

E
Ablation of the Gromov-Hausdorf Coefficients
We evaluate the impact of Cromov-Hausdorfy coefficients on the graph search space by conducting an additional set of experiments. We compare the use of an unweighted, pruned graph search space with the introduction Gromov- coefficients the datasets considered in this paper. The results are show in Figures 19 to 22 below.
6110
tonties
tonsors
Figure 19: Results (mean and standard deviation 10 runs) for candidate latent geometries involving product manifolds composed of 13 model spaces. For each plot a different ground truth product manifold Py is used generate the reference signal.
64:00
G+
S 80
0 20
Oigi 80
Meseum
Figure 20: Results (mean and standard deviation over 10 runs) for candidate latent geometries involving product manifolds composed of 15 model spaces. For each plot different ground truth product manifold Pr is used to generate the reference signal.
196ST
80
Figure 21: Results (mean and standard deviation over 10 runs) on image reconstruction tasks, for Up 7.
80 00:00
Figure 22: Results (mean and standard deviation over 10 runs) latent graph inference tasks.
The plots above show that the majority of cases, the GH weights appear to either match or enhance the performance of the search algorithm. This observation is particularly pronounced
33

---

in synthetic tasks, which can be intuitively justified due to the significant impact of changes in the latent geometry on the network's output in such a setup. However, we highlight that a large amount of the performance gains in the algorithm seem to come from the inductive bias endowed to the search algorithm through the graph search space. This could due to the fact that in image reconstruction tasks, searching for the optimal dimension is more important for reconstruction than finding the optimal latent geometry within a particular dimension. Since the GH coefficients are unity between dimensions, this makes the performance of the two search spaces similar in this context.
It should be noted that setting the inverse of the Gromov-Hausdorfi distance as the edges of search graph only one potential way add geometric inductive bias into the search space. Future work could use similar mechanisms as those developed in this paper but in a different context in order to improve optimization performance. Finally, we would like highlight that a more thorough analysis of the links between the performance of neural networks and the similarity of their latent geometries a relevant question that merits further study, which left for future work.
34


Reviews:
[{"review_id": "zXyBVGnKeW", "reviewer": "Reviewer_cFVp", "summary": "In the past few years, there's been an interesting trend in the ML literature where people have proposed non-Euclidean latent space geometries to improve network performance. However, to my knowledge geometries have always been picked on an ad hoc basis or tuned naively as a hyperparameter - the most common being hyperbolic and hyperspherical spaces. This evokes a natural question: is there an efficient way to choose the best latent geometry? The authors formulate this choice as a task on its own: neural latent geometry search (NLGS).\n\nThe authors proceed to propose their own method for NLGS. The method chooses a latent geometry from the set of products of up to a fixed number $n_\\mathcal{P}$ of manifolds, where $n_\\mathcal{P}$ is a hyperparameter. At a high level, the authors metrize the set of product manifolds, use this metric to structure a graph over the set, and then use this graph as a search space for Bayesian optimization.\n- The metric is derived from the Gromov-Hausdorff distance between manifolds. The Gromov-Hausdorff distance isometrically embeds two manifolds into a common space ($6n - 6$ dimensional Euclidean space, for this work) and computes the maximum distance from a point on one manifold to the other manifold. Some effort is involved in actually estimating these distance values.\n- The authors then create a graph whose vertices are product manifolds and whose edges connect those product manifolds which differ by a single term in the product. The edges are weighted according to the inverse Gromov-Hausdorff distance.\n- The authors then proceed to use Bayesian optimization on the resulting graph to pick the latent geometry which minimizes the loss for the problem of interest.\n\nThis Gromov-Hausdorff Bayesian optimization (GH BO) method is tested against a naive baseline and Bayesian optimization on a fully connected graph. It outperforms the two methods on both synthetic and real-world experiments.\n", "questions": "What is your opinion on my concern #3? Do you think there's reason to expect GH BO will perform at scale?\n\nCan you please clarify this statement regarding other benchmarks: \"Furthermore, as there are no existing algorithms to compute the distance between arbitrary product manifolds, we have not included these methods in our benchmark as they would simply converge to random search.\"\n- Can you not metrize arbitrary product manifolds using distances within the graph structure you've designed?\n- Why would they converge to random search?", "limitations": "The authors identify *constant curvature* model spaces as a limitation. They also point to limits in the field of Bayesian optimization on graphs.\n\nHowever, I believe some of the simplifications I pointed out in the weaknesses section are also pertinent. It would be helpful to either (1) explicitly raise these as limitations or (2) further justify them as good choices for the algorithm.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- NLGS is an interesting and well-motivated problem. Based on the broader literature's interest in latent geometries, NLGS could end up being a very impactful framework for model tuning.\n  - As a result, GH BO might also end up being a common tool in the hyperparameter search toolbox.\n- The theory and machinery used to develop their Gromov-Hausdorff search space is creative and interesting; this is a very original paper.\n- The whole paper is well-written and easy to follow. At a glance, even the appendices are a nice read.\n- Their approach very clearly empirically outperforms the baselines, which are reasonable in my opinion.\n", "weaknesses": "1. The choice of Gromov-Hausdorff distance as the metric strikes me as a bit arbitrary. Sure, it's *a* way to metrize the model spaces, but I do not think the paper very clearly justifies it as being the right way for the purpose of ML model performance. Why should I expect two models whose latent geometries are closer in Gromov-Hausdorff distance to perform more similarly? As a general principle of optimization, there needs to be some relationship between the objective function and the search space. My concern here is that this relationship is embodied by the structure of the graph (which is discussed for 1 paragraph), rather than the Gromov-Hausdorff weights themselves (which are developed over more than 2 pages + Appendix B).\n   - A natural ablation here would be to test BO with the same graph as is used in GH BO algorithm, except with uniform edge weights of 1 instead of the GH weights. This experiment would test whether the GH weights, and not just the choice of edges, are responsible for performance. Thus far the authors have only ablated against a complete graph.\n\n2. The experiments are not at all reproducible.\n   - The authors did not describe their network architectures. The appendix only contains a vague description of the autoencoder architecture without details on the number of layers, weights, filters, etc. Even fewer details are available for the other experiments.\n   - The authors did not provide their code. Since their method is somewhat involved, this will also hamper adoption by future work.\n\n3. Clearly GH BO performs better than other methods for the models given, but it's not clear whether these are actually *decent models*. As a result, it's unclear to me how much GH BO will help at scale in realistic settings. Clarifying the model architectures used in the experiments might help.\n\n4. On the theoretical side, several simplifications are made (mostly for computional purposes) which make the final algorithm much less interesting:\n   - The search space is restricted to products between manifolds of dimension 2.\n   - Only unitary (-1, 1) curvature values are used for the non-Euclidean model spaces due to the discrete setup, whereas other works tune the curvature through gradient descent (perhaps that can be done post hoc, but it would be nicer if it were integrated into NLGS).\n   - The authors pruned their search graph by removing edges wherein manifold products differ by more than one model space.\n   - As a result, they only needed to compute 3 distinct Gromov-Hausdorff distance values, resulting in only 3 distinct graph weights.\n\nI'm open to raising my score if the authors satisfactorily address concerns #1 and #2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very positive comments regarding our work. We also believe that the concept of NLGS could have far-reaching ramifications in the context of model tuning. We hope that the following explanations and changes will be sufficiently convincing to raise the preliminary score.\n\n**[Weakness 1]** We thank the reviewer for this suggestion.\n\nIn this work, we operate under the assumption that if two models have similar latent space geometries, they might capture similar underlying data structures, which could lead to similar performance on downstream tasks. This is a similar assumption as that of *smoothness* in the Neural Architecture Search (NAS) community [1], where neural network architectures with few changes in their choice of parameters and layers are assumed to yield similar performance, even though this may not always be the case. As the reviewer correctly points out, this relationship is not guaranteed (either in our case or in the case of general NAS), and the effectiveness of the Gromov-Hausdorff distance as a proxy measure depends on the specific application and the nature of the models being compared. In fact, we believe that a more thorough analysis of how changes in the latent geometry affect the performance of neural networks is a topic of its own right that could be explored in future work as a standalone idea that we could pursue.\n\nTo empirically validate this assumption, we have added the ablation suggested by the reviewer, which shows that the Gromov-Haussdorff graph search space outperforms the search space constructed with the same graph but unit coefficients.\n\nIn terms of the reasons behind choosing the Gromov-Hausdorf distance over alternative metrics, we highlight that while traditional metrics like Euclidean or cosine distances might suffice for some cases, they do not always capture the complex and nonlinear relationships between data points in high-dimensional spaces. The Gromov-Hausdorff distance considers the overall structure and shape of these spaces, rather than just individual distances, which can lead to better generalization and robustness. Moreover, another appealing aspect of Gromov-Hausdorff distance is its invariance to isometric transformations. This is a desirable property when comparing latent spaces, as the metric used should reflect the similarity in shape and structure, rather than being influenced by trivial transformations. An explanation of the motivations behind the use of the GH distance has been included in Appendix B.7.\n\nFinally, we would like to highlight that the key contribution of this paper is to introduce NLGS to the machine learning community and provide an initial solution (under some assumptions that we believe to be a reasonable starting point) based on *both* the construction of the graph search space and Gromov-Hausdorff coefficients. However, we hope that this initial attempt will be improved upon in future work.\n\n**[Weakness 2]** We agree with the reviewer on this point. To facilitate the reproducibility and adoption of the proposed framework, we have added a more thorough description of the architectures used in the attached pdf and added the code through an anonymous link sent to the ACs. \n\n**[Weakness 3/Question 1]** We have added more details with regard to the models in the Appendix. Further, we highlight that mixed-curvature autoencoders [2] and latent graph inference with product manifolds [3] have achieved state-of-the-art performance in both reconstruction and downstream tasks, respectively.\n\n**[Weakness 4]** We agree with the reviewer that certain simplifying assumptions have been made to the search space. However, we emphasize that this is a necessary step to be able to run an optimization procedure over it, as is the case with Neural Architecture Search (NAS), which motivated our assumptions. Product manifolds are the current state-of-the-art approach [2,3,4] for adding geometric inductive biases to the latent space, so we choose it as our main focus. This could be generalized in future work and an Appendix A.4 has been added to discuss the rationale behind this further. We would like to emphasize that the main contribution of this work is to bring NLGS to the community and provide a potential solution to the problem.  From this perspective, we believe the assumptions being made constitute a reasonable starting point. We hope that future research can explore further along this direction by relaxing some of the assumptions made in this initial approach.\n\n**[Question 2]** Executing search algorithms on a fully-connected graph would effectively mean treating all geometries as equidistant from one another. When applying a search algorithm to a continuum like the real line, the x values possess orientation and interconnectivity. Through discretization, this could be envisioned as a path graph. However, when completely connecting the x-axis, the algorithm loses directional guidance during searches and becomes unable to differentiate between localities or sectors, due to the uniform proximity of all points. Most search algorithms involve some sort of measure of distance between points, which is not available unless it is modified to be compatible with the proposed graph search space, which would imply that each point chosen would essentially be \"random\". Note that we focus on search space (and not the optimization procedure) which is the main focus of our work, we focus on a single optimization algorithm.\n\n**[Limitation 1]** We will add a small section in the conclusion to take these points into account. \n\n**References**\n(see overall response)"}, {"review_id": "NlzUHnhJGG", "reviewer": "Reviewer_v4xj", "summary": "A variety of recent advances have provided evidence that learning in the\nmanifold setting (i.e., assuming that high dimensional data lie along\nlow-dimensional latent manifolds, aka. the manifold hypothesis) is advantageous. However, determining the optimal geometry of the latent space often requires expertise or the use of expensive hyperparameter tuning.\nThe present work addresses the problem of learning the optimal latent space\ngeometry for a variety of latent variable models and downstream tasks as well as\nprovides a formal approach to the topic. To solve the problem, the authors\nintroduce the construction of a novel graph search space as a basis for applying\nBayesian optimization. The nodes carry information about different latent space\ngeometries and the edges carry weights from the calculation of the\nGromov-Hausdorff distance between them. As an example to highlight the details of the approach, product manifolds are used. Experiments with toy and real data sets complete the work and provide empirical evidence for the functionality of the approach.", "questions": "- Q1: In Section 4.1, can you give more insight into the \"Ground Truth\" product manifold $\\mathcal{P}_{T}$? Even if this was chosen arbitrarily, what is its dimension?\n- Q2: What measure is used for evaluation in the experiments; i.e., what does\n  \"Minimum\" at the y-axis in all plots refer to? And are there any data\n  normalization's involved since I identify no changes in the scale of the plots.\n- Q3: I would say I got a good intuition for closed balls of radius one in spaces\n  of curvature zero or one, at least for dimensions lower 3. What's a good\n  intuition for closed balls in spaces of curvature smaller zero? Would be cool\n  to have a small section on this, e.g., as part of the appendix. Especially,\n  because those balls are used for comparison of model spaces (see l. 137ff.).\n  \n- Minor stuff:\n    - l. 129 check wording.\n    - l. 134 $b_0 \\in B$\n    - l. 145 $d_{GH}(B,B\\times B')$?\n    - l. 180 $\\mathbb{E}^{6n-6}$ ?\n    - l. 199 $d_{GH}(B_{\\mathbb{E}²},B_{\\mathbb{H²}})$ twice?\n    - l. 352 \"Figure 6._In\"\n", "limitations": "Limitations are addressed (see e.g. Chapter 5) and mainly concern the focus on a\nlimited class of models, namely product manifolds formed by spaces of constant\ncurvature. From my perspective there are important aspects that need to be\ndiscussed additionally, e.g., computational limitations in the case of high\ndimensional latent spaces, since here the estimation of the Gromov-Hausdorff\ndistance in combination with the application of Bayesian optimization usually\ncauses an enormous computational effort.  According to section B.4 in the\nappendix, restrictions related to estimating the Gromov-Hausdorff distance for\nthe (arguably rather simple) model class under consideration are already\nimplied. In the case of arbitrarily complex Riemannian manifolds, do these\ndifficulties even lead to intractability?\n\nNo evidence was found that the authors addressed potential negative societal impact. Please make the effort to include this mandatory information in your work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I greatly appreciate any attempt to incorporate geometrically sensitive analysis\ninto machine learning. From my perspective it is advantageous, to assume that\ndata incorporates structural information and to give models the ability to\naddress them.\n\nI really like the careful, clear and independent introduction to the theoretical\ncontent; It makes it easy (I would assume even for non-experts) to easily follow the\nauthors' arguments.\n\nFurther, the content is well organized and, with very few exceptions, has no spelling or grammatical flaws. Empirical evidence is accessed through experiments on toy and real world data and the authors make effort to extend the small-scale experimental setting.", "weaknesses": "- W1: The theory is presented in a rather narrow setting of product\n  manifolds and the authors simplify the problem setting (e.g., see\n  Section 3.1) by choosing a rather low dimension (i.e., $d_i=2$) of each of the model spaces and restrict themselves\n  to a (very) limited selection of different possible curvatures $K_i \\in \\lbrace-1,\n  0, 1\\rbrace$. According to Section B in the appendix, estimation of the\n  Gromov-Hausdorff distance in those cases, i.e., in the class of\n  product manifolds parameterized by a limited number of signatures, already seems\n  laborious. The presented method is very model space specific and seems\n  difficult to generalize to arbitrary manifolds. Overall, this impression\n  contradicts my expectations of an automatically optimal latent geometry\n  recognition when learning any latent space models. However, will there always be the need of expert knowledge in the field of\n  differential geometry to first choose a class of manifolds, then derive a\n  class-specific Gromov-Hausdorff distance formula (if tractable or derive\n  sufficient upper bounds) and finally construct a graph search space?\n\n- W2: In fact, the authors are aware of, that the class of product manifolds are far from\n  being as rich as, e.g., the class of arbitrary Riemannian manifolds (cf. l.\n  112ff.). Is it possible to quantify how well this class of manifolds works as\n  a proxy for arbitrary latent space model classes? In other words, in the case\n  of product manifolds as a model class, how severe are the constraints imposed\n  when learning a latent space model, e.g., in the setting of variable Bayesian\n  inference using a variational auto-encoder model?\n\n- W3: Considering the points (i) I have rather a broad knowledge on the subject\n  and (ii) the authors admit that to their knowledge no computational method for\n  the mathematical comparison of product manifolds exists (see l. 58), it is\n  nevertheless questionable that no related work exists. I would cherish the\n  effort to address prior work even it is in the broader sense of latent space\n  geometry or addressed to single properties only, e.g., curvature (see Lubold\n  et al., Identifying the latent space geometry of network models through analysis of curvature).\n\n- W4: I'm interested in implementation details, in particular how the presented\n  geometry identification fits into a neural network pipeline of a latent space\n  model, for instance in a variation auto-encoder. A structured description of the\n  methodology in form of a code listing as well as code examples for the purpose\n  of reproducibility would be desirable.\n\nHowever, I would like to reiterate my appreciation for the authors' interest in\nexpanding the field of latent space models from a geometrical point of view. I'm\nsure that by working on the deficiencies mentioned, the work will gain in value\nand will deliver a valuable contribution to the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the importance of the topic that we target in our work. We hope that the following responses to the weaknesses raised by the reviewer will merit a raise in the score.\n\n**[W1]** We thank the reviewer for raising this issue. In the same spirit as our previous response (W4) to Reviewer cFVp, we would first like to stress the fact that simplifying assumptions have been made to the search space in order to run a tractable optimization procedure (similar approaches have been adopted in Neural Architecture Search, see [1]) over it to find the best latent geometry out of those available. Our goal with this work is to introduce the idea of NLGS to the machine learning community (which we believe could enrich the AutoML toolbox in a previously unexplored way) and provide an initial solution that could be improved upon by future work. One such example of improvement could be that of curvature learning (first introduced in section 3 in [2]). We believe, however, that the assumptions made are a reasonable starting point that could be built upon in the future. \n\n**[W2]** Thank you for this comment. It is indeed the case that product manifolds are not as rich as the class of arbitrary Riemannian manifolds. To the best of our knowledge, there is no principled way of quantifying how well product manifolds of constant curvature model spaces cover all the possible latent manifolds. However, product manifolds of constant curvature model spaces offer significant advantages in that there exist closed-form solutions for their exponential maps and geodesic distances between points, which is not the case for arbitrary Riemannian manifolds. In the latter case, these quantities must typically be computed numerically or approximated using specialized algorithms. These computations often involve solving differential equations, and depending on the manifold's curvature and geometry, these solutions can be quite complex and may not have closed-form expressions. This is the main reason behind this assumption, which we have detailed more explicitly in Appendix A.4 (shown in the comment to all reviewers). \n\nWith regard to the performance of models operating under these assumptions, we highlight that current state-of-the-art approaches in the literature [2,3] employ product manifolds for geometric modelling of the latent space.\n\n**[W3]** We thank the reviewer for bringing this to our attention. We have added the suggested citation and some others ([6,7]) to address prior work in the field. Nevertheless, we highlight that the use of the GH distance to compare latent spaces is introduced for the first time in our work, to the best of our knowledge. \n\n**[W4]** As mentioned in (W2) addressed to Reviewer cFVp, we have added more details related to the architectures in the Appendix (see additional pdf) and provided the code in an anonymous repository.\n\n**[Q1]** We apologize to the reviewer for the lack of clarity regarding this point. Given that the individual model spaces are of dimension two, product manifolds of 13 and 20 model spaces will have dimensions of 26 and 40 respectively.\n\n**[Q2]** Here, minimum refers to the minimum loss encountered by the model thus far. Indeed, all datasets are normalized for the maximum to be at 1 and the minimum to be at 0. \n\n**[Q3]** Essentially, we should think of $B_1^{\\mathbb{H}_3}$ as a manifold that is isomorphic to $B_1^{\\mathbb{E}^3}$ which is difficult to cover by balls of smaller radius, in fact, it will be coarsely approximated by a finite tree of balls of small radius (the underlying tree being bigger as the chosen radius is smaller).  Furthermore, the philosophy that traveling around $B_1^{\\mathbb{H}_3}$ becomes more and more expensive as we leave the origin is made accurate with the consideration of divergence functions: geodesics in $\\mathbb{H}^3$ witness exponential divergence. More precisely, there exists a constant $c>1$ such that for every $r>0$ and for every geodesic $\\gamma: \\mathbb{R}\\rightarrow \\mathbb{H}^3$ passing through the origin at time $t=0$ it is true that if $\\alpha$ is a path that connects $\\gamma(-r)$ and $\\gamma(r)$ and that lies entirely outside of the ball of radius $d(0, \\gamma(r))$, then $\\alpha$ must have length at least $c^r$. Again, this is better appreciated when put in comparison with the Euclidean space $\\mathbb{E}^3$. Any geodesic (any line) $\\gamma: \\mathbb{R}\\rightarrow \\mathbb{E}^3$ passing through the origin at time $t=0$ will have the property that for all $r>0$ there is a path of length $\\pi r+\\epsilon$, with $\\epsilon>0$ arbitrarily small, that lies entirely outside of the ball of radius $r$ with centre at the origin. The length $\\pi r+\\epsilon$ is linear in $r$, as opposed to the exponential length that is required in hyperbolic spaces. \n\n**[Q4]**\nl. 129 check wording. Done\nl. 134 fixed.\nl. 145 The GH distance between a product manifold with an additional model space and the original model space is 1, the maximum.\nL.180 yes, fixed\nL.199, fixed in parenthesis should be the distance between S^2 and H^2 instead, thanks for pointing it out\nLine 352, fixed\n\n**[Lim1]** Scalability: In this case, we would like to highlight some of the previously mentioned points regarding the necessity of using product manifolds of constant curvature model spaces due to the closed-form solutions of their geodesics and exponential maps. In terms of tractability, it is important to note that given the way that our graph search space is constructed, there is no need to repeatedly compute the GH distance, as it has been pre-computed and can now just be applied to the graph search space. This makes the algorithm highly efficient and scalable. For additional motivation on the use of the GH distance, we refer the reviewer to Appendix B.7 (overall response), which we have added post-review.\n\n**[Lim2]** Societal Impact: We thank the reviewer and have included this discussion int he paper.\n\n**References**\n(see overall comment)"}, {"review_id": "HMuotl3OvO", "reviewer": "Reviewer_6Vwv", "summary": "\nThe paper introduces an approach to estimate the best latent geometry structure for a machine learning model, trained on a data distribution and for a specific task. The approach assumes a product manifold structure of the latent space of subspace of constant curvature, being able to model product spaces of euclidean, spherical and hyperbolic subspaces. The search over candidate geometries is performed by constructing a graph search space where each node in the graph represents a candidate product manifold, associated with the model performance. Edges represent the inverse of the Gromov- Hausdorff distance between candidate spaces. The search is then conducted using Bayesian optimization algorithms. Experiments on synthetic and real scenarios demonstrate that assuming a graph space which models the topology and geometry of the spaces (with the right distances on the edges) entail finding the correct minima and being more efficient in the optimization process.\n", "questions": "\n- Does the method scales well with growing dimensions of the latent space or the requirement to embed everything isometrically for hyperbolic latent space is a bottleneck in scaling up everything?\n\n- Can the isometric embedding requirement be relaxed and allow for some bounded distortion in the embedding space? Could this help in reducing the dimensionality constraints on the embedding space?\n\n- Is the latent space search always made post hoc (after having trained a model)? Do you think that the model training would benefit for enforcing a given geometry during training as well?\n\n- The graph pruning strategy (considering models which differ by a single type of latent space and with similar number of subspaces) also loses something in the latent geometry selection strategy? I.e. would it more beneficial in some cases to have two models that differs of more than one component connected in the graph?\n\n- How is the neural network employed to decode the signals in experiment in section 4.1 trained?\n\n- Are the experiments performed always on a single vector projected on the different product manifold geometries? Especially for real case scenario would not be better to work with multiple samples or some averaged statistics of the latent space?\n\n- For real dataset  (e.g. MNIST autoencoder) it is correct to say that the result of the graph search provides the best geometry for the latent space among the one considered? Does this tell something specific about the geometry and topology of the data? What happens if the model is trained from scratch assuming the selected geometry, will the resulting model have better performances?\n\n- The diffusion kernels signature is nearly invariant to isometric transformation of the graph: does this bring any problem in the graph search (e.g. there are cases in which it would be useful to distinguish two nodes with isometrically similar signatures)? How many components (eigenvalues, eigenvectors pairs) are used to approximate the kernel?\n", "limitations": "The limitations are correctly addressed in the manuscript.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n\n- The paper experimental results highlight the importance of correctly modeling topology and geometry of the latent space in the search for the product latent space that better suits the data and the task at hand, outlining the importance of modeling the geometry in embedding spaces.\n\n- The paper is well written and clear.\n\n- The ideas developed and explained in the paper are relevant to the representation learning and geometric deep learning communities: they could be used for new modeling strategies of latent spaces of NNs, developing constraints, and posing the product space search itself as a learning problem to find the geometry that better represents the data for a given task.\n", "weaknesses": "- The approach seems not very efficient to scale up to real case scenarios due to the requirement of embedding everything  isometrically in a space of 6n-n dimensions when hyperbolic components are considered. Also it is not entirely evident how the approach would scale up in real scenarios, when growing the number of components (subspaces)  and their dimensions considered. \n\n- The experiment primarily shows that modeling the topology and geometry is important to search for the right geometry. While this is a good contribution by itself, in my opinion another great potential outcome of the pape would be interesting in showing  how the resulting product manifold approximates the latent space, to understand if this tells us something about the data, especially for real dataset (e.g. does the MNIST latent space is better modeled by an euclidean product manifold or different ones, how many components better approximate it, etc..)\n\n- It would be good to add some additional works modeling the latent space of NNs as a product manifold, e.g. : \n\n     - *Fumero, Marco, et al. \"Learning disentangled representations via product manifold projection.\" International conference on machine learning. PMLR, 2021.*\n\n     - *Pfau, David, et al. \"Disentangling by subspace diffusion.\" Advances in Neural Information Processing Systems 33 (2020): 17403-17415.*\n\n     - *Zhang, Sharon, Amit Moscovich, and Amit Singer. \"Product manifold learning.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review. We hope that any outstanding concerns will be addressed with the answers provided below.\n\n**[W1/Q1]** We would like to thank the reviewer for raising this concern. As mentioned in (L1) for Reviewer v4xj, there are no scalability issues associated with the proposed method, as the GH coefficients do not have to be re-computed. If more dimensions are required in the problem considered, one can simply construct additional “graph slices” (see Figure 2) of higher dimension and connect the nodes to the lower dimensional slices using the procedure described in the paper. In either case, the number of coefficients will remain to be 3, and no additional computational cost will be incurred.\n\nAdditionally, we would like to mention that we consider the optimal latent geometry, where, under the manifold hypothesis, there is an assumption that the data should lie in a low-dimensional manifold. This would discourage the use of higher dimensions to model the latent space, but this could be done if necessary. It is worthy of mention that relatively large search spaces have already been explored in the paper, which has not led to an explosion in computational requirements.\n\n**[W2]** We thank the reviewer for this positive comment. We do agree that an exploration of the latent space in reconstruction tasks would be a useful study to gain further insight. For example, discussion with other reviewers has motivated the idea of exploring the links between geometric deformations of the latent space results in different neural network outputs. We do note, however, that prior work (see Appendix E of [2]) has already focussed on these ideas. While we do think this point is relevant and well raised, we believe it would be better to thoroughly analyze this idea as a standalone piece, and not distract the reader from the focus of our proposal, which is related to search. \n\n**[W3]** Thank you for pointing us to relevant works that might have been originally overlooked. We have added the rest of recommended citations.\n\n**[Q2]** We have provided the formula to compute the GH coefficient for any arbitrary embedding space dimensionality and hence, there is no need for relaxing the requirements. Practically speaking we apply our algorithm for model spaces of d=2, but we provide the tools to extend the approach to higher-dimensional embedding spaces.\n\n**[Q3]** Thank you for your question. The choice of latent geometry does not have to be made post-hoc. In fact, recent works [2,3] show that there are significant benefits of adopting a correct geometry in the latent space can lead to state-of-the-art performance. The key idea of our method is to introduce the idea of searching for the optimal geometry that will lead to optimal training and better generalization, and propose a potential solution to the problem by leveraging ideas related to product manifolds (see [4]). The search procedure, however, does require training several architectures with different latent geometries, which our method tries to do with a minimal amount of evaluations by virtue of our geometry-informed search space.\n\n**[Q4]** We thank the reviewer for raising this concern. While additional connections could be added to the graph search space, we believe that an overly-connected graph would effectively destroy the topological information, thereby leading to decreased search performance, as is the case in the fully connected (naive) graph search space. The current choice in structure is made such that there are individual “clusters” of nodes in the same dimension, and the algorithm is not encouraged to make large jumps between these dimensional clusters. This choice ensures the graph has a sense of directionality and that search algorithms can leverage local sectors in the search space. \n\n**[Q5]** This setting is supposed to allow us to control the optimal latent geometry, and to gauge how well our method can find it using the output of a neural network as a signal. For this exercise, training of the neural network is not relevant given that our interest is to test the model’s ability to find the (known) optimal geometry. For that reason, the network weights are actually frozen. We have added some additional explanatory diagrams in Appendix D.1 to clarify this, which is included in the additional PDF. \n\n**[Q6]** We thank the reviewer for pointing this out. For each optimal sub-manifold experiment, the random vector is initialized randomly again, which prevents it from being the same one. In real-world experiments, the latent vector will be different at initialization due to the random initialization of network weights. \n\n**[Q7]** (i) Yes, the model would provide the best-performing latent geometry among those available. Under the manifold hypothesis, this would be suggestive of the fact that the data is approximated by a low manifold dimensional structure in the latent space, whose shape is captured best by said latent product manifold. (ii) If the model was trained from scratch with this inductive bias, it could lead to the architecture having improved performance, which could even be state-of-the-art, see [2,3].\n\n**[Q8]** (i) Yes, this is indeed the case. While it is true that the diffusion kernel is in some cases not ideal, it has been previously used effectively in NAS (see [1]), which motivated its use for our problem. As mentioned, this choice could be refined to account for more recent work [5]. However, we would like to highlight that the optimization procedure itself is not the core contribution of our work, which led to this decision.\n(ii) The whole Laplacian is used to approximate the kernel (i.e. all its eigenvalues and eigenvectors)\n\n**References**\n(see overall comment)\n"}, {"review_id": "ZFo8b8IlLN", "reviewer": "Reviewer_1UmR", "summary": "The goal of this paper is to learn the geometry of underlying given data points after mapping them into latent space. Rather than learning a general (nonparametric) Riemannian manifold from the embedding data, the paper models this manifold as a product of certain prototypical manifolds with constant curvature (Euclidean, spherical, hyperbolic). Perhaps the idea is that a general nonlinear manifold can be stitched together using patches from these three types of manifolds. The paper describes a way to map points on these prototypical manifolds to their tangent spaces and back, i.e., the inverse exponential and the exponential map. This way one can express an embedded data point as elements of manifolds using their Euclidean coordinates. Patching together of prototype manifolds is done using a graph representation. \n\nWith this geometry, the paper derives a framework to fit a latent manifold to the training data. The loss function is built using the Gromov-Hausdorff distance. This requires isometrically embedding the prototypical manifolds into larger Euclidean spaces, and then utilize Euclidean distances to specific Gromov-Hausdorff distances. The search for optimal graph representation of latent space manifold is conducted through Bayesian optimization.\n\nSeveral experiments on synthetic and real-world datasets are presented in support of this theory. \n", "questions": "\nPlease see my comment above. ", "limitations": "The paper does not discuss any limitation specifically. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- Tackles a difficult problem of manifold learning by doing to in the latent space. \n\n-- The idea of representing arbitrary manifolds by a patching of some constant curvature pieces is interesting. (However, one has to show mathematically that this construction covers all or most of the interesting scenarios.)\n\n-- The use of graph representation to patch together the pieces is also interesting and feasible. \n\n-- The experimental results are reasonably demonstrative of the ideas although I have not evaluated them too carefully. ", "weaknesses": "\n-- Why is the manifold learning or fitting performed in the latent space and not the original data space. This involves a choice of embedding which could be arbitrary. Perhaps the isometric nature of the embedding allows this arbitrariness to be introduced. \n\n-- Is it not possible to provided a simple minded sketch of the overall framework first using basic mathematics? There are too many symbols introduced right away and that distracts from a quick, preliminary understanding of the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments on our work. We hope that the following will address some of the issues raised. \n\n**[Weakness 1]** We thank the reviewer for raising this concern. The reason why the manifold is fitted in the latent space is a consequence of the manifold hypothesis, which operates under the assumption that the data resides in a low-dimensional latent manifold inside the high-dimensional data space. Therefore, we focus on directly modelling the latent space by finding the optimal product of constant curvature model spaces, which is the current state-of-the-art method to model the geometry of the latent space (see [4]).\n\n**[Weakness 2]** In order to improve clarity, we have added a comment in the introduction which points readers to Section 3.1, which provides a high-level sketch of the problem formulation without going into any of the mathematical details. Specifically, the text added is:\n\n*For a high level description of the proposed framework in simple mathematical terms, we recommend skipping to Section 3.1.*\n\n**References**\n(see overall comment)"}, {"review_id": "NEbqk1WCJv", "reviewer": "Reviewer_aQg4", "summary": "This paper addresses the problem of automatically identifying the optimal latent geometry for machine learning models. The neural latent geometry search (NLGS) searches for a latent geometry composed of a product of constant curvature model spaces with minimal query evaluations.The proposed method utilizes the Gromov-Hausdorff distance as a measure to compare and evaluate the distance between different metric spaces. By constructing a graph search space and leveraging Bayesian optimization, the method efficiently searches for the optimal latent geometry.Experimental results on synthetic and real-world datasets demonstrate the effectiveness of the proposed approach in identifying the optimal latent geometry for various machine learning problems.", "questions": "1. Is it possible to utilize the Gromov-Wasserstein (GW) distance, which is a reformulation of the Gromov-Hausdorff (GH) distance [A], as an alternative for practical computation, given the known computational challenges associated with the GH distance?\n  \n2. Why was the diffusion kernel chosen to compute similarity in lines 284-287 instead of using the inverse of the Gromov-Hausdorff distance directly?\n  \n3. In the experiment, what does the \"Naive BO\" method use to be the weight on the fully-connected graph?\n  \n\n[A] Mémoli, Facundo. \"Gromov–Wasserstein distances and the metric approach to object matching.\" *Foundations of computational mathematics* 11 (2011): 417-487.", "limitations": "The authors address the limitations of their work. This work does not have a negative societal impact.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors propose a modeling approach where the latent geometry is represented using the Cartesian product of Riemannian manifolds with constant curvature.\n- This paper constructs a graph search space where each node represents a latent space product manifold. The inverse GH distance is used as the weight on the edges, and Bayesian optimization is employed to find the minimum within this space.\n- Appropriate figures are included in the paper, effectively illustrating the concepts and results.", "weaknesses": "- There is a concern regarding Section 3.1 regarding the potential for overfitting due to the similarity between the objective function $L_{T,A}(g)$ used to determine the latent geometry $g$ and the loss function in the downstream task.\n- The paper lacks adequate illustration and explanation for the rationale behind utilizing the Cartesian product of Riemannian manifolds with constant curvature, such as Euclidean space, hyperbolic space, and sphere. It would be beneficial for the authors to provide a more comprehensive insight into the reasons for this choice, including potential advantages like efficient computation or other relevant considerations.\n- Although I am not an expert in image reconstruction and latent graph inference, it could be beneficial to compare the GH-BO method with existing benchmarks for these tasks in order to evaluate its performance and assess its effectiveness and competitiveness.\n\n**Other comments:**\n\nIt seems that the pdf submitted includes the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments with regard to our work. We hope the additional explanations for some of the weaknesses identified will help to raise the score.\n\n**[Weakness 1]** We thank the reviewer for this comment, but in our opinion, the proposed minimization problem would not necessarily lead to overfitting. The proposed minimization problem has many parallels with the Neural Architecture Search (NAS) problem. In our case, instead of seeking to optimize downstream performance by changing architectural components, we seek to optimize it by changing the latent geometry. Overfitting is prevented by evaluating the performance of the architecture on unseen (validation) data.\n\n**[Weakness 2]** We thank the reviewer for pointing us to this issue. In order to remedy this, we have added a detailed explanation of the motivations behind the use of products of constant curvature model spaces (with ample references to the previous literature) in Appendix A.4, which is included in the overall comment to all reviewers.\n\n**[Weakness 3]** We thank the reviewer for raising this point. We would like to mention that, to the best of our knowledge, there are not benchmarks in the literature focusing on leveraging geometric inductive biases to compare latent geometries in the context of image reconstruction and latent graph inference. Rather, in state-of-the-art papers, most authors focus on manually searching for the most appropriate geometry by exploring different classes of manifolds (mostly product manifolds) to encode the latent space. In our case, we reproduce these state-of-the-art results, and at the same time introduce a principled way to automatically search for the optimal geometry rather than relying on random search. In terms of performance on downstream tasks, we would like to highlight that the use of an advantageous latent geometry has previously led to state-of-the-art performance [2,3]. \n\n**[Question 1]** We thank the reviewer for raising this point. In the literature, one can find techniques to estimate both distances, however, it seems that the Gromov-Wasserstein gives better results when it comes to comparing discrete (e.g. finite) objects, as opposed to the continuous infinitesimal objects that we consider in our work (manifolds). In fact, the Gromov-Wasserstein is more technical because it requires endowing your objects with an additional structure: a measure. For the Riemannian manifolds M we consider, a natural choice for this measure would be to assign (to every bounded measurable subset C in M), the volume of C. It is not clear to the authors that the estimation of the corresponding Gromov-Wasserstein distances presents fewer difficulties than the Gromov-Hausdorff distance. Furthermore, given that our manifolds are homogeneous, the estimation of the Gromov-Hausdorff distances is not too bad, and we introduce several simplifications for this task that come from this symmetry (which would be harder to take advantage of when considering other distances).\n\n**[Question 2]** The GH distance is used to generate the graph search space. BO, which utilizes the diffusion kernel, is performed based on the search space, and hence, the GH distance is implicitly utilized by the diffusion kernel (which is a node-wise kernel defined on the graph search space) as an inductive bias when searching through the graph search space.\n\n(Question 3) Naive BO in the main text considers performing BO over a fully-connected unweighted graph (all weights are set to one). Such a graph is latent geometry agnostic, that is, it does not include any metric geometry inductive bias to provide the optimization algorithm with a sense of closeness between the candidate latent geometries. We have included an additional appendix in the paper to explain this further.\n\n**References**\n(see overall comment)"}]

Author rebuttal:
We thank all reviewers for their feedback. We are glad that they acknowledged the novelty of the problem being addressed and the proposed framework. In the overall response, we would like to address some of the concerns raised in multiple reviews, which we have added in different appendices. 

**[Motivations of the GH distance] - Appendix B.7**

The Gromov-Hausdorff distance offers a way to gauge the similarity between metric spaces. This is achieved by casting their representation within a shared space, all the while striving to maintain the original pairwise distances. In the context of machine learning models, particularly those involving generative models (such as VAEs or GANs), the latent space assumes a pivotal role in shaping the characteristics of the generated data. This also holds true in the case of reconstruction tasks, such as those carried out by autoencoders. Through the application of a metric that factors in their inherent geometries, the aim is to capture the concept of resemblance in data generation, reconstruction, and other downstream tasks which will be heavily affected by the choice of latent geometry.

While the direct relationship between Gromov-Hausdorff distance and model performance might not be immediately obvious, we argue that if two models have similar latent space geometries, this suggests that they might capture similar underlying data structures, which could lead to similar performance on downstream tasks. However, it is important to note that this relationship is not guaranteed, and the effectiveness of Gromov-Hausdorff distance as a proxy measure depends on the specific application and the nature of the models being compared.

In summary, the selection of the Gromov-Hausdorff distance as a metric for comparing latent spaces in machine learning models originates from the intention to encompass profound structural resemblances that conventional distance metrics might overlook. Although its direct correlation with model performance could be intricate, it offers a systematic approach to evaluate the similarity of latent geometries within a wider framework.

**[Motivation of using Product Manifolds of model spaces] - Appendix A.4**

Most arbitrary Riemannian manifolds do not have closed-form solutions for their exponential maps and geodesic distances between points, as well as for other relevant mathematical notions. The exponential map takes a point on the manifold and exponentiates a tangent vector at that point to produce another point on the manifold. The geodesic distance between two points on a manifold is the shortest path along the manifold connecting those points.

For simple, well-studied manifolds like Euclidean space, hyperbolic space, and spherical space, closed-form solutions for exponential maps and geodesic distances are available. This is because these manifolds have constant curvature, which allows for more straightforward calculations. However, for most other manifolds, the exponential map and geodesic distances must typically be computed numerically or approximated using specialized algorithms. These computations often involve solving differential equations, and depending on the manifold's curvature and geometry, these solutions can be quite complex and may not have closed-form expressions. For certain types of manifolds, like product manifolds where each factor is a well-understood manifold, the computation of exponential maps and geodesic distances can sometimes be simplified due to the separability of the metric. However, scaling these methods to more general, complex manifolds can be challenging and computationally intensive.

**[Scalability of the GH coefficients] Appendix B.8**

The proposed approach sidesteps scalability concerns by obviating the need to recalibrate GH coefficients, as long as one adheres to latent spaces derived from products of model spaces. Should the situation necessitate a higher-dimensional latent space, the solution lies in effortlessly crafting supplementary "graph slices" with augmented dimensions. These slices can be seamlessly integrated with lower-dimensional counterparts following the procedure described in the paper. The count of coefficients remains constant at 3, rendering the Gromov-Hausdorff distances free from extra computation overhead during method upscaling. Moreover, in alignment with the manifold hypothesis, data is anticipated to exist within a low-dimensional manifold. This discourages an inclination towards higher dimensions for latent space modeling, though the possibility persists for situations demanding it. It is important to note, nonetheless, that the paper has already delved into relatively large search spaces.

**References**

[1] Oh, C., Tomczak, J., Gavves, E. and Welling, M., 2019. Combinatorial bayesian optimization using the graph cartesian product. Advances in Neural Information Processing Systems, 32.

[2] Skopek, O., Ganea, O. E., & Bécigneul, G., 2020. Mixed-curvature variational autoencoders. In 8th International Conference on Learning Representations.

[3] de Ocáriz Borde, H.S., Kazi, A., Barbero, F. and Lio, P., 2022, September. Latent graph inference using product manifolds. In The Eleventh International Conference on Learning Representations (ICLR 2023).

[4] Gu, A., Sala, F., Gunel, B. and Ré, C., 2018, September. Learning mixed-curvature representations in product spaces. In International conference on learning representations.

[5] Zhi, Y.C., Ng, Y.C. and Dong, X., 2023. Gaussian processes on graphs via spectral kernel learning. IEEE Transactions on Signal and Information Processing over Networks.

[6] Hauberg, Søren, Oren Freifeld, and Michael Black. "A geometric take on metric learning." Advances in Neural Information Processing Systems 25 (2012).
 
[7] Arvanitidis, G., Hansen, L.K. and Hauberg, S., 2017. Latent space oddity: on the curvature of deep generative models. arXiv preprint arXiv:1710.11379


Meta-review: This paper addresses a Bayesian optimization for neural latent geometry search. All reviewers agree that the current paper tackles an important problem, well formulating it as NLGS, where the search is carried out using Bayesian optimization algorithms. The authors did a good job in their rebuttal, resulting in an increase in the overall score.  All reviewers feel that the paper has interesting contributions and is deserved for publication.
Decision: Accept (poster)