SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 2: Slice of the graph search space for latent geometries of dimension 4: product manifolds obtained using 2 models spaces of dimension 2. The graph edges are shown in different colours to depict a different degree of connectivity (black:  $w_{\mathbb{E}^2,\mathbb{S}^2}$ , red:  $w_{\mathbb{E}^2,\mathbb{H}^2}$ , blue:  $w_{\mathbb{S}^2,\mathbb{H}^2}$ ), this is determined by the inverse of the Gromov-Hausdorff distance between the different product manifolds.
We model the search space as a graph over which we perform BO. More formally, we consider a graph  $\mathcal{G}$  given by  $(\mathcal{V},\mathcal{E})$ , where  $\mathcal{V} = \{v_{i}\}_{i = 1}^{N}$  are the nodes of the graph and  $\mathcal{E} = \{e_j\}_{j = 1}^M$  is the set of edges, where edge  $e_j = (v_i,v_j)$  connects nodes  $v_{i}$  and  $v_{j}$ . The topology of the graph is encoded in its adjacency matrix, which we denote with  $\mathbf{A}\in \mathbb{R}^{N\times N}$ . In our case, we focus on a weighted and undirected graph, meaning that  $\mathbf{A}_{i,j} = \mathbf{A}_{j,i}$ , and  $\mathbf{A}_{i,j}$  can take any positive real values. We consider a setting in which the function  $\mathfrak{f}(\cdot)$  to minimize is defined over the nodes of the graph, and the objective of using BO is to find the node associated to the minimum value  $v^{*} = \arg \min_{v\in \mathcal{V}}\mathfrak{f}(v)$ .
In our setting, each node in the graph represents a different latent space product manifold, and the value at that node is the validation set performance of a neural network architecture using this latent geometry. We use the inverse of the Gromov-Hausdorff distance between product manifolds to obtain edge weights.
We denote by  $w_{\mathcal{M}_1,\mathcal{M}_2}$  (edge weights) the inverse Gromov-Hausdorff distance between model spaces  $\mathcal{M}_1$  and  $\mathcal{M}_2$ . The approximate values of these coefficients are presented in Table 1. In particular, the Gromov-Hausdorff distance  $\mathrm{d}_{\mathrm{GH}}(\mathbb{E}^2,\mathbb{S}^2)$  used to compute  $w_{\mathbb{E}^2,\mathbb{S}^2}$  is derived analytically in Appendix B.3, while the remaining coefficients are obtained through computational approximations of Equation 6, which are depicted in Appendix B.4. Further, the Gromov-Hausdorff distance between manifolds of different dimensions is one (see Section 2).
In order to impose additional structure on the search space, we only allow connections between nodes corresponding to product manifolds which differ by a single model space. For example, within the same dimension,  $\mathbb{S}^2\times \mathbb{H}^2$  and  $\mathbb{E}^2\times \mathbb{H}^2$  would be connected with edge weighting  $w_{\mathbb{E}^2,\mathbb{S}^2}$  while  $\mathbb{S}^2\times \mathbb{H}^2$  and  $\mathbb{E}^2\times \mathbb{E}^2$  would have no connection in the graph. Furthermore, product manifolds in different dimensions follow the same rule. For instance, we would have a connection of strength one between  $\mathbb{E}^2\times \mathbb{H}^2$  and  $\mathbb{E}^2\times \mathbb{H}^2\times \mathbb{H}^2$ , but no connection between, for instance,  $\mathbb{E}^2$  and  $\mathbb{E}^2\times \mathbb{H}^2\times \mathbb{H}^2$  or  $\mathbb{S}^2$  and  $\mathbb{E}^2\times \mathbb{H}^2$ . This construction induces a sense of directionality into the graph and generates clusters of product manifolds of the same dimension. Finally, it should be mentioned that in practice there are only four edge weights. For example, the connectivity strength between  $\mathbb{E}^2\times \mathbb{H}^2$  and  $\mathbb{H}^2\times \mathbb{H}^2$  is  $w_{\mathbb{E}^2,\mathbb{H}^2}$  since  $\mathrm{d}_{\mathrm{GH}}(\mathbb{E}^2\times \mathbb{H}^2,\mathbb{H}^2\times \mathbb{H}^2) = \mathrm{d}_{\mathrm{GH}}(\mathbb{E}^2,\mathbb{H}^2)$  given that  $\mathrm{d}_{\mathrm{GH}}(\mathbb{H}^2,\mathbb{H}^2) = 0$ . Visual representations of the graph search space can be found Figures 2 and 3. Note that both figures use the same colour scheme, and in Figure 3 there is an increase in the dimen
Figure 3: Example graph search space for product manifolds composed of up to seven model spaces. Manifolds of different dimensionality are connected with edges coloured grey. Node labels have been omitted for visual clarity.
sionality of the product manifolds from left to right
(e.g. on the top left corner, one can see a triangle
corresponding to the three model spaces). We refer the reader to Appendix B.6 for additional visualizations.
Bayesian Optimization over the Graph Search Space. We aim to find the minimum point within the graph search space through the use of BO (see Appendix C). While performing BO on graphs allows us to search for the minimum over a categorical search space, some of the key notions used in BO over Euclidean spaces do not translate directly when operating over a graph. For example, notions of similarity or "closeness" which are trivially found in Euclidean space through the  $\ell_1$  or  $\ell_2$  norms require more careful consideration when using graphs.
In our setting, we employ a diffusion kernel [Smola and Kondor, 2003, Kondor and Vert, 2004] to compute the similarity between the nodes in the graph. The diffusion kernel is based on the eigendecomposition of the graph Laplacian  $\mathbf{L} \in \mathbb{R}^{N \times N}$ , defined as  $\mathbf{L} = \mathbf{D} - \mathbf{A}$  where  $\mathbf{D}$  is the degree matrix of the graph. In particular, the eigendecomposition of the Laplacian is given by  $\mathbf{L} = \mathbf{U} \boldsymbol{\Lambda} \mathbf{U}^{\mathrm{T}}$ , where  $\mathbf{U} = [\mathbf{u}_1, \dots, \mathbf{u}_N]$  is a matrix containing eigenvectors as columns and  $\boldsymbol{\Lambda} = \mathrm{diag}(\lambda_1, \dots, \lambda_N)$  is a diagonal matrix containing increasingly ordered eigenvalues. The covariance matrix used to define the GP over the graph is given by
$$
\mathcal {K} (\mathcal {V}, \mathcal {V}) = \mathbf {U} e ^ {- \beta \boldsymbol {\Lambda}} \mathbf {U} ^ {\mathrm {T}}, \tag {7}
$$
where  $\beta$  is the lengthscale parameter. Our approach is therefore conceptually similar to that of [Oh et al., 2019], which employ a diffusion kernel to carry out a NAS procedure on a graph Cartesian product. We highlight, however, that the main contribution of this work is the construction of the graph search space, and we employ this search procedure to showcase the suitability of our method in finding the optimal latent product manifold. For reference, we note that other works employing Bayesian optimization in graph-related settings include Cui and Yang [2018], Como et al. [2020], Ma et al. [2019], Ru et al. [2020], Cui et al. [2020], Wan et al. [2021].
In this section, a detailed outline of the experimental results is provided. It comprises of experiments performed on synthetic datasets, as well as experimental validation on custom-designed real-world datasets obtained by morphing the latent space of mixed-curvature autoencoders and latent graph inference. The results demonstrate that the Gromov-Hausdorff-informed graph search space can be leveraged to perform NLGS across a variety of tasks and datasets.
4.1 Synthetic Experiments on Product Manifold Inference
In order to evaluate the effectiveness of the proposed graph search space, we conduct a series of tests using synthetic data. We wish to present a setting for which the latent optimal product manifold is known by construction. To do so, we start by generating a random vector  $\mathbf{x}$  and mapping it to a "ground truth" product manifold  $\mathcal{P}_T$ , which we choose arbitrarily, using the corresponding exponential map. The resulting projected vector is then decoded using a neural network with frozen weights,  $f_{\theta}$ , to obtain a reference signal  $y^{\mathcal{P}_T}$ , which we wish to recover using NLGS. To generate the rest of the dataset, we then consider the same random vector but map it to a number of other product manifolds  $\{\mathcal{P}_i\}_{i\in n_{\mathcal{P}}}$  with the same number of model spaces as  $\mathcal{P}_T$  but different signatures. Decoding the projected vector through the same neural network yields a set of signal,  $\{y^{\mathcal{P}_i}\}_{i\in n_{\mathcal{P}}}$ . We use the aforementioned signals, to set the value associated with each node of the graph search space to be  $MSE(y^{\mathcal{P}_T},y^{\mathcal{P}_i})$ . In this way, our search algorithm should aim to find the node in the graph that minimizes the error and hence find the latent geometry that recovers the original reference signal.
Our method consists in using BO over our Gromov-Hausdorff-informed search space. We compare it against random search, and what we call "Naive BO", which performs BO over a fully-connected graph which disregards the Gromov-Hausdorff distance between candidate latent geometries. The figures presented, namely Figures 4 and 5, display the results. Each plot illustrates the performance of the algorithms and baselines as they select different optimal latent product manifolds denoted as  $\mathcal{P}_T$ . Notably, the figures demonstrate that the algorithm utilizing the Gromov-Hausdorff-informed search space surpasses all other baselines under consideration and consistently achieves the true
function minimum. Figure 4 generally requires fewer iterations compared to Figure 5 to attain the global minimum, primarily due to the smaller size of the search space. It is important to note that our benchmark solely involves the same algorithm but with a change in the search space to a topologically uninformative one. This modification allows us to evaluate the impact of incorporating this information into the optimization process. We have not considered other benchmarks such as evolutionary algorithms since they always rely on a notion of distance between sampled points. Furthermore, as there are no existing algorithms to compute the distance between arbitrary product manifolds, we have not included these methods in our benchmark as they would simply converge to random search. All results are shown on a log scale, and we apply a small offset of  $\varepsilon = 10^{-3}$  to the plots to avoid computing  $\log (0)$  when the reference signal is found.
Figure 4: Results (mean and standard deviation over 10 runs) for candidate latent geometries involving product manifolds composed of 13 model spaces. For each plot a different ground truth product manifold  $\mathcal{P}_T$  is used to generate the reference signal.
Figure 5: Results (mean and standard deviation over 10 runs) for candidate latent geometries involving product manifolds composed of 20 model spaces. Same setup as above.
4.2 Experiments on Real-World Datasets
To further validate our method, we conduct additional tests using image and graph datasets. Specifically, we focus on image reconstruction and node classification tasks.
Image Reconstruction with Autoencoder
We consider four well-known image datasets: MNIST [Deng, 2012], CIFAR-10 [Krizhevsky et al., 2014], Fashion MNIST [Xiao et al., 2017] and eMNIST [Cohen et al., 2017]. Some of these datasets have been shown to benefit from additional topological priors, see Khrulkov et al. [2020] and Moor et al. [2020]. We use an autoencoder, detailed more thoroughly in Appendix D, to encode the image in a low dimensional latent space and then reconstruct it based on its latent representation. To test the performance of different latent geometries, we project the latent vector onto the product manifold being evaluated and use the reconstruction loss at the end of training as the reference signal to use in the graph search space. We consider a search space size of  $n_p = 7$  for MNIST and  $n_p = 8$  for CIFAR-10, Fashion MNIST and eMNIST. The results are displayed in Figure 6. In line with previous experiments, the Gromov-Hausdorff-informed search graph enables us to find better solutions in a smaller amount of evaluations.
Figure 6: Results (mean and standard deviation over 10 runs) for image reconstruction tasks.
Latent Graph Inference
Finally, we consider searching for the optimal product manifold for a graph neural network node classification task using latent graph inference. In particular, we build upon previous work by Sáez de Ocariz Borde et al. [2023b] which used product manifolds to produce richer embeddings spaces for latent graph inference. We consider the Cora [Sen et al., 2008] and Citeseer [Giles et al., 1998] citation network datasets, and a search space consisting of product manifolds of up to seven model spaces  $n_p = 7$ . The aim is to find the latent space which gives the best results for the node classification problem using minimal query evaluations. Performing BO alongside the Gromov-Hausdorff informed search space gives a clear competitive advantage over Naive BO and random search in the case of Cora. For Citeseer, the experiments do not give such a clear-cut improvement, which can be attributed to the lack of smoothness of the signal on the graph search space and its incompatibility with the intrinsic limitations of the diffusion kernel.
Figure 7: Results (mean and standard deviation over 10 runs) for latent graph inference datasets.
In this work, we have introduced neural latent geometry search (NLGS), a novel problem formulation that consists in finding the optimal latent space geometry of machine learning algorithms using minimal query evaluations. In particular, we have modeled the latent space using product manifolds based on Cartesian products of constant curvature model spaces. To find the optimal product manifold, we propose using Bayesian Optimization over a graph search space. The graph is constructed based on a principled measure of similarity, utilizing the Gromov-Hausdorff distance from metric geometry. The effectiveness of the proposed method is demonstrated through our experiments conducted on a variety of tasks, based on custom-designed synthetic and real-world datasets.
Limitations and Future Work. While the NLGS framework is general, we have restricted ourselves to using product manifolds of constant curvature model spaces to model the geometry of the latent space. Furthermore, we have only considered curvatures  $\{-1,0,1\}$ , and model spaces of dimension two in order for the optimization problem to be tractable. In future research, there is potential to explore alternative approaches for modeling the latent space manifold. Additionally, the field of Bayesian Optimization over graphs is still in its early stages. Incorporating recent advancements in the area of kernels on graphs [Borovitskiy et al., 2021, Zhi et al., 2023] could lead to improved performance. In our current research, our emphasis is on introducing NLGS and providing an initial solution under a set of simplifying assumptions related to the potential latent manifolds available and the optimization algorithm. The investigation of the impact of using different similarity measures to compare latent structures also remains a subject for future research.
Societal Impact Statement
This work is unlikely to result in any harmful societal repercussions. Its primary potential lies in its ability to enhance and advance existing data modelling and machine learning methods.
AA thanks the Rafael del Pino Foundation for financial support. AA and HSOB thank the Oxford-Man Institute for computational resources. AA acknowledges the G-Research grant for travel assistance. XD acknowledges support from the Oxford-Man Institute and the EPSRC (EP/T023333/1).
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. arXiv preprint arXiv:1710.11379, 2017.
Federico Barbero, Cristian Bodnar, Haitz Sáez de Ocariz Borde, Michael Bronstein, Petar Velicković, and Pietro Lio. Sheaf neural networks with connection laplacians. In Topological, Algebraic and Geometric Learning Workshops 2022, pages 28-36. PMLR, 2022a.
Federico Barbero, Cristian Bodnar, Haitz Sáez de Ocariz Borde, and Pietro Liò. Sheaf attention networks. In NeurIPS 2022 Workshop on Symmetry and Geometry in Neural Representations, 2022b.
Danilo Blanus. Über die Einbettung hyperbolischer Räume in euklidische Räume. Monatsh. Math., 59:217-229, 1955.
Viacheslav Borovitskiy, Iskander Azangulov, Alexander Terenin, Peter Mostowsky, Marc Deisenroth, and Nicolas Durrande. Matérn gaussian processes on graphs. In International Conference on Artificial Intelligence and Statistics, pages 2593-2601. PMLR, 2021.
Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and A. Doucet. Riemannian score-based generative modeling. ArXiv, abs/2202.02763, 2022.
David Brander. Isometric embeddings between space forms. 2003. Master thesis.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velicković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021.
Alexander J Smola and Risi Kondor. Kernels and regularization on graphs. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, pages 144-158. Springer, 2003.
Abdel Aziz Taha and Allan Hanbury. An efficient algorithm for calculating the exact Hausdorff distance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37:2153-2163, 2015.
Joshua B. Tenenbaum, Vin De Silva, and John C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 2000.
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. *ArXiv*, abs/2111.14522, 2021.
X. Wan, H. Kenlay, B. Ru, A. Blaas, M. A. Osborne, and X. Dong. Adversarial attacks on graph classifiers via bayesian optimisation. Conference on Neural Information Processing Systems, 2021.
Melanie Weber. Curvature and representation learning: Identifying embedding spaces for relational data. 2019.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. MIT press Cambridge, MA, 2006.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
Sharon Zhang, Amit Moscovich, and Amit Singer. Product manifold learning. In International Conference on Artificial Intelligence and Statistics, 2020.
Deli Zhao, Jiapeng Zhu, and Bo Zhang. Latent variables on spheres for autoencoders in high dimensions. arXiv: Learning, 2019.
Yin-Cong Zhi, Yin Cheng Ng, and Xiaowen Dong. Gaussian processes on graphs via spectral kernel learning. IEEE Transactions on Signal and Information Processing over Networks, 2023.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. *ArXiv*, abs/1611.01578, 2016.
A.1 Differential Geometry, Riemannian Manifolds, and Product Manifolds
Differential geometry is a mathematical discipline that employs rigorous mathematical techniques, including calculus and other mathematical tools, to investigate the properties and structure of smooth manifolds. These manifolds, despite bearing a local resemblance to Euclidean space, may exhibit global topological features that distinguish them from Euclidean space. The fundamental objective of differential geometry is to elucidate the geometric properties of these spaces, such as curvature, volume, and length, through rigorous mathematical analysis and inquiry.
A Riemannian manifold generalizes Euclidean space to more curved spaces, and it plays a critical role in differential geometry and general relativity. A Riemannian manifold is a smooth manifold  $M$  equipped with a Riemannian metric  $g$ , which assigns to each point  $p$  in  $M$  an inner product on the tangent space  $T_{p}M$ . This inner product is a smoothly varying positive definite bilinear form on the tangent space given by  $g_{p}: T_{p}M \times T_{p}M \to \mathbb{R}$ , which satisfies the following properties: symmetry,  $g_{p}(X,Y) = g_{p}(Y,X)$  for all  $X,Y \in T_{p}M$ ; linearity,  $g_{p}(aX + bY,Z) = ag_{p}(X,Z) + bg_{p}(Y,Z)$  for all  $X,Y,Z \in T_{p}M$  and scalars  $a,b \in \mathbb{R}$ ; positive definiteness:  $g_{p}(X,X) > 0$  for all non-zero  $X \in T_{p}M$ . The metric  $g_{p}$  induces a norm  $||\cdot||_{p}$  on  $T_{p}M$  given by  $||X||_{p} = \sqrt{g_{p}(X,X)}$ . This norm allows us to define the length of curves on the manifold, and hence a notion of distance between points. Specifically, given a curve  $c:[0,1] \to M$  and a partition  $0 = t_0 < t_1 < \ldots < t_n = 1$ , the length of the curve  $c$  over the partition is given by:  $L(c) = \sum ||c'(t_i)||c(t_i)$ , where  $c'(t_i)$  denotes the tangent vector to  $c$  at time  $t_i$ , and  $||\cdot||c(t_i)$  denotes the norm induced by the metric at the point  $c(t_i)$ . The total length of the curve is obtained by taking the limit of the sum as the partition becomes finer.
On the other hand, a product manifold results from taking the Cartesian product of two or more manifolds, resulting in a manifold with a natural product structure. This enables researchers to examine the local and global geometry of the product manifold by studying the geometry of its individual factors. Riemannian manifolds and product manifolds are essential mathematical concepts that have far-reaching applications in diverse fields such as physics, engineering, and computer science. In the appendix of this paper, we provide a more in-depth look at these concepts, including their properties and significance.
A.2 Constant Curvature Model Spaces
A manifold  $\mathcal{M}$  is a topological space that can be locally identified with Euclidean space using smooth maps. In Riemannian geometry, a Riemannian manifold or Riemannian space  $(\mathcal{M},g)$  is a real and differentiable manifold  $\mathcal{M}$  where each tangent space has an associated inner product  $g$  known as the Riemannian metric. This metric varies smoothly from point to point on the manifold and allows for the definition of geometric properties such as lengths of curves, curvature, and angles. The Riemannian manifold  $\mathcal{M} \subseteq \mathbb{R}^N$  is a collection of real vectors that is locally similar to a linear space and exists within the larger ambient space  $\mathbb{R}^N$ .
Curvature is effectively a measure of geodesic dispersion. When there is no curvature geodesics stay parallel, with negative curvature they diverge, and with positive curvature they converge. Constant curvature model spaces are Riemannian manifolds with a constant sectional curvature, which means that the curvature is constant in all directions in a 2D plane (this can be generalized to higher dimensions). These spaces include the Euclidean space, the hyperbolic space, and the sphere. The Euclidean space has zero curvature, while the hyperbolic space has negative curvature and the sphere has positive curvature. Constant curvature model spaces are often used as reference spaces for comparison in geometric analysis and machine learning on non-Euclidean data.
Euclidean space,
$$
\mathbb {E} _ {K _ {\mathbb {E}}} ^ {d _ {\mathbb {E}}} = \mathbb {R} ^ {d _ {\mathbb {E}}} \tag {8}
$$
is a flat space with curvature  $K_{\mathbb{E}} = 0$ . We note that in this context, the notation  $d_{\mathbb{E}}$  is used to represent the dimensionality. In contrast, hyperbolic and spherical spaces possess negative and positive curvature, respectively. We define hyperboloids  $\mathbb{H}_{K_{\mathbb{H}}}^{d_{\mathbb{H}}}$  as
$$
\mathbb {H} _ {K _ {\mathbb {H}}} ^ {d _ {\mathbb {H}}} = \left\{\mathbf {x} _ {p} \in \mathbb {R} ^ {d _ {\mathbb {H}} + 1}: \langle \mathbf {x} _ {p}, \mathbf {x} _ {p} \rangle_ {\mathcal {L}} = 1 / K _ {\mathbb {H}} \right\}, \tag {9}
$$
where  $K_{\mathbb{H}} < 0$  and  $\langle \cdot ,\cdot \rangle_{\mathcal{L}}$  is the Lorentz inner product
$$
\langle \mathbf {x}, \mathbf {y} \rangle_ {\mathcal {L}} = - x _ {1} y _ {1} + \sum_ {j = 2} ^ {d _ {\mathbb {H}} + 1} x _ {j} y _ {j}, \forall \mathbf {x}, \mathbf {y} \in \mathbb {R} ^ {d _ {\mathbb {H}} + 1}, \tag {10}
$$
and hyperspheres  $\mathbb{S}_{K_{\mathbb{S}}}^{d_{\mathbb{S}}}$  as