SlowGuess/NeurIPS_2023_OCR huggingface dataset:
spaces are continuous manifolds. To approximate the Gromov-Hausdorff distance between them we must discretize the spaces. A more fine-grained discretization results in a better approximation. As previously mentioned in Section 2, to compare  $\mathbb{E}^n$ ,  $\mathbb{S}^n$ , and  $\mathbb{H}^n$ , we can take closed balls of radius one in each space. Since these spaces are homogeneous Riemannian manifolds, every ball of radius one is isometric. We can estimate or provide an upper bound for their Gromov-Hausdorff distance by using this method.
In the case of the Euclidean plane, we sample points from the ball,  $B_{\mathbb{E}^2} = \{(r\cos (t),r\sin (t)):r\in [0,1],t\in [0,2\pi)\}$ , with a discretization of 10,000 points in both  $r$  and  $t$ . To obtain points in  $B_{\mathbb{H}^2}$  we will use the points in  $B_{\mathbb{E}^2}$  as a reference, and apply the exponential map, and the change of coordinates described in Appendix B.1 to convert points in  $B_{\mathbb{E}^2}$  to points in  $B_{\mathbb{H}^2}$ . However, due to numerical instabilities instead of sampling from  $B_{\mathbb{E}^2}$ , we will restrict ourselves to  $B_{\mathbb{E}^2}' = \{(r\cos (t),r\sin (t)):r\in [0.00000001,0.97],t\in [0,2\pi)\}$  and use a discretization of 10,000 as before. Lastly to sample points for the spherical space,  $B_{\mathbb{S}^2} = \{(\sin (\beta)\cos (\alpha),\sin (\beta)\sin (\alpha),\cos (\beta)): \alpha \in [0,2\pi),\beta \in [0,1]\}$ , we use a discretization of 100 for both  $\alpha$  and  $\beta$ . The granularity of the discretization was chosen as a trade-off between resolution and computational time. We observed that the Gromov-Hausdorff distance stabilized for our discretization. However, given the nature of the Gromov-Hausdorff distance, it is difficult to conclude whether some unexpected behaviour could be observed with greater discretization.
B.4.2 Calculating Constants for the Embedding Function
The next step is to obtain a computational approximation of the mapping function defined in Appendix B.2. The embedding of  $\mathbb{H}^n$  into  $\mathbb{E}^{6n - 6}$  requires computing several constants. Using a standard integral solvers  $A\approx 0.141328$ , (Equation 25) can be approximated. To calculate the constant  $c\approx 10.255014502464228$ , we discretize the input  $x\in [-2,2]$  using a step size of  $10^{-8}$  to compute  $G_{1}$  and  $G_{2}$  and use a loop to calculate the max in Equation 28. Note that this requires to constantly reevaluate the integrals for  $\psi_1(x)$  and  $\psi_{2}(x)$ . Likewise,  $G_{1}$  and  $G_{2}$  are also used to obtain  $\varepsilon$  in Equation 33, which is in turn used to calculate the mapping function that maps points in  $\mathbb{H}^2$  to the higher dimensional embedding Euclidean space  $\mathbb{E}^6$ . We use  $F$  to map  $B_{\mathbb{H}^2}$  to  $\mathbb{E}^6$ , and we keep those points fixed in space.
B.4.3 Optimizing the Embedding Functions for the Euclidean and Spherical Spaces
Next to approximate the Gromov-Hausdorff distances:
$$
\mathrm {d} _ {\mathrm {G H}} \left(B _ {\mathbb {E} ^ {2}}, B _ {\mathbb {H} ^ {2}}\right) \approx \min  _ {i, j, k} \mathrm {d} _ {\mathrm {H}} ^ {\mathbb {R} ^ {6}, f _ {k}, F} \left(P _ {i} ^ {H}, Q _ {j}\right) = \min  _ {i, j, k} \mathrm {d} _ {\mathrm {H}} ^ {\mathbb {R} ^ {6}} \left(f _ {k} \left(P _ {i} ^ {H}\right), F \left(Q _ {j}\right)\right), \tag {38}
$$
$$
\mathrm {d} _ {\mathrm {G H}} \left(B _ {\mathbb {S} ^ {2}}, B _ {\mathbb {H} ^ {2}}\right) \approx \min  _ {i, j, k} \mathrm {d} _ {\mathrm {H}} ^ {\mathbb {R} ^ {6}, g _ {k}, F} \left(P _ {i} ^ {S}, Q _ {j}\right) = \min  _ {i, j, k} \mathrm {d} _ {\mathrm {H}} ^ {\mathbb {R} ^ {6}} \left(g _ {k} \left(P _ {i} ^ {S}\right), F \left(Q _ {j}\right)\right), \tag {39}
$$
we must optimize  $f_{k}$  and  $g_{k}$ . These are the functions used to embed  $B_{\mathbb{E}^2}$  and  $B_{\mathbb{S}^2}$  in  $\mathbb{E}^{6n - 6}$ . Note that in practice,  $B_{\mathbb{E}^2}$  and  $B_{\mathbb{S}^2}$  are discretized into  $P_i^H$  and  $P_i^S$ , respectively.
To optimize for  $f_{k}$  we consider all possible permutations of the basis vectors of  $\mathbb{E}^6$ :  $\{e_1, e_2, e_3, e_4, e_5, e_6\}$  for each  $f_{k}$  we consider two elements  $\{e_i, e_j\}$  and use those dimensions to embed  $\mathbb{E}^2$  in  $\mathbb{E}^6$ . In principle, we should also consider a small offset given by  $F(\mathbf{0}) = \mathbf{0}$ , but it is zero regardless. Additionally, during the optimization we also add a small vector (with all entries but a single dimension between zero) to the mapping function to translate the plane in different directions by an offset of between -0.5 and 0.5, with a total of a 100 steps between these two quantities.
To optimize for  $g_{k}$  we follow a similar procedure in which we consider all permutations of the basis vectors. Note however, that in this case we would have three basis vectors instead of two, given how  $B_{\mathbb{S}^2} = \{(\sin (\beta)\cos (\alpha),\sin (\beta)\sin (\alpha),\cos (\beta)): \alpha \in [0,2\pi), \beta \in [0,1]\}$  is sampled. For each permutation family,  $P_{i}^{S}$ , we also consider its negative counterpart,  $-P_{i}^{S}$ , to compute the Gromov-Hausdorff distance as well as experimenting with offsetting the mapping function.
B.5 Derivation of Number of Product Manifold Combinations in the Search Space
Here, we derive the exact number of nodes in the graph search space in our setting. In particular, we consider the case with three model spaces (the Euclidean plane, the hyperboloid and the hypersphere). The growth of the search space can be modelled with growth of a tree, as depicted in Figure 11.
Figure 11: The growth of the graph search space as a function of the number of model spaces used to generate product manifolds can be represented as a tree. Note that as discussed in Section 3.3 we assume commutativity:  $\mathcal{M}_i \times \mathcal{M}_j = \mathcal{M}_j \times \mathcal{M}_i$ .
To calculate the number of elements in the search space, we define the total number of products at level  $h$  of the tree as  $N$ . We have that  $N(h) = N_{E}(h) + N_{H}(h) + N_{S}(h)$ , where  $N_{E}(h)$  is the number of Euclidean spaces added to the product manifolds at depth  $h$  of the tree, and  $N_{H}(h)$  and  $N_{S}(h)$  represent the same for the hyperboloid and the hypersphere respectively. By recursion, we can write
$$
N _ {E} (h) = N _ {E} (h - 1) = 1 \tag {40}
$$
$$
N _ {H} (h) = N _ {E} (h - 1) + N _ {H} (h - 1) \tag {41}
$$
$$
= 1 + h \tag {42}
$$
$$
N _ {S} = N _ {E} (h - 1) + N _ {H} (h - 1) + N _ {S} (h - 1) \tag {43}
$$
$$
= \frac {h (h + 1)}{2} N _ {E} (1) + h N _ {H} (1) + N _ {S} (1) \tag {44}
$$
$$
= \frac {h (h + 1)}{2} + h + 1 \tag {45}
$$
hence we have that
$$
N (h) = 1 + (1 + h) + (1 + h + \frac {h (h + 1)}{2}) \tag {46}
$$
$$
= 3 + \frac {5}{2} h + \frac {1}{2} h ^ {2} \tag {47}
$$
To be consistent with the previous notation, we write the above in terms of the number of products  $n_p$
$$
N \left(n _ {p}\right) = 3 + \frac {5}{2} n _ {p} + \frac {1}{2} n _ {p} ^ {2} \tag {48}
$$
The total number of nodes in the graph search space for a number of product  $n_p$  is then
$$
N _ {T} = \sum_ {i = 1} ^ {n _ {p}} N (i) \tag {49}
$$
B.6 Visualizing the Graph Search Space
In this appendix, we give a visual depiction of the graph search space we construct for neural latent geometry search. Figure 12 and Figure 13 provide plots of the graph search space as the size of the product manifolds considered increases. For product manifolds composed of a high number of model spaces, visualizing the search space becomes difficult. We omit the strengths of the connections for visual clarity in this plots.
If we focus on Figure 12, we use e, h, and s to refer to the model spaces of constant curvature  $\mathbb{E}^2$ ,  $\mathbb{H}^2$ , and  $\mathbb{S}^2$ . In this work, we have considered model spaces of dimensionality two, but the graph search space would be analogous if we were to change the dimensionality of the model spaces. Nodes that include more than one letter, such as hh, ss, ee, etc, refer to product manifolds. For example, hh would correspond to the product manifold  $\mathbb{H}^2 \times \mathbb{H}^2$ . As discussed in Section 3.3, we can see that connections are only allowed between product manifolds that differ by a single model space. To try to clarify this further, we can see that e, h and s are all interconnected since they only differ by a single model space (one deletion and one addition). Likewise, e is connected to ee, eh, and ee since they only differ by a single model space (one addition). However, e is not connected to hs (one deletion and two additions) nor is ee connected to ss (two deletions and two additions).
Figure 12: Graph search spaces for  $n_p = 2$  (left) and  $n_p = 3$  (right). Strength of connectivity is not depicted in the graph.
Figure 13: Graph search spaces for  $n_p = 4$  (left) and  $n_p = 5$  (right). Strength of connectivity is not depicted in the graph.
Figure 14: Graph search spaces for  $n_p = 8$ .
Figure 15: Graph search spaces for  $n_p = 10$ .
Figure 16: Graph search spaces for  $n_p = 12$ .
Figure 17: Graph search spaces for  $n_p = 14$ .
B.7 Motivating Gromov-Hausdorff Distances for Comparing Latent Geometries
The use of Gromov-Hausdorff distances can be motivated from a theoretical and practical perspective. From a theoretical perspective, the Gromov-Hausdorff distance offers a way to gauge the similarity between metric spaces. This is achieved by casting their representation within a shared space, all the while maintaining the original pairwise distances. In the context of machine learning models, particularly those involving generative models (such as VAEs or GANs), the latent space assumes a pivotal role in shaping the characteristics of the generated data. This also holds true in the case of reconstruction tasks, such as those carried out by autoencoders. Through the application of a metric that factors in their inherent geometries, the aim is to capture the concept of resemblance in data generation, reconstruction, and other downstream tasks which will be heavily affected by the choice of latent geometry.
While traditional metrics like Euclidean or cosine distances might suffice for some cases, they do not always capture the complex and nonlinear relationships between data points in high-dimensional spaces. The Gromov-Hausdorff distance considers the overall structure and shape of these spaces, rather than just individual distances, which can lead to better generalization and robustness. This is especially relevant when dealing with complex data distributions or high-dimensional latent spaces. Moreover, another appealing aspect of Gromov-Hausdorff distance is its invariance to isometric transformations, such as rotations, translations, and reflections. This is a desirable property when comparing latent spaces, as the metric used should reflect the similarity in shape and structure, rather than being influenced by trivial transformations.
While the direct relationship between Gromov-Hausdorff distance and model performance might not be immediately obvious, we argue that if two models have similar latent space geometries, this suggests that they might capture similar underlying data structures, which could lead to similar performance on downstream tasks. This is known as an assumption of smoothness in the NAS literature, see Oh et al. [2019]. However, it is important to note that this relationship is not guaranteed, and the effectiveness of Gromov-Hausdorff distance as a proxy measure depends on the specific application and the nature of the models being compared.
B.8 Method Scalability
The proposed approach sidesteps scalability concerns by preventing the need to recalibrate GH coefficients, as long as one adheres to latent spaces derived from products of model spaces of dimension two. If a higher-dimensional latent space was needed, this could be solved by adding additional "graph slices" with augmented dimensions (as depicted in Figure 2). These slices can be integrated with the lower-dimensional counterpart of the graph search space following the procedure described in the paper. The number of coefficients remains constant at 3, rendering the Gromov-Hausdorff distances free from extra computation overhead when increasing the dimension.
Moreover, in alignment with the manifold hypothesis, data is expected to exist within a low-dimensional manifold. This discourages the exploration of higher dimensions for latent space modeling. It is important to note, nonetheless, that the paper has already delved into relatively large search spaces, which include the exploration of high-dimensional latent spaces.
C Background on Bayesian Optimization
Bayesian optimization (BO) is a query-based optimization framework for black-box functions that are expensive to evaluate. It builds a probabilistic model of the objective function using past queries to automate the selection of meta-parameters and minimize computational costs. BO seeks to find the global optimum of a black-box function  $\mathfrak{f}:\mathcal{X}\to \mathbb{R}$  by querying a point  $\mathbf{x}_n\in \mathcal{X}$  at each iteration  $n$  and obtaining the value  $y_{n} = \mathfrak{f}(\mathbf{x}_{n}) + \varepsilon$ , where  $\varepsilon \sim \mathcal{N}(0,\sigma^2)$  is the noise in the observation. To do so, BO employs a surrogate to model the function  $\mathfrak{f}(\cdot)$  being minimized given the input and output pairs  $\mathcal{D}_t = \{(x_i,y_i = \mathfrak{f}(x_i)\}_{i = 1}^N$ , and selects the next point to query by maximizing an acquisition function. In our work, we use the Gaussian Process (GP)[Williams and Rasmussen, 2006] as the surrogate model and expected improvement (EI)[Mockus et al., 1978] as the acquisition function.
Preliminaries. Bayesian optimization (BO) is particularly beneficial in problems for which evaluation is costly, behave as a black box, and for which it is impossible to compute gradients with respect to the loss function. This is the case when tuning the hyperparameters of machine learning models. In our
case, we will use it to find the optimal product manifold signature. Effectively, Bayesian optimization allows us to automate the selection of critical meta-parameters while trying to minimize computational cost. This is done building a probabilistic proxy model for the objective using outcomes recorded in past experiments as training data. More formally, BO tries to find the global optimum of a black-box function  $\mathfrak{f}:\mathcal{X}\to \mathbb{R}$ . To do this, a point  $\mathbf{x}_n\in \mathcal{X}$  is queried at every iteration  $n$  and yields the value  $y_{n} = \mathfrak{f}(\mathbf{x}_{n}) + \varepsilon$ , where  $\varepsilon \sim \mathcal{N}(0,\sigma^2)$ , is a noisy observation of the function evaluation at that point. BO can be phrased using decision theory
$$
L \left(\mathfrak {f}, \left\{\mathbf {x} _ {n} \right\} _ {n = 1} ^ {N}\right) = N \times c + \min  _ {1 \leqslant n \leqslant N} \mathfrak {f} \left(\mathbf {x} _ {n}\right), \tag {50}
$$
where  $N$  is the maximum number of evaluations,  $c$  is the cost incurred by each evaluation, and the loss  $L$  is minimized by finding a lower  $\mathfrak{f}(\mathbf{x})$ . In general, Equation 50 is intractable and the closed-form optimum cannot be found, so heuristic approaches must be used. Instead of directly minimizing the loss function, BO employs a surrogate model to model the function  $\mathfrak{f}(\cdot)$  being minimized given the input and output pairs  $\mathcal{D}_t = \{\left(\mathbf{x}_i,y_i = \mathfrak{f}(x_i)\right)_{i = 1}^N\}$ . Furthermore, in order to select the next point to query, an acquisition function is maximized. In this work, the surrogate model is chosen to be a Gaussian Process (GP) [Williams and Rasmussen, 2006] and expected improvement (EI) [Mockus et al., 1978] is used as the acquisition function.
Gaussian Processes. A GP is a collection of random variables such that every finite collection of those random variables has a multivariate normal distribution. A GP is fully specified by a mean function,  $\mu (\mathbf{x})$ , and covariance function (or kernel), denoted as  $k(\mathbf{x},\mathbf{x}^{\prime})$ . The prior over the mean function is typically set to zero, and most of the complexity of the model hence stems from the kernel function. Given  $t$  iterations of the optimization algorithm, with an input  $\mathbf{x}_t = [x_1,\dots ,x_t]^T$  and output  $\mathbf{y}_t = [y_1,\dots ,y_t]^T$ , the posterior mean and variance are given by
$$
\mu \left(x _ {t + 1} \mid \mathcal {D} _ {t}\right) = \mathbf {k} \left(x _ {t + 1}, \mathbf {x} _ {t}\right) \left[ \mathbf {K} _ {1: t} + \sigma_ {n} ^ {2} \mathbf {I} _ {t} \right] ^ {- 1} \mathbf {y} _ {t}, \tag {51}
$$
$$
\sigma \left(x _ {t + 1} \mid \mathcal {D} _ {t}\right) = \mathbf {k} \left(x _ {t + 1}, x _ {t + 1}\right) - \mathbf {k} \left(x _ {t + 1}, \mathbf {x} _ {t}\right) \left[ \mathbf {K} _ {1: t} + \sigma_ {n} ^ {2} \mathbf {I} _ {t} \right] ^ {- 1} \mathbf {k} \left(\mathbf {x} _ {t}, x _ {t + 1}\right), \tag {52}
$$
where we define  $[\mathbf{K}_{1:t}]_{i,j} = k(x_i,x_j)$  is the  $(i,j)$ -th element of the Gram matrix.
Expected Improvement. EI is a widely used acquisition function for Bayesian optimization. EI improves the objective function through a greedy heuristic which chooses the point which provides the greatest expected improvement over the current best sample point. EI is calculated as
$$
\alpha_ {\mathrm {E I}} (\mathbf {x}) = \sigma (\mathbf {x}) \left[ \Gamma (\mathbf {x}) \Phi (\Gamma (\mathbf {x})) + \mathcal {N} (\Gamma (\mathbf {x}) | 0, 1) \right], \tag {53}
$$
$$
\Gamma (\mathbf {x}) = \frac {\mathrm {f} (\mathbf {x} _ {\text {b e s t}}) - \mu (\mathbf {x})}{\sigma (\mathbf {x})}, \tag {54}
$$
and  $\Phi (\cdot)$  denotes the CDF of a standard normal distribution.
In this section we provide additional details for the implementation of the experimental setup, including how the datasets are generated, relevant theoretical background, and the hyperparameters used for BO over our discrete graph search space.
D.1 Synthetic Experiments
To assess the effectiveness of the proposed graph search space, we conduct a series of tests using synthetic data. Our objective is to establish a scenario in which we have control over the latent space and manually construct the optimal latent space product manifold. To achieve this, we initiated the process by generating a random vector, denoted as  $\mathbf{x}$ , and mapping it to a predetermined "ground truth" product manifold  $\mathcal{P}_T$ . This mapping is performed using the exponential map for the product manifold  $\mathcal{P}_T$ , which can be derived based on the model spaces of constant curvature that constitute it. Subsequently, the resulting projected vector is decoded via a neural network with fixed weights, denoted as  $f_{\theta}$ , to yield a reference signal  $y^{\mathcal{P}_T}$ . The rationale behind employing a frozen network is
to introduce a non-linear mapping from the latent space to the signal space. This approach aims to mimic the behavior of a trained network in a downstream task, while disregarding the specific task or model weights involved. By utilizing a frozen network, we can capture the essence of a non-linear mapping without relying on the exact details of the task or specific model weights. The primary goal is to recover this reference signal using the neural latent geometry search (NLGS) approach. To generate the remaining dataset, we employ the same random vector and mapped it to several other product manifolds, denoted as  $\{\mathcal{P}_i\}_{i\in n_{\mathcal{P}}}$ , comprising an equivalent number of model spaces as  $\mathcal{P}_T$  but with distinct signatures. Decoding the projected vector using the same neural network produces a set of signals, denoted as  $\{y^{\mathcal{P}_i}\}_{i\in n_{\mathcal{P}}}$ . To populate the nodes of the graph search space, we assign the corresponding value of mean squared error (MSE) between  $y^{\mathcal{P}_T}$  and  $y^{\mathcal{P}_i}$  for each pair. In this manner, our search algorithm aims to identify the node within the graph that minimizes the error, effectively determining the latent geometry capable of recovering the original reference signal. A schematic of the proposed method and specifics on the construction of the network used to decode the signal are shown in Figure 18 and Table 3.
Figure 18: Schematic of procedure used to generate synthetic datasets. We model  $f$  as an MLP.
Table 3: Summary of network to generate synthetic datasets.
Model
Linear (data dim, 100) - ELU
Linear (100, 100) - ELU
Linear (100, 100) - ELU
Linear (100, 5) - ELU
D.2 Autoencoders
Autoencoders are a type of neural network architecture that can be used for unsupervised learning tasks. They are composed of two main parts: an encoder and a decoder. The encoder takes an input and compresses it into a low-dimensional representation, while the decoder takes that representation and generates an output that tries to match the original input. The goal of an autoencoder is to learn a compressed representation of the input data that captures the most important features, and can be used for tasks such as image denoising, dimensionality reduction, and anomaly detection. They have been used in a wide range of applications, including natural language processing, computer vision, and audio analysis.
Table 4: Autoencoder architecture summary.
Component	Layers
Encoder	Conv2d (1, 20, 3) - BatchNorm2d - SiLUConv2d (20, 20, 3) - BatchNorm2d - SiLU
(9 repetitions)
Conv2d (20, 2, 3) - BatchNorm2d - SiLUFlatten - Linear - SiLU
Decoder	Linear - SiLU - UnflattenConvTranspose2d (2, 20, 3) - BatchNorm2d - SiLUConvTranspose2d (20, 20, 3) - BatchNorm2d - SiLU
(9 repetitions)
ConvTranspose2d (20, 1, 3) - BatchNorm2d - Sigmoid
The autoencoder's objective is to learn a compressed representation (latent space) of the input data and use it to reconstruct the original input as accurately as possible. In our experiments, the encoder takes an input image and applies a series of convolutional layers with batch normalization and SiLU activation functions, reducing the image's dimensions while increasing the number of filters or feature maps. The final output of the encoder is a tensor of shape (batchsize, latentdim, 6, 6) or (batchsize, latentdim, 10, 10), where latentdim is the desired size of the latent space. After flattening this tensor, a fully connected layer is used to map it to the desired latent space size. The encoder's output is a tensor of shape (batchsize, latentdim), which represents the compressed representation of the input image. The decoder takes the latent space representation as input and applies a series of transpose convolutional layers with batch normalization and SiLU activation functions, gradually increasing the image's dimensions while decreasing the number of feature maps. The final output of the decoder is an image tensor of the same shape as the input image. The loss functions used for training are the MSELoss (mean squared error) and BCELoss (binary cross-entropy) from the PyTorch library. A summary of the autoencoder is provided in Table 4.
D.3 Latent Graph Inference