SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Graph Neural Networks (GNNs) leverage the connectivity structure of graph data to achieve state-of-the-art performance in various applications. Most current GNN architectures assume a fixed topology of the input graph during training. Research has focused on improving diffusion using different types of GNN layers, but discovering an optimal graph topology that can help diffusion has only recently gained attention. In many real-world applications, data may only be accessible as a point cloud of data, making it challenging to access the underlying but unknown graph structure. The majority of Geometric Deep Learning research has relied on human annotators or simplistic pre-processing algorithms to generate the graph structure, and the correct graph may often be suboptimal for the task at hand, which may benefit from rewiring. Latent graph inference refers to the process of inferring the underlying graph structure of data when it is not explicitly available. In many real-world applications, data may only be represented as a point cloud, without any knowledge of the graph structure. However, this does not mean that the data is not intrinsically related, and its connectivity can be utilized to make more accurate predictions.
For these experiments we reproduce the architectures described in Sáez de Ocariz Borde et al. [2023b], see Table 5 and Table 6 for the original architectures. In our case, we use the GCN-dDGM model leveraging the original input graph inductive bias.
Table 5: Summary of model architectures for experiments for Cora and CiteSeer.
Model
MLP	GCN	GCN-dDGM
No. Layer parameters	Activation	Layer type
N/A	N/A	dDGM
(No. features, 32)	ELU	Linear	Graph Conv	Graph Conv
N/A	N/A	dDGM
(32, 16)	ELU	Linear	Graph Conv	Graph Conv
N/A	N/A	dDGM
(16, 8)	ELU	Linear	Graph Conv	Graph Conv
(8, 8)	ELU	Linear	Linear	Linear
(8, 8)	ELU	Linear	Linear	Linear
(8, No. classes)	-	Linear	Linear	Linear
Table 6: dGMM* and dGMM architectures for Cora and CiteSeer.
dDGM*	dDGM
No. Layer parameters	Activation	Layer type
(No. features, 32)	ELU	Linear	Linear
(32, 16 per model space)	ELU	Linear	Graph Conv
(16 per model space, 4 per model space)	Sigmoid	Linear	Graph Conv
D.3.1 Differentiable Graph Module
In their work, Kazi et al. [2022] presented a general method for learning the latent graph by leveraging the output features of each layer. They also introduced a technique to optimize the parameters responsible for generating the latent graph. The key concept is to use a similarity metric between the latent node features to generate optimal latent graphs for each layer  $l$ . In this context,  $\mathbf{X}^{(0)}$  and  $\mathbf{A}^{(0)}$  represent the original input node feature matrix and adjacency matrix, respectively. So that
$$
\mathbf {X} ^ {(0)} = \left[ \begin{array}{c} - \mathbf {x} _ {1} ^ {(0)} - \\ - \mathbf {x} _ {2} ^ {(0)} - \\ \vdots \\ - \mathbf {x} _ {n} ^ {(0)} - \end{array} \right], \tag {55}
$$
and  $\mathbf{A}^{(0)} = \mathbf{A}$  if the adjacency matrix from the dataset, or  $\mathbf{A}^{(0)} = \mathbf{I}$  if  $\mathcal{G} = (\mathcal{V},\emptyset)$ . The proposed architecture in Kazi et al. [2022] consists of two primary components: the Differentiable Graph Module (DGM) and the Diffusion Module. The Diffusion Module,  $\mathbf{X}^{(l + 1)} = g_{\phi}(\mathbf{X}^{(l)},\mathbf{A}^{(l)})$ , may be one (or multiple) standard GNN layers. The first DGM module takes the original node features and connectivity information and produces an updated adjacency matrix
$$
\mathbf {X} ^ {\prime (l = 1)}, \mathbf {A} ^ {(l = 1)} = \operatorname {D G M} \left(\mathbf {X} ^ {(0)}, \mathbf {A} ^ {(0)}\right). \tag {56}
$$
In principle, the DGM module utilizes information from previous layers to generate adjacency matrices at each layer
$$
\mathbf {X} ^ {\prime (l + 1)}, \mathbf {A} ^ {(l + 1)} = \operatorname {D G M} \left(\operatorname {c o n c a t} \left(\mathbf {X} ^ {(l)}, \mathbf {X} ^ {\prime (l)}\right), \mathbf {A} ^ {(l)}\right). \tag {57}
$$
To do so, a measure of similarity is used
$$
\varphi \left(\mathbf {x} _ {i} ^ {\prime (l + 1)}, \mathbf {x} _ {j} ^ {\prime (l + 1)}\right) = \varphi \left(f _ {\boldsymbol {\Theta}} ^ {(l)} \left(\mathbf {x} _ {i} ^ {(l)}\right), f _ {\boldsymbol {\Theta}} ^ {(l)} \left(\mathbf {x} _ {j} ^ {(l)}\right)\right). \tag {58}
$$
In summary, the proposed approach utilizes a parameterized function  $f_{\Theta^{(t)}}$  with learnable parameters to transform node features and a similarity measure  $\varphi$  to compare them. The function can be an MLP or composed of GNN layers if connectivity information is available. The output of the similarity measure is used to create a fully-connected weighted adjacency matrix for the continuous
differentiable graph module (cDGM) approach or a sparse and discrete adjacency matrix for the discrete Differentiable Graph Module (dDGM) approach, with the latter being more computationally efficient and recommended by the authors of the DGM paper [Kazi et al., 2022]. To improve the similarity measure  $\varphi$  and construct better latent graphs, the approach employs product spaces and Riemannian geometry. Additionally, an extra loss term is used to update the learnable parameters of the dDGM module.
Lastly, we will examine the dDGM module, which utilizes the Gumbel Top-k [Kool et al., 2019] technique to generate a sparse  $k$ -degree graph by stochastically sampling edges from the probability matrix  $\mathbf{P}^{(l)}(\mathbf{X}^{(l)};\boldsymbol{\Theta}^{(l)},T)$ , which is a stochastic relaxation of the kNN rule, where each entry corresponds to
$$
p _ {i j} ^ {(l)} = \exp (- \varphi (T) \left(\mathbf {x} _ {i} ^ {\prime (l + 1)}, \mathbf {x} _ {j} ^ {\prime (l + 1)}\right)). \tag {59}
$$
$T$  being a learnable parameter. The primary similarity measure utilized in Kazi et al. [2022] involved computing the distance between the features of two nodes in the graph embedding space
$$
p _ {i j} ^ {(l)} = \exp \left(- T \Delta \left(\mathbf {x} _ {i} ^ {\prime (l + 1)}, \mathbf {x} _ {j} ^ {\prime (l + 1)}\right)\right) = \exp \left(- T \Delta \left(f _ {\boldsymbol {\Theta}} ^ {(l)} \left(\mathbf {x} _ {i} ^ {(l)}\right), f _ {\boldsymbol {\Theta}} ^ {(l)} \left(\mathbf {x} _ {j} ^ {(l)}\right)\right), \right. \tag {60}
$$
where  $\Delta (\cdot ,\cdot)$  denotes a generic measure of distance between two points. They assumed that the latent features laid in an Euclidean plane of constant curvature  $K_{\mathbb{E}} = 0$  , so that
$$
p _ {i j} ^ {(l)} = \exp \left(- T \mathfrak {d} _ {\mathbb {E}} \left(f _ {\boldsymbol {\Theta}} ^ {(l)} \left(\mathbf {x} _ {i} ^ {(l)}\right), f _ {\boldsymbol {\Theta}} ^ {(l)} \left(\mathbf {x} _ {j} ^ {(l)}\right)\right)\right), \tag {61}
$$
where  $\mathfrak{d}_{\mathbb{E}}$  is the distance in Euclidean space. Based on
$$
\operatorname {a r g s o r t} \left(\log \left(\mathbf {p} _ {i} ^ {(l)}\right) - \log (- \log (\mathbf {q}))\right) \tag {62}
$$
where  $\mathbf{q} \in \mathbb{R}^N$  is uniform i.i.d in the interval  $[0, 1]$ , we can sample the edges
$$
\mathcal {E} ^ {(l)} \left(\mathbf {X} ^ {(l)}; \boldsymbol {\Theta} ^ {(l)}, T, k\right) = \left\{\left(i, j _ {i, 1}\right), \left(i, j _ {i, 2}\right), \dots , \left(i, j _ {i, k}\right): i = 1, \dots , N \right\}, \tag {63}
$$
$k$  being the number of sampled connections using the Gumbel Top-k trick. The Gumbel Top-k approach utilizes the categorical distribution  $\frac{p_{ij}^{(l)}}{\Sigma_r p_{ir}^{(l)}}$  for sampling, and the resulting unweighted adjacency matrix  $\mathbf{A}^{(l)}(\mathbf{X}^{(l)};\boldsymbol{\Theta}^{(l)},T,k)$  is used to represent  $\mathcal{E}(\mathbf{X}^{(l)};\boldsymbol{\Theta}^{(l)},T,k)$ . It is worth noting that including noise in the edge sampling process can generate random edges in the latent graphs, which can serve as a form of regularization.
Finally, we can summarize a multi-layer GNN using the dDGM module as
$$
\mathbf {X} ^ {\prime} ^ {(l + 1)} = f _ {\boldsymbol {\Theta}} ^ {(l)} \left(\operatorname {c o n c a t} \left(\mathbf {X} ^ {(l)}, \mathbf {X} ^ {\prime (l)}\right), \mathbf {A} ^ {(l)}\right), \tag {64}
$$
$$
\mathbf {A} ^ {(l + 1)} \sim \mathbf {P} ^ {(l)} \left(\mathbf {X} ^ {\prime (l + 1)}\right), \tag {65}
$$
$$
\mathbf {X} ^ {(l + 1)} = g _ {\phi} \left(\mathbf {X} ^ {(l)}, \mathbf {A} ^ {(l + 1)}\right). \tag {66}
$$
Equations 64 and 65 belong to the dDGM module, while Equation 66 is associated with the Diffusion Module. In our study, we extend Equation 59 to measure distances without relying on the assumption utilized in Kazi et al. [2022], which restricts the analysis to fixed-curvature spaces, specifically to Euclidean space where  $K_{\mathbb{E}} = 0$ .
D.4 Naive Bayesian Optimization
Naive BO in the main text considers performing BO over a fully-connected unweighted graph. Such a graph is latent geometry agnostic, that is, it does not include any metric geometry inductive bias to provide the optimization algorithm with a sense of closeness between the candidate latent geometries.
E Ablation of the Gromov-Hausdorff Coefficients
We evaluate the impact of Gromov-Hausdorff coefficients on the graph search space by conducting an additional set of experiments. We compare the use of an unweighted, pruned graph search space with the introduction of Gromov-Hausdorff coefficients in the datasets considered in this paper. The results are show in Figures 19 to 22 below.
Figure 19: Results (mean and standard deviation over 10 runs) for candidate latent geometries involving product manifolds composed of 13 model spaces. For each plot a different ground truth product manifold  $\mathcal{P}_T$  is used to generate the reference signal.
Figure 20: Results (mean and standard deviation over 10 runs) for candidate latent geometries involving product manifolds composed of 15 model spaces. For each plot a different ground truth product manifold  $\mathcal{P}_T$  is used to generate the reference signal.
Figure 21: Results (mean and standard deviation over 10 runs) on image reconstruction tasks, for  $n_p = 7$ .
Figure 22: Results (mean and standard deviation over 10 runs) on latent graph inference tasks.
The plots above show that in the majority of cases, the GH weights appear to either match or enhance the performance of the search algorithm. This observation is particularly pronounced
in synthetic tasks, which can be intuitively justified due to the significant impact of changes in the latent geometry on the network's output in such a setup. However, we highlight that a large amount of the performance gains in the algorithm seem to come from the inductive bias endowed to the search algorithm through the graph search space. This could be due to the fact that in image reconstruction tasks, searching for the optimal dimension is more important for reconstruction than finding the optimal latent geometry within a particular dimension. Since the GH coefficients are unity between dimensions, this makes the performance of the two search spaces similar in this context.
It should be noted that setting the inverse of the Gromov-Hausdorff distance as the edges of a search graph is only one potential way to add a geometric inductive bias into the search space. Future work could use similar mechanisms as those developed in this paper but in a different context in order to improve optimization performance. Finally, we would like to highlight that a more thorough analysis of the links between the performance of neural networks and the similarity of their latent geometries is a relevant question that merits further study, which is left for future work.