SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CT slices. For IMGP, we use  $\nu = 3$ . In Table 5 we report results for IMGP-S (full) and IMGP-SS (full). These are based on the "exact" eigenpairs, computed by torch.linalg.eigh, as opposed to the standard Lanczos implementation we use by default. Additionally, these include the hyperparameter re-optimization step, as described in Section 5.2 and discussed below. Regarding RMSE, all three compared methods—IMGP-S (full) and IMGP-SS (full) and EGP—exhibit similar performance, with a slight advantage for SS-IMGP (full) as the number of training points increases. When considering NLL, as previously observed with MNIST, SS-IMGP performs best in all settings. However, NLL decreases for larger values of  $n / N$ , indicating a probable overfit in this regime.
Hyperparameter re-optimization. We observed this step to serve two important purposes: (1) fixing overly small values of the signal variance  $\sigma^2$ , potentially caused by the absence of covariance normalization in the optimization process and poor convergence; (2) adjusting the length scale parameter to take into account the loss of high-frequency components due to truncation.
Implementation. Currently, the implementation faces two significant limitations that might restrict its usability to high-memory hardware setups. Firstly, due to the absence of rich enough sparse matrix routines in PyTorch, we had to develop our own custom implementation of differentiable sparse operators. We kept to high-level routines, which forced us to strike a balance between performance and memory efficiency. In particular, for matrix-vector sparse product operations, our approach relies on highly optimized vectorized code, delivering high performance on GPU at the expense of increased memory allocation. Secondly, we faced challenges with sparse eigen-solvers in the PyTorch ecosystem. Our attempts of using the PyTorch implementation of the LOBPCG algorithm, torch.lobpcg, yielded relatively poor results. We had similar experience with the Lanczos implementation from GPyTorch (Gardner et al., 2018). In light of this, when extracting higher-frequency components was needed, we resorted to two alternative solutions: PyTorch dense matrix eigen-decomposition torch.linalg.eigh and the SciPy Arpack wrapper scipy.linalg.eigsh. The first approach, while benefiting from GPU acceleration, can be infeasible because of limited GPU memory. The second approach, known for its efficiency in memory usage due to its Krylov subspace-based nature, is constrained to CPU utilization, significantly impacting the algorithm's overall performance.
C Hyperparameter Priors and Initialization
Here we describe hyperpameter priors which might be of help when using implicit manifold Gaussian process regression.
Graph Bandwidth  $\alpha$ . Graph Laplacian converges to the Laplace-Beltrami operator when  $\alpha$  tends to zero, motivating smaller  $\alpha$ . However, in a non-asymptotic setting it is impractical to have  $\alpha$  overly small as it will render the graph effectively disconnected and cause numerical instabilities. One appropriate prior could thus be gamma distribution, whose right tail discourages high values of  $\alpha$ , and which, given an appropriate choice of the parameters, encourages  $\alpha$  to align with the scale of pairwise distances between graph nodes. Specifically, we choose the parameters of the gamma distribution so as to (1) match its mode with the median  $(Q_{2})$  of pairwise average distance between  $K$ -th nearest neighbors because such an  $\alpha$  would give reasonable weights in the KNN graph and (2) so as to place its standard 0.95 confidence interval to the right of a certain lower bound  $\bar{\alpha}$  we define further that ensures the graph is numerically not disconnected. Let
$$
\mathcal {D} = \left\{\max  _ {\boldsymbol {x} _ {i} \in \operatorname {K N N} (\boldsymbol {x} _ {j})} \| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} \| \text {w h e r e} j = 1,.., N \right\}. \tag {47}
$$
Dataset	n/N	RMSE	NLL
S-IMGP	SS-IMGP	EGP	S-IMGP	SS-IMGP	EGP
SR-MNIST	10%	0.04 ± 0.01	0.01 ± 0.00	0.12 ± 0.01	-1.42 ± 0.01	-1.52 ± 0.01	-0.54 ± 0.01
MR-MNIST	1%	0.74 ± 0.02	0.43 ± 0.02	0.74 ± 0.02	2.24 ± 0.20	-0.59 ± 0.01	-0.20 ± 0.01
MR-MNIST	10%	0.14 ± 0.05	0.03 ± 0.00	0.13 ± 0.00	-0.68 ± 0.08	-0.79 ± 0.00	-0.43 ± 0.01
Table 4: Results for the rotated MNIST dataset.
n/N	RMSE	NLL
S-IMGP (full)	SS-IMGP (full)	EGP	S-IMGP (full)	SS-IMGP (full)	EGP
5%	0.27 ± 0.02	0.39 ± 0.04	0.24 ± 0.02	0.64 ± 0.83	-2.48 ± 0.08	-0.80 ± 0.02
10%	0.22 ± 0.02	0.19 ± 0.01	0.16 ± 0.00	0.88 ± 0.29	-2.35 ± 0.04	-0.96 ± 0.00
25%	0.14 ± 0.01	0.08 ± 0.02	0.09 ± 0.01	-0.42 ± 0.10	-1.99 ± 0.04	-1.20 ± 0.09
50%	0.08 ± 0.01	0.07 ± 0.01	0.08 ± 0.04	-0.96 ± 0.11	-2.04 ± 0.02	-1.02 ± 0.04
Table 5: Results for Relative location of CT slices on axial axis ( $d = 385$ ,  $N = 48150$ ) from UCI Machine Learning Repository.
(a) Prior for the semi-supervised case
(b) Prior for the supervised case
Figure 7: The histogram of  $\mathcal{D}$  and the prior for the bandwidth hyperparameter  $\alpha$ .
The bandwidth's lower bound we compute as
$$
\bar {\alpha} = \min  _ {d \in \mathcal {D}} \sqrt {- \frac {d ^ {2}}{4 \log \tau}}, \tag {48}
$$
where  $\tau$  is a user-defined parameter.
Let  $\eta$  and  $\beta$  the shape and rate parameters of the gamma distribution. In order to achieve (1), we define
$$
\eta = \rho Q _ {2} + 1 \quad \text {a n d} \quad \beta = \rho \tag {49}
$$
where  $\rho$  is used to achieve (2) and is given by
$$
\rho \approx \frac {4 Q _ {2}}{\left(Q _ {2} - \bar {\alpha}\right) ^ {2}}. \tag {50}
$$
Considering the S-MNIST dataset, Figure 7 shows the bandwidth prior distribution for the semi-supervised (labeled and unlabeled points) and the supervised (labeled points) scenarios.
Signal Variance  $\sigma_f^2$ . Assuming normalized  $y_{i}$ , the signal variance  $\sigma_f^2$  should be close to 1. Because of this a natural prior for  $\sigma^2$  is the truncated normal (onto the set of positive reals  $\sigma^2 >0$ ), with mode at 1. The pre-truncation variance can be chosen, for example, to have 0 at 3 standard deviations away from the mode, i.e. it can be chosen to be  $1 / 9$ . Note that setting a prior over the signal variance parameter requires evaluating the normalization constant  $C_{\nu ,\kappa}^{7}$  for the kernel at each hyperparameter optimization step, something that can be avoided otherwise.
Noise Variance  $\sigma_{\varepsilon}^{2}$ . Choosing a prior for the noise variance  $\sigma_{\varepsilon}^{2}$  is heavily problem-dependent. Truncated normal (onto the set  $\sigma_{\varepsilon}^{2} > 0$ ) with mode at 0 could be a reasonable option. The truncation variance can be chosen, for example, to be  $1, 1/4$  or  $1/9$ , placing the value 1, which is the variance of the normalized observations  $y_{i}$ , at 1, 2 or 3 standard deviations away from the mode.
${}^{7}$  Covariance matrix normalization constant  ${C}_{\nu ,\kappa }$  can be approximated as  ${C}_{\nu ,\kappa } \approx  \frac{1}{M}\mathop{\sum }\limits_{{i = 1}}^{M}{\mathbf{e}}_{i}^{T}{\mathbf{P}}_{\mathbf{{XX}}}^{-1}{\mathbf{e}}_{i}$  , where  $M$  is relatively small,  ${\mathbf{e}}_{i}$  are random standard basis vectors and the solve is performed by running the conjugate gradients. Differentiability of the model is preserved. Note however that we did not use this normalization in our tests since the performance improvement we observed was not enough to justify the additional computation overhead. This trade-off can vary considerably from case to case, which is why in our implementation, the covariance normalization is optional.
Length scale The interpretation and the scale of the length scale parameter is manifold-specific. This makes it very difficult to come up with any reasonable prior. Because of this, we suggest actually leaving the length scale parameter free.
Parameter initialization When doing MAP estimation, one can initialize parameters randomly, sampling them from respective priors.