SlowGuess/NeurIPS_2023_OCR huggingface dataset:
larger this value is, the more distant from zero the values of that row are. Overall, these results suggest that TDGP leans towards learning sparser representations than its competitors.
4.2 Bathymetry case study
Data. As a case-study, we also apply TDGP to the bathymetry dataset GEBCO. This dataset contains a global terrain model (elevation data) for ocean and land. As an example of a non-stationary task, we selected an especially challenging subset of the data covering the Andes mountain range, ocean, and land (see Appendix D.2 for details). We subsample 1,000 points from this region and compare the methods via five-fold cross-validation.
Figure 6: Results for GEBCO bathymetry dataset. The leftmost plot shows the dataset, while the remaining plots show the mean predictions and NLPD of each method. Note how, despite having the best mean posteriors, DKL has an overfitting problem, as shown in the NLPD. TDGP is the most balanced, especially when capturing the transitions between ocean/mountains/plains.
Figure 7: GEBCO dataset: visualization of the latent spaces learned by each model (top) and their inverse lengthscale fields (bottom). Models that do not induce a latent space or a lengthscale field are marked in red crosses. TDGP is the only model that allows both interpretations. For expert users, the lengthscale field is the only one on which informative priors can be put.
Results. The NLPD and MRAE results are listed in Table 2, where we observed our method to be slightly better than others. However, more importantly, this dataset can be thoroughly examined and interpreted. Figure 8 shows correlation plots for a point located on the lower slope of the Andes. We observe more sensible correlations for TDGP and CDGP compared to other methods — the correlation is high in the locations across the slope. This plot also highlights a recurring problem with DNSGP: Despite a high inverse lengthscale barrier in the Andes, there is still a correlation between the slope of the moun
tain and the sea level. Additionally, Fig. 7 shows the domain coloring of learned latent spaces and the sum of the eigenvalues of the lengthscale fields. Once again, we note that only TDGP can be analyzed in both ways. This is an advantage in settings where expert priors on the smoothness exist, as these can be placed directly on the lengthscale field instead of the less accessible latent space mapping. As expected, methods that learn the lengthscale field place high inverse lengthscale values around the mountain range and low values in the smooth oceans and plains.
Table 2: Performance of each model on the GEBCO dataset (avg±std). Lower is better.
NLPD	MRAE
SGP	-0.13 ± 0.09	1.19 ± 0.63
DKL	3.85 ± 0.92	0.59 ± 0.31
CDGP	-0.44 ± 0.12	0.83 ± 0.56
DNSGP	-0.31 ± 0.12	1.12 ± 0.75
TDGP	-0.53 ± 0.10	0.66 ± 0.43
4.3 Benchmark datasets
Data. We also compare the methods on four well-known regression datasets from the UCI repository. To assess each model fairly, we adopt a ten-fold separation of the datasets into training and testing.
Results. Figure 9 shows the average NLPD for each method in the UCI datasets along with their respective standard deviations. TDGP performs either on par with other methods or outperforms them. Figure 10 also shows the relative relevance of each method's latent dimensions. Similarly
Figure 8: Correlation plots on GEBCO dataset for a datapoint marked  $\times$  (1.7, -0.9), which is located close to mountains. We observe better interpretability with TDGP (the zone of high correlation coincides with mountains) compared to DNSGP (high correlation extends into the sea and plains).
Figure 9: Benchmark datasets: test NLPD (lower is better) of each model for 10 folds. Each dot represents the result of a fold. The cross and bar represent the mean and standard deviation across all folds. TDGP is as good or better than the compared models.
to what we observed in the synthetic experiments, TDGP's inductive bias leads to a sharp split between relevant and irrelevant dimensions compared to the other approaches. Even for the cases with similar generalization errors, TDGP shows better interpretability and bias towards learning lower-dimensional representations.
Figure 10: Benchmark datasets: comparison of the relevances of the latent dimensions identified by each model (mean and  $1\sigma$  interval). TDGP consistently identifies low-dimensional latent spaces (most dimensions are irrelevant).
Expressive non-stationary GPs can be constructed in a variety of ways. We give a brief overview of some contributions available in the literature below.
Convolution of SE and scale mixture kernels. Non-stationary covariance learning can be done by convolving two SE kernels with different lengthscales. This well-studied approach was later extended to arbitrary kernels by Paciorek and Schervish [19]. Gibbs [12] introduced this formulation,
representing the lengthscale as the sum of squares of an arbitrary function applied to each covariance input. Later, Higdon et al. [14] introduced this approach in the spatial context, varying the focus points of the ellipse representing the  $2 \times 2$  lengthscale matrix of a SE covariance with independent GPs, which constitutes a two-layer DGP in 2D. Heinonen et al. [13] introduced a method to jointly vary the covariance amplitude and lengthscale, and the observation noise over the domain, using Hamiltonian Monte Carlo for inference. As discussed in Section 2, all these methods inherit the limitations of their kernel, i.e. issues with the pre-factor and violation of the triangle inequality, which hinders manifold learning.
Compositional deep GPs. Compositional DGPs [15, 3] construct priors over complex functions by repeatedly applying a sequence of non-parametric transformations to the input, with each transformation being represented by a GP with a stationary covariance. One of the most significant issues with CDGPs is that, unlike neural networks which compose linear functions through nonlinear scalar activations, every latent layer is a non-linear function. Therefore, several authors have suggested improvements to DGPs by adding skip connections that directly concatenate the hidden layer with the input space [9] or the use of linear mean functions [16, 23] for improved performance and capability. CDGP research also explores improvements in inference, such as the use of mini-batching and more efficient approximations to intractable integrals [23], and the inclusion of auxiliary latent variables to allow for more flexible posteriors [21]. Since TDGP is based on deformation kernels and is related to CDGPs, we believe it can leverage many of these advances in inference, which is left to future investigations.
Deep kernel learning. An alternative to CDGPs is to use layers of parametric transformations to learn arbitrarily complex kernels. Deep kernel learning [1, 26] parameterizes such transformations with a deterministic neural network. Since these deep kernels have many parameters and are not constructed in a Bayesian way, they are prone to overfitting and miscalibrated uncertainties, and can perform poorly in practice [17]. Nevertheless, by using Bayesian injective warping functions that are well-suited for spatial applications, Zammit-Mangion et al. [27] were able to achieve good results in low-dimensional datasets.
This work presented Thin and Deep GP (TDGP), a new hierarchical architecture for DGPs. TDGP can recover non-stationary functions through a locally linear deformation of stationary kernels. Importantly, this construction allows us to interpret TDGP in terms of the latent embeddings and the lengthscale field it induces. Additionally, while TDGP shares a connection with CDGP, our prior does not concentrate on "flat" samples when the number of layers increases — as is the case with CDGP. Furthermore, our experiments show that TDGP performs as well as or better than previous approaches. Moreover, TDGP has a robust inductive bias toward learning low-dimensional embeddings, which is instrumental for better interpretability.
That being said, we expect TDGP will be especially useful for geospatial modeling in cases where we are modeling non-stationary functions and when we have expert knowledge on how this function should vary over space/time, which could be inserted as priors over the locally linear transformations. We also believe that recent improvements in inference available in the literature could greatly enhance the expressivity and ease of training of TDGP.
This work was supported in part by the CONFAP-CNPq-THE UK Academies program (grant UKA-00160-00003.01.00/19). Diego Mesquita was supported by the Silicon Valley Community Foundation (SVCF) through the Ripple impact fund, the Fundação de Amparo à Pesquisa do Estado do Rio de Janeiro (FAPERJ) through the Jovem Cientista do Nosso Estado program, and the Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) through the grant 2023/00815-6.
[24] Michalis K. Titsias and Neil D. Lawrence. "Bayesian Gaussian Process Latent Variable Model". In: Artificial Intelligence and Statistics (AISTATS). 2010.
[25] Michalis K. Titsias and Miguel Lázaro-Gredilla. "Variational Inference for Mahalanobis Distance Metrics in Gaussian Process Regression". In: Advances in Neural Information Processing Systems (NeurIPS). 2013.
[26] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. "Deep Kernel Learning". In: Artificial Intelligence and Statistics (AISTATS). 2016.
[27] Andrew Zammit-Mangion, Tin Lok James Ng, Quan Vu, and Maurizio Filippone. “Deep Compositional Spatial Models”. In: Journal of the American Statistical Association 117.540 (2022).