SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the second simulation, our aim is to assess parameter identifiability in a GP with a mixture kernel consisting of Matérn kernels with distinct smoothness. The mixture kernel is denoted as  $K = \sum_{l=1}^{3} w_l K_l$ , where  $K_l$  is a Matérn kernel with parameters  $(\sigma_l^2, \alpha_l, \nu_l)$ . For this simulation, we set  $\nu_1 = 1/2, \nu_2 = 3/2, \nu_3 = 5/2$ . Theorem 2 indicates that the only identifiable parameter is
$w_{l}\sigma_{l}^{2}\alpha_{l}$ , while all other parameters remain unidentifiable. In this scenario, the identifiable parameter, also known as the microergodic parameter, is associated with the least smooth component, i.e., the Matérn  $1/2$  kernel. We will assess parameter inference by comparing estimated values to their true values across different training sample sizes. For this experiment, we generate  $n$  samples, ranging from 20, 50, 100, 500. For each sample size, we replicate the simulation 100 times.
Figure 2: Parameter estimation. (A) Only the microergodic parameter  $w_{1}\sigma_{1}^{2}\alpha_{1}$  is identifiable. (B-F) other parameters are not identifiable. The red line indicates the true value.
The results provide persuasive support for Theorem 2 that, in the context of additive mixture of Matérn kernels with distinct smoothness, only the MLE of the microergodic parameter  $w_{1}\sigma_{1}^{2}\alpha_{1}$  for the least smooth component converges to the true value. The MLEs for all other parameters do not converge to their respective true values, highlighting the unique identifiability of the parameters in the least smooth component within the additive mixture kernel framework. Such results are robust in terms of optimizer choice and consistent as the sample size increases to 3000 (details in the Supplement).
5.3 Simulation 3 - Multivariate GP
Next, we turn our attention to multivariate GPs with a separable kernel. We define the separable kernel as  $K = AK_{0}$ . Here, we select Matérn  $1/2$  as  $K_{0}$ , characterized by the parameters  $(\sigma^{2},\alpha)$ . Our Theorem 3 suggests that only  $A\alpha^{2\nu}\sigma^{2}$  is identifiable. To verify this, we adopt a bivariate setup where  $A$  is a  $2\times 2$  positive definite matrix. In this simulation, the sample size  $n$  varies from 50, 100, 200, 400. For each sample size, we replicate the simulation 100 times.
The parameter estimates are summarized in Figure 3. Evidently, only the MLE of the microergodic parameter  $A\sigma^2\alpha^{2\nu}$  converges to its true value. These findings reinforce the identifiability and interpretability of the separable kernel. This understanding is critical for the interpretation and application of separable kernels in real-world situations.
6 Application
Motivated by the findings in our theoretical and simulation studies, we proceed to compare the performance of the mixture kernel with varying smoothness and the least smooth component within that mixture only, in real-world data applications. In this section, we will revisit several applications in previous studies (Rasmussen and Williams (2006), Wilson et al. (2014), Sun et al. (2018)), illustrating that the mixture kernel achieves prediction performance comparable to the least smooth kernel component within the mixture.
Figure 3: Parameter estimation. (A-C) Only the microergodic parameter  $A\sigma^2\alpha^{2\nu}$  is identifiable (D-E). All other parameters are not identifiable. The red dotted line indicates the true value.
6.1 Application 1 - Image prediction
For our first real-data application, we delve into the domain of image analysis. GP has been widely used to predict missing image (Wilson et al. (2014), Sun et al. (2018)). This task of image prediction is an exemplary context for assessing kernel performance in discerning local correlation patterns, providing a visually intuitive framework for comparative analysis.
We employ a handwritten zero image from the MNIST dataset (LeCun et al. (1998)) for this analysis; we extract a  $20 \times 20$  section from the center of the original  $100 \times 100$  image to serve as our test image. This results in a training dataset of 9600 pixels and a testing dataset with 400 pixels (Figure 4A). The input for this analysis comprises the 2-D pixel locations, with the corresponding pixel intensities serving as the response. To reconstruct the missing area, we train GP regression models with both the mixture kernel with varying smoothness and a single Matérn  $1/2$  kernel on the training dataset, and subsequently make predictions on the testing dataset. Specifically, we employ a mixture of three Matérn kernels of smoothness  $1/2$ ,  $3/2$ ,  $5/2$ . Our analysis seeks to demonstrate the similarities in performance between the mixture kernel and the least smooth component within that mixture, i.e., Matérn  $1/2$ , in terms of prediction in the missing image area.
A Training Image
B Matérn mixture
Figure 4: Texture exploration in diamond plate. (A) Training image. (B) Prediction of the mixture kernel. The area in the pink box represents the testing image area. (C) Predication of the Matern  $1 / 2$  kernel.
C Matérn 1/2
The predictions generated by both the mixture kernel and the Matérn 1/2 kernel demonstrate a remarkable similarity, as shown in Figure 4. This observation suggests that the mixture kernel and a single Matérn 1/2 kernel have comparable performance. The close alignment between their predictions implies that using the mixture kernel does not yield significant advantages over the least smooth kernel component in this specific application, further reinforcing our theoretical findings (Theorem 2) regarding the dominance of the least smooth kernel component in the mixture kernel.
6.2 Application 2 - Manua Loa  $\mathrm{CO}_{2}$
In this section, we extend our comparative analysis to the widely recognized Moana Loa  $\mathrm{CO}_{2}$  dataset from the Global Monitoring Laboratory's Repository as our benchmark for regression analysis (Tans and Keeling (2023)). Rasmussen and Williams (2006) employed this dataset to demonstrate that constructing a complex covariance function by combining several different types of simple covariance functions could yield an excellent fit to complex data. Consequently, we consider it to be a suitable choice for testing our theorem.
Specifically, we used the decimal date from 1960 to 2020 as our input and the monthly average  $\mathrm{CO}_{2}$  as our response. In this context, we compare the performance of Matérn mixture  $1/2 + 3/2$ ,  $1/2 + 3/2 + 5/2$ ,  $3/2 + 5/2$  and three single Matérn kernels  $(1/2, 3/2, 5/2)$ . The dataset consists of 720 records. We undertook an extensive assessment, varying the training sample size from  $5\%$  to  $95\%$  and keeping the test sample size as the remaining data. For each training sample size, we conducted 10 simulations using different random splits of the dataset as replications. This strategy allows for a robust and thorough evaluation of the prediction accuracy of both the mixture kernel and the single kernel component in the context of a regression problem with varying training sample sizes. We use the MSE to quantify the prediction accuracy (Figure 5).
Figure 5: Manua Loa  $\mathrm{CO}_{2}$  prediction MSE in test dataset with  $20\%$ ,  $50\%$  and  $75\%$  training dataset. Boxplot are colored by different Matérn kernels.
The Matérn mixture kernel  $1/2 + 3/2 + 5/2$  aligned closely with  $1/2 + 3/2$  across all experiments. Furthermore, these three Matérn mixture kernels  $(1/2 + 3/2 + 5/2, 1/2 + 3/2, 1/2)$  converge when the training sample size ratio reaches  $50\%$ . This empirical finding supports the theoretical equivalence of these three GPs as posited in Theorem 2. Similarly, the Matérn mixture  $3/2 + 5/2$  kernel and the standalone Matérn  $3/2$  kernel converge from a  $65\%$  sample size, lending empirical weight to their theoretical equivalence and their equivalence in MSE as stated in Corollary 1.
Matérn  $3/2$  kernel has exceptional performance across all experiments. However, its influential performance is diluted when coupled with the less smooth component, Matérn  $1/2$ . On the contrary, when mixed with the smoother Matérn  $5/2$  component, the worse performance of the latter is overshadowed by Matérn  $3/2$ . These behaviors lend empirical weight to our Theorem 2, suggesting that in the asymptotic sense, the mixture kernel is dominated by its least smooth component.
A noteworthy trend is the inequivalence of the Matérn mixture kernel and its least component with a limited training sample (training sample size  $< 50\%$ ). We consider such scenario as the "finite" sample scenario in contrast to our focus on the infinite sample scenario, i.e. asymptotic theory. Theoretically, while the Matérn mixture kernels  $1/2 + 3/2 + 5/2$ ,  $1/2 + 3/2$ , and the standalone Matérn  $1/2$  kernel are deemed equivalent, and the Matérn mixture kernel  $3/2 + 5/2$  is akin to the Matérn  $3/2$  kernel, their practical agreement points, i.e.  $50\%$  and  $65\%$ , differed with limited training samples. Although securing a conclusive theoretical support for the finite sample regime is challenging, we managed to provide some insights to interpret our findings. To do so, we have to delve further into the proof of Stein (1993)'s Theorem 1, which guides us to Theorem 3.1 in Stein (1990). The proof suggests that the relative difference between the MSEs rests on the tail of the series. This difference is influenced by both the sample size (denoted as  $N$  in Stein (1990)) and by  $b_{jk}$  and  $\mu_j$ , as defined on the same page. For a fixed sample size, smaller values of  $(b_{jk} + \mu_j\mu_k)^2$  result in a
smaller relative difference between MSEs. By the definition of  $b_{jk}$  and  $\mu_j$ , the more "different" the two kernels are, in other words, the more "significant" the additional smoother components become — the greater the difference between the MSEs will be.
7 Discussion and future work
In this study, we delved into the identifiability and interpretability of parameters in GPs with various mixture kernels in the asymptotic scenario and fixed domain, including the additive kernel and separable kernels. We formulated a series of theorems clarifying the identifiable parameters in these kernel structures, and further corroborated our theorems through multiple simulations and real-data applications. Our simulation results convincingly demonstrated that in GPs with mixture kernels, the only identifiable parameter, known as the microergodic parameter, is associated with the least smooth kernel component. This discovery has profound implications for parameter interpretation in the context of GPs with mixture kernels. Empirical evidence from image data analysis and Manua Loa  $\mathrm{CO}_{2}$  regression studies further reinforce these discoveries. Despite the inclusion of kernels with varying smoothness in the mixture kernel, the performance of the mixture kernel closely paralleled that of the least smooth kernel component within it when the sample size is large enough, regardless of its performance. This observation suggests that the inclusion of kernels with different smoothness does not necessarily improve the prediction accuracy. In fact, due to various real world factors including limited training samples, optimization and more, determining a clear winner in performance between Matérn mixture kernel and single Matérn kernel proves to be a challenging task. Lastly, in the case of multivariate GPs with separable kernels, our theoretical and simulation results show that the correlation structure is identifiable up to a multiplicative constant. This result underlines that the interpretability of separable kernels mainly resides in the relative correlation structure, rather than individual parameters.
Although our study has provided substantial insight on the identifiability and interpretability of parameters in GPs with various mixture kernel types, it also opens several exciting avenues for future research. First, our analysis primarily focuses on Matérn kernels, and it would be intriguing to extend this framework to other families of kernels, such as the periodic kernel. Such an extension would provide a more comprehensive understanding of parameter identifiability and predictive performance across a broader spectrum of kernel types. Second, our work has so far considered the cases where  $p \leq 3$  or  $p \geq 5$ . However, extending this to  $p = 4$  presents a substantial challenge due to the lack of mathematical tools to determine whether two Gaussian random measures are equivalent or not. Third, our observations in the Mauna Loa CO $_2$  example underscore the need for further exploration in the finite sample scenario. Potential avenues for future research include investigating the convergence rates of both the Matérn mixture kernel and the single Matérn kernel. Lastly, while we have identified the microergodic parameters that are theoretically identifiable in mixture kernels, an important direction for future work involves finding consistent estimators for these kernel parameters. This would entail developing novel estimation techniques or adapting existing ones to reliably estimate the parameters in practice, thereby enhancing the practical utility of our theoretical findings. These endeavors will not only extend the theoretical foundations of GPs with mixture kernels, but will also broaden their applicability across various real-world scenarios.
We express our sincere gratitude to anonymous reviewers and the Area Chair for their insightful comments and constructive feedback, which significantly enhanced the quality and clarity of this work. DL was supported by the NIH/NCATS award UL1 TR002489, NIH/NHLBI award R01 HL149683 and NIH/NIEHS award P30 ES010126. YL was partially supported by NIH awards R01AG079291 and U01HG011720.
Remes, S., M. Heinonen, and S. Kaski (2017). Non-stationary spectral kernels. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Advances in Neural Information Processing Systems, Volume 30. Curran Associates, Inc.
Samo, Y.-L. K. and S. Roberts (2015). Generalized spectral kernels.
Simpson, F., I. Davies, V. Lalchand, A. Vullo, N. Durrande, and C. E. Rasmussen (2021). Kernel identification through transformers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems, Volume 34, pp. 10483-10495. Curran Associates, Inc.
Stein, M. (1990). Uniform asymptotic optimality of linear predictions of a random field using an incorrect second-order structure. The Annals of Statistics 18(2), 850-872.
Stein, M. L. (1993). A simple condition for asymptotic optimality of linear predictions of random fields. Statistics & Probability Letters 17(5), 399-404.
Stein, M. L. (1999). Interpolation of spatial data: some theory for kriging. Springer Science & Business Media.
Stein, M. L. (2005). Space-time covariance functions. Journal of the American Statistical Association 100(469), 310-321.
Sun, S., G. Zhang, C. Wang, W. Zeng, J. Li, and R. Grosse (2018, 10-15 Jul). Differentiable compositional kernel learning for Gaussian processes. In J. Dy and A. Krause (Eds.), Proceedings of the 35th International Conference on Machine Learning, Volume 80 of Proceedings of Machine Learning Research, pp. 4828-4837. PMLR.
Tang, W., L. Zhang, and S. Banerjee (2022). On identifiability and consistency of the nugget in gaussian spatial process models. Journal of the Royal Statistical Society Series B: Statistical Methodology 83(5), 1044-1070.
Tans, P. and R. Keeling (2023). Trends in atmospheric carbon dioxide. https://gml.noaa.gov/ccgg/trends/data.html. Accessed: 2023-08-01.
Verma, A. and B. E. Engelhardt (2020). A robust nonlinear low-dimensional manifold for single cell RNA-seq data. BMC Bioinformatics 21(1), 324.
Wilson, A. and R. Adams (2013, 17-19 Jun). Gaussian process kernels for pattern discovery and extrapolation. In S. Dasgupta and D. McAllester (Eds.), Proceedings of the 30th International Conference on Machine Learning, Volume 28 of Proceedings of Machine Learning Research, Atlanta, Georgia, USA, pp. 1067-1075. PMLR.
Wilson, A. G., E. Gilboa, A. Nehorai, and J. P. Cunningham (2014). Fast kernel learning for multidimensional pattern extrapolation. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger (Eds.), Advances in Neural Information Processing Systems, Volume 27. Curran Associates, Inc.
Zhang, H. (2004). Inconsistent estimation and asymptotically equal interpolations in model-based geostatistics. Journal of the American Statistical Association 99(465), 250-261.