On the Identifiability and Interpretability of Gaussian Process Models
Authors: Jiawen Chen
Wancen Mu
Yun Li
Didong Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Gaussian process
Identifiability
Interpretability
Mixture kernel
Separable kernel
TL;DR: Gaussian process with Matérn mixture kernel is asymptotically equivalent as its least smooth component.

Abstract: In this paper, we critically examine the prevalent practice of using additive mixtures of Mat\'ern kernels in single-output Gaussian process (GP) models and explore the properties of multiplicative mixtures of Mat\'ern kernels for multi-output GP models. For the single-output case, we derive a series of theoretical results showing that the smoothness of a mixture of Mat\'ern kernels is determined by the least smooth component and that a GP with such a kernel is effectively equivalent to the least smooth kernel component. Furthermore, we demonstrate that none of the mixing weights or parameters within individual kernel components are identifiable. We then turn our attention to multi-output GP models and analyze the identifiability of the covariance matrix $A$ in the multiplicative kernel $K(x,y) = AK_0(x,y)$, where $K_0$ is a standard single output kernel such as Mat\'ern. We show that $A$ is identifiable up to a multiplicative constant, suggesting that multiplicative mixtures are well suited for multi-output tasks. Our findings are supported by extensive simulations and real applications for both single- and multi-output settings. This work provides insight into kernel selection and interpretation for GP models, emphasizing the importance of choosing appropriate kernel structures for different tasks.

On the Identifiability and Interpretability of Gaussian Process Models
Jiawen Chen Department of Biostatistics University of North Carolina at Chapel Hill j jiawenn@emall. unc. edu
Wancen Mu
Department of Biostatistics University of North Carolina at Chapel Hill wancen@live. unc.
Yun Li
Didong Li Department Biostatistics University of North Carolina at Chapel Hill didongli@unc. edu
Department of Biostatistics University of North Carolina at Chapel Hill yun_ 1i Smed. unc edu
Abstract
In this paper, critically cxamine the prevalent practice of using additive mixtures of Matérn kernels in single-output Gaussian process (GP) models and explore the properties of multiplica- tive mixtures of Matérn kernels for multi-output GP models. For the single-output case, we derive a series of theoretical results showing that the smoothness is determined by the least smooth component and that GP with such kernel is effectively equivalent to the least smooth kernel component. Furthermore, we demonstrate that none of the mixing weights
mixture of Matérn kernels
parameters within individual kernel components turn our attention to multi-output GP models and analyze the identifiability of the covariance matrix A in the multiplicative kernel K (I, y) (Ko y). where Ko is a standard single output kernel such as Matérn. We show that A is identifiable up to a multiplicative constant, suggesting that multiplicative mixtures are well suited for multi-output tasks. Our findings are supported by extensive simulations and real applications for both single- and multi nulti-output settings. This work provides insight into kernel selection and interpretation for GP models, emphasizing the importance of choosing appropriate kernel structures for different tasks.
identifiable. We then
Introduction
Gaussian processes (GPs) have emerged as a powerful and popular tool in machine learning, spatial statistics, functional data analysis, etc., due their versatility, flexibility, and interpretability nonparametric method (Rasmussen and Williams) 2006 Banerjee et al. 2014). GPs provide an intuitive means of modeling uncertainty, allowing the generation of predictive distributions for unseen data points without requiring explicit model specification. Consequently, GPs have found applications in variety of contexts.
The key to harnessing the power of GPs lies in choice of kernel functions, also known as covariance functions. Rather than specifying model, the GP framework revolves around selecting an appropriate kernel, transforming the problem into one of parameter estimation. Over time, researchers have developed various kernels, each tailored to specific scenarios, such as spatial data, time series data, and others (Genton) 2001). For instance, for spatiotemporal, variety kernels have been developed to capture unique characteristics and patterns (Gneiting 2002 Stein 2005).
The Radial Basis Function (RBF) kernel and Matérn kernels serve as prime examples of the diversity within the kernel family (Cressie and Wikle (013). The RBF kemnl,, renowned for its infinite differentiability, yields smooth functions, while Matérn kernels control the degree smoothness by kernel parameter, thereby accommodating both smooth and nonsmooth functions (Stein (9999)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

With each kernel bearing its own set of advantages and disadvantages, kernel selection, as a nontrivial task, often demands extensive domain knowledge.
Inspired by the potential of harnessing the strengths of multiple kernels concurrently, researchers have ventured into methods that involve kernel combinations, such as spectral mixtures (Wilson and Adams 2013; Samo and Roberts, 2015) Remes et al. 2017). addition and/or multiplication of kernels (Duvenaud et al. 2011)) such as mixture of RBF (Duvenaud et al. 2013), mixture of RBF, periodic (Per), linear (Lin), and rational quadratic (RQ) (Kronberger and Kommenda| 2013), mixture
of RBF, RQ, Matérn and Per (Cheng et al. 2019 Verma and Engelhardt| (020). mixture of RBF.
Matérn 1/2, 3/2, 5/2, as well as more sophisticated methods like Neural Kerne Networks (NK
al. (2018) and Automatic Bayesian Covariance Discovery (ABCD, Lloyd et al. (2014). For instance, within the realm of spectral mixtures, Remes et al. (2017) proposed the Generalized Spectral Mixture (GSM) kernel, which is product of three components all parametrized by GPs, representing wwwpbeeciess.nnghh-eeeee
and mixture weights. In the category of summation and/or multiplication, Verma and Engelhardi (2020) utilized the sum of the RBF kernel and Matérn kernel with varying smoothness as the final kernel in their t-distribution Gaussian process latent variable model (IGPLVM), an extension to GPLVM (Lawrence) 2003; Lalchand et al. 2022), to characterize the latent features in single-cell RNA sequencing data.
In addition, methodological advances have facilitated the efficient discovery of optimal kernel mixtures. For example, NKN is reported to be more efficient than the Automatic Statistician, a gradient-based method. [Simpson et al. (202T) utilized a transformer- mixture kernel recommendations. These studies underscore the advantages of employing mixed kernels in GP modeling, showcasing improved model fitting and more accurate predictions. By combining multiple kernels, the strengths of individual kernels can be leveraged to capture complex data patterns and relationships, potentially exceeding the capabilities of single kernels. Furthermore, the automatic selection of kernels optimizes model performance, reducing the reliance on extensive prior knowledge.
framework to generate
Another essential benefit of using mixture kernels in GP modeling lies in the improved interpretability. In contrast to complex. data-driven kemelss, mixture kernels allow the decomposition of intricate patterns into simpler, distinct base kernels that are more readily interpretable. This technique, often termed decomposition, enables a more comprehensive understanding of the underlying data structure by simplifying complex patterns into their constitutive components. An illustrative example of this approach the work of Duvenaud et al.] (2013), who applied this decomposition technique for structure discovery in time series data. Their proposed mixture kernel comprised several base kernels, including RBF, periodic, linear, and rational quadratic kernels, enabling the dissection of time-series data patterns into components such as long-term trends, annual periodicity, and medium- term deviations. similar vein, the ABCD approach also employed decomposition, albeit with a different set of base kernels.
The identifiability of parameters within single Matérn kernel has previously been explored, with the microergodic parameter uniquely identified as the only identifiable parameter, as outlined in Stein (999))- (Tang et al.) 2022) examined the identifiablity of parameters within single Matérn kernel with nuggets. Despite the prevalent use and interpretation of mixture kernels, their theoretical properties including identifiability and interpretability of parameters within kernel components, to the best of our knowledge, remain underexplored. A related critical question pertains to the common practice of using kernels with varying degrees of smoothness for enhanced flexibility. In this work, we turn our attention to the additive kernel in univariate GPs and multiplicative (separable) kernel in multivariate GPs, specifically focusing on the mixture of the widely-used Matérn kernel. We highlight the following novel findings:
The smoothness of an additive mixture kernel is completely determined by the least smooth component.
For additive mixture of Matérn kernels, the identifiability is confined only to a single parameter, also known as the microergodic parameter that is associated with the least smooth kernel component.
For multivariate GPs with multiplicative separable kernels, the multiplicative matrix that controls the correlation structure among the response variables is identifiable up a multiplicative constant.
Our conclusions extend beyond the specific case of Matérn kernels, demonstrating applicability to a wider range of mixture kernels.
2

---

Our study aims to deepen the understanding of mixture kernel identifiability and interpretability, as well as to clarify the practical benefits of using kernels with varying degrees of smoothness, Our theoretical assertions are supported by both simulations and real-world applications. Details regarding the proofs of our theories and numerical experiments can be found in the supplementary material.
2 Gaussian process, kernels and parameter inference
This section serves to define key terms and outline the parameter inference algorithm integral to the forthcoming theory and simulation sections.
A GP is a random function where any finite set of its realizations follows multivariate normal distribution, characterized by mean function μ and covariance function K (Rasmussen and Williams 2006).
Definition 1. f is said to follow a Gaussian process domain Q with a mean function p Ω IR and a covariance function K St x Q2 + R for any 21,...;: In E S2
f(xx)] ~ (e, E), where v = [u(21)...... ((x,)] Σij = (x),,
For the purpose of this study, we assume = 0, adhering to common practice and for the sake of simplicity. If the mean is not zero, a data transformation can be applied to achieve this. This approach often adopted in machine learning and statistical analysis to simplify calculations and comply with certain algorithmic requirements (Rasmussen and Williams 2006 Murphy 2012).
As consequence, the behavior of a GP is primarily determined by its kernel function. this study, we assume the domain f = IP and our focus is on the selection of kernels that are widely recognized and commonly used machine learning.
Definition 2. The RBF kernel, also known as the squared exponential kernel of Gaussian kernel is given by:
K(x,x') exp(-a | x )1²)
The parameter 8² is called the spatial variance or partial sill that controls the point-wise variance, while a is called the (length) scale, range, or decay that controls the spatial dependency. A key characteristic of RBF is its smoothness, defined as follows:
Definition 3. A Gaussian process I is said to be mean-square continuous (MSC) if [x h) (xx)² h-0,, O for any x E RP. I is said to be mean-square differentiable (MSD) if lim;-00 ((oth)-(()) exists in the mean-square topology, and the limiting process is called the derivative process of f, denoted by f'. Similarly, the d-times mean-square differentiable (d-MSD) GPs can be defined inductively.
In particular, the above-defined RBF kernel is infinitely differentiable. However, oversmoothing can be problematic in prediction (Stein) 1999), leading the popularity of the following flexible family of kernels that allow for varying smoothness:
Definition 4. The Matérn kernel is given by
K K(x, 022 ( -''1))
Ky(a || z'1l)
I(v)
where Ko is the modified Bessel function of the second kind. When 1/2, the Matérn kernel becomes the exponential kernel:
K(z,z') = 0² exp((=a z'
The additional parameter v is called the smoothness, since the smoothness of a Matérn GP is exactly [v] - 1. Inferring v is known to be a particularly challenging problem, both from the theoretical and empirical perspectives (Zhang 2004). In practice, is usually set to 3/2,3/2. 5/2, ... due to the simplified analytic form the modified Bessel function.
The most straightforward estimator of the parameters o² and is the maximum likelihood estimator (MLE). In practice, optimizers such as Adam (Kingma and Ba 2015) or stochastic gradient descent (SGD) are used to maximize the log-likelihood.
3

---

3 Univariate GP: additive kernels
In this section, we investigate the smoothness of mixture kernels and the identifiability of parameters in the additive mixture of Matérn kernels in the context of a univariate response variable. We note here that all the theorems presented are framed in an asymptotic context.
Consider KI,'.. ,K as kernlls with dr-MSD. Define K Σ wiKi where Ei= in and wi 2 O.
Theorem 1. determined by the least smooth component.
= mnin((d)--iime MSD, but not + 1-times MSD, so the smoothness of K is
Beyond smoothness, we discuss the identifiability of the parameters in the mixture kernel, which relies on the notion of equivalence of measures.
Two measures F1 and P22 are said to be equivalent, i.e. P1 = P2, if they are absolutely continuous with respect each other. i.e., (B 0 = (2( 0. Two GPs are equivalent if the corresponding Gaussian random measures are equivalent.
Definition 5.
As consequence, two equivalentt GPs cannot be distinguished by any finite number of realiza- tions (Stein 1999). Specifically, given a family of GPs parametrized by 8, if Pø, < Po, with 81 * 82 then 0 is not identifiable since cannot distinguish between 0 and 0z. As corollary, there does not exist any consistent estimator for 0.
Now we can study the identifiability of the mixture Matérn kernel K = E'= WyK,, where K2 is the Matérn kernel with parameters (o an v1), given known vis in ascending order V1 < V/ < UL with MI 11-1 for different smoothness.
Theorem 2. Let K = Ei Wt and K = -, Eg i be two kernels represented as linear combina- tions of Matérn kernels Ki 's and K₁ with parameters (oî, on, v)) and (0? āt, vi).
(i) When p- 1,2,3, Then K = K ifjaa 201 a
(ii) When p 2 5, Then K = K if
wj 2v क 2v: w202022
202 viwio o 2(m = w203022 -1 -/
22-2(+1+1)
Consequently, single parameter is identifiable. The only identifiable parameter when p is wioja?" 201 known as the microergodic parameter SStein 1999).
Interestingly, the case of p = 4 is missing here. We hypothesize that it aligns with the case of p 2 5. However, providing proof remains challenging. It is also worth noting that the issue of identifiability persists even for a single Matérn kernel, as highlighted in the existing literature (Zhang
2004 Anderes 2010 Li et al. 2023).
Corollary 1. The mixture kernel K is equivalent to K1, that is, the mixture kernel is equivalent to its least smooth component. Furthermore, the mean squared error (MSE) of the mixture kernel K is asymptotically equal to the MSE of K1.
Theorem 2 limplies that interpreting W/ the weight of each component is often misleading due to its nonidentifiability. Furthermore, Corollary [SsglaryssSsgstssss the use of a single Matérn kemel in lieu of the linear mixture kernel. Both assertions are supported by the simulation study in Section 5 and the real-world application in Section
An alternative strategy is to assume the same smoothness across mixing components, i.c., 11 1/2 ... VL. In this case, we prove that no single parameter is identifiable. The complete theorem, proof, and simulation are in the Supplement.
When considering real- world data, is often the case that the observed outcomes are noisy, which is frequently modeled as an i.i.d. Gaussian with variance denoted by 7?, commonly referred as the "nugget". The inclusion of such a noise term is essential to capture the inherent variability in the data. Equivalently, the model can be formulated as noiseless by adjusting the kernel to be K(x, + " The following corollary shows that the noise, or nugget, does not impact the identifiability and interpretability of the mixture of Matérn kernels as shown in Theorem

---

Corollary 2. Let T² and F2 be the noise variance of K and K. If # then K If the previous results in Theorem hold.
In essence. while the presence of noise or "nugget" captures real-world data variability, it does not obscure or alter the fundamental characteristics of the mixture of Matérn kernels as established in our theorems.
4 Multivariate GP: multiplicative kernels
In this section, we shift our focus to multivariate GPs, also known as multi--utput or multi-task GPs, which are applicable to datasets with multiple response variables.
Definition 6. f Ω + R is said to follow m-variate GP in domain Q with zero mean and cross pross-covariance function K Q x + PD(m), where PD(m) the space of all m by m positive definite matrices, for any T1 In="
(f(zz), ... f(r.)) (M Σ).
where MN represents matrix Gaussian distribution and E consists of m m blocks Enj = K (Ix,, T.).
The construction of valid cross-covariance kernels is pivotal yet challenging aspect of multivariate GPs (Gneiting, 2002; Apanasovich and Genton 2010 Genton and Kleiber 2015). A popular kernel that is widely used due to its simplicity and seemingly interpretability is the multiplicative kernel, also known as a separable kernel. This kernel admits the form K (x, y) = AKo (z, y) where Ko is a standard kernel for univariate GP like Matérn while A PD(m) is positive definite matrix reflecting the correlation between response variables. The following theorem investigates the identifiability of the multiplicative roos-covariiance kernel.
Theorem 3. (i) When Ko is Matérn with parameters (o²,,x, v) where v is given, then K K if 2²a2" A = 2" That is, the identifiable parameter o² A. Hence A is identifiable up multiplicative constant and the correlation structure is identifiable
(ii) Let Ko be arbitrary kernel with spectral density po satisfying:
By E Wib,,6m7 0 < oc, C1, > 0, s.t., cr7(w)² S po(w) S co>(w)?
where W, is the space of Fourier transforms of T22 T22()) functions with compact support b-b, bjd (see the Supplement for more details). Of is a mircroergodic parameter of Ko, such that Sir [سw)
pla)) de < 0 if = then K = K if 0A ŌA. That is, the identifiable parameter 9A, hence identifiable up to a multiplicative constant and the correlation structure is identifiable.
p(w)
Note that (i) is a special case of (ii), as the Matérn kernel satisfies the additional assumptions in (ii). This theorem positively supports the utilization of the multiplicative kernel and the interpretation that Ai measures the correlation between the i-th response variable and the j-th response variable.
Simulation
In this section, we will validate the proposed theories by conducting three simulation studies. The first simulation demonstrates Theorem I using a mixture of Matérn kernels, highlighting that the smoothness of the sample path (e.g., realization of the GP) is determined by the least smooth component. The second simulation supports Theorem 2 using a mixture kernel of Matérn kernels with varying smoothness. It shows that none of the single parameters are identifiable, and only the proposed microergodic parameter can be identified. Finally, the third simulation showcases Theorem suggesting that the multiplicative matrix is identifiable up to multiplicative constant.
5.1 Simulation Univariate GP: Smoothness
In the first simulation, we show that the smoothness of the sample path determined by the smoothness of the least smooth kernel. We generate 200 samples equidistant sequence ranging
5

---

from 0 - 10. We consider the following mixture kernel: K = Ei=1 wi Ki, where K2 is the Matérn kernel with smoothness parameter 14 being د1ا1 and Ei=1 W/ 1, wi 2 We randomly sample Y from multivariate normal distribution with kernel function as the mixture kernel, Matérn with 1/2,3/2, 5/2. The resulting sample paths are summarized Figure
Mixture
Matérn 1/2
Matérn 3/2
Motern 5/2
24
15
Matém 1/2-3/2
Matém 1/2
Matérn 3/2
2.6
200
25
380
35
150
100
Figure 1: Smoothness of Matérn kernels. Sample path of (A) Mixture kernel (B) Matérn 1/2 (C)
Matérn 3/2 (D) Matérn 5/2. (E) Numerical examination of Theorem !
Our results clearly indicate that the Matérn kernel with smoothness (continuous but not differentiable), and the smoothness increases with v. The smoothness of the mixture kernel is predominantly influenced by the Matérn kernel with comparing the sample path of the mixture kernel to those of the Matérn kernels with v = 3/2 and
1/2 exhibits the least degree of
1/2. When
5/2, is evident that the mixture kernel demonstrates degree of smoothness similar to the Matérn kernels with y = 1/2.
To better demonstrate the smoothness of the mixture kernel and its least component, further examine the continuity and MSD of the mixture kernel and its mixing components empirically. For a fixed Ty 0, let x; 1/i for i 1,2..... and we generate y from the GP, where I 1 1...,, denotes the index of replicates. This allows us to approximate lim [ff f(0))
by A, - (v) 2. As per Definition the GP is continuous if and only if 3 + 0. Similarly,
we can approximate lim E f(=)-/10) by 74 = +
The GP is mean-square
differentiable if and only lim " exists. Specifically, for the mixture Matérn 1/2 and 3/2 and Matérn 1/2, while 3: + 0, % does not converge (Figure [ first two columns), which indicates that both the mixture kernel and Matérn 1/2 are continuous but not differentiable. However, for Matérn 3/2 (Figure I third column), B, 0 and " converges, implying that Matérn 3/2 is continuous and differentiable.
These empirical observations strongly support the claims made in Theorem which suggests that the inclusion of smoother kernel components in the mixture does not inherently enhance the overall smoothness of the mixture kernel.
5.2 Simulation Univariate GP: Components with different smoothness
In the second simulation, our aim is to assess parameter identifiability in GP with mixture kernel consisting of Matérn kernels with distinct smoothness. The mixture kernel is denoted as K = Σ wy K1, where Kr is a Matérn kernel with parameters (of on, v1). For this simulation, set 11 1/2, 3/2, 5/2. Theorem 2 indicates that the only identifiable parameter is

---

wizfa, while all other parameters remain unidentifiable. In this scenario, the identifiable parameter, also known as the microergodic parameter, is associated with the least smooth component, i.e., the Matérn 1/2 kernel. We will assess parameter inference by comparing estimated values to their true values across different training sample sizes. For this experiment, we generate n samples, ranging from 20, 50, 100, 500. For each sample size, we replicate the simulation 100 times.
A
0y0ja]
B
0,85a2"
C
0y0jag"
10.0
0.0
0.4-
20
5.0-
0.2
10-
0.01
0.0
20
100
50c
20
50
102
20
1.00
500
Sample Size
Sa mpmll $220
Sample s/20
A:
6g
8
1.00
12-
12-
0.25
20
30
100
500
20
50
100
500
100
300
Sample size
Sample size
Sample size
Figure 2: Parameter estimation. (A) Only the microergodic parameter w1 viaan is identifiable. (B-F) other parameters are not identifiable. The red line indicates the true value.
The results provide persuasive support for Theorem 2 that, in the context of additive mixture of Matérn kernels with distinct smoothness, only the MLE of the microergodic parameter Wiofaj for the least smooth component converges to the true value. The MI.Es for all other parameters do not converge to their respective true values, highlighting the unique identifiability of the parameters in the least smooth component within the additive mixture kernel framework. Such results are robust in terms of optimizer choice and consistent as the sample size increases to 3000 (details in the Supplement).
5.3 Simulation 3 Multivariate GP
Next, we turn our attention to multivariate GPs with separable kernel. We define the separable kernel as K AKg. Here, we select Matérn 1/2 as Kg. characterized by the parameters (a² a).. Our Theorem suggests that only 2az"g" is identifiable. To verify this, we adopt a bivariate setup where A is a 2 x 2 positive definite matrix. In this simulation, the sample size varies from 50, 100, 200, 400. For each sample size, we replicate the simulation 100 times.
The parameter estimates are summarized in Figure 3 Evidently, only the MLE of the microergodic parameter 2 converges to its true value. These findings reinforce the identifiability and interpretability of the separable kernel. This understanding is critical for the interpretation and application of separable kernels in real-world situations.
Application
Motivated by the findings in our theoretical and simulation studies, we proceed compare the performance of the mixture kernel with varying smoothness and the least smooth component within that mixture only, real-world data applications.
this section, we will revisit several applications in (illson that the mixture kernel achieves prediction performance comparable to the least smooth kernel component within the mixture.
previous studies (Rasmussen and Williams (2006), Wilson et al. (2014), Sun et al. (201 8)), illustrating
7

---

A, 1878
Rg 28
266 355
Ra
8'
256
250
355
155
Figure 3: Parameter estimation. (A-C) Only the microergodic parameter A is identifiable (D-E) All other parameters are not identifiable. The red dotted line indicates the true value.
6.1 Application Image prediction
For our first real-data application, we delve into the domain of image analysis. GP has been widely used to predict missing image (Wilson et al. (2014), [Sun et al. (2018)). This task of image prediction an exemplary context for assessing kernel performance discerning local correlation patterns, providing visually intuitive framework for comparative analysis.
We employ handwritten zero image from the MNIST dataset (LeCun al. 1998)) for this analysis; we extract a 20 x 20 section from the center of the original 100 x 100 image to serve as our test image. This results in training dataset of 9600 pixels and testing dataset with 400 pixels (Figure A The input for this analysis comprises the 2-D pixel locations, with the corresponding pixel intensities serving as the response. To reconstruct the missing area, we train GP regression models with both the mixture kernel with varying smoothness and single Matérn 1/2 kernel on the training dataset, and subsequently make predictions the testing dataset. Specifically, we employ mixture of three Matérn kernels of smoothness 1/2, 3/2, 5/2. Our analysis seeks to demonstrate the similarities in performance between the mixture kernel and the least smooth component within that mixture, i.e. Matérn 1/2, in terms of prediction in the missing image area.
Training image
Motérn mixture
Matern
Figure 4: Texture exploration in diamond plate. (A) Training image. (B) Prediction of mixture kernel. The area in the pink box represents the testing image area. (C) Predication of the Matérn 1/2 kernel.
The predictions generated by both the mixture kernel and the Matérn 1/2 kernel demonstrate remarkable similarity. as shown in Figure [ This observation suggests that the mixture kernel and a single Matérn 1/2 kernel have comparable performance. The close alignment between their predictions implies that using the mixture kernel does not yield significant advantages over the least smooth kernel component in this specific application, further reinforcing our theoretical findings (Theorem) regarding the dominance of the least smooth kernel component in the mixture kernel.

---

6.2 Application 2 Manua Loa CO2
In this section, we extend our comparative analysis to the widely recognized Moana Loa CO2 dataset from the Global Monitoring Laboratory's i Repository as our benchmark for regression analysis (Tans) and Keeling (023)). Rasmussen and Williams (2066) employed this dataset to demonstrate that constructing complex covariance function by combining several different types of simple covariance functions could yield an excellent fit to complex data. Consequently, we consider it to be a suitable
choice for testing our theorem.
Specifically, we used the decimal date from 1960 to 2020 as our input and the monthly average CO2 as our response. In this context, we compare the performance of Matérn mixture 1/2 + 3/2, 1/2 - 3/2 + 5/2, 3/2 + 5/2 and three single Matérn kernels (1/2, 3/2, 5/2). The dataset consists of 720 records. We undertook an extensive assessment, varying the training sample size from 5% to 95% and keeping the test sample size as the remaining data. For each training sample size, we conducted 10 simulations using different random splits of the dataset as replications. This strategy allows for robust and thorough evaluation of the prediction accuracy of both the mixture kernel and the single kernel component in the context of regression problem with varying training sample sizes. We use the MSE to quantify the prediction accuracy (Figure 5).
0.2
0.5
0.75
Matérn
1/2+3/2+5/2
1/2+3/2
3/2+5/2
5/2
Figure 5: Manua Loa CO2 prediction MSE in test dataset with 20%, 50% and 75% training dataset. Boxplot are colored by different Matérn kemelss.
The Matérn mixture kernel 1/2 + 3/2 + 5/2 aligned closely with 1/2 + 3/2 across all experiments. Furthermore, these three Matérn mixture kernels (1/2 3/2 + 5/2 3/2, 1/2) converge when the training sample size ratio reaches 50%. This empirical finding supports the theoretical equivalence of these three GPs as posited in Theorem 2 Similarly, the Matérn mixture 3/2 + 5/2 kernel and the standalone Matérn 3/2 kernel converge from a 65% sample size, lending empirical weight to their theoretical equivalence and their equivalence in MSE as stated in Corollary [
Matérn 3/2 kernel has exceptional performance across all experiments. However, its influential performance is diluted when coupled with the less smooth component, Matérn 1/2. On the contrary, when mixed with the smoother Matérn 5/2 component, the worse performance of the latter is overshadowed by Matérn 3/2. These behaviors lend empirical weight to our Theorem 2 suggesting that in the asymptotic sense, the mixture kernel is dominated by its least smooth component.
A noteworthy trend is the inequivalance of the Matérn mixture kernel and its least component with a limited training sample (training sample size < 50%). We consider such scenario as the "finite" sample scenario in contrast to our focus on the infinite sample scenario, i.e. asymptotic theory. Theoretically, while the Matérn mixture kernels 1/2 + 3/2 + 5/2, 1/2 + 3/2, and the standalone Matérn 1/2 kernel are deemed equivalent, and the Matérn mixture kernel 3/2 + 5/2 is akin to the Matérn 3/2 kernel, their practical agreement points, i.c. 50% and 65%. differed with limited training samples. Although securing a conclusive theoretical support for the finite sample regime is challenging, we managed to provide some insights to interpret our findings. To do so, we have to delve further into the proof of |Stein (1993 's Theorem which guides us to Theorem 3.1 in Stein (1990). The proof suggests that the relative difference between the MSEs rests on the tail of the series. This difference is influenced by both the sample size (denoted as N in Stein (1990) and by бІą and Mj, defined on the same page. For fixed sample size, smaller values of (byk Mypp)) ) result in

---

smaller relative difference between MSEs. By the definition of bys and pj, the more "different" the two kernels are, in other words, the more "significant" the additional smoother components become the greater the difference between the MSEs will be.
7 Discussion and future work
In this study, we delved into the identifiability and interpretability of parameters in GPs with various mixture kernels in the asymptotic scenario and fixed domain, including the additive kernel and separable kernels. We formulated a series of theorems clarifying the identifiable parameters in these kernel structures, and further corroborated our theorems through multiple simulations and real-data applications. Our simulation results convincingly demonstrated that in GPs with mixture kernels, the only identifiable parameter, known the microergodic parameter, is associated with the least smooth kernel component. This discovery has profound implications for parameter interpretation in the context of GPs with mixture kernels. Empirical evidence from image data analysis and Manua Loa CO2 regression studies further reinforce these discoveries. Despite the inclusion of kernels with varying smoothness in the mixture kernel, the performance of the mixture kernel closely paralleled that of the least smooth kernel component within it when the sample size is large enough, regardless of its performance. This observation suggests that the inclusion of kernels with different smoothness does not necessarily improve the prediction accuracy. In fact, due to various real world factors including limited training samples, optimization and more, determining clear winner performance between Matérn mixture kernel and single Matérn kernel proves to be a challenging task. Lastly, in the case of multivariate GPs with separable kernels, our theoretical and simulation results show that the correlation structure is identifiable up to a multiplicative constant. This result underlines that the interpretability of separable kernels mainly resides in the relative correlation structure, rather than individual parameters.
Although our study has provided substantial insight on the identifiability and interpretability of parameters in GPs with various mixture kernel types, it also opens several exciting avenues for future research. First, our analysis primarily focuses on Matérn kernels, and would be intriguing to extend this framework to other families of kernels, such as the periodic kernel. Such an extension would provide more comprehensive understanding of parameter identifiability and predictive performance across a broader spectrum of kernel types. Second, our work has so far considered the cases where p S 3 or p 2 5. However, extending this to = presents a substantial challenge due to the lack of mathematical tools to determine whether two Gaussian random measures are equivalent or not. Third, our observations in the Mauna Loa CO2 example underscore the need for further exploration the finite sample scenario. Potential avenues for future research include investigating the convergence rates both the Matérn mixture kernel and the single Matérn kernel. Lastly, while we have identified the microergodic parameters that are theoretically identifiable in mixture kernels, an important direction for future work involves finding consistent estimators for these kernel parameters. This would entail developing novel estimation techniques adapting existing the parameters in practice, thereby enhancing the practical utility of our theoretical findings. These endeavors will not only extend the theoretical foundations of GPs with mixture kernels, but will also broaden their applicability across various real-world scenarios.
reliably estimate
Acknowledgments and Disclosure of Funding
We express our sincere gratitude to anonymous reviewers and the Area Chair for their insightful comments and constructive feedback, which significantly enhanced the quality and clarity of this work. DL was supported by the NIH/NCATS award ULI TR002489, NIH/NHLBI award RO1 HL149683 and NIH/NTEHS award P30 ES010126. YL. was partially supported by NIH awards RO1 AG079291 and U01 HG01 1720.
10

---

References
Anderes, E. (2010). On the consistent separation of scale and variance for Gaussian random fields. The Annals of Statistics.
Apanasovich, T. V. and M. Genton (2010). Cross-covariance functions for multivariate random fields based on latent dimensions. Biometrika 97(1), 15-30.
Banerjee, S., B. P. Carlin, and A. E. Gelfand (2014). Hierarchical modeling and analysis for spatial data. CRC press.
Cheng, L., S. Ramchandran, T. Vatanen, N. Lietzén, R. Lahesmaa, A. Vehtari, and H. Lähdesmäki (2019). An additive Gaussian process regression model for interpretable non-paran con-parametric analysis of longitudinal data. Nature Communications 10(1), 1798.
Cressie, N. and C. K. Wikle (2015). Statistics for spatio-temporal data. John Wiley & Sons.
Duvenaud, D., J. Lloyd, R. Grosse, Tenenbaum, and G. Zoubin (2013, 17-19 Jun). Structure discovery in nonparametric regression through compositional kernel search. In S. Dasgupta and D. McAllester (Eds.), Proceedings of the 30th International Conference on Machine Learning, Volume 28 of Proceedings of Machine Learning Research, Atlanta, Georgia, USA, pp. 1166-1174. PMLR.
Duvenaud, D. K., H. Nickisch, and C. Rasmussen (2011). Additive Gaussian processes. In J. Shawe- Taylor, Zemel, Bartlett, Pereira, and K. Weinberger (Bds.), Advances in Neural Information Processing Systems, Volume 24. Curran Associates, Inc.
Genton, M. G. (2001). Classes of kernels for machine learning: statistics perspective. Journal of machine learning research 2(Dec), 299-312.
Genton, M. G. and W. Kleiber (2015). Cross-Covariance Functions for Multivariate Geostatistics. Statistical Science 30(2), 147 163.
Gneiting, (2002). Nonseparable, stationary covariance functions for space-time data. Journal of the American Statistical Association 97(458), 590-600.
Kingma, D. P. and Ba (2015). Adam: method for stochastic optimization. In Y. Bengio and Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Kronberger, G. and M. Kommenda (2013). Evolution of covariance functions for Gaussian process regression using genetic programming.
Lalchand, V., A. Ravuri, and N. D. Lawrence (2022). Generalised gplvm with stochastic variational inference. In International Conference Artificial Intelligence and Statistics, pp. 7841-7864. PMLR.
Lawrence, N. (2003). Gaussian process latent variable models for visualisation of high dimensional data. In S. Thrun, L.. Saul, and B. Schölkopf (Eds.), Advances in Neural Information Processing Systems, Volume 16. MIT Press.
LeCun, Y., C. Cortes, and C. J. Burges (1998). The mnist database of handwritten digits. y/yann. lecun. htmm/exdb/mmnn
http:
Accessed: 2023-08-01.
Li D., W. Tang, and S. Banerjee (2023). Inference for gaussian processes with matérn covariogram on compact riemannian manifolds. Journal of Machine Learning Research 24(101), 1-26.
Lloyd, J. R., D. Duvenaud, R. Grosse, J. B. Tenenbaum, and Z. Ghahramani (2014). Automatic construction and natural-language description of nonparametric regression models.
Murphy, K. P. (2012). Machine learning: probabilistic perspective. MIT press.
Rasmussen, C. and Williams (2006). Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press.

---

Remes, S., M. Heinonen, and S. Kaski (2017). Non-stationary spectral kernels. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and Garnett (Eds.), Advances in Neural Information Processing Systems, Volume 30. Curran Associates, Inc.
Samo, Y.-L. K. and S. Roberts (2015). Generalized spectral kernels.
Simpson, F., I. Davies, V. Lalchand, A. Vullo, N. Durrande, and C. E. Rasmussen (2021). Kernel identification through transformers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and W. Vaughan (Eds.), Advances in Neural Information Processing Systems, Volume 34, pp. 10483-10495. Curran Associates, Inc.
Stein, M. (1990). Uniform asymptotic optimality of linear predictions of a random field using an incorrect second-orde structure. The Annals of Statistics 18(2), 850-872.
Stein, M. L.. (1993). A simple condition for asymptotic optimality of linear predictions of random fields. Statistics & Probability Letters 17(5), 404.
404.
Stein, M. L. (1999). Interpolation of spatial data: some theory for kriging. Springer Science & Business Media.
Stein, M. L. (2005). Space-tine time covariance functions. Journal of the American Statistical Associa- fion 100(469), 310-321.
Sun, S., G. Zhang, C. Wang, W. Zeng, Li, and R. Grosse (2018, 10-15 Jul). Differentiable compositional kernel learning for Gaussian processes. J. Dy and A. Krause (Eds.), Proceedings of the 35th International Conference on Machine Learning, Volume 80 of Proceedings of Machine Learning Research, pp. 4828 4837. PMLR.
Tang, W., L. Zhang, and S. Banerjee (2022). On identifiability and consistency of the nugget in gaussian spatial process models. Journal of the Royal Statistical Society Series B: Statistical Methodology 83(5), 1044-1070.
Tans, P. and R. Keeling (2023). Trends in atmospheric carbon dioxide. https://gml now.. gov/ ccgg/ wrendd/ddaii /data.d//aaa... html Accessed: 2023-08-01.
Verma, A. and B. E. Engelhardt (2020). A robust nonlinear low-dimensional manifold for single cell RNA-seq data. BMC Bioinformatics 2/(1), 324.
Wilson, A. and Adams (2013, 17-19 Jun). Gaussian process kernels for pattern discovery and extrapolation. In S. Dasgupta and D. McAllester (Eds.), Proceedings of the 30th International Conference on Machine Learning, Volume 28 of Proceedings of Machine Learning Research. Atlanta, Georgia, USA, pp. 1067-1075. PMLR.
Wilson, A. G., E. Gilboa, A. Nehorai, and J. P. Cunningham (2014). Fast kernel learning for multidimensional pattern extrapolation. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger (Eds.), Advances in Neural Information Processing Systems, Volume 27. Curran Associates, Inc.
Zhang, H. (2004). Inconsistent estimation and asymptotically equal interpolations in model-based geostatistics. Journal of the American Statistical Association 99(455) 250-261.
12


Reviews:
[{"review_id": "0ybQDy0V9y", "reviewer": "Reviewer_PY7Z", "summary": "This work looks at idenfiability in the context of mixture kernels in GP regression. They look at additive mixtures of Matern kernels and multivariate separable kernels in the context of multivariate GPs (multiple outputs). They show theoretically and through simulation that ML-II learning cannot discover the hyperparameters in the mixture corresponding to the different components as they are not microergodic (Stein, 1999). On the contrary, only products of hyperparameters in specific combinations is identifiable. The simulation studies on synthetic data are interesting - through repeated experiments they show that ML-II does not converge to the true hyperparametes in a mixture kernel and only the microergodic parameters are discovered.  ", "questions": "- Theorem 2 'p' is introduced without defining. I've had to figure that it is the dimension. \n- Perhaps tangential to this work but it would be interesting to reason about the identifiability of the noise variance when a mixture kernel is used. \n- How does the additive GPs work where the constructed kernel is additive and considers all combinations of dimensions in the data  fit with the identifiability insights of this work. \n- About the dominance of the least smooth kernel component, how does one extend the insight to mixtures of different kernel types like Matern + RQ or periodic + Matern. \n- Line 281 / 282 pls cite Lalchand et al, Generalised GPLVM with Stochastic Variational Inference, AISTATS 2022.\n- Line 47/ 48 pls cite Simpson et al, Kernel Identification with Transformers, NeurIPS, 2021.\n", "limitations": "\nThe most obvious one - which the authors mention in conclusions is the extension of these insights to mixture of kernel families which are used very frequently to model different effects in univariate functions. eg. CO2 Mauna Loa kernel.\n ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A lot of the theory on identifiability has been scattered in literature like Zhang (2004) and Stein's textbook (1999) but I would argue that a succint and targetted presentation such as this one might benefit the community and hence can be significant. ", "weaknesses": "This work begs the question - what happens in the large data limit when n -> \\infty, as the simulations and applications only address the finite data limit. My guess is that the parameters which are not microergodic remain unidentifiable but some comment is needed. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for presenting us with such insightful questions and observations regarding our manuscript. Given the submission constraints, we're unable to provide an updated version of the full manuscript at this juncture. However, to facilitate a more tangible understanding of our clarifications and to provide visual support, we've attached a 1-page PDF with addditional figures. In the subsequent sections, we proceed with a point-to-point response to each of your comments.\n\n**Weakness about large sample**\n\nA: We appreciate this observation. Our theoretical framework is indeed designed for the infinite sample scenario. Consequently, as $n \\to \\infty$, the conclusion should remain consistent, wherein only the proposed microergodic parameter is identifiable, while others are not. We have clarified this in the manuscript and conducted additional simulations for $n=1000,3000$. The conclusion remains consistent (Figure 1 in the one-page PDF).\n\n\n**Question 1 about $p$**\n\nA: Thank you for pointing this out. Indeed, we defined all domains in this manuscript as $\\Omega = \\mathbb{R}^p$ in line 94. Recognizing the distance between this and Theorem 2, we have explicitly defined $p$ as the dimension of the domain within Theorem 2 in our revised manuscript. \n\n**Question 2 about noise variance**\n\nThank you for raising this intriguing point. In the majority of existing literature, researchers often work under the assumption of zero noise (no nugget), primarily due to the complexities associated with noise inclusion (nowhere continuous). As of our current knowledge, the most recent investigations into this matter can be found in papers by Tang, Zhang, Banerjee (JRSSB 2022) and Loh and Sun (Bernoulli 2023). These studies specifically focus on a single Mat'ern kernel rather than a mixture kernel. To encapsulate their findings briefly, they concluded that the noise variance is indeed identifiable, and the presence of noise does not undermine the identifiability of the Mat'ern parameters. Extending these findings to our mixture kernel remains challenging but is a fruitful avenue for future research.\n\n**Question 3 about adding up all dimensions**\n\nA: This question delves deep into an intriguing facet of our research. We suppose you're referring to a kernel of the form $K(x,x') = \\sum_{l=1}^p w_l \\text{Matern}(x_l-x_l';\\sigma^2_l,\\alpha_l,\\nu)$, where $x_l$ represents the $l$-th coordinate of $x$. \n\nIf so, regrettably, the primary tool we utilized for proving equivalence of measures isn't directly applicable here. This is primarily due to the non-fulfillment of the first condition in the integral test (Yadrenko 1983), which expects the spectral density to behave as $1/\\omega^r$ for some positive $r$. Our conjecture, based on our current understanding, is as follows: Individual parameters may not be identifiable. However, the term $w_l\\sigma_l^2\\alpha_l^{2\\nu}$ remains identifiable, for each $l=1,...,p$. \n\nTo underpin this conjecture, we employed distinct Mat'ern $1/2$ kernels for every dimension of a synthesized 2-dimensional dataset ($p=2$). Our observations indicate that both $w_1\\sigma_1^2\\alpha_1^{2\\nu}$ and $w_2\\sigma_2^2\\alpha_2^{2\\nu}$ are identifiable (Figure 2 in the one-page PDF).\n\nOne can interpret these findings as follows: When our domain is restricted to a 1-D line in 2-D, for instance where $x_1\\neq 0$ and all other dimension are set to zero ($x_2=...=x_p=0$), the kernel morphs into a 1-D Matern supported solely on the $x_1$-axis. Based on established theory, $w_1\\sigma_1^2\\alpha_1^{2\\nu}$ becomes identifiable. \n\nThis logic can be analogously applied to other dimensions. However, it's imperative to note that this isn't a rigorous proof given the absence of a conclusive integral test. Your question has certainly shed light on a promising direction for future research. We're grateful for this thought-provoking inquiry.\n\n\n\n**Question 4 about RQ and periodic kernels**\n\nA: Thank you for posing this insightful query. Notably, RQ and periodic kernels do not have analytic spectral densities, which are essential for a theoretical analysis of GPs. Extending our insights to such kernel mixtures would necessitate the application of more advanced mathematical techniques, particularly within probability theory. We recognize this as a promising area for future research. \n\n\n**Question 5 and 6 about references**\n\nThank you for drawing our attention to these relevant works. We have now incorporated citations to both Lalchand et al. (AISTATS 2022) and Simpson et al. (NeurIPS 2021) in the appropriate sections of our revised manuscript.\n\n\n**Limitation about Co2 Mauna Loa kernel**\n\nA: Thank you for highlighting this pertinent point. The CO2 Mauna Loa dataset and its associated kernel combination serve as iconic examples in the Gaussian Processes literature, demonstrating the power of combining different kernels to encapsulate diverse features of a dataset. While our study emphasizes the theoretical underpinnings of mixture kernel types, integrating these findings into practical, often-used kernel combinations like those for the CO2 Mauna Loa dataset would undeniably deepen our understanding. As we noted in our discussion, broadening our framework to encapsulate other kernel families is an intriguing and vital future research direction.\n\nHowever, we'd like to underscore that extending these insights to some kernel families can be particularly challenging due to the absence of an analytic form of their spectral densities. Instead, we performed experiments on this dataset with Matérn kernels, see Figure 3 in the one-page PDF for rebuttal. On this new dataset, we can see that Matérn 1/2 performs similarly with the mixture of Matérn 1/2, 3/2 and 5/2; and Matérn 3/2 performs similarly with the mixture of Matérn 3/2 and 5/2, in terms of MSE, further justifying our Theorem 2.\n"}, {"review_id": "PVe6Exp8Oa", "reviewer": "Reviewer_dNVU", "summary": "This paper investigates additive and separable mixture kernels in the context of Gaussian process regression. Concretely, it tests the intuition behind the convex combination of Matern processes and identifies limitations in the interpretability of the resulting mixture kernel that might contradict the intuition of the modeler.\nIt further analyzes the identifiability of parameters of separable Kernels in multi-output GPs.", "questions": "- I would like to understand the impact of the \"smoothness-of-mixed-Matern-kernel-is-determined-by-its-least-smooth-component\" argument better:\n\t- Is it really only about smoothness? This seems rather obvious to me (and I am certain this has been known before). But Theorem 2 states that the kernel is equivalent, and hence makes a more general statement about the structure of the functions from this kernel's RKHS, than merely smoothness, right? As somebody who is not familiar with proofs about identifiability and equivalence of kernels: apart from smoothness of the resulting sample paths, does mixing in more Matern components not add *any* structure to a single (least-smooth) Matern component? I could imagine that other readers would also appreciate a clarifying statement that brings together the \"same-smoothness\" and \"equivalence\" arguments.\n\t- What reinforces me in my endeavor to understand the impact of this statement better is also the fact that in Section 7, you write that \"[...] the inclusion of kernels with different smoothness does not necessarily improve prediction accuracy\".\n\t  On the other hand, Theorem 2 states that it does absolutely make no difference (not only \"not *necessarily*\"). Could you elaborate more on the impact of this statement and its strength?\n- Suggestion: the paper includes a comparison between (i) a mixture of kernels with varying smoothness parameters and (ii) a kernel consisting only of the least-smooth component. It would be interesting to see the comparison to (iii) the other (two?) components, as well. I would expect these results would be different then. But *how* they would be different (by how much or - e.g. in the case of Figure 6 - how they differ structurally) would be quite interesting, measured by the expected effort of this addition.", "limitations": "Limitations of the analysis presented in the text are adequately addressed in the discussion section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I believe this paper is a useful contribution to understanding when intuition about modeling decisions diverges from the actual impact of these decisions. It thus earns its place at a ML conference in that it strictly extends the pool of resources that should be considered when designing a GP model\n- NeurIPS seems to me as the right venue to publish these results in that format, since it manages a good balance between mathematical rigor and intuition, which makes the text very approachable for more application-oriented researchers, as well.\n- the overall presentation follows a very clear structure that is easy to follow. I appreciate that the authors do not obscure meaningful results behind unnecessarily complicated formulations.", "weaknesses": "- The plots are aesthetically rather unpleasing and the presentation of the results would, in my opinion, profit quite significantly from a polishing of the figures.\n- The authors focus solely on Matern processes, to the point where the notion of \"kernel\" is used interchangeably with \"Matern-kernel\". This limitation, however, is understandable in light of the page limit and is also addressed in the discussion section. (It does not affect my score.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe are deeply grateful to the reviewer for your thoughtful and thorough examination of our manuscript. Your observations and suggestions have been invaluable in refining the narrative and content of our paper. While we're unable to present the full revised manuscript at this time, please find our detailed responses to your comments below. Further clarifications, especially in relation to the figures, can be found in the accompanying 1-page PDF for rebuttal. Your feedback has been instrumental, and we genuinely appreciate the effort and expertise you have devoted to our work.\n\n**Weakness 1: about the plots**\n\nA: Thank you for the comment. We have meticulously revised and polished all the figures. Since we are not allowed to upload the revised manuscript, please see the additional figures provided in the one-page PDF for rebuttal. \n\n**Weakness 2: about the use of \"kernel\" vs \"Matérn kernel\"**\n\nA: Thank you for understanding the scope of our work within the constraints of the page limit. To ensure clarity and to avoid any possible confusion, we have revised the manuscript to consistently use \"Matérn kernel\" instead of the more generic term \"kernel.\" \n\n**Question 1.1: about smoothness**\n\nA: Thank you for this insightful comment. While Theorem 1 primarily emphasizes the smoothness aspect, Theorem 2 delves into a deeper equivalence, which articulates that the Gaussian random measure associated with the mixture of Matérn kernels is fundamentally equivalent to that of the least smooth single Matérn. This means that beyond the superficial similarities in smoothness, the underlying structures of functions from the kernel's RKHS are also equivalent. In essence, incorporating additional Matérn components doesn't introduce new structural nuances beyond what the least-smooth component already encapsulates.\n\nTo aid understanding for our readers, we've appended a paragraph after Theorem 2 to bridge the connection between the smoothness result in Theorem 1 and the deeper equivalence presented in Theorem 2. Your feedback has illuminated an important aspect, and we believe this addition will offer more clarity to the readers.\n\n**Question 1.2: about the conclusion in Section 7**\n\nA: Thank you for highlighting this discrepancy. You are absolutely right that there seems to be a contrast between the theoretical implications of Theorem 2 and the conclusion in Section 7.\n\nThe fundamental insight from Theorem 2 is that, from a purely theoretical standpoint, the inclusion of additional Matérn components with varying smoothness levels does not bring about any change in the underlying structure or representation capabilities of the kernel. Hence, when our model assumptions hold, one shouldn't expect improvements in prediction accuracy merely by adding kernels of different smoothness.\n\nHowever, real-world data and applications often exhibit nuances and complexities that might not align perfectly with our theoretical model assumptions. One potential reason for the discrepancy in performance between a single kernel and a mixture kernel in real data settings could be the increased parameter space in the mixture model, making parameter estimation more challenging. As a result, despite the potential for greater flexibility, the mixture kernel might not always result in improved prediction due to potential sub-optimal parameter estimations.\n\nAnother aspect to consider is the potential mismatch between our chosen model (GP with specific kernels) and the true underlying process of the real-world data. If the real-world function doesn't strictly adhere to a Gaussian Process or doesn't exactly reside in the presumed RKHS, discrepancies between theoretical predictions and empirical observations can emerge.\n\nIn summary, while Theorem 2 provides a strong theoretical foundation asserting the equivalence of a single least-smooth Matérn kernel and its mixture counterpart, real-world applications introduce complexities that can sometimes deviate from our theoretical insights. We have further elucidated this point in the revised manuscript to ensure that the relationship between theoretical findings and empirical observations is clear to our readers.\n\n**Question 2: suggestion about additional experiments related to Figure 6**\n\nA: We are grateful for the reviewer's insightful suggestion, emphasizing the potential to further elucidate our arguments. The proposed comparison between the additional components is not only intriguing but will also serve to provide more comprehensive insight into our research.\n\nAcknowledging the feedback, and in alignment with remarks from reviewer j39y, we recognize that the Pollen dataset might not be the most optimal choice for demonstrating the nuances of the Matérn kernel. Taking this into account, we have transitioned to utilizing the Mauna Loa $CO_2$ time series data (Figure 3 in the 1-page PDF for rebuttal). We observe similar performance between Matérn mixture $1/2+3/2+5/2$ and Matérn $1/2$, as well as the resemblance between Matérn mixture $3/2+5/2$ and Matérn $3/2$. At the same time, while the performance of Matérn $3/2$ exceeds that of Matérn $1/2$, underscoring the dataset's inherent differentiability, it's noteworthy that the integration of the $3/2$ component to $1/2$ in a mixture kernel doesn't elevate its performance to match that of Matérn $3/2$. This observation supports our contention: the performance of a Matérn mixture kernel is chiefly dictated by its least smooth component. \n"}, {"review_id": "hoKZGqdPeP", "reviewer": "Reviewer_LXB2", "summary": "This paper intoduces a few results on identifiability of Gaussian Processes. \nIt is shown that a Gaussian process with a kernel that is a mixture of kernels, the smoothness is that \nof the least smooth component.  \nIt is shown that for two mixtures of Matern kernels, the induces processes are equivalent if certain\nproducts involving only parameters of the least smooth kernels in the mixture coincide. Consequently, \nhigher smoothness components do not affect equivalence. \nFinally, for a multi output GP with covariance  of the form $A \\cdot k$, where A is a fixed matrix and k is a scalar Matern kernel, the matrix A is identifiable. \n\n\nExperiments that are intended to support the above results are included. \n\n", "questions": "Please see above. \nMy recommendation for the rating is temporary and I plan to revisit it following author's responses. ", "limitations": "The subject of this works involves some abstract properties of Gaussain Processes. No specific societal impact discussion is required. ", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "The general subject of identifiability in GPs is important. \nThe non technical parts of the paper are well written and it is possible get a general idea of what the paper is about. \nThe experiment 5.2 is illustrative in showing that the prdouct can be identified while the individual components are not identified properly. (however, a much more detailed description of the experiment is needed). \n", "weaknesses": "My main question is regarding the notion of equivalence used in this paper. It is not clear why this is \na useful notion. The questions are as follows:\n\n* Clairty regarding identifiability 1. First,  what is definition of the $\\equiv$ symbol that is used for kernels in all the theorems? I couldn't find this definition for some reason. Where is it defined? Does it mean that the GPs induced by the corresponding kernels are equivalent in the sense of Definition 5? \n\n* Clairty regarding identifiability 2: I'm assuming the answer to the question above is yes. \nThen, why is this equivalence interesting and relevant to identifiability? Please explain why this notion is relevant. \nIndeed, if my index set is finite, I can have two GPs on this set with covraiances C_1, C_2. As long as C_1,C_2 have full rank, the processes will be absolutely continuous w.r.t each other, but they are very different processes and C_1 and C_2 can be learned.  How this related to the above notion of equivalence? \n\n* In anycase, this notion of equivalence is by no means standard in the ML commmunity, and needs to be introduced and discussed in more detail. \n\n\nAdditional Questions:\n\n\n* Relevance:  While the general subject of identifiability is imporatnt, this is concerned with a very specific case of Matern linear mixtures. It is not clear this specific case has been used in applications before.  \nSome references are provided about the use of mixtures on lines 37-38, but how many of these use specifically *linear mixtures*, and  *Matren mixtures*?\n\n* Literature discussion: There is no Literature section in the paper.  It is my understanding that there is previous work on identifibility, even for a single Matren kernel. This needs to be discussed in the paper, and comparison provided. Are there new mathematical ideas involved in the results in this paper compared to existing work? \n\n* Inverse implication in results: The results state that if parameters satisfy something, then there is equivalence. Is the reverse direction true? Is it obvious? This again is related to the somewhat \n\n\n* Experiments: \nAs mentioned earlier, Experimen 5.2 is illustartive. However, it is not specified how the optimization is made. It may also have happened that there is no reconstruction due to local maxima of the method, rather than general lack of identifiability.  \n\n* As for the other experiments: the experiments 5.1, 6.1, 6.2 do not illustrate anything that I would find informative.  6.2 is a 1d version of an already small multidimensional dataset. It is not clear how 6.1 is realted to  ML use cases. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to express our gratitude to the reviewer for your meticulous feedback and constructive comments.  Enclosed within the 1-page PDF for additional figures, we address each point as follows. \n\n**Weakness 1: definition of equivalence of measures**\n\nA: Thanks for highlighting this. The symbol $\\equiv$ denotes the equivalence of the Gaussian Processes induced by the two kernels (Definition 5). We've now defined it in the context of Theorem 2. \n\n**Weakness 2: finite domain**\n\nA: Thank you for your insightful comment. The equivalence of GPs in the context of our study is pivotal because if two kernels within the same parametric family produce equivalent GPs, then the parameters of those kernels become non-identifiable. This, in essence, underlines that one cannot uniquely determine the parameters given observed data, which is a vital understanding for model interpretation and estimation.\n\nYou're correct in noting the differences in the finite and infinite domains. When working in a finite domain, GPs are indeed just multivariate Gaussian distributions, and as you rightly pointed out, any two finite dimensional Gaussian measures are equivalent. However, we focused on infinite subset of Euclidean space (we've clarified this in the revision). \n\n**Weakness 3: the notion of equivalence**\n\nA: We've elaborated on equivalence post Definition 5 and emphasized challenges of parameter identifiability in infinite domains.\n\n**Q1: about specific case of Matérn mixtures**\n\nA: For additive methods, Duvenaud et al. (NeurIPS 2011) used a mixture of RBF; Duvenaud et al. (ICML 2013) applied a mixture of RBF, periodic (Per), linear (Lin), and rational quadratic (RQ); Kronberger and Kommenda (LNTCS, 2013) worked with RBF+RQ, RBF+Matérn, RBF+Per; Verma and Engelhardt (BMC Bioinformatics 2020) used RBF, Matérn 1/2, 3/2, 5/2. For more complex methods with additive and multiplication methods, Sun et al. (ICML 2018) used RBF, Per, Lin, and RQ; Lloyd et al. (AAAI 2014) employed RBF, Lin, Per, RQ, white noise, and constant.\n\nWe concur that our study focuses on the additive/multiplicative aspect of the Matérn kernel. We've incorporated this focus into the limitations section of our manuscript and we anticipate that this exploration will stimulate future research concerning identifiability and interpretability.\n\n**Q2: literature discussion**\n\nA: We've expanded our literature review in the introduction, highlighting the progress in the area of identifiability for single Matérn kernels. While these works provide foundational insights, our paper primarily fills the gap in literature concerning mixtures of Matérn kernels.\n\nIn terms of the mathematical intricacies, while we utilize established probability theorems on Gaussian random measures, the application and verification of these conditions for our research context presented non-trivial challenges, detailed further in our supplementary material. The introduction has now been updated to highlight this comparative discussion, which only underlines our contributions but also contextualizes our approach within the broader GP kernel identifiability landscape.\n\n\n\n**Q3: inverse implication**\n\nA: Your observation is sharp: The inverse implication is not immediately obvious. While we believe that the reverse direction is true based on our understanding and intuition, its proof poses substantial challenges. In prior work on single Matérn kernels, such as Zhang (JASA 2004), the key in the proof is that two Matérn kernels differing only in their microergodic parameter define the same correlogram. However, this does not hold for mixture kernels anymore. Thus, existing probability tools are not directly applicable. This open question is certainly intriguing and worthy of future exploration. \n\n**Q4: optimization in experiment 5.2**\n\nA: We understand your optimization concerns. Our extensive analysis (more experiment details can be found in the supplement), including L-BFGS (as also suggested by reviewer j39y), supports our main conclusions. We also included a loss curve for Simulation 2, see Figure 5 in the PDF,  implying that our findings are not the result of local maxima but are indicative of the identifiability issue we are investigating. Another supporting evidence is the distinct behavior of the microergodic parameter. It uniquely converges to its true value, even when other parameters do not. \n\n**Q5: other experiments**\n\nThe goal of 5.1 is to support that the smoothness of the mixture kernel is driven by the least smooth kernel. As also suggested by Reviewer j39y, we performed more experiments to support Theorem 1, see Figure 6 in the PDF. \n\nFor applications, we aim to replicate experiments performed in other analysis in ML literature. 6.1 comes from Figure 1(a) in Wilson et al. (NIPS 2014), Figure 4(a) in Remes et al., (NIPS 2017) and Figure 5(a) in Sun et al., (ICML 2018). 6.2 comes from Table 1-2 in Sun et al., (2018). To better claim our points, we've implemented the Matérn mixture kernel on MNIST and Mauna Loa $CO_2$. For MNIST, the Matérn kernel has demonstrated satisfactory prediction accuracy (Figure 4 in the PDF). Consistent with the results in our manuscript, the mixture kernel exhibits comparable performance to the Matérn 1/2 kernel, evidenced by a Pearson correlation coefficient of 0.99. $CO_2$, suggested by reviewer dNVU and employed in Wilson and Adams (ICML 2013) (Figure 3 in the PDF), shows analogous performance between the Matérn mixture 1/2+3/2+5/2 and Matérn 1/2, as well as the resemblance between Matérn mixture 3/2+5/2 and Matérn 3/2. \n\nWe agree that Matérn kernel may not apply ideally on some of the dataset. Our intention was to investigate the comparative performance of GP regression using a Matérn mixture kernel and its least smooth kernel. We understand the reviewer's concerns and are open to conducting further experiments to address any specific interests. \n"}, {"review_id": "YXbZJnLVA1", "reviewer": "Reviewer_j39y", "summary": "The authors carefully examine additive mixture of Matern kernels for Gaussian processes. For the single-output case, the authors show that a mixture of Matern kernels is equivalent to the least smooth component of the mixture (Theorem 2). Consequently, \"there is no advantage in including any other component apart from the least smooth component\" (Corollary 1). For the multi-output case, the authors leverage results from Bachoc et al. (2022) to identify microergodic parameters of separable multi-output kernels. These theoretical results are accompanied by demonstrations on simulated data. In addition, in three experiments on real-world data, the authors provide evidence that a mixture Matern kernels performs as well as just the least smooth components.", "questions": "## Conclusion\n\nI think that this is great submission with very interesting results. However, in my opinion the authors have not provided sufficient evidence for the central message of the paper, which is that the inclusion of kernels with different length scales does not improve prediction accuracy. (Please correct me if this is not the central message of the paper.) The authors have not provided sufficient evidence because of the following two reasons:\n\n* I believe that the conclusion in Corollary 1 is not justified. Please see the reasoning above.\n\n* I believe that the data sets chosen in the experiments do not contain any smooth components, so obviously modelling smooth components in the data will not get you any improvement in predictive accuracy. Please see the foregoing section. Therefore, I think that the experimental results do not support the central message of the paper.\n\nIn addition, the proof of Theorem 2 might have an error (please point out whether I'm right or wrong about this!), Simulations 1 and 2 (please see above) have flaws, and I think that Theorem 3 is only a minor contribution. If I add all this up, I'm afraid that I must recommend rejection at this point.\n\n_However_, I'm very willing to debate the above arguments and revise my score if necessary. Therefore, authors, please carefully argue the above, because I would love to be wrong and change my recommendation to accept.\n\nEDIT\n\nIn the discussion below, the authors have outlined changes that address the above shortcomings. These changes are substantial. Since uploading revisions is not possible, normally I would recommend that the revised PDF go through another round of review. However, I feel that the authors and I are now aligned sufficiently well, so this time I am willing to recommend a \"borderline accept\".\n\nI should emphasise that I believe that this submission likely deserves a better score than \"borderline accept\", but I am not willing to recommend any higher score before reviewing the revised PDF.", "limitations": "See above.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I would like to thank the authors for their submission. I enjoyed reading the submission and certainly learned few new things.\n\n## Strengths\n\n* The paper is very well written and overall the presentation is good. I found hardly any typos.\n\n* Additive mixtures of kernels are very important in GP modelling, since it allows the practitioner to model multiple length scales in the data at various levels of smoothness, so a careful assessment of additive mixtures like this work does is important.\n\n* The main result of the paper, Theorem 2, is very interesting and might have important implications. I've read through the proof, and, modulo my comment below in the weaknesses, I think that the proof is otherwise likely alright.\n\n* The experiments appear to nicely support the central conclusion of the paper: \"the inclusion of kernels with different smoothness does not necessarily include prediction accuracy\" (Section 7). (However, I have doubts about this. Please see below.)", "weaknesses": "## Weaknesses in the Method and Theory\n\n### Claim of Theorem 1 Not Fully Supported by the Proof, but a Fix Appears Possible\nThe claim of Theorem 1 consists of two parts: (1) $K$ is $d$-times MSD, and (2) \"so the smoothness of $K$ is determined by the least smooth component\". For (2) to follow from (1), it is also necessary that $K$ is _not_ smoother than $d$-times MSD! That is, it is necessary that $K$ is _not_ $(d + 1)$-times MSD.\n\nFor functions, consider the following example. Suppose that $f_1$ is differentiable and that $f_2$ is non-differentiable. Is $f_1 + f_2$ then non-differentiable, so the smoothness of the sum is determined by the least smooth component? The answer is no: if $f_1 - f_2$ is non-differentiable, then a counter-example is found with $g_1 = f_1$ and $g_2 = -f_2$; and if $f_1 - f_2$ is differentiable, then a counter-example is found with $g_1 = f_1 - f_2$ and $g_2 = f_2$. Therefore, for sums of functions, the smoothness is _not necessarily_ determined by the least smooth component, because the sum can be smoother than the least smooth component! Although the claim is false for functions, kernels have very different properties, and I think the claim is true in that case.\n\nHence, to fully support the claim in Theorem 1, it is necessary that the author prove that $K$ is not $(d + 1)$-times MSD, which is currently not shown in the proof. Fortunately, I think that the proof is easily extended by making use of the following property: if $f_1$ is integrable and $f_2$ is non-integrable, then, by $|f_1 + f_2| >= |f_2| - |f_1|$, $f_1 + f_2$ is also non-integrable.\n\n### Question About Proof of Theorem 2\n\nIn the proof of Theorem 2, the fourth inequality on line 25 of the appendix seems to use the following property:\n\n$ \\frac{f(\\omega) + o(f(\\omega))}{g(\\omega) + o(g(\\omega))} = \\frac{f(\\omega)}{g(\\omega)}(1 + O(\\|\\omega\\|^{-2})) $\n\nwhere $f(\\omega)$ and $g(\\omega)$ are the first terms of the particular mixtures of $p_l$ and $\\tilde p_l$. Now,\n\n$ \\left| \\frac{(f(\\omega) + o(f(\\omega)))/(g(\\omega) + o(g(\\omega)))}{f(\\omega)/g(\\omega)} - 1\\right| = \\left|\\frac{1 + o(1)}{1 + o(1)}\\right| = |o(1)|$,\n\nso\n\n$ \\frac{f(\\omega) + o(f(\\omega))}{g(\\omega) + o(g(\\omega))} = \\frac{f(\\omega)}{g(\\omega)}(1 + o(1)) $,\n\nwhich is a weaker result! If you're a little more careful, instead of $o(1)$ you might get a $O(\\|\\omega\\|^{-q})$ where $q$ depends on the ratio of the leading term and the second term in the particular mixtures of $p_l$ and $\\tilde p_l$. Importantly, $q$ might be very small, smaller than $2$. It's not at all obvious to me that $q=2$ is true! This may seem like a minor detail, but it could mean that the integrand actually depends on $O(\\|\\omega\\|^{-2q})$ for $q$ potentially very small, which would affect the whether the integrand is integrable or not.\n\nI would like the authors to reply to this question. Since Theorem 2 is the central result of the paper, it is important that the proof is bulletproof.\n\n### I Disagree With Corollary 1\n\nCorollary 1 states that \"there is no advantage in including any other components apart from the least smooth component\".\nI strongly disagree with this conclusion, because I think that it misinterprets the concept of equivalence.\n\nIf two GPs are equivalent, then they produce _asymptotically_ similar predictions in the limit of infinite data under fixed-domain asymptotics.\n_However_, for finite data, equivalent GPs may give different predictions, and this difference may be practically very significant!\nTherefore, to conclude that \"there is no advantage in including any other components apart from the least smooth component\", in my opinion, is wrong.\nLet me give a simple counter-example.\n\nLet $k_1$ be a Matern-$1/2$ kernel with length scale $1$ and let $k_2$ a Matern-$3/2$ kernel with length scale $1000$.\nConsider the mixture $k = k_1 + k_2$, and suppose that data is generated from $k$.\nSuppose that we have $3$ observations randomly sampled from $[0, 2000]$.\n\nIf we make predictions with just the least smooth component, $k_1$, then, because of the short length scale of $k_1$, the predictive mean is basically zero on all of $[0, 2000]$ except near the observations.\nOn the other hand, if we make predictions with the mixture $k$, then, because of the long length scale of $k_2$, the predictions are much more reasonable.\n\n(You can concoct a similar scenario by considering a Matern-$1/2$ kernel with length scale $1$ and a Matern-$v$ kernel for very high $v$ (so basically a squared-exponential kernel) also with length scale $1$. For finite data, the difference in smoothness will give very different predictive uncertainty.)\n\n### Attribution of Theorem 3\n\nIn the main body, the authors present Theorem 3 as if it is a completely novel result.\nHowever, Theorem 3.(i) is basically Theorem 3 by Bachoc et al. (2022), and Theorem (3).(ii) follows nearly immediately from Theorem 2 by Bachoc et al. (2022).\nThe authors acknowledge this in the proof in the supplement, but they should really state this attribution in the main body.\nBecause of the similarity to the theorems by Bachoc et al., I consider Theorem 3 by the authors only a minor contribution.\n\n## Weaknesses in Experiments\n\n* In Simulation 1, the authors state that the \"it is evident that the mixture kernel demonstrates a degree of smoothnes similar to the Matern kernels with $\\nu = 1/2$\". Although I think that the result is true, I think that this is not necessarily so obvious from Figure 1. To really determine the smoothness of the samples of the mixture kernel, the authors should provide more conclusive evidence, e.g. by numerically determining whether the function is differentiable: numerically compute $|f(x) - f(x + h)|/|h|^{\\alpha}$ for $h \\to 0$ and _e.g._ $\\alpha = 1$ (for once differentiability).\n\n* In Simulation 2, the authors demonstrate that $w_1 \\sigma_1^2 \\alpha_1$ is identifiable and they claim that _only_ this parameter is identifiable. To really give evidence that only  $w_1 \\sigma_1^2 \\alpha_1$ is identifiable, the authors should show the parameters that would be microergodic if you would consider the kernels separately: $w_2 \\sigma_2^2 \\alpha_2$ and $w_3 \\sigma_3^2 \\alpha_3$. These parameters are also not shown in Figure S1. Without showing these additional parameters, I'm not convinced that the figure really prove that only  $w_1 \\sigma_1^2 \\alpha_1$ is identifiable.\n\n* Throughout the real-world experiments, the authors use Adam/SGD. I wonder why the authors wouldn't just run L-BFGS-B until convergence at a reasonable tolerance. This would likely eliminate some difficulties in optimising the mixture kernel. In addition, the supplement says that the kernel matrices are _heavily_ regularised by a diagonal of magnitude $0.01$ or $0.1$. I think that is _much more_ than should be necessary if one uses double precision. I'm not sure if it would change any results, but it does make me suspicious.\n\n* In Application 1, the authors compare a Matern mixture kernel against a Matern-1/2 kernel to perform texture extrapolation. Although I don't disagree with the result that the mixture Matern kernel performs similar to the Matern-1/2 kernel, I think both the mixture kernel and the Matern-1/2 kernel are a very poor fit for the data, which can be seen from the prediction in Figure 4. The point of these textures is that extrapolation is possible because they are periodic, so one should really use a (weakly) periodic kernel. I understand that Theorem 2 does not apply to mixtures of periodic Matern kernels, just to regular Matern kernels, but to me that suggests that this is not a good data set to demonstrate Theorem 2 on.\n\n### Experiments Do Not Strongly Support Main Claim\n\nMy main issue with the experimental section is that the authors take some data sets, compare a mixture of Matern kernels versus a Matern-1/2 kernel, observe that the performance of the two is very similar, and conclude that, generally, there might be no advantage in adding additional components to the mixture. In my opinion, this is not the right conclusion. The right conclusion is that either (a) generally, there might be no advantage in adding additional components to the mixture; or (b) specifically in the chosen experiments, there is no advantage in adding additional components to the mixture.\n\n_To really conclude (a), the authors should compare the mixture kernel to the Matern-1/2 kernel on a data set where conventially one would agree that a Matern mixture kernel would be a good choice._ An example of such a data set is a time series that has a noisy short-length-scale components and a smooth long-length-scale component. In constrast, economical data sets such as prices are traditionally modelled with an Ornstein-Uhlenbeck process (Matern-1/2 kernel), so I'm not surprised additional smooth kernels won't help there. Moreover, if you visualise the gene expression data, you will see that the data is _very_ erratic and definitely doesn't contain and smooth components. Therefore, I am not at all convinced that the authors have made a good case for (a).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe sincerely thank the reviewer for the meticulous and insightful examination of our work. Your comprehensive feedback has been instrumental in enhancing the quality and clarity of our paper. In response, we have addressed each of the concerns raised and revised the manuscript accordingly. While we are unable to upload the revised manuscript at this juncture, please find below our point-by-point response to your comments. Additional figures that further elucidate our points are provided in the attached 1-page PDF for rebuttal. We deeply appreciate the time and effort you've invested in guiding our revisions.\n\n**About Theorem 1**\n\nA: Thank you for your keen observation. It's pivotal to also ensure that the kernel doesn't exceed this smoothness level. Your suggestion aligns with the proof's direction: since $\\rho_1(\\omega)\\omega^{2d+2}$ is non-integrable, it implies that $\\rho(\\omega)\\omega^{2d+2}$ is similarly non-integrable.\n\nYour example effectively distinguishes the differentiability of a deterministic function and mean-squared differentiability of random functions. We value your comprehensive feedback and have provided a clear proof of Theorem 1 in our revised paper. \n\n**About Theorem 2**\n\nA: You've identified a crucial detail. We omitted a key assumption about smoothness: $\\nu_{l+1}-\\nu_l\\geq 1$.  This assumption is reasonable since if two consecutive $\\nu$’s are too close, then the corresponding kernel components $K_1$ and $K_2$ admit the same smoothness, negating the need for a mixed kernel (see Theorem 4 in the supplement).\nAs detailed in line 23 in the supplement, the order of the leading term $p_1(w)$ is $2\\gamma_1$, while the order of all other terms, $2\\gamma_2,\\cdots,2\\gamma_L$, are all smaller than $2\\gamma_1$ by at least 2. As a result, if we divide both the numerator and the denominator by $\\|\\omega\\|^{2\\gamma_1}$, the denominator becomes \n$\\sum_{l}{w_l}{\\sigma}_l^2{\\alpha}_l^{2\\nu_l}{p}_l(\\omega)/\\|\\omega\\|^{2\\gamma_1}=w_1\\sigma_1^2\\alpha_1^{2\\nu_1}+O(\\|\\omega\\|^{-2})$.\nApplying the same trick to the numerator, the ratio simplifies to $\\frac{\\widetilde{w}_1\\widetilde{\\sigma}_1^2\\widetilde{\\alpha}_1^{2\\nu_1}}{{w_1}{\\sigma}_1^2{\\alpha}_1^{2\\nu_1}}(1+O(\\|\\omega\\|^{-2}))$.\n\n\n**About Corollary 1**\n\nA: We agree with your perspective on the significant difference between a model's asymptotic behavior and its finite sample performance.  Corollary 1 emphasized asymptotic equivalence, possibly downplaying the importance of finite sample scenarios. Our revision clarifies this.\n\n**About Theorem 3**\n\nA: We recognize the need for proper attribution. While Theorem 3.(i) is indeed a direct corollary of the result by Bachoc et al., Theorem 3.(ii) is a distinct extension requiring nuanced verification. We have revised the presentation of Theorem 3 to explicitly mention its relationship to Bachoc et al.\n\n\n**About Simulation 1**\n \nA: Thank you for the insight. In our context, the differentiability of GP is in the mean-square sense (Definition 3). The numerical difference of the sample path doesn’t inherently imply mean-square differentiability. Our method (Figure 6 in the 1-page PDF) appears novel and involves the following steps:\n\nFor a given fixed $x_0=0$, let $x_i=1/i$ for $i=1,2,…$, and we generate $y_i^l$ from the GP, where $l=1,…,T$ denotes the index of replicates. This allows us to approximate $\\lim_{x\\to 0}\\mathbb{E}(f(x)-f(0))^2$ with $\\beta_i=\\frac{1}{T}\\sum_{l=1}^T (y_i^l-y_0^l)^2$. As per Definition 3, the GP is continuous if and only if $\\beta_i\\to0$.\n Similarly, we can approximate $\\lim_{x\\to 0}\\mathbb{E}\\left(\\frac{f(x)-f(0)}{x}\\right)^2$ with $\\gamma_i=\\frac{1}{T}\\sum_{l=1}^T \\left(\\frac{y_i^l-y_0^l}{x_i}\\right)^2$. The GP is mean-square differentiable if and only if $\\lim_{i\\to\\infty}\\gamma_i $ exists. \n \nSpecifically, for the mixture of Matérn 1/2 and 3/2 (the first column) and Matérn 1/2 (the second column), while $\\beta_i\\to0$, $\\gamma_i$ does not converge. However, for Matérn $3/2$ (the third column), $\\beta_i\\to0$ and $\\gamma_i$ converges,. This empirical evidence bolsters the claims made in Theorem 1.\n\n**About Simulation 2**\n\nA: Figure 7 in the 1-page PDF now includes two panels showing inconsistencies of $w_2\\sigma^2_2\\alpha_2^{2\\nu_2}$ and $w_3\\sigma^2_3\\alpha_3^{2\\nu_3}$.\n \n**About  L-BFGS**\n\nA: We appreciate the reviewer's keen observation regarding the use of the L-BFGS optimizer in our real-world experiments. Following the GPyTorch's guideline, our initial approach employed the Adam and SGD optimizers. However, upon recommendation, we conducted the analysis again using the L-BFGS optimizer.  This adjustment enhanced our optimization process, enabling us to diminish the diagonal magnitude to $1e-04$ (Figure 7 in the 1-page PDF was obtained by L-BFGS). It reaffirms our manuscript's conclusion that only the term $w_1\\sigma^2_1\\alpha_1^{2\\nu_1}$ is identifiable.\n\n**About Application 1**\n\nA: We have conducted a new experiment utilizing an MNIST image (Figure 4 in the 1-page PDF) due to its absence of periodic patterns. The results demonstrate pleasing prediction accuracy for Matérn kernels, thereby reinforcing our understanding of their applicability and effectiveness in this particular context.\n\n**About experiments do not strongly support main claim.**\n\nA: We are in full agreement with the suggestion that we should include a dataset with a smooth component. Also suggested by reviewer dNVU, we conducted a new analysis on the Mauna Loa $CO_2$ dataset (Figure 3 in the 1-page PDF). In this context, the Matérn $3/2$ appears to exhibit better prediction accuracy compare to Matérn $1/2$. However, we observe similar performance between the Matérn mixture $1/2+3/2+5/2$ and Matérn $1/2$, as well as the resemblance between Matérn mixture $3/2+5/2$ and Matérn $3/2$. This observation further substantiates our argument that the performance of a Matérn mixture kernel is primarily dictated by the least smooth Matérn kernel within it.\n"}]

Author rebuttal:
We express our sincere gratitude to the reviewers for their meticulous examination of our manuscript and their insightful comments. Your observations and recommendations have significantly enhanced the quality and content of our paper, and we have learned much from your comments. In this global response, we will discuss some common interest and concern across all the comments, go through the new changes in the manuscript. And at the end, we will explain the detailed information about the figures in the one-page PDF.

**1. (Reviewer PY7Z, LXB2, j39y)** We wish to address the concern regarding the large data limit. Our theory is built on the asymptotic side, and as such, our results remain consistent as $n\to\infty$. 

**2. (Reviewer PY7Z, dNVU, LXB2)** We noted the shared interest in extending our methods to other kernels, such as RQ and periodic kernels. These particular kernels lack the analytic spectral densities necessary for a theoretical analysis of Gaussian Processes, which would necessitate more complex mathematical approaches. While our current study does not encompass these kernels, we have initiated a new simulation in response to reviewer PY7Z's interest in the kernel $K(x,x')=\sum_{l=1}^pw_l\text{Matern}(x_l-x_l';\sigma^2_l,\alpha_l,\nu)$, where $x_l$ represents the $l$-th coordinate of $x$. Although the primary tool we used to prove equivalence of measures is not directly applicable in this context, we have formulated a conjecture that the microergodic parameter of each dimension remains identifiable, while other individual parameters may not be identifiable, along with simulation study to support this conjecture.

**3. (Reviewer dNVU, LXB2, j39y)** The importance of smoothness in our paper was another focal point. We would like to reiterate that the smoothness of a mixture kernel is influenced by the least smooth kernel within it. This property has direct implications for the conditions in our proof, and we've conducted new experiments to numerically verify Theorem 1. 

**4. (Reviewer LXB2, j39y)** We agree that the limited performance of the Matérn kernel may affect the presentations and raise concern. In acknowledgment of this issue, we have conducted two new experiments: MNIST image recovery and prediction on Mauna Loa CO$_2$. For the MNIST dataset, Matérn kernels exhibit satisfactory performance in recovering the image. Mauna Loa CO$_2$ is a well known example that has been widely used on Matérn kernels and other mixture kernels. Interestingly, Matérn 3/2 performed better than Matérn 1/2 in our experiments. However, in Matérn mixture of $\nu=1/2,3/2,5/2$, adding Matérn 3/2 did not improve performance, and the mixture kernel's performance remained similar to Matérn 1/2. This finding further confirms our point. We appreciate your feedback, which has assisted in refining our study.

In response to the feedback received, we have made substantial changes to our manuscript to address the concerns raised. These revisions include the addition and clarification of terminology uncommon in the machine learning community. We have polished all the figures to enhance their clarity and presentation. While we are unable to submit a revised manuscript due to submission rules, the styles of the revised figures can be referred to the new figures included in the one-page PDF. Furthermore, we have expanded our literature review to include detailed information about previous work on the application of mixture kernels, such as the specific types of kernels used in various analyses, and the identifiability work concerning Matérn kernels. This added context should strengthen the connection between our work and existing research. We also included 6 new experiments. 

Our one-page PDF includes the visual presentation of our newly conducted experiments. We conducted four new simulations and two real data applications. 

**Figure 1 (Reviewer PY7Z)** Simulation 2 with larger sample sizes, where we added experiments with sample sizes of n=1000,3000 to validate our conclusions in much larger sample case, thus reinforcing our results.  

**Figure 2 (Reviewer PY7Z)** New simulation concerning the kernel $K(x,x')=\sum_{l=1}^p w_l\text{Matern}(x_l-x_l';\sigma^2_l,\alpha_l,\nu)$, leading us to conjecture that microergodic parameters for each dimension are identifiable while all other parameters remain unidentifiable.  

**Figure 3,4 (Reviewer PY7Z,dNVU, LXB2, j39y)** Mauna Loa CO$_2$ data in 2022 (smoother) and MNIST data (non-periodic), as suggested by reviewers. Our experiments support our claim that the performance of mixture kernel is determined by the least smooth component rather than the best-performed component. 

**Figure 6 (Reviewer dNVU, LXB2, j39y)** New simulation to support Theorem 1, allowing us to confirm that Matérn 3/2 is continuous and differentiable, while Matérn 1/2 and the Matérn mixture 1/2+3/2 are continuous but not differentiable. 

**Figure 5,7 (Reviewer LXB2, j39y)** Simulation 2 with the L-BFGS optimizer, responding to reviewer concerns by verifying our theorem across various optimizers and learning rates, thus reinforcing our confidence in the results. Additionally, following the reviewers' suggestions, two new panels about $w_2\sigma_2^2\alpha_2^{2\nu_2},w_3\sigma_3^2\alpha_3^{2\nu_3}$ were added to the figures of all simulations to better substantiate our point that only the microergodic parameter for the least smooth component is identifiable. 

We have undertaken extensive work in this rebuttal and devoted significant effort to refining our manuscript. We hope that the newly added explanation and experiments could answer your questions and concerns. We would be more than happy to discuss more in the discussion period. Once again, we thank the reviewers for their thoughtful examination and invaluable feedback. Your contributions have not only enhanced our current work but also promise to influence and inspire further research in this field.

Meta-review: Three reviewers recommend borderline accept, weak accept or accept. Reviewer LXB2 recommends Reject. The reviewer disputes the definition of equivalence used in the paper and its practical use in ML. The authors have sufficiently clarified in the rebuttal that their study applies to the infinite domain and that future work can consider the more difficult setting, the finite case, that the reviewer discusses. I believe the paper makes a sufficient contribution to the state of the art of identifiability of GPs and they have backed up their claims with extensive empirical studies. Although the reviewer's concern is valid, I do not believe it guarantees the rejection of the paper. Still, instead, I ask the authors to include such discussion in the supplemental material for the paper.
Decision: Accept (poster)