Title: On the Identifiability and Interpretability of Gaussian Process Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian process

Identifiability

Interpretability

Mixture kernel

Separable kernel
Abstract: In this paper, we critically examine the prevalent practice of using additive mixtures of Mat\'ern kernels in single-output Gaussian process (GP) models and explore the properties of multiplicative mixtures of Mat\'ern kernels for multi-output GP models. For the single-output case, we derive a series of theoretical results showing that the smoothness of a mixture of Mat\'ern kernels is determined by the least smooth component and that a GP with such a kernel is effectively equivalent to the least smooth kernel component. Furthermore, we demonstrate that none of the mixing weights or parameters within individual kernel components are identifiable. We then turn our attention to multi-output GP models and analyze the identifiability of the covariance matrix $A$ in the multiplicative kernel $K(x,y) = AK_0(x,y)$, where $K_0$ is a standard single output kernel such as Mat\'ern. We show that $A$ is identifiable up to a multiplicative constant, suggesting that multiplicative mixtures are well suited for multi-output tasks. Our findings are supported by extensive simulations and real applications for both single- and multi-output settings. This work provides insight into kernel selection and interpretation for GP models, emphasizing the importance of choosing appropriate kernel structures for different tasks.

On the Identifiability and Interpretability of Gaussian Process Models

Abstract

In this paper, we critically examine the prevalent practice of using additive mixtures of Matérn kernels in single-output Gaussian process (GP) models and explore the properties of multiplicative mixtures of Matérn kernels for multi-output GP models. For the single-output case, we derive a series of theoretical results showing that the smoothness of a mixture of Matérn kernels is determined by the least smooth component and that a GP with such a kernel is effectively equivalent to the least smooth kernel component. Furthermore, we demonstrate that none of the mixing weights or parameters within individual kernel components are identifiable. We then turn our attention to multi-output GP models and analyze the identifiability of the covariance matrix A in the multiplicative kernel K ( x, y ) = AK 0 ( x, y ) , where K 0 is a standard single output kernel such as Matérn. We show that A is identifiable up to a multiplicative constant, suggesting that multiplicative mixtures are well suited for multi-output tasks. Our findings are supported by extensive simulations and real applications for both single- and multi-output settings. This work provides insight into kernel selection and interpretation for GP models, emphasizing the importance of choosing appropriate kernel structures for different tasks.

1 Introduction

Gaussian processes (GPs) have emerged as a powerful and popular tool in machine learning, spatial statistics, functional data analysis, etc., due to their versatility, flexibility, and interpretability as a nonparametric method (Rasmussen and Williams, 2006; Banerjee et al., 2014). GPs provide an intuitive means of modeling uncertainty, allowing the generation of predictive distributions for unseen data points without requiring explicit model specification. Consequently, GPs have found applications in a variety of contexts.

The key to harnessing the power of GPs lies in the choice of kernel functions, also known as covariance functions. Rather than specifying a model, the GP framework revolves around selecting an appropriate kernel, transforming the problem into one of parameter estimation. Over time, researchers have developed various kernels, each tailored to specific scenarios, such as spatial data, time series data, and others (Genton, 2001). For instance, for spatiotemporal, a variety of kernels have been developed to capture unique characteristics and patterns (Gneiting, 2002; Stein, 2005).

The Radial Basis Function (RBF) kernel and Matérn kernels serve as prime examples of the diversity within the kernel family (Cressie and Wikle, 2015). The RBF kernel, renowned for its infinite differentiability, yields smooth functions, while Matérn kernels control the degree of smoothness by a kernel parameter, thereby accommodating both smooth and nonsmooth functions (Stein (1999)).

With each kernel bearing its own set of advantages and disadvantages, kernel selection, as a nontrivial task, often demands extensive domain knowledge.

Inspired by the potential of harnessing the strengths of multiple kernels concurrently, researchers have ventured into methods that involve kernel combinations, such as spectral mixtures (Wilson and Adams, 2013; Samo and Roberts, 2015; Remes et al., 2017), addition and/or multiplication of kernels (Duvenaud et al., 2011) such as mixture of RBF (Duvenaud et al., 2013), mixture of RBF, periodic (Per), linear (Lin), and rational quadratic (RQ) (Kronberger and Kommenda, 2013), mixture of RBF, RQ, Matérn and Per (Cheng et al., 2019; Verma and Engelhardt, 2020), mixture of RBF, Matérn 1/2, 3/2, 5/2, as well as more sophisticated methods like Neural Kernel Networks (NKN,Sun et al. (2018) and Automatic Bayesian Covariance Discovery (ABCD, Lloyd et al. (2014). For instance, within the realm of spectral mixtures, Remes et al. (2017) proposed the Generalized Spectral Mixture (GSM) kernel, which is a product of three components all parametrized by GPs, representing frequencies,length-scales and mixture weights. In the category of summation and/or multiplication, Verma and Engelhardt (2020) utilized the sum of the RBF kernel and Matérn kernel with varying smoothness as the final kernel in their t-distribution Gaussian process latent variable model (tGPLVM), an extension to GPLVM (Lawrence, 2003; Lalchand et al., 2022), to characterize the latent features in single-cell RNA sequencing data.

In addition, methodological advances have facilitated the efficient discovery of optimal kernel mixtures. For example, NKN is reported to be more efficient than the Automatic Statistician, a gradient-based method. Simpson et al. (2021) utilized a transformer-based framework to generate mixture kernel recommendations. These studies underscore the advantages of employing mixed kernels in GP modeling, showcasing improved model fitting and more accurate predictions. By combining multiple kernels, the strengths of individual kernels can be leveraged to capture complex data patterns and relationships, potentially exceeding the capabilities of single kernels. Furthermore, the automatic selection of kernels optimizes model performance, reducing the reliance on extensive prior knowledge.

Another essential benefit of using mixture kernels in GP modeling lies in the improved interpretability. In contrast to complex, data-driven kernels, mixture kernels allow the decomposition of intricate patterns into simpler, distinct base kernels that are more readily interpretable. This technique, often termed decomposition, enables a more comprehensive understanding of the underlying data structure by simplifying complex patterns into their constitutive components. An illustrative example of this approach is the work of Duvenaud et al. (2013), who applied this decomposition technique for structure discovery in time series data. Their proposed mixture kernel comprised several base kernels, including RBF, periodic, linear, and rational quadratic kernels, enabling the dissection of time-series data patterns into components such as long-term trends, annual periodicity, and mediumterm deviations. In a similar vein, the ABCD approach also employed decomposition, albeit with a different set of base kernels.

The identifiability of parameters within a single Matérn kernel has previously been explored, with the microergodic parameter uniquely identified as the only identifiable parameter, as outlined in Stein (1999). Tang et al. (2022) examined the identifiablity of parameters within a single Matérn kernel with nuggets. Despite the prevalent use and interpretation of mixture kernels, their theoretical properties including identifiability and interpretability of parameters within kernel components, to the best of our knowledge, remain underexplored. A related critical question pertains to the common practice of using kernels with varying degrees of smoothness for enhanced flexibility. In this work, we turn our attention to the additive kernel in univariate GPs and multiplicative (separable) kernel in multivariate GPs, specifically focusing on the mixture of the widely-used Matérn kernel. We highlight the following novel findings:

The smoothness of an additive mixture kernel is completely determined by the least smooth component.

For additive mixture of Matérn kernels, the identifiability is confined only to a single parameter, also known as the microergodic parameter that is associated with the least smooth kernel component.

For multivariate GPs with multiplicative separable kernels, the multiplicative matrix that controls the correlation structure among the response variables is identifiable up to a multiplicative constant.

Our conclusions extend beyond the specific case of Matérn kernels, demonstrating applicability to a wider range of mixture kernels.

Our study aims to deepen the understanding of mixture kernel identifiability and interpretability, as well as to clarify the practical benefits of using kernels with varying degrees of smoothness. Our theoretical assertions are supported by both simulations and real-world applications. Details regarding the proofs of our theories and numerical experiments can be found in the supplementary material.

2 Gaussian process, kernels and parameter inference

This section serves to define key terms and outline the parameter inference algorithm integral to the forthcoming theory and simulation sections.

A GP is a random function where any finite set of its realizations follows a multivariate normal distribution, characterized by a mean function µ and a covariance function K (Rasmussen and Williams, 2006).

Definition 1. f is said to follow a Gaussian process in domain Ω with a mean function µ : Ω → R and a covariance function K : Ω × Ω → R if for any x 1 , . . . , x n ∈ Ω ,

For the purpose of this study, we assume µ = 0 , adhering to common practice and for the sake of simplicity. If the mean is not zero, a data transformation can be applied to achieve this. This approach is often adopted in machine learning and statistical analysis to simplify calculations and comply with certain algorithmic requirements (Rasmussen and Williams, 2006; Murphy, 2012).

As a consequence, the behavior of a GP is primarily determined by its kernel function. In this study, we assume the domain Ω = R p and our focus is on the selection of kernels that are widely recognized and commonly used in machine learning.

Definition 2. The RBF kernel, also known as the squared exponential kernel of Gaussian kernel is given by:

The parameter σ 2 is called the spatial variance or partial sill that controls the point-wise variance, while α is called the (length) scale, range, or decay that controls the spatial dependency. A key characteristic of RBF is its smoothness, defined as follows:

Definition 3. A Gaussian process f is said to be mean-square continuous (MSC) if E ( f ( x + h ) -f ( x )) 2 h → 0 - - - → 0 for any x ∈ R p . f is said to be mean-square differentiable (MSD) if lim h → 0 f ( x + h ) -f ( x ) h exists in the mean-square topology, and the limiting process is called the derivative process of f , denoted by f ′ . Similarly, the d -times mean-square differentiable ( d -MSD) GPs can be defined inductively.

In particular, the above-defined RBF kernel is infinitely differentiable. However, oversmoothing can be problematic in prediction (Stein, 1999), leading to the popularity of the following flexible family of kernels that allow for varying smoothness:

Definition 4. The Matérn kernel is given by

where K ν is the modified Bessel function of the second kind. When ν = 1 / 2 , the Matérn kernel becomes the exponential kernel:

The additional parameter ν is called the smoothness, since the smoothness of a Matérn GP is exactly ⌈ ν ⌉ -1 . Inferring ν is known to be a particularly challenging problem, both from the theoretical and empirical perspectives (Zhang, 2004). In practice, ν is usually set to 1 / 2 , 3 / 2 , 5 / 2 , · · · due to the simplified analytic form of the modified Bessel function.

The most straightforward estimator of the parameters σ 2 and α is the maximum likelihood estimator (MLE). In practice, optimizers such as Adam (Kingma and Ba, 2015) or stochastic gradient descent (SGD) are used to maximize the log-likelihood.

3 Univariate GP: additive kernels

In this section, we investigate the smoothness of mixture kernels and the identifiability of parameters in the additive mixture of Matérn kernels in the context of a univariate response variable. We note here that all the theorems presented are framed in an asymptotic context.

Consider K 1 , · · · , K L as L kernels with d l -MSD. Define K = ∑ L l =1 w l K l where ∑ L l =1 w l = 1 and w l ≥ 0 .

Theorem 1. K is d := min l { d l } -times MSD, but not d +1 -times MSD, so the smoothness of K is determined by the least smooth component.

Beyond smoothness, we discuss the identifiability of the parameters in the mixture kernel, which relies on the notion of equivalence of measures.

Definition 5. Two measures P 1 and P 2 are said to be equivalent, i.e., P 1 ≡ P 2 , if they are absolutely continuous with respect to each other, i.e., P 1 ( B ) = 0 ⇐⇒ P 2 ( B ) = 0 . Two GPs are equivalent if the corresponding Gaussian random measures are equivalent.

̸

As a consequence, two equivalent GPs cannot be distinguished by any finite number of realizations (Stein, 1999). Specifically, given a family of GPs parametrized by θ , if P θ 1 ⇐⇒ P θ 2 with θ 1 = θ 2 , then θ is not identifiable since we cannot distinguish between θ 1 and θ 2 . As a corollary, there does not exist any consistent estimator for θ .

Now we can study the identifiability of the mixture Matérn kernel K = ∑ L l =1 w l K l , where K l is the Matérn kernel with parameters ( σ 2 l , α l , ν l ) , given known ν l 's in ascending order ν 1 < ν 2 · · · < ν L with ν l -ν l -1 ≥ 1 for different smoothness.

Theorem 2. Let K = ∑ l w l K l and ˜ K = ∑ l ˜ w l ˜ K l be two kernels represented as linear combinations of Matérn kernels K l 's and ˜ K l with parameters ( σ 2 l , α l , ν l ) and ( ˜ σ 2 l , ˜ α l , ν l ) .

(i) When p = 1 , 2 , 3 , Then K ≡ ˜ K if w 1 σ 2 1 α 2 ν 1 1 = ˜ w 1 ˜ σ 2 1 ˜ α 2 ν 1 1 .

(ii) When p ≥ 5 , Then K ≡ K if

Consequently, no single parameter is identifiable. The only identifiable parameter when p ≤ 3 is w 1 σ 2 1 α 2 ν 1 1 , known as the microergodic parameter (Stein, 1999).

Interestingly, the case of p = 4 is missing here. We hypothesize that it aligns with the case of p ≥ 5 . However, providing a proof remains challenging. It is also worth noting that the issue of identifiability persists even for a single Matérn kernel, as highlighted in the existing literature (Zhang, 2004; Anderes, 2010; Li et al., 2023).

Corollary 1. The mixture kernel K is equivalent to K 1 , that is, the mixture kernel is equivalent to its least smooth component. Furthermore, the mean squared error (MSE) of the mixture kernel K is asymptotically equal to the MSE of K 1 .

Theorem 2 implies that interpreting w l as the weight of each component is often misleading due to its nonidentifiability. Furthermore, Corollary 1 suggests the use of a single Matérn kernel in lieu of the linear mixture kernel. Both assertions are supported by the simulation study in Section 5 and the real-world application in Section 6.

An alternative strategy is to assume the same smoothness across mixing components, i.e., ν 1 = ν 2 , · · · , ν L . In this case, we prove that no single parameter is identifiable. The complete theorem, proof, and simulation are in the Supplement.

When considering real-world data, it is often the case that the observed outcomes are noisy, which is frequently modeled as an i.i.d. Gaussian with variance denoted by τ 2 , commonly referred as the "nugget". The inclusion of such a noise term is essential to capture the inherent variability in the data. Equivalently, the model can be formulated as noiseless by adjusting the kernel to be K ( x, x ′ ) + τ 2 1 { x = x ′ } . The following corollary shows that the noise, or nugget, does not impact the identifiability and interpretability of the mixture of Matérn kernels as shown in Theorem 2.

̸

Corollary 2. Let τ 2 and ˜ τ 2 be the noise variance of K and ˜ K . If τ 2 = ˜ τ 2 , then K ̸≡ ˜ K . If τ 2 = ˜ τ 2 , the previous results in Theorem 2 hold.

In essence, while the presence of noise or "nugget" captures real-world data variability, it does not obscure or alter the fundamental characteristics of the mixture of Matérn kernels as established in our theorems.

4 Multivariate GP: multiplicative kernels

In this section, we shift our focus to multivariate GPs, also known as multi-output or multi-task GPs, which are applicable to datasets with multiple response variables.

Definition 6. f : Ω → R m is said to follow a m -variate GP in domain Ω with zero mean and cross-covariance function K : Ω × Ω → PD( m ) , where PD( m ) is the space of all m by m positive definite matrices, if for any x 1 , · · · , x n :

where MN represents matrix Gaussian distribution and Σ consists of m × m blocks Σ ij = K ( x i , x j ) .

The construction of valid cross-covariance kernels is a pivotal yet challenging aspect of multivariate GPs (Gneiting, 2002; Apanasovich and Genton, 2010; Genton and Kleiber, 2015). A popular kernel that is widely used due to its simplicity and seemingly interpretability is the multiplicative kernel, also known as a separable kernel. This kernel admits the form K ( x, y ) = AK 0 ( x, y ) where K 0 is a standard kernel for univariate GP like Matérn while A ∈ PD( m ) is a positive definite matrix reflecting the correlation between response variables. The following theorem investigates the identifiability of the multiplicative cross-covariance kernel.

Theorem 3. (i) When K 0 is Matérn with parameters ( σ 2 , α, ν ) where ν is given, then K ≡ ˜ K if σ 2 α 2 ν A = ˜ σ 2 ˜ α 2 ν ˜ A . That is, the identifiable parameter is σ 2 α 2 ν A . Hence A is identifiable up to a multiplicative constant and the correlation structure is identifiable.

(ii) Let K 0 be an arbitrary kernel with spectral density ρ 0 satisfying:

where W [ -b,b ] d is the space of Fourier transforms of L 2 ( R p ) functions with compact support [ -b, b ] d (see the Supplement for more details). If θ is a mircroergodic parameter of K 0 , such that ∫ R p 1 γ 4 ( ω ) ( ρ ( ω ) θ -˜ ρ ( ω ) ˜ θ ) 2 d ω < 0 if θ = ˜ θ , then K ≡ ˜ K if θA = ˜ θ ˜ A . That is, the identifiable parameter is θA , hence A is identifiable up to a multiplicative constant and the correlation structure is identifiable.

Note that (i) is a special case of (ii), as the Matérn kernel satisfies the additional assumptions in (ii). This theorem positively supports the utilization of the multiplicative kernel and the interpretation that A ij measures the correlation between the i -th response variable and the j -th response variable.

5 Simulation

In this section, we will validate the proposed theories by conducting three simulation studies. The first simulation demonstrates Theorem 1 using a mixture of Matérn kernels, highlighting that the smoothness of the sample path (e.g., a realization of the GP) is determined by the least smooth component. The second simulation supports Theorem 2 using a mixture kernel of Matérn kernels with varying smoothness. It shows that none of the single parameters are identifiable, and only the proposed microergodic parameter can be identified. Finally, the third simulation showcases Theorem 3, suggesting that the multiplicative matrix is identifiable up to a multiplicative constant.

5.1 Simulation 1 - Univariate GP: Smoothness

In the first simulation, we show that the smoothness of the sample path is determined by the smoothness of the least smooth kernel. We generate 200 samples as an equidistant sequence ranging

from 0 -10 . We consider the following mixture kernel: K = ∑ 3 l =1 w l K l , where K l is the Matérn kernel with smoothness parameter ν l being 2 l -1 2 and ∑ L l =1 w l = 1 , w l ≥ 0 . We randomly sample Y from multivariate normal distribution with kernel function as the mixture kernel, Matérn with ν = 1 / 2 , 3 / 2 , 5 / 2 . The resulting sample paths are summarized in Figure 1.

A

Mixturekernel

Matérn 3/2

2.5

0.0

2.5

-5.0

4

0.0

5.0

7.510.0

0.02.55.0

7.510.0

0:0

2.5

5.0

7.510.0

0.0

2.5

5.0

7.510.0

X

X

X

X

B

Matérn 1/2+3/2

Matérn 3/2

2.0 -

1.5

1.0

B

- S'0

0:0

200

150

100-

50

0

25

50

75

100

125 0

25

50

75

100

125 0

50

75

100

125

Our results clearly indicate that the Matérn kernel with ν = 1 / 2 exhibits the least degree of smoothness (continuous but not differentiable), and the smoothness increases with ν . The smoothness of the mixture kernel is predominantly influenced by the Matérn kernel with ν = 1 / 2 . When comparing the sample path of the mixture kernel to those of the Matérn kernels with ν = 3 / 2 and ν = 5 / 2 , it is evident that the mixture kernel demonstrates a degree of smoothness similar to the Matérn kernels with ν = 1 / 2 .

To better demonstrate the smoothness of the mixture kernel and its least component, we further examine the continuity and MSD of the mixture kernel and its mixing components empirically. For a fixed x 0 = 0 , let x i = 1 /i for i = 1 , 2 , . . . , and we generate y l i from the GP, where l = 1 , . . . , T denotes the index of replicates. This allows us to approximate lim x → 0 E ( f ( x ) -f (0)) 2 by β i = 1 T ∑ T l =1 ( y l i -y l 0 ) 2 . As per Definition 3, the GP is continuous if and only if β i → 0 . Similarly, we can approximate lim x → 0 E ( f ( x ) -f (0) x ) 2 by γ i = 1 T ∑ T l =1 ( y l i -y l 0 x i ) 2 . The GP is mean-square differentiable if and only if lim i →∞ γ i exists. Specifically, for the mixture of Matérn 1 / 2 and 3 / 2 and Matérn 1 / 2 , while β i → 0 , γ i does not converge (Figure 1 first two columns), which indicates that both the mixture kernel and Matérn 1 / 2 are continuous but not differentiable. However, for Matérn 3 / 2 (Figure 1 third column), β i → 0 and γ i converges, implying that Matérn 3 / 2 is continuous and differentiable.

These empirical observations strongly support the claims made in Theorem 1, which suggests that the inclusion of smoother kernel components in the mixture does not inherently enhance the overall smoothness of the mixture kernel.

5.2 Simulation 2 - Univariate GP: Components with different smoothness

In the second simulation, our aim is to assess parameter identifiability in a GP with a mixture kernel consisting of Matérn kernels with distinct smoothness. The mixture kernel is denoted as K = ∑ 3 l =1 w l K l , where K l is a Matérn kernel with parameters ( σ 2 l , α l , ν l ) . For this simulation, we set ν 1 = 1 / 2 , ν 2 = 3 / 2 , ν 3 = 5 / 2 . Theorem 2 indicates that the only identifiable parameter is

w l σ 2 l α l , while all other parameters remain unidentifiable. In this scenario, the identifiable parameter, also known as the microergodic parameter, is associated with the least smooth component, i.e., the Matérn 1 / 2 kernel. We will assess parameter inference by comparing estimated values to their true values across different training sample sizes. For this experiment, we generate n samples, ranging from 20 , 50 , 100 , 500 . For each sample size, we replicate the simulation 100 times.

A

W002W

B

C

10.0

0.6

FOE

7.5

·

0.4

20

5.0

0.2

10 -

2.5

0

0.0

一

F0°0

20

50

100

500

20

50

100

500

20

50

100

500

Sample size

Sample size

Sample size

D

W1

E

α1

F

6

1.00

15

·

16

1

0.75

!

12

10 -

·

·

.

0.50

-8

·

5

0.25

00°0

20

50

100

500

20

50

100

500

20

50

100

500

Sample size

Sample size

Sample size

The results provide persuasive support for Theorem 2 that, in the context of additive mixture of Matérn kernels with distinct smoothness, only the MLE of the microergodic parameter w 1 σ 2 1 α 1 for the least smooth component converges to the true value. The MLEs for all other parameters do not converge to their respective true values, highlighting the unique identifiability of the parameters in the least smooth component within the additive mixture kernel framework. Such results are robust in terms of optimizer choice and consistent as the sample size increases to 3000 (details in the Supplement).

5.3 Simulation 3 - Multivariate GP

Next, we turn our attention to multivariate GPs with a separable kernel. We define the separable kernel as K = AK 0 . Here, we select Matérn 1 / 2 as K 0 , characterized by the parameters ( σ 2 , α ) . Our Theorem 3 suggests that only Aα 2 ν σ 2 is identifiable. To verify this, we adopt a bivariate setup where A is a 2 × 2 positive definite matrix. In this simulation, the sample size n varies from 50 , 100 , 200 , 400 . For each sample size, we replicate the simulation 100 times.

The parameter estimates are summarized in Figure 3. Evidently, only the MLE of the microergodic parameter Aσ 2 α 2 ν converges to its true value. These findings reinforce the identifiability and interpretability of the separable kernel. This understanding is critical for the interpretation and application of separable kernels in real-world situations.

6 Application

Motivated by the findings in our theoretical and simulation studies, we proceed to compare the performance of the mixture kernel with varying smoothness and the least smooth component within that mixture only, in real-world data applications. In this section, we will revisit several applications in previous studies (Rasmussen and Williams (2006), Wilson et al. (2014), Sun et al. (2018)), illustrating that the mixture kernel achieves prediction performance comparable to the least smooth kernel component within the mixture.

A

A1.102α2v

B

A1.202α2v

C

A2.202α2w

D

α

40

100-

30

100

75

20

10

50

50

25

50

Sample size

100

200

400

-10 -

50

Sample size

100

200

400

50

Sample size

100

200

400

50

Sample size

100

200

400

E

A1.1

F

A1.2

A2.2

H

2

：

30

10

30

10

8

：

20 -

20

10 -

10-

50

100200

400

50

100200

400

50

100200

400

50

100

200

400

Sample size

Sample size

Sample size

Sample size

6.1 Application 1 - Image prediction

For our first real-data application, we delve into the domain of image analysis. GP has been widely used to predict missing image (Wilson et al. (2014), Sun et al. (2018)). This task of image prediction is an exemplary context for assessing kernel performance in discerning local correlation patterns, providing a visually intuitive framework for comparative analysis.

We employ a handwritten zero image from the MNIST dataset (LeCun et al. (1998)) for this analysis; we extract a 20 × 20 section from the center of the original 100 × 100 image to serve as our test image. This results in a training dataset of 9600 pixels and a testing dataset with 400 pixels (Figure 4A). The input for this analysis comprises the 2-D pixel locations, with the corresponding pixel intensities serving as the response. To reconstruct the missing area, we train GP regression models with both the mixture kernel with varying smoothness and a single Matérn 1 / 2 kernel on the training dataset, and subsequently make predictions on the testing dataset. Specifically, we employ a mixture of three Matérn kernels of smoothness 1 / 2 , 3 / 2 , 5 / 2 . Our analysis seeks to demonstrate the similarities in performance between the mixture kernel and the least smooth component within that mixture, i.e., Matérn 1 / 2 , in terms of prediction in the missing image area.

ATrainingImage

BMaternmixture

C

The predictions generated by both the mixture kernel and the Matérn 1/2 kernel demonstrate a remarkable similarity, as shown in Figure 4. This observation suggests that the mixture kernel and a single Matérn 1 / 2 kernel have comparable performance. The close alignment between their predictions implies that using the mixture kernel does not yield significant advantages over the least smooth kernel component in this specific application, further reinforcing our theoretical findings (Theorem 2) regarding the dominance of the least smooth kernel component in the mixture kernel.

6.2 Application 2 - Manua Loa CO 2

In this section, we extend our comparative analysis to the widely recognized Moana Loa CO 2 dataset from the Global Monitoring Laboratory's Repository as our benchmark for regression analysis (Tans and Keeling (2023)). Rasmussen and Williams (2006) employed this dataset to demonstrate that constructing a complex covariance function by combining several different types of simple covariance functions could yield an excellent fit to complex data. Consequently, we consider it to be a suitable choice for testing our theorem.

Specifically, we used the decimal date from 1960 to 2020 as our input and the monthly average CO 2 as our response. In this context, we compare the performance of Matérn mixture 1 / 2 + 3 / 2 , 1 / 2 + 3 / 2 + 5 / 2 , 3 / 2 + 5 / 2 and three single Matérn kernels ( 1 / 2 , 3 / 2 , 5 / 2 ). The dataset consists of 720 records. We undertook an extensive assessment, varying the training sample size from 5% to 95% and keeping the test sample size as the remaining data. For each training sample size, we conducted 10 simulations using different random splits of the dataset as replications. This strategy allows for a robust and thorough evaluation of the prediction accuracy of both the mixture kernel and the single kernel component in the context of a regression problem with varying training sample sizes. We use the MSE to quantify the prediction accuracy (Figure 5).

0.2

0.5

0.75

12.5

10.0 -

Matern

7.5

1/2+3/2+5/2

MSE

1/2

1/2+3/2

3/2

5.0

3/2+5/2

5/2

2.5

0.0

1/2 +3/2+5/2 1/2+3/2

1/2

3/2

3/2 +5/2

5/2

/E+/ /S+/+/

1/2

3/2

3/2 + 5/2

5/2

/E+/ /S+/+/

1/2

3/2

3/2 +5/2

5/2

The Matérn mixture kernel 1 / 2 + 3 / 2 + 5 / 2 aligned closely with 1 / 2 + 3 / 2 across all experiments. Furthermore, these three Matérn mixture kernels ( 1 / 2 + 3 / 2 + 5 / 2 , 1 / 2 + 3 / 2 , 1 / 2 ) converge when the training sample size ratio reaches 50% . This empirical finding supports the theoretical equivalence of these three GPs as posited in Theorem 2. Similarly, the Matérn mixture 3 / 2 + 5 / 2 kernel and the standalone Matérn 3 / 2 kernel converge from a 65% sample size, lending empirical weight to their theoretical equivalence and their equivalence in MSE as stated in Corollary 1.

Matérn 3 / 2 kernel has exceptional performance across all experiments. However, its influential performance is diluted when coupled with the less smooth component, Matérn 1 / 2 . On the contrary, when mixed with the smoother Matérn 5 / 2 component, the worse performance of the latter is overshadowed by Matérn 3 / 2 . These behaviors lend empirical weight to our Theorem 2, suggesting that in the asymptotic sense, the mixture kernel is dominated by its least smooth component.

A noteworthy trend is the inequivalance of the Matérn mixture kernel and its least component with a limited training sample (training sample size < 50% ). We consider such scenario as the "finite" sample scenario in contrast to our focus on the infinite sample scenario, i.e. asymptotic theory. Theoretically, while the Matérn mixture kernels 1 / 2 + 3 / 2 + 5 / 2 , 1 / 2 + 3 / 2 , and the standalone Matérn 1 / 2 kernel are deemed equivalent, and the Matérn mixture kernel 3 / 2 + 5 / 2 is akin to the Matérn 3 / 2 kernel, their practical agreement points, i.e. 50% and 65% , differed with limited training samples. Although securing a conclusive theoretical support for the finite sample regime is challenging, we managed to provide some insights to interpret our findings. To do so, we have to delve further into the proof of Stein (1993)'s Theorem 1, which guides us to Theorem 3.1 in Stein (1990). The proof suggests that the relative difference between the MSEs rests on the tail of the series. This difference is influenced by both the sample size (denoted as N in Stein (1990)) and by b jk and µ j , as defined on the same page. For a fixed sample size, smaller values of ( b jk + µ j µ k ) 2 result in a

smaller relative difference between MSEs. By the definition of b jk and µ j , the more "different" the two kernels are, in other words, the more "significant" the additional smoother components become -the greater the difference between the MSEs will be.

7 Discussion and future work

In this study, we delved into the identifiability and interpretability of parameters in GPs with various mixture kernels in the asymptotic scenario and fixed domain, including the additive kernel and separable kernels. We formulated a series of theorems clarifying the identifiable parameters in these kernel structures, and further corroborated our theorems through multiple simulations and real-data applications. Our simulation results convincingly demonstrated that in GPs with mixture kernels, the only identifiable parameter, known as the microergodic parameter, is associated with the least smooth kernel component. This discovery has profound implications for parameter interpretation in the context of GPs with mixture kernels. Empirical evidence from image data analysis and Manua Loa CO 2 regression studies further reinforce these discoveries. Despite the inclusion of kernels with varying smoothness in the mixture kernel, the performance of the mixture kernel closely paralleled that of the least smooth kernel component within it when the sample size is large enough, regardless of its performance. This observation suggests that the inclusion of kernels with different smoothness does not necessarily improve the prediction accuracy. In fact, due to various real world factors including limited training samples, optimization and more, determining a clear winner in performance between Matérn mixture kernel and single Matérn kernel proves to be a challenging task. Lastly, in the case of multivariate GPs with separable kernels, our theoretical and simulation results show that the correlation structure is identifiable up to a multiplicative constant. This result underlines that the interpretability of separable kernels mainly resides in the relative correlation structure, rather than individual parameters.

Although our study has provided substantial insight on the identifiability and interpretability of parameters in GPs with various mixture kernel types, it also opens several exciting avenues for future research. First, our analysis primarily focuses on Matérn kernels, and it would be intriguing to extend this framework to other families of kernels, such as the periodic kernel. Such an extension would provide a more comprehensive understanding of parameter identifiability and predictive performance across a broader spectrum of kernel types. Second, our work has so far considered the cases where p ≤ 3 or p ≥ 5 . However, extending this to p = 4 presents a substantial challenge due to the lack of mathematical tools to determine whether two Gaussian random measures are equivalent or not. Third, our observations in the Mauna Loa CO 2 example underscore the need for further exploration in the finite sample scenario. Potential avenues for future research include investigating the convergence rates of both the Matérn mixture kernel and the single Matérn kernel. Lastly, while we have identified the microergodic parameters that are theoretically identifiable in mixture kernels, an important direction for future work involves finding consistent estimators for these kernel parameters. This would entail developing novel estimation techniques or adapting existing ones to reliably estimate the parameters in practice, thereby enhancing the practical utility of our theoretical findings. These endeavors will not only extend the theoretical foundations of GPs with mixture kernels, but will also broaden their applicability across various real-world scenarios.

Acknowledgments and Disclosure of Funding

We express our sincere gratitude to anonymous reviewers and the Area Chair for their insightful comments and constructive feedback, which significantly enhanced the quality and clarity of this work. DL was supported by the NIH/NCATS award UL1 TR002489, NIH/NHLBI award R01 HL149683 and NIH/NIEHS award P30 ES010126. YL was partially supported by NIH awards R01AG079291 and U01HG011720.

References

Anderes, E. (2010). On the consistent separation of scale and variance for Gaussian random fields. The Annals of Statistics .

Apanasovich, T. V. and M. G. Genton (2010). Cross-covariance functions for multivariate random fields based on latent dimensions. Biometrika 97 (1), 15-30.

Banerjee, S., B. P. Carlin, and A. E. Gelfand (2014). Hierarchical modeling and analysis for spatial data . CRC press.

Cheng, L., S. Ramchandran, T. Vatanen, N. Lietzén, R. Lahesmaa, A. Vehtari, and H. Lähdesmäki (2019). An additive Gaussian process regression model for interpretable non-parametric analysis of longitudinal data. Nature Communications 10 (1), 1798.

Cressie, N. and C. K. Wikle (2015). Statistics for spatio-temporal data . John Wiley & Sons.

Duvenaud, D., J. Lloyd, R. Grosse, J. Tenenbaum, and G. Zoubin (2013, 17-19 Jun). Structure discovery in nonparametric regression through compositional kernel search. In S. Dasgupta and D. McAllester (Eds.), Proceedings of the 30th International Conference on Machine Learning , Volume 28 of Proceedings of Machine Learning Research , Atlanta, Georgia, USA, pp. 1166-1174. PMLR.

Duvenaud, D. K., H. Nickisch, and C. Rasmussen (2011). Additive Gaussian processes. In J. ShaweTaylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger (Eds.), Advances in Neural Information Processing Systems , Volume 24. Curran Associates, Inc.

Genton, M. G. (2001). Classes of kernels for machine learning: a statistics perspective. Journal of machine learning research 2 (Dec), 299-312.

Genton, M. G. and W. Kleiber (2015). Cross-Covariance Functions for Multivariate Geostatistics. Statistical Science 30 (2), 147 - 163.

Gneiting, T. (2002). Nonseparable, stationary covariance functions for space-time data. Journal of the American Statistical Association 97 (458), 590-600.

Kingma, D. P. and J. Ba (2015). Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings .

Kronberger, G. and M. Kommenda (2013). Evolution of covariance functions for Gaussian process regression using genetic programming.

Lalchand, V., A. Ravuri, and N. D. Lawrence (2022). Generalised gplvm with stochastic variational inference. In International Conference on Artificial Intelligence and Statistics , pp. 7841-7864. PMLR.

Lawrence, N. (2003). Gaussian process latent variable models for visualisation of high dimensional data. In S. Thrun, L. Saul, and B. Schölkopf (Eds.), Advances in Neural Information Processing Systems , Volume 16. MIT Press.

LeCun, Y., C. Cortes, and C. J. Burges (1998). The mnist database of handwritten digits. http: //yann.lecun.com/exdb/mnist/ . Accessed: 2023-08-01.

Li, D., W. Tang, and S. Banerjee (2023). Inference for gaussian processes with matérn covariogram on compact riemannian manifolds. Journal of Machine Learning Research 24 (101), 1-26.

Lloyd, J. R., D. Duvenaud, R. Grosse, J. B. Tenenbaum, and Z. Ghahramani (2014). Automatic construction and natural-language description of nonparametric regression models.

Murphy, K. P. (2012). Machine learning: a probabilistic perspective . MIT press.

Rasmussen, C. E. and C. K. I. Williams (2006). Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press.

Remes, S., M. Heinonen, and S. Kaski (2017). Non-stationary spectral kernels. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Advances in Neural Information Processing Systems , Volume 30. Curran Associates, Inc.

Samo, Y.-L. K. and S. Roberts (2015). Generalized spectral kernels.

Simpson, F., I. Davies, V. Lalchand, A. Vullo, N. Durrande, and C. E. Rasmussen (2021). Kernel identification through transformers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems , V olume 34, pp. 10483-10495. Curran Associates, Inc.

Stein, M. (1990). Uniform asymptotic optimality of linear predictions of a random field using an incorrect second-order structure. The Annals of Statistics 18 (2), 850-872.

Stein, M. L. (1993). A simple condition for asymptotic optimality of linear predictions of random fields. Statistics & Probability Letters 17 (5), 399-404.

Stein, M. L. (1999). Interpolation of spatial data: some theory for kriging . Springer Science & Business Media.

Stein, M. L. (2005). Space-time covariance functions. Journal of the American Statistical Association 100 (469), 310-321.

Sun, S., G. Zhang, C. Wang, W. Zeng, J. Li, and R. Grosse (2018, 10-15 Jul). Differentiable compositional kernel learning for Gaussian processes. In J. Dy and A. Krause (Eds.), Proceedings of the 35th International Conference on Machine Learning , Volume 80 of Proceedings of Machine Learning Research , pp. 4828-4837. PMLR.

Tang, W., L. Zhang, and S. Banerjee (2022). On identifiability and consistency of the nugget in gaussian spatial process models. Journal of the Royal Statistical Society Series B: Statistical Methodology 83 (5), 1044-1070.

Tans, P. and R. Keeling (2023). Trends in atmospheric carbon dioxide. https://gml.noaa.gov/ ccgg/trends/data.html . Accessed: 2023-08-01.

Verma, A. and B. E. Engelhardt (2020). A robust nonlinear low-dimensional manifold for single cell RNA-seq data. BMC Bioinformatics 21 (1), 324.

Wilson, A. and R. Adams (2013, 17-19 Jun). Gaussian process kernels for pattern discovery and extrapolation. In S. Dasgupta and D. McAllester (Eds.), Proceedings of the 30th International Conference on Machine Learning , Volume 28 of Proceedings of Machine Learning Research , Atlanta, Georgia, USA, pp. 1067-1075. PMLR.

Wilson, A. G., E. Gilboa, A. Nehorai, and J. P. Cunningham (2014). Fast kernel learning for multidimensional pattern extrapolation. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger (Eds.), Advances in Neural Information Processing Systems , Volume 27. Curran Associates, Inc.

Zhang, H. (2004). Inconsistent estimation and asymptotically equal interpolations in model-based geostatistics. Journal of the American Statistical Association 99 (465), 250-261.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors carefully examine additive mixture of Matern kernels for Gaussian processes. For the single-output case, the authors show that a mixture of Matern kernels is equivalent to the least smooth component of the mixture (Theorem 2). Consequently, \"there is no advantage in including any other component apart from the least smooth component\" (Corollary 1). For the multi-output case, the authors leverage results from Bachoc et al. (2022) to identify microergodic parameters of separable multi-output kernels. These theoretical results are accompanied by demonstrations on simulated data. In addition, in three experiments on real-world data, the authors provide evidence that a mixture Matern kernels performs as well as just the least smooth components.\n\nSTRENGTHS:\nI would like to thank the authors for their submission. I enjoyed reading the submission and certainly learned few new things.\n\n## Strengths\n\n* The paper is very well written and overall the presentation is good. I found hardly any typos.\n\n* Additive mixtures of kernels are very important in GP modelling, since it allows the practitioner to model multiple length scales in the data at various levels of smoothness, so a careful assessment of additive mixtures like this work does is important.\n\n* The main result of the paper, Theorem 2, is very interesting and might have important implications. I've read through the proof, and, modulo my comment below in the weaknesses, I think that the proof is otherwise likely alright.\n\n* The experiments appear to nicely support the central conclusion of the paper: \"the inclusion of kernels with different smoothness does not necessarily include prediction accuracy\" (Section 7). (However, I have doubts about this. Please see below.)\n\nWEAKNESSES:\n## Weaknesses in the Method and Theory\n\n### Claim of Theorem 1 Not Fully Supported by the Proof, but a Fix Appears Possible\nThe claim of Theorem 1 consists of two parts: (1) $K$ is $d$-times MSD, and (2) \"so the smoothness of $K$ is determined by the least smooth component\". For (2) to follow from (1), it is also necessary that $K$ is _not_ smoother than $d$-times MSD! That is, it is necessary that $K$ is _not_ $(d + 1)$-times MSD.\n\nFor functions, consider the following example. Suppose that $f_1$ is differentiable and that $f_2$ is non-differentiable. Is $f_1 + f_2$ then non-differentiable, so the smoothness of the sum is determined by the least smooth component? The answer is no: if $f_1 - f_2$ is non-differentiable, then a counter-example is found with $g_1 = f_1$ and $g_2 = -f_2$; and if $f_1 - f_2$ is differentiable, then a counter-example is found with $g_1 = f_1 - f_2$ and $g_2 = f_2$. Therefore, for sums of functions, the smoothness is _not necessarily_ determined by the least smooth component, because the sum can be smoother than the least smooth component! Although the claim is false for functions, kernels have very different properties, and I think the claim is true in that case.\n\nHence, to fully support the claim in Theorem 1, it is necessary that the author prove that $K$ is not $(d + 1)$-times MSD, which is currently not shown in the proof. Fortunately, I think that the proof is easily extended by making use of the following property: if $f_1$ is integrable and $f_2$ is non-integrable, then, by $|f_1 + f_2| >= |f_2| - |f_1|$, $f_1 + f_2$ is also non-integrable.\n\n### Question About Proof of Theorem 2\n\nIn the proof of Theorem 2, the fourth inequality on line 25 of the appendix seems to use the following property:\n\n$ \\frac{f(\\omega) + o(f(\\omega))}{g(\\omega) + o(g(\\omega))} = \\frac{f(\\omega)}{g(\\omega)}(1 + O(\\|\\omega\\|^{-2})) $\n\nwhere $f(\\omega)$ and $g(\\omega)$ are the first terms of the particular mixtures of $p_l$ and $\\tilde p_l$. Now,\n\n$ \\left| \\frac{(f(\\omega) + o(f(\\omega)))/(g(\\omega) + o(g(\\omega)))}{f(\\omega)/g(\\omega)} - 1\\right| = \\left|\\frac{1 + o(1)}{1 + o(1)}\\right| = |o(1)|$,\n\nso\n\n$ \\frac{f(\\omega) + o(f(\\omega))}{g(\\omega) + o(g(\\omega))} = \\frac{f(\\omega)}{g(\\omega)}(1 + o(1)) $,\n\nwhich is a weaker result! If you're a little more careful, instead of $o(1)$ you might get a $O(\\|\\omega\\|^{-q})$ where $q$ depends on the ratio of the leading term and the second term in the particular mixtures of $p_l$ and $\\tilde p_l$. Importantly, $q$ might be very small, smaller than $2$. It's not at all obvious to me that $q=2$ is true! This may seem like a minor detail, but it could mean that the integrand actually depends on $O(\\|\\omega\\|^{-2q})$ for $q$ potentially very small, which would affect the whether the integrand is integrable or not.\n\nI would like the authors to reply to this question. Since Theorem 2 is the central result of the paper, it is important that the proof is bulletproof.\n\n### I Disagree With Corollary 1\n\nCorollary 1 states that \"there is no advantage in including any other components apart from the least smooth component\".\nI strongly disagree with this conclusion, because I think that it misinterprets the concept of equivalence.\n\nIf two GPs are equivalent, then they produce _asymptotically_ similar predictions in the limit of infinite data under fixed-domain asymptotics.\n_However_, for finite data, equivalent GPs may give different predictions, and this difference may be practically very significant!\nTherefore, to conclude that \"there is no advantage in including any other components apart from the least smooth component\", in my opinion, is wrong.\nLet me give a simple counter-example.\n\nLet $k_1$ be a Matern-$1/2$ kernel with length scale $1$ and let $k_2$ a Matern-$3/2$ kernel with length scale $1000$.\nConsider the mixture $k = k_1 + k_2$, and suppose that data is generated from $k$.\nSuppose that we have $3$ observations randomly sampled from $[0, 2000]$.\n\nIf we make predictions with just the least smooth component, $k_1$, then, because of the short length scale of $k_1$, the predictive mean is basically zero on all of $[0, 2000]$ except near the observations.\nOn the other hand, if we make predictions with the mixture $k$, then, because of the long length scale of $k_2$, the predictions are much more reasonable.\n\n(You can concoct a similar scenario by considering a Matern-$1/2$ kernel with length scale $1$ and a Matern-$v$ kernel for very high $v$ (so basically a squared-exponential kernel) also with length scale $1$. For finite data, the difference in smoothness will give very different predictive uncertainty.)\n\n### Attribution of Theorem 3\n\nIn the main body, the authors present Theorem 3 as if it is a completely novel result.\nHowever, Theorem 3.(i) is basically Theorem 3 by Bachoc et al. (2022), and Theorem (3).(ii) follows nearly immediately from Theorem 2 by Bachoc et al. (2022).\nThe authors acknowledge this in the proof in the supplement, but they should really state this attribution in the main body.\nBecause of the similarity to the theorems by Bachoc et al., I consider Theorem 3 by the authors only a minor contribution.\n\n## Weaknesses in Experiments\n\n* In Simulation 1, the authors state that the \"it is evident that the mixture kernel demonstrates a degree of smoothnes similar to the Matern kernels with $\\nu = 1/2$\". Although I think that the result is true, I think that this is not necessarily so obvious from Figure 1. To really determine the smoothness of the samples of the mixture kernel, the authors should provide more conclusive evidence, e.g. by numerically determining whether the function is differentiable: numerically compute $|f(x) - f(x + h)|/|h|^{\\alpha}$ for $h \\to 0$ and _e.g._ $\\alpha = 1$ (for once differentiability).\n\n* In Simulation 2, the authors demonstrate that $w_1 \\sigma_1^2 \\alpha_1$ is identifiable and they claim that _only_ this parameter is identifiable. To really give evidence that only  $w_1 \\sigma_1^2 \\alpha_1$ is identifiable, the authors should show the parameters that would be microergodic if you would consider the kernels separately: $w_2 \\sigma_2^2 \\alpha_2$ and $w_3 \\sigma_3^2 \\alpha_3$. These parameters are also not shown in Figure S1. Without showing these additional parameters, I'm not convinced that the figure really prove that only  $w_1 \\sigma_1^2 \\alpha_1$ is identifiable.\n\n* Throughout the real-world experiments, the authors use Adam/SGD. I wonder why the authors wouldn't just run L-BFGS-B until convergence at a reasonable tolerance. This would likely eliminate some difficulties in optimising the mixture kernel. In addition, the supplement says that the kernel matrices are _heavily_ regularised by a diagonal of magnitude $0.01$ or $0.1$. I think that is _much more_ than should be necessary if one uses double precision. I'm not sure if it would change any results, but it does make me suspicious.\n\n* In Application 1, the authors compare a Matern mixture kernel against a Matern-1/2 kernel to perform texture extrapolation. Although I don't disagree with the result that the mixture Matern kernel performs similar to the Matern-1/2 kernel, I think both the mixture kernel and the Matern-1/2 kernel are a very poor fit for the data, which can be seen from the prediction in Figure 4. The point of these textures is that extrapolation is possible because they are periodic, so one should really use a (weakly) periodic kernel. I understand that Theorem 2 does not apply to mixtures of periodic Matern kernels, just to regular Matern kernels, but to me that suggests that this is not a good data set to demonstrate Theorem 2 on.\n\n### Experiments Do Not Strongly Support Main Claim\n\nMy main issue with the experimental section is that the authors take some data sets, compare a mixture of Matern kernels versus a Matern-1/2 kernel, observe that the performance of the two is very similar, and conclude that, generally, there might be no advantage in adding additional components to the mixture. In my opinion, this is not the right conclusion. The right conclusion is that either (a) generally, there might be no advantage in adding additional components to the mixture; or (b) specifically in the chosen experiments, there is no advantage in adding additional components to the mixture.\n\n_To really conclude (a), the authors should compare the mixture kernel to the Matern-1/2 kernel on a data set where conventially one would agree that a Matern mixture kernel would be a good choice._ An example of such a data set is a time series that has a noisy short-length-scale components and a smooth long-length-scale component. In constrast, economical data sets such as prices are traditionally modelled with an Ornstein-Uhlenbeck process (Matern-1/2 kernel), so I'm not surprised additional smooth kernels won't help there. Moreover, if you visualise the gene expression data, you will see that the data is _very_ erratic and definitely doesn't contain and smooth components. Therefore, I am not at all convinced that the authors have made a good case for (a).\n\nQUESTIONS:\n## Conclusion\n\nI think that this is great submission with very interesting results. However, in my opinion the authors have not provided sufficient evidence for the central message of the paper, which is that the inclusion of kernels with different length scales does not improve prediction accuracy. (Please correct me if this is not the central message of the paper.) The authors have not provided sufficient evidence because of the following two reasons:\n\n* I believe that the conclusion in Corollary 1 is not justified. Please see the reasoning above.\n\n* I believe that the data sets chosen in the experiments do not contain any smooth components, so obviously modelling smooth components in the data will not get you any improvement in predictive accuracy. Please see the foregoing section. Therefore, I think that the experimental results do not support the central message of the paper.\n\nIn addition, the proof of Theorem 2 might have an error (please point out whether I'm right or wrong about this!), Simulations 1 and 2 (please see above) have flaws, and I think that Theorem 3 is only a minor contribution. If I add all this up, I'm afraid that I must recommend rejection at this point.\n\n_However_, I'm very willing to debate the above arguments and revise my score if necessary. Therefore, authors, please carefully argue the above, because I would love to be wrong and change my recommendation to accept.\n\nEDIT\n\nIn the discussion below, the authors have outlined changes that address the above shortcomings. These changes are substantial. Since uploading revisions is not possible, normally I would recommend that the revised PDF go through another round of review. However, I feel that the authors and I are now aligned sufficiently well, so this time I am willing to recommend a \"borderline accept\".\n\nI should emphasise that I believe that this submission likely deserves a better score than \"borderline accept\", but I am not willing to recommend any higher score before reviewing the revised PDF.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper intoduces a few results on identifiability of Gaussian Processes. \nIt is shown that a Gaussian process with a kernel that is a mixture of kernels, the smoothness is that \nof the least smooth component.  \nIt is shown that for two mixtures of Matern kernels, the induces processes are equivalent if certain\nproducts involving only parameters of the least smooth kernels in the mixture coincide. Consequently, \nhigher smoothness components do not affect equivalence. \nFinally, for a multi output GP with covariance  of the form $A \\cdot k$, where A is a fixed matrix and k is a scalar Matern kernel, the matrix A is identifiable. \n\n\nExperiments that are intended to support the above results are included.\n\nSTRENGTHS:\nThe general subject of identifiability in GPs is important. \nThe non technical parts of the paper are well written and it is possible get a general idea of what the paper is about. \nThe experiment 5.2 is illustrative in showing that the prdouct can be identified while the individual components are not identified properly. (however, a much more detailed description of the experiment is needed).\n\nWEAKNESSES:\nMy main question is regarding the notion of equivalence used in this paper. It is not clear why this is \na useful notion. The questions are as follows:\n\n* Clairty regarding identifiability 1. First,  what is definition of the $\\equiv$ symbol that is used for kernels in all the theorems? I couldn't find this definition for some reason. Where is it defined? Does it mean that the GPs induced by the corresponding kernels are equivalent in the sense of Definition 5? \n\n* Clairty regarding identifiability 2: I'm assuming the answer to the question above is yes. \nThen, why is this equivalence interesting and relevant to identifiability? Please explain why this notion is relevant. \nIndeed, if my index set is finite, I can have two GPs on this set with covraiances C_1, C_2. As long as C_1,C_2 have full rank, the processes will be absolutely continuous w.r.t each other, but they are very different processes and C_1 and C_2 can be learned.  How this related to the above notion of equivalence? \n\n* In anycase, this notion of equivalence is by no means standard in the ML commmunity, and needs to be introduced and discussed in more detail. \n\n\nAdditional Questions:\n\n\n* Relevance:  While the general subject of identifiability is imporatnt, this is concerned with a very specific case of Matern linear mixtures. It is not clear this specific case has been used in applications before.  \nSome references are provided about the use of mixtures on lines 37-38, but how many of these use specifically *linear mixtures*, and  *Matren mixtures*?\n\n* Literature discussion: There is no Literature section in the paper.  It is my understanding that there is previous work on identifibility, even for a single Matren kernel. This needs to be discussed in the paper, and comparison provided. Are there new mathematical ideas involved in the results in this paper compared to existing work? \n\n* Inverse implication in results: The results state that if parameters satisfy something, then there is equivalence. Is the reverse direction true? Is it obvious? This again is related to the somewhat \n\n\n* Experiments: \nAs mentioned earlier, Experimen 5.2 is illustartive. However, it is not specified how the optimization is made. It may also have happened that there is no reconstruction due to local maxima of the method, rather than general lack of identifiability.  \n\n* As for the other experiments: the experiments 5.1, 6.1, 6.2 do not illustrate anything that I would find informative.  6.2 is a 1d version of an already small multidimensional dataset. It is not clear how 6.1 is realted to  ML use cases.\n\nQUESTIONS:\nPlease see above. \nMy recommendation for the rating is temporary and I plan to revisit it following author's responses.\n\nLIMITATIONS:\nThe subject of this works involves some abstract properties of Gaussain Processes. No specific societal impact discussion is required.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates additive and separable mixture kernels in the context of Gaussian process regression. Concretely, it tests the intuition behind the convex combination of Matern processes and identifies limitations in the interpretability of the resulting mixture kernel that might contradict the intuition of the modeler.\nIt further analyzes the identifiability of parameters of separable Kernels in multi-output GPs.\n\nSTRENGTHS:\n- I believe this paper is a useful contribution to understanding when intuition about modeling decisions diverges from the actual impact of these decisions. It thus earns its place at a ML conference in that it strictly extends the pool of resources that should be considered when designing a GP model\n- NeurIPS seems to me as the right venue to publish these results in that format, since it manages a good balance between mathematical rigor and intuition, which makes the text very approachable for more application-oriented researchers, as well.\n- the overall presentation follows a very clear structure that is easy to follow. I appreciate that the authors do not obscure meaningful results behind unnecessarily complicated formulations.\n\nWEAKNESSES:\n- The plots are aesthetically rather unpleasing and the presentation of the results would, in my opinion, profit quite significantly from a polishing of the figures.\n- The authors focus solely on Matern processes, to the point where the notion of \"kernel\" is used interchangeably with \"Matern-kernel\". This limitation, however, is understandable in light of the page limit and is also addressed in the discussion section. (It does not affect my score.)\n\nQUESTIONS:\n- I would like to understand the impact of the \"smoothness-of-mixed-Matern-kernel-is-determined-by-its-least-smooth-component\" argument better:\n\t- Is it really only about smoothness? This seems rather obvious to me (and I am certain this has been known before). But Theorem 2 states that the kernel is equivalent, and hence makes a more general statement about the structure of the functions from this kernel's RKHS, than merely smoothness, right? As somebody who is not familiar with proofs about identifiability and equivalence of kernels: apart from smoothness of the resulting sample paths, does mixing in more Matern components not add *any* structure to a single (least-smooth) Matern component? I could imagine that other readers would also appreciate a clarifying statement that brings together the \"same-smoothness\" and \"equivalence\" arguments.\n\t- What reinforces me in my endeavor to understand the impact of this statement better is also the fact that in Section 7, you write that \"[...] the inclusion of kernels with different smoothness does not necessarily improve prediction accuracy\".\n\t  On the other hand, Theorem 2 states that it does absolutely make no difference (not only \"not *necessarily*\"). Could you elaborate more on the impact of this statement and its strength?\n- Suggestion: the paper includes a comparison between (i) a mixture of kernels with varying smoothness parameters and (ii) a kernel consisting only of the least-smooth component. It would be interesting to see the comparison to (iii) the other (two?) components, as well. I would expect these results would be different then. But *how* they would be different (by how much or - e.g. in the case of Figure 6 - how they differ structurally) would be quite interesting, measured by the expected effort of this addition.\n\nLIMITATIONS:\nLimitations of the analysis presented in the text are adequately addressed in the discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work looks at idenfiability in the context of mixture kernels in GP regression. They look at additive mixtures of Matern kernels and multivariate separable kernels in the context of multivariate GPs (multiple outputs). They show theoretically and through simulation that ML-II learning cannot discover the hyperparameters in the mixture corresponding to the different components as they are not microergodic (Stein, 1999). On the contrary, only products of hyperparameters in specific combinations is identifiable. The simulation studies on synthetic data are interesting - through repeated experiments they show that ML-II does not converge to the true hyperparametes in a mixture kernel and only the microergodic parameters are discovered.\n\nSTRENGTHS:\nA lot of the theory on identifiability has been scattered in literature like Zhang (2004) and Stein's textbook (1999) but I would argue that a succint and targetted presentation such as this one might benefit the community and hence can be significant.\n\nWEAKNESSES:\nThis work begs the question - what happens in the large data limit when n -> \\infty, as the simulations and applications only address the finite data limit. My guess is that the parameters which are not microergodic remain unidentifiable but some comment is needed.\n\nQUESTIONS:\n- Theorem 2 'p' is introduced without defining. I've had to figure that it is the dimension. \n- Perhaps tangential to this work but it would be interesting to reason about the identifiability of the noise variance when a mixture kernel is used. \n- How does the additive GPs work where the constructed kernel is additive and considers all combinations of dimensions in the data  fit with the identifiability insights of this work. \n- About the dominance of the least smooth kernel component, how does one extend the insight to mixtures of different kernel types like Matern + RQ or periodic + Matern. \n- Line 281 / 282 pls cite Lalchand et al, Generalised GPLVM with Stochastic Variational Inference, AISTATS 2022.\n- Line 47/ 48 pls cite Simpson et al, Kernel Identification with Transformers, NeurIPS, 2021.\n\nLIMITATIONS:\nThe most obvious one - which the authors mention in conclusions is the extension of these insights to mixture of kernel families which are used very frequently to model different effects in univariate functions. eg. CO2 Mauna Loa kernel.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}