Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel
Authors: Ziyang Jiang
Tongshu Zheng
Yiling Liu
David Carlson
Venue: Accepted by TMLR ()

Abstract: It is challenging to guide neural network (NN) learning with prior knowledge. In contrast, many known properties, such as spatial smoothness or seasonality, are straightforward to model by choosing an appropriate kernel in a Gaussian process (GP). Many deep learning applications could be enhanced by modeling such known properties. For example, convolutional neural networks (CNNs) are frequently used in remote sensing, which is subject to strong seasonal effects. We propose to blend the strengths of NNs and the clear modeling capabilities of GPs by using a composite kernel that combines a kernel implicitly defined by a neural network with a second kernel function chosen to model known properties (e.g., seasonality). We implement this idea by combining a deep network and an efficient mapping function based on either Nystrom approximation or random Fourier features, which we call Implicit Composite Kernel (ICK). We then adopt a sample-then-optimize approach to approximate the full GP posterior distribution. We demonstrate that ICK has superior performance and flexibility on both synthetic and real-world datasets including a remote sensing dataset. The ICK framework can be used to include prior information into neural networks in many applications.

Published in Transactions on Machine Learning Research (02/2024)
Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel
Ziyang Jiang
ziyang. joyonngiiong@dduueeeei
Deportment of Civil and Environmental Engineering
Duke University
Tongshu Zheng
tongghucchengg220@maai.cco
Division of Natural and Applied Science Duke Kunshan University
Yiling Liu
gilling yillng.liu@ddukee liw@duke edu
Program in Computational Biology and Bioinformatics Duke University School of Medicine
David Carlson
david. www.sonntnnooooooo Bdwkondddduueec edu
Deportment of Civil and Enwironmental Engineering
Department of Biostatislics and Bioinformatics
Department of Computer Science
Duke University
Reviewed on OpenReview: https: // openreview. net/ forum? id= ddHhSSalvwie
Abstract
It is challenging to guide neural network (NN) learning with prior knowledge. In contrast, many known propertics, such as spatial smoothness or scasonality, are straightforward to model by choosing an appropriate kernel a Gaussian process (GP). Many deep learning applications could be enhanced by modeling such known properties. For example, convolu- tional neural networks (CNNs) are frequently used in remote sensing, which is subject to strong seasonal effects. We propose blend the strengths of NNs and the clear modeling capabilities of GPs by using composite kernel that combines a kernel implicitly defined by neural network with a second kernel function chosen to model known properties (c.g., seasonality). We implement this idea by combining a deep nelwork and an efficient mapping function based on either Nyström approximation or random Fourier features, which we call Implicit Composite Kernel (ICK). We then adopt sample-then-optimize approach to approximate the full GP posterior distribution. We demonstrate that ICK has superior performance and flexibility both synthetic and real-world datasets including remote sensing dataset. The ICK framework can be used to include prior information into neural networks in many applications.
1 Introduction
In complex regression tasks, input data often contains multiple sources of information. These sources be presented in both high-dimensional (c.g. images, audios, texts, etc.) and low-dimensional (c.g. timestamps, spatial locations, etc.) forms. A common approach to learn from high-dimensional information is to use neural networks (NNs) [Goodfellow et al. 2016 LeCun et al. 2015), as NNs are powerful enough to capture the relationship between complex high-dimensional data and target variables of interest. In many areas, NNs are standard practice, such the dominance of Convolutional Neural Networks (CNNs) for image analysis (Jiang et al. 2022 Zheng et al. 2021 2020 Tn contrast, for low-dinensional information, we usually have

---

Published in Transactions on Machine Learning Research (02/2024)
some prior knowledge how the information relates to the predictions. As concrete example, consider remote sensing problem where we predict ground measurements from satellite imagery with associated timestamps, A priori, we expect the ground measurements to vary periodically with respect to time between summer and winter due to seasonal effects. We would typically use a CNN to capture the complex relationship between the imagery and the ground measuremennts. In this case, we want to guide the learning of the CNN with our prior knowledge about the scasonality. This is challenging because knowledge represented NNs pertains mainly to correlation between network units instead of quantifiable statements (Marcus 2018
Conversely, Gaussian processes (GPs) have been used historically to incorporate relevant prior beliefs by specifying the appropriate form of its kernel (or covariance) function (Bishop & Nasrabadi, 2006; Williams & Rasmussen 2006). One approach to modeling multiple sources of information is to assign a relevant kernel function to each source of information respectively and combine them through addition or multiplication, resulting in a composite kernel function [Duvenaud] 2014). This formulation means that specifying a kernel to match prior beliefs on one source of information straightforward. Such composite kernel learning tochniques are extensively used in many application areas such multi-media data (McFee et al. 2011), neuroimaging
(Zhang et al. 2011), spatial data analysis, and environmental data analysis (Kim et al. 2005 Petelin et al. 2013). In view of the clear modeling capabilities of GP, it is desirable to examine how NN could be imbuod with the same modeling ease.
In recent years, researchers have come up with a variety of methods to incorporate prior knowledge into NNs. These efforts can be broken into many categories, such as those that add prior information through loss terms like puyysics-informee
NNs Lagaris et al., 1998 Moseley et al., 2020). Here, we focus on the major category of those methods that build integrated models of NNs and GPs with various structures (Van der Wilk et al.,
2017 Wilson et al. 2016b 2011). Related to our proposed methodology, Pearce et al. (2020) exploited the fact that Bayesian neural network (BNN) approximates GP to construct additive and multiplicative kernels, but they were limited to specific predefined kernels. Matsubara et al. (2020) then resolved this limitation by constructing priors of ENN paramoters based on the ridgelct transform and its dual, but they did not explicitly show how their approach works for data with multiple sources of information. To our knowledge, no existing approach allows a modeler choose any appropriate kernel over multiple sources.
We address this limitation by presenting a simple yet novel Implicit Composite Kernel (ICK) framework, which processes high-dinensional information using a kernel implicitly defined by neural network and low-dimensional information using a chosen kernel function. The low-dimelsional kernels are mapped into the neural network framework to create a straightforward and simple-to-lean learrn implementation. Our key results and contributions are:
We analytically show our ICK framework, under reasonable assumptions, is approximately equivalent to sampling from a Gaussian process regression (GPR) model with composite kernel priori.
We adopt a sample-hen-optimize procedure to ICK to approximate the full posterior distribution of GP with a composite kernel.
We show that ICK yields better performance on prediction and forecasting tasks, even with limited data.
We show that ICK can flexibly capture the patterns of the low-dimensional information without bespoke pre-processing procedures or complex NN structures.
Based these contributions, we believe ICK is useful in learning from complex hybrid data with prior knowledge, especially in remote sensing and spatial statistics.
2
Related Work
Equivalence between NNs and GPs The equivalence between GPs and randomly initialized single-layer NNs with infinite width was first shown by Neal (1966). With the development of modern deep learning, rescarchers further extended this relationship to deep networks (Lee et al., 2017 Matthews et al.) (018)) and convolutional neural networks (CNNs) (Garriga- GGariga-Alonsoo Alonso et al. 2018 Novak al. 2018).. This relationship is crucial for showing the resemblance between GPR and our ICK framework, which is discussed in Section 4.1
2

---

Published in Transactions on Machine Learning Research (02/2024)
NNs with prior knowledge As mentioned before, one approach to equip NNs with prior knowledge is to modify the loss function. For example, Lagaris et al. (9988) solved differential equations (DEs) using NNs by setting the loss to be a function whose derivative satisfies the DE conditions. Another approach is to build integrated models of NNs and kernel-based models. For example, Wilson et al. (2011) implemented a regression work with GP priors over latent variables and made inference by approximaling the posterior using Variational Bayes or sampling from the posterior using Gibbs sampling scheme. [aarnelo al. (2018)
introduced class of neural latent variable models called Neural Processes (NPs) which are capable of learning efficiently from the data and adapting rapidly to new observations. In addition to these, various studies have explored the integration of Neural Networks (NNs) and kernel methods (Hinton & Salakhutdinov 2007;
Wilson et al.) 2016a Adlam et al. 2020). Our ICK framework fuses prior knowledge into NNs by modulating the learnt features using another set of features outpulted from a kernel-based mapping, which can also be viewed as an integrated model of NNs and kernel machines.
GP with composite kernels Composite kernel GPs are widely used in both machine learning Duvenaud
2014 Williams & Rasmussen, 2006) and geostatistical modeling (Datta et al.) 2016 Gelfand & Schliep 2016).
GPR in geostatistical modeling is also known as kriging (Journel & Huijbregts/ 1976 Krige 1951 which
serves surrogate model to replace expensive function evaluations. The inputs for composite GP are usually low-dimensional (e.g. spatial distance) as GPs do not scale well with the number of samples for
high-dimensional inputs (Bouhlel & Martins 2019 Bouhlel et al. (016). To overcome this issue, Pearce et al..
(2020) and Matsubara et al. (2020) developed BNN analogues for composite GPs. Similar to these studies, our ICK framework can also be viewed as a simulation for composite GPs.
GP for large datasets Since training and inference of exact GP scales O(N*), either parallel computing Wang et al. (2019): Adlam et al. (2023) or kernel approximation are needed to scale GP large datasets. Nyström low-rank matrix approximation (Drincas et al., 2005 Williams & Socger 2000) and Random Fourier Features (Rahimi & Recht, 2007:
2008) are two commonly used approximation methods. Building upon these concepts, several popular frameworks, such as sparse GPs (Snelson & Ghahramani 2005 Titsias 2009;
Hensman et al. 2013), have been developed to facilitate GP inference on large datasets. In our research, we draw inspiration from these approximation methods and utilize them as transformation functions to project
the kernel matrix into latent space representations, as discussed in Section 4.2
3 Background
3.1 Problem Setup
To formalize the problem, we have a training data set which contains N data points X = [o:)]* = (zi, F2, F2,, and the corresponding labels of these data points are y - - = [y1, 3/2, where Vi E R. Each data point x; (x (1) (2)
_x!") ...,,;; ...,, is composed of information from M different sources where the th source of information the 7th data point is denoted as x."' (m)) E RR",, Our goal is to learn a
function ú, f(x,) RD1, x D2 x ... RD - which takes in a data point æ, and outputs a predicted value g..
3.2 Composite GPs
A Gaussian process (GP) describes a distribution over functions (Williams & Rasmussen 2006). A key property of GP is that it is completely defined by a mean function ((()) and a kernel function K K(x, (.) where x and ø' represent different samples from the training dataset. The mean function u(x) is often assumed to be zero for simplicity. In that case, the outcome function
f(x) ~ GP (0, ((x,())))).
(1)
Any finite subset of random variables has a multivariate Gaussian distribution with mean and kernel matrix K whose entries can be calculated as K2 KZ,,,2,) where 1 1 i,j S N. In many situations, the full kernel function is built. by a composite kernel by combining simple kernels through addition Keomp(x,, x²)= =
3

---

Published in Transactions on Machine Learning Research (02/2024)
x0)
x
x
Rd
e R
(
g()
K,(x,x))
Ky(x,x"
Given data containing 2 sources of information x()) and x(²), we can process the data using either (Left) a composite Gaussian process regression (GPR) model or (Right) our ICK framework where x(1) is pro- cossed with neural network fNN(.) and x( is processed with g(-) where g(-) consists of kernel function K₂ and some transformation which maps the kernel matrix K₂ into the latent space.
Figure 1:
K
2
Kg6x,x))
K₂
Combine
K cnmp
2
x
9- --f(x(1 22) (..(p. Kammp)
Inner product
9
9
(a + K2 æ') or multiplication Koomp (x, x²) = (1 x²) (2 a') Duvenaud 2014). A useful property that ICK exploits is that K₁ and K₂ can take different subparts of a as their inputs. For example, (comp(xxx')) = (x))), (0))) + (2(x(3), x(2)) or Kcomp(x, x') = K₁ (x())) x01) (2)))(),x
3.3 Correspondence between GPs and NNs
Neal (1996) proved that a single-hidee hidden layer network with infinite width is evactly equivalent a GP over data indices i = 1,2, ..., N under the assumption that the weight and bias parameters of the hidden layer are i.idd Gaussian with sero mean. This statement was then extended to deep NNs (Lee et al. 2017 Matthews et al. 2018) and convolutional NNs (Garriga-Alonso et al. 2018 Novak et al. 2018). Specifically, let z = JNN x D. + Tp' be the latent representation extracted from x!1) where is the dimension of the extracted representation and JNN is a neural network with zero-mean i.i.d. parameters and continuous activation function & which satisfies the linear envelope property
(o())l c m/ul Vu E IR,
(2)
if there exists c, m 2 0. This includes many standard nonlinearities. The 4h entry of this representation will converge in distribution to a WN-implied GP in the infinite width limit
(x 4 KNN (20))))00))))) 2a)))
33)
Here NN in Equation 3 denotes the covariance function of the equivalent GP the width of fNN goes
to infinity and can be computed numerically in recursive manner (Lee et al. 2017). That is to say, the tth component 2 of the representation extracted by the network has zero mean (p(()))
(1) = 0 for all
2ik
i = 1,2, N where represents the network paramcters. The covariance between 2/k
and 2jk for different
(1)
data indices i,j = 1,,, .... N can be approxinated as cov(= (1) ( = (ooo Fik 9jk ~ KNN (a xz)
where x. (1) and a; are the corresponding inputs in case the network width is finite.
4 Implicit Composite Kernel
We show the structure of composite GPR model and our ICK framework in Figure 1 To make the illustration clear, limit ourselves to data with information from 2 different sources æ (0)) (²)) where (()) dige-dimen and (²)) low-dimensional (i.e. D1 > D2) with some known relationship with the farget y. We are inspired by composite GPR, which computes 2 different kernel matrices K₁ and K2 and
4

---

Published in Transactions on Machine Learning Research (02/2024)
x R
x R
xIMi PM
Figure 2: Given data containing M sources of information a (x(1),x(2) ...,,((Me)) (,(,,,, the data using our ICK framework where high- nigh-dimensional information (e.g. x(1) in the figure) is pro- cessed using neural network and low- dimensional information (e.g. 2(2) in the figure) is processed using a ker- nel function and some transformation which maps the kernel matrix into the latent space.
Ka(s,z))
K1
Kи
we can process
29
200
E
R
Inner product: ý Y2 2
then combines them into a single composite kernel matrix Kroom However, as discussed before, it is more suitable to use a NN to learn from the high dimensional information x(1), In our ICK framework, we process (11) with a NN fNN(.) (-) RP + I' and x(2) with a mapping g(.) R + RP, which consists of a kernel function K2 followed by kernel-to-latent-spacee transformation (described in Section 4.2), resulting two latent representations =0)),((2) E IR". Then, we make prediction 9 by doing an inner product between these JNN (ax11) g (z=22). Finally, the parameters of both the NN and the kernel function are learned via gradient-based optimization methods (Bottou et al. 2018
two representations
Besides the formulation in Figure 1
ICK can also process data æ
{x(1) 1 x12
(M) with M > 2 sources
of information as shown in Figure 2 Here Ki,..., KM represent different types of kernels with trainable parameters. The final prediction is calculated by a chained inner product of all extracted representations i 3 m=1 (m))
M
In the sections below, we first show the relationship between ICK and a composite GPR model with a multiplicative kernel a priori in Section 4.1 which is used to motivate the model form. We then show how we implement the kernel-to-latent-sppac transformation Section 4.2 To enable a full GP posterior approzimation for ICK, will provide insight into how we derive uncertainty estimates from ICK. This can be achieved either through direct variance calculation (Section 4.3.1 through the utilization of an ensemble approach (Section 4.3.2
4.1 Resemblance between Composite GPR and ICK
We will analytically prove the following theorem for data with information from different sources (x:11, (2)] for clarity, and we note this theorem can be straightforwardly extended to M > 2.
Theorem 1. Let JNN: Dr + R be a NN function with random weights and g RD2 - TRP be mapping function, and define an inner product between the representations 9 = fICK (2) x(21) SNN (x(1)
22)) =2)) Then fIck will converge in distribution to GP in the infinite width limit a priori 9 (x(2)) =
fICk 4 (N (1))) (2) (())))
(4)
if INN is a neural network with zero-mean i.i.d. parameters and continuous activation function & which satisfies the linear envelope property in Equation and g includes the following determénistic kernel- to to-latent-ttee latent-space space transformation for all 1 S i.j
(2) a Ka(w!
(2)
x (2)
2
g(x; gg( (2)
(5)
where K NN is NN-implied kernel and K2 is any valid kernel of our choice.
5

---

Published in Transactions on Machine Learning Research (02/2024)
To prove Theorem 1 we first state following lemma.
Lemma 2. For latent representations 21t) and 21)) extracted from different data points ai and æ where
j, the interactions between different entries of 2, and z( can be reasonably ignored. In other words, let ((1) be the parameters of the neural network which takes in x!1)) and oulputs 2(1), we have Regern (2) (1)) 0
Pik jj
for all k L..
A detailed proof of Lemma 2 is provided in Appendix With Lemma let o {ol1) 0(0} represent the parameters of ICK, we can calculate the covariance between di and Øg for different data indices i + j:
cov(Di,0))
= p [0:9,] (ppee))))
(6)
(Ek-1 Fike 46k
(1) (2) (2))
(Ein 2jk - (3
(7)
= pe))
Ek=1 E
(1)
(1)
(2) (2)
(8)
P 31
ik
= E))))
E
Fike Pjk Pike 3j
(1) (1) (2)
(9)
Ex=n E
(((())))
[1) (1)
Fik Ei
E (((2))
Fik Pjk
(2)
(10)
8 KNN
a; 2
(1) (1)
Ek=1
E ((())))
(2) (2) Pik 9jk
(11)
Here, from Equation which leads to (e) U.
6 to Equation
we use the statement E pp(0)))
0 in Section 3.3 and ( 1 (
[, 0. From Equation 8 to Equation
we get rid of all the cross terms using
Lemma and ((2) IL 0 Specifically, we have E
in Fjz 4sh Pjz Enp (1) (1)
(1) (2)
Zj p Pih 2j
(2) 0
for all * L. From Equation
Equation 10 we again make use of ( 1 e From Equation 10
Equation 11 we use the statement Ep(om) ik 3k =
K (NN(x!") (z))) from Section 3.3 If the kernel-to-
latent-space transformation in g(-) is deterministic, we can remove the expectation sign from the summation term in Equation 11 and the covariance can be further expressod as
(1) (1) cov 0j. = KNN(2 (z) .T) (2) (2) 200)
(12)
KIN(2x) 2 K2(0 22)
which means that approximately follows a GP with multiplicative composite kernel (comp 2z)) KNN (1)
(a)" fr) K₂ (z!²) æ.
(2)
(2) priori. This serves consise proof of Theorem 1. We also give more
detailed proof in Appendix
4.2 Bernel-to-latent-spaa Transformation
We now show how we can construct an appropriate mapping g(-) that approximately satisfies the assumed form of equation 5 and is used in the derivation of ICK from equation [ to equation I2 Here we adopt two methods, Nyström approximation and Random Fourier Features (RFF), to map the kernel matrix into the latent space. Below, we elaborate the formulations for both methods, and give the methods and results for the Nyström method. The results of RFF will be presented in Appendix C We name our framework with N Nyström method and random Fourier Features ICKy and ICKr, respectively.
According to Yang et al. (2012), the Nyström method will yield much better performance than RFF if there exists a large gap in the cigen-spectrum of the kernel matrix. This phenomenon is mainly caused by how these two methods construct their basis functions. In particular, the basis functions used by RFF are sampled from a Gaussian distribution that is independent from the training examples, while the basis functions used by the Nyström method are sampled from the training samples so they are data-dependent. In our synthetic data experiments, we train our ICK framework using a batch size of 50. The cigenvalues of the kernel matrices computed from the first batches of the senthetic data set displayed in Figure 3 It can be observed that.
6

---

Published in Transactions on Machine Learning Research (02/2024)
the first few cigenvalues of the kernel matrix are much larger than the remaining eigenvalues. Namely, we observe large gap in the eigen-spectrum of the kernel matrix, and Nyström method does generalize much better than RFF in our experiments.
4.2.1 Nyström Approximation
The main idea of Nyström approximation (Williams & Seeger 2000 is to approximate the kernel matrix K E RX with much smaller low-rank matrix Kg E Raxq where < N so both the computa-
0.6
Batch
Batch
0.5
Batch
tional and space complexity kernel learning can be significantly roducod, yiclding
Batch
Elgenvalues/N
0.4
K K k K, K KT
(13)
0.3
The entries of Kg and Kng ng can be calculated as (Ka) = K( ((;,, (fi, i,j E 21,,,... of (I,,,ty). E (1,2,..,,N). E (1,2....9), respec- tively. æ represents the original data points and & represents pre-defined inducing points (or pseudo- inputs (Snelson Ghahramani 2005)). In our study, these inducing points are chosen by defin- ing evenly spaced vector over the range of original data points. By performing Cholesky decomposition K ---=-0 UUU, where U E RR98, K is expressed
2,....... and (K =
0.2
0.1
0.0
10
30
40
Rank
Figure 3: Eigenvalues of the kernel matrix computed from the first 4 batches of training data where N is the total number of data points.
K- KnnKK K nq
(14)
= Kn (UK
Therefore, if set q = p: then each element in K approximately satisfies equation 5 as stated in Theorem 1: K (z,, xz) = Kij R KG x zj. Conveniently, modern deep learning frameworks can propagate gradients through the Cholesky operation, making it forwarl forward to update the kernel parameters with gradient methods. Note that as we increase the number of inducing points p, the approximation error between K and K decreases. However, it is not recommended to set p very large as updating the Cholesky decomposition requires O(p"). The empirical impact of p computational time and performance is shown in Appendix D In our experiments, only mild values of are necessary and the impact on computational is relatively small. The full training procedure of ICKy with M 2 is presented in Algorithm 1
2; UKKT) m kernc-to-latent-spacc transformation because
4.2.2 Random Fourier Features
Random Fourier Features (RFF) is another popular approximation method used for kernel learning (Rahimi & Recht 2007). Unlike the Nyström method which approximates the entire kernel matrix, RFF directly approximates the kernel function K using some randomized feature mapping @ R - 2ddm such that
K (m.(m)) .ry (m)
(a("m))
Jr (m)) To obtain the feature mapping o, based Bochner's theorem, we
first compute the Fourier transform p(w) of kernel K
p(w) = (2 Pm
e 6 K
(15)
where 0 = x (m) (m²) mapping & as follows
Then we draw dm i.i.d. samples w1, w2, Wim from p(w) and construct the feature
o (x( = 1/11/2 [cos (wf ((m) cos (wf (m))) sin (wf (m)) .... ,...,sin sin (wff [m)))]]
(16)

---

Published in Transactions on Machine Learning Research (02/2024)
Algorithm Implicit Composite Kernel-Nystróm (ICKy)
N
Input: data X =
x= (1) (2) ==(z)) 1 2
targets y = [uv] JIN,, 0(),, 1(2), learning rate 8
1=1"
Sample a total of Ng minibatches {XB, yB B=1
Na
for B from to NB do
for z (1) (2)
2; in BB 1, RB do
ANN
Define inducing points 20
Compute K,: (Kp),t ((2)
+ 2
(2) (2)
Do Cholesky decomposition (K,) U
Compute Kap: (Knp)zk = (2)
(2) (2)
(2) U (KTT),
D1 (1)
end for
ÚB = concat (91 ... Inn)
Compute loss L (yn, 98) ((1) ((1) 3V 0(1) L
((2) - ( V L
end for
Return: Predictions ÿ and updated parameters ((1),(2))
Since @ (z(")) E 2dm we need to set dm = p/2 when using RFF as a kernel- poatell-tollteenn lo-laaent-sspce transformation. In addition, since RFF involves sampling from distribution, the kernel parameters are thus not directly disferentiable and we need to apply reparameeeriization trick (Maddison el al. 2016) to learn those parameters. The full training procedure of ICKr with M is presented in Algorithm
4.3
Uncertainty Estimation
Since the motivation of ICK discussed in Section closcly related to GPs, we present two dis- finct approaches for estimating uncertainty in the subsequent sections: ensembling method and direct calculation posterior variance.
2.5
4.1
2.0
1.5
1.0
4.3.1 Ensembling
0.5
One approach for estimating the uncertainty is to adopt a sample-then-optiniee approach (Matthews et al. 2017) and construct a deep ensemble posterior approximation for ICK. There are several ways to en- able GP posterior interpretation for a deep ensemble trained with SGD. Specifically, let the final layer of each baselearner NN be C-dimensional and denote the deep ensemble F J.N. where fn. is basclcarner NN and N. is the total number of base- learners in the ensemble. After training, each fn. can viewed as i.id. samples from a multi-output GP posterior with kernel function Kp in the infinite width limit and the ensemble thus represents Ne independent draws from the GP posterior, where Kr
0.0
-055
train
-1.0
ICK ensemble
ICK-NNGP analytic
-1.5 -6
-4
-2
2
X2
Figure 4: Predictive distribution from ICKy ensemble and its GP posterior counterpart on a ID regression task.

---

Published in Transactions on Machine Learning Research (02/2024)
Algorithm Implicit Composite Kernel-RFF (ICKr) x= (1) (2) targets y = [yi] JINN, O(),, K2, 0(2), learning rate 8 Input: data X = {=(0))
N
2
1=1"
Sample a total of Ng minibatches {XB. ye) Na B=1 for B from to NB do
for z; (1)
2 in x 1, RB do
(2)
ANN
Compute the Fourier transform p(w) of kernel K2:
p(w) = (m 1 e j- K (5)dd
where 8 0= 24
(2)
Draw 2/2 i.idd samples
from p(w).
Construct feature mapping
(2) vp/2 cos (wf ((2)), ..., cos (w (2)))
sin (wf
sin 21))
(1) T (2)
0. 2,
z;
end for
ÜB = concat (91. .... 9nn)
Compute loss L = (yn, ÿn) ((1) (1)) VV 1() C
((2) 2 3V go C
end for
Return: Predictions ÿ and updated parameters 011,000
Algorithm 3 ICKy Ensemble
Input: data X { {r ! x. (2) ' targets y = [ ensemble Ficky {Ii,.., JN.. where each function
in F consists of fNN, K2 with parameters (1), ( respectively
for = I.., do
Apply proper initialization strategy to Perform Algorithm on function fs end for
Return: predictive mean P = N N'' fa(X), predictive variance 6² = updated parameters ((1), ( for all functions in FicKy
1 N.
LN.. (fa(X) - -A)² and
can be either the Newral Network Gaussian Process (NNGP) kernel (eee al. 2017) if we make only the
last layer trainable the Neural Tangent Kernel (NTK) (He et al. 2020 Jacot et al. 2018 if we add randomized and untrainable function to each baselearner.
This deep ensemble mechanism can be easily applied to ICKy. Specifically, the final prediction of ICKy = fIcky (z(1), (²) can be viewed
weighted sum of ÿ Eh-n ax2k where a 2. g (a (2) Hence, if construct ensemble of ICKy, FicK, {fi, ..., fN. (as shown in Algorithm and fNN is appropriately initialized for all baselearners f. E FiCKy, s = 1,.., Ne, then all trained basclcarners in FICKy can be approzimately viewed as i.i.d. posterior samples from a single-output GP with a multiplicative kernel KgK2. KF will either be an NNGP or an NTK and K₂ comes from our defined kernel. the parameters of INN (i.c. @(I) are independently drawn from a zero-mean Gaussian and are fixed except the last layer (as specified by Lee al. (2019)) for all basclearners in the ensemble, this corresponds an NNGP. A detailed proof is in Appendix

---

Published in Transactions on Machine Learning Research (02/2024)
Observed Data
Observed Data
True mean
True mean
GP mean
ICKy mean
Std of
Std of g
GP uncertainty
ICKy uncertainty
-5
-15
(a)
(b)
Figure 5: Plots of predicted mean and uncertainties for sinusoidal function with exponential damping with noise f(x) e sin(2 + using (a) exact GP with RBF kernel and (b) ICKy with RBF kernel. Here ICKy predicts the uncertainty by directly calculating the covariance matrix E in Equation 17
To verify our argument, we train an ICKy ensemble FICKy containing 300 baselearners on the same ID regression task as provided by He et al. (2020) using SGD optimizer with a learning rate of 5 * 10 and the mean-squared error (MSE) loss until convergence. We then compare the predictive distribution to its GP posterior counterpart based the NNGP implementation in the neural langents (Novak et al. 2020) package. The weights for each baselearner and variance equal to ow//Vd where dy is the width of the corresponding NN layer. The GP posterior has zero mcan and kernel function KNNGP K2, where KNNGP is the corresponding NNGP kernel of fun and K₂ is the chosen kernel function for the mapping in the ICKy ensemble in Figure 4 Here, K₂ is set to be an exponential-siin wwweeentiaaa-ii---eeeeeeeeeeeeee sine--uurre-- kernel with period T = 2n. The predictive distribution ICKy ensemble is very close to the analytic GP posterior, demonstrating the approzimate equivalence between the two models.
initialized to be drawn from i.i.d. Gaussian with
4.3.2 Computing Variance
Another approach to estimate the uncertainty is to directly compute the covariance matrix of the posterior distribution
Σ = K Kurx (Kex + 0²1) K
(17)
where Katz- = KX'', is the kernel matrix evaluated on the test dataset, Kgg K(X,X) is the kernel matrix evaluated on the training dataset, Kz-z K(X*, X) is the cross-covariance matrix evaluated on both training and test datasets, and K K" 22.. Here K refers the composite kernel function implied by ICK. Note that if we get rid of fNN(-) and only keep g(-), then K K₂ and the computation of Σ is straightforward. We believe the true uncertainty can be reasonably approximated by ICKy, indicated by diag(E), while maintaining a reasonable computational complexity. To demonstrate this, we sample 100 data points from a sinusoidal function with exponential decay f(x) = = e - sin(2 + where ~ NO, 0.5) and see how an exact GP and ICKy with radial basis function (RBF) kernel can capture the mean and uncertainty of this function by lcarning from these samples and directly calculating the covariance Σ according to Equation 17. In Figure 3 our results reveal that both GP and ICKy accurately capture the mean function. However, ICKy exhibits prediction of uncertainty closer to the ground truth when compared to GP.
10

---

Published in Transactions on Machine Learning Research (02/2024)
15
Prediction y with plain HLP
Prediction y with M.P-RF
1.5
Prodiction o with ICRV
Thue
0.8
0.8
0.8
0.1
0.2
01
-66
-0.6
-0.
-06
-133
-133
-13
-20
0.0
0.5
1.0
2.0
-2.0
0.3
0.5
10
1.5
2.0
-2.0
0.0
0.5
1.5
2.
0.0
0.5
1.5
25
Figure 6: Prediction of GP(O, K1 K2) where ( is input to a linear kernel K1 and x( is input to a spectral mixture kernel K2. We plot (2) against the predicted We show results from a plain MLP (left), MLP-RF (middle left), and ICKy framework (middle right), and we compare to the true values (right).
Prddition of with plain MLP
Pred ction of with MLP-R
Prediction of with ICS)
Tru values of
1.0
15
2.00
33
07
0.13
X-550
1.8
-25
1.0 xm1
1.5
0.0
1.3
1.3
05
0.0 0.5 3.5
2.0
Km
Figure 7: Prediction of y ~ GP(O, K2 + K2), where x01) is input to linear kernel K₁ and (2) is input to spectral mixture kernel K2. We plot (2)) against. the predicted We show results from plain MLP (left), MLP-RF (middle left), and ICKy framework (middle right), and we compare to the true values of (right).
5
Experiments
We evaluate ICKy on different data sets: synthetic datascts and 4 real-world datasets In all the experiments, our ICKy framework only consists of 2 kernels (i.e. M = 2), one NN-implied kernel and one chosen kernel function with trainable parameters. The implementation details of all the experiments this section are provided in Appendix F and the data accessibility and restrictions are provided in Appendix G
5.1 Synthetic Data
5.1.1 Case when M 2
verify that ICKy can simulate sampling from GP with multiplicative kernel, we create a synthetic data set CP(O, K2) containing 3000 data points where x(1) e '0, 1] the input for the linear kernel K₁ and ((2) [0, 2] is the input for a spectral mixture kernel (ilsson & Adams 2013) K₂ with 2 components.
We compare ICKy with two models: multi-layer perceptron (MLP) applied to the concalenated features and a novel multi-layer perceptron-random forest (MT.P-RF) joint model employed by Zhenn et al. 2021 where MLP learns from (1) and RF learns from (²2), We believe MLP-RE serves as a good benchmark model as is a joint model with similar architecture to our ICKy framework. To see how ICKy simulates the spectral mixture kernel, we plot only (2) against the predicted value of y as shown in Figure As can be seen from the figure, plain MLP only captures the linear trend. MLP-RF only captures the mean of the spectral mixture components. In contrast, our ICKy framework captures both the mean and the variance of the spectral mixture kernel. We also repeat this experiment by sampling from GP with additive kernel and we obtain similar observations as shown in Figure
The code for all the experiments presented in this paper can accessed http:///gittubb.cc///eeeaee cow///yy/3///100/
11

---

Published in Transactions on Machine Learning Research (02/2024)
1.25
1.25
1.25
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
Figure 8: Visualization of (a) True matrix (b) estimated matrix by ICKy framework, and (c) absolute difference between the true and estimated matrix for the spectral mixture kernel
50- Spearman == 0.86
40
Spearman r 0.94
60
Spearman == 0.94
Pearson -- 0.76
Pearson 0.86
Pearson 0.93
RMSE
30
MAE 6.9
MAE 4.86
40
MAE 3.27
20
20
10
Yared
Yprwd
Yared
-10-
-20
30
20
60
Yiue
Viue
Yeue
(a)
(b)
(c)
Figure 9: Scatter plots of the true values of against the predicted values of y using our ICKy framework with information from (a) one source ÿ = fINN (I(1)), (b) 2 sources ÿ = fICKy (a(1), x())), and (c) 3 sources 0 fICky (x(1), (2(2),(())
We also examine whether ICKy can retrieve the spectral mixture kernel in this task. A After fitting the parameters the spectral mixture kernel in ICKy, we compute the kernel matrix KICK, using these learned parameters and compare with the true kernel matrix Kiree by calculating the absolute difference between them as displayed in Figure As can be observed, KICK, and K are similar and their absolute difference is relatively small, indicating that ICKy approximately retrieve the spectral mixture kernel.
5.1.2 Case when M 3
We then construct another synthetic data set with 3000 data points where each input æ = (2)),, 2x(2) (.()) con- tains information from 3 difference sources. The output is generated by 2(3) tanh (2x(1) cos? (TTZ(2) (oxo(2)/55) /50)) + where is Gaussian noise term. We process α(1) with small hngll-hhideenllaer NN, xx(2) with expo- nential sine squared kernel, and (3) with radial-basis function (RBF) kernel. Figure g shows the prediction results as we progressively add more sources of information into our ICKy framework with corresponding kernel functions. It can be observed that ICKy yields both smallest. error and highest correlation with information from all 3 different sources. Hence, ICKy works well with the M = 3 case and the regression performance improved as we add in more information related to the target.
12

---

Published in Transactions on Machine Learning Research (02/2024)
5.1.3 Cases when M
To assess the robustness of ICKy in handling diverse information sources, we conduct "stress tests" by evaluating its performance under vary- ing numbers of information sources, i.e, when M (1, 4, 5). In this analysis, we generate synthetic data following methodology similar to that outlined in Section 5.1 Specifically, we model the data as y ~ GP 0, I K:), -.)KKKK), where K2; K3, K1, and K₁, are set be the spectral mixture kernel, linear kernel, RBF kernel, exponential sinc-squared kernel, and Matérn kernel with 5/2,
Table 1: Prediction errors of ICKy on synthetic dataset generated by GP with multiplicative kernel, i.e., y GP (0.11 K.).
M =
M = 2
M =3 3
M
M =
RMSE MAE
0.0334
0.0440
0.0466
0.2108
0.2560
0.0266
0.0347
0.0353
0.1565
0.1790
respectively. As presented in Table [ when M > 3, the performance of ICKy starts to degrade. We posit that this degradation is primarily attributed to vanishing gradient, phenomenon we will discuss further in the Discussion section.
5.2 Remote Sensing Data
We believe ICKy will be particularly useful for remote sensing applications. In this experiment, evaluate ICKy y on remote sensing datasets where cach data point x = {x, t) contains 2 sources of information: a three-band natural color (red-blue-green) satellite image x as the high-dimensional information and the corresponding timestamp as the low-dimensional information. Our goal is to forecast the ground- PM2.5 concentration 0 = (z, t) using both sources of information.
As PM2.5 varies with time on a yearly basis, we use an cxponenlial-sine-ssuuaa kernel with a period of T = 365 (days) to process the low-dimensional information t. The satellite images processed with a CNN. The results of ICKy are then compared with 2 benchmarks as shown in Table 2 Convolutional Neural Network-Random Forest (CNN-RF) joint model (Zheng et al. 2020 2021 (similar to the MLP-RF model in Section 5.1 and a carefully designed Seasonal CNN-RF model that maps into two new features, sin(2nt/335) and (os(22t/365) to explicitly model seasonality.
We note that the inner product operation in ICK is similar in mathematical structure to attention- based mechanisms (Vswann et al., 2017), which are popular in many deep learning frameworks. Therefore, we introduce 2 variants of ICKy and compare them with another 4 benchmarks where the CNN is replaced by an allention- mecha- nism based off a Vision Transformer (ViT) (Doso- vitskiy et al., 2020) and Dee (Zhou et al. 2021] architectures. These models include VIT- RF, Seasonal ViT-RF, Seasonal DeepViT-RF, and Seasonal MAE- ViT-RF, where ViT is pre- trained by a Masked Autoencoder (He et al.) 2022 "Seasonal" here denotes that we use the transformed periodic representation defined pre- viously. We note that we are unaware of Vision Transformers being used in this manner, and we want to evaluate whether it is the model structure the prior information that is improving the results. Hence, these attention-baser models, while not primary contribution, are novel and represent a good-faith effort to define models with similar forms. As displayed in Table 2 both standard CNN-RE and VIT-RE models yields very large errors on predicting PM2.5 values. After including seasonality, CNN-RF performs significantly better and shares similar predictive performance with CNN-ICKy. After replacing CNN with VIT, we observe slight improvement in both RF-joint models (especially when pre-trained by MAE) and ICKy variants. Among the models present here, ViT-ICKy and DeepVrT-ICKy achieve the highest correlation
Table Correlation and error statistics of ICKy and other joint deep models with both convolutional and attention- based urchitectures the PM2.5 forecasting task. "S." denotes seasonal variants.
RSpear
RMSE
MAE
MSLL
CNN-RF
0.00
194.63
185.83
VIT-RF
0.07
190.82
181.63
CNN-RF
0.62
53.36
39.38
96.77
S. VIT-RF
0.66
56.45
41.73
14.69
S. Deep- VYT-RF
0.65
56.36
42.46
17.63
S. MAE-VIT-RF
0.67
53.87
40.78
31.09
CNN-ICKy
0.62
53.46
39.76
10.92
VIT-ICKy DeepViT-ICKy
0.68
56.56
41.41
12208
0.66
52.41
35.93
38220
13

---

Published in Transactions on Machine Learning Research (02/2024)
Table 3: Prediction error of actual worker produc- tivity on the test data set with ICKy and other benchmark models.
Table 4: Roob-meaa--square--rro (RMSE) and nega- live log-likelihood (NLL) of ICKy with Matern-3/2 kernel and periodic kernel and two GP benchmarks on power consumption data.
MSE 4 (+10 3)
4 3)
MLP Cyclic MLP
20.16 + 1.26
9.93 0.36
20.97 + 1.98
10.16 +: 0.77
RMSE
NLL
Cyclic MLP-RF
19.05 + 1.36
9.70 = 0.48
Exact GP SVGP
0.055 + 0.000
-0.152 = 0.001
DKL
21.40 1 2.85
11.14 1 0.94
0.084 + 0.005
-1.010 = 0.039
ICKy, 2
3.43 1 1.42
4.85 = 1.00
ICKy, Matérn-3/2
0.036 1 0.000
-1.424 + 0.000
ICKy, T=
0.44 + 0.13
1.43 0.15
ICKy, periodic
0.033 + 0.000
-1.737 + 0.000
ICKy. =
0.31 + 0.09
1.17 + 0.14
and the smallest prediction error, respectively. To evaluate the uncertainty calibration capability of ICKy, we construct ICKy ensembles by following Algorithm X and compare them with ensemble formulations of RF-joint model beuchmarks. We use a criterion called Mean Standardized Log Loss (MSLL) defined in Williams & Rasmuussn (2006) to evaluate the uncertainty calibration. From Table 2 it can be observed that CNN-ICKy ensemble achieves the smallest MSLL. In addition, we realize ViT-based ICKy variants yield very large MSLL as they make predictions which are far from the truc PM2.5 labels with high confidence (i.c.
small variance). We provide details and possible explanations in Appendix H
5.3 Other Regression Datasets
To see if our ICKy framework generalizes to other domains, we acquired two additional regression datasets from the UCI machine learning repository, one for predicting worker productivity (with 1197 samples and 15 features) and the other for predicting power consumption (with 2075259 samples and 9 features). For the worker productivity dataset, we employ ICKy with an exponential-sine-squuee
kernel with different periods and compare them with benchmarks including MLPs (Al Imran et al. 2019 and Deep Kernel Learning (DKL) WWilson et al.) 2016b) as shown in Table For the power consumption dataset, we use ICKy with both an exponential- wewponenniillsine--sppeeee sine-squared kernel and a Malérn 3/2 kernel and compare them with scalable exact GP (Wang al. 2019) and stochastic variational GP (Hensman al. 2013) as shown in Table From the results, observe that ICKy outperforms all the benchmarks in both experiments (especially the worker productivity data with margin of almost one order of magnitude).
5.4 Adapting ICK for Classification
While regression tasks are the primary motivation for this paper, there are many ways to adapt GPR for classification tasks. For example, a binary classification model can be created by using a sigmoid (Williams & Barber 1998) or probit link (Choudhuri et al., (007)) on the output of the GP. Succinetly, given a function [(x) GP (0, (xxx a²),, the binary outcome probability is be given as p(y 1x)())) (x(x))) Likewise, multiple classification model can be constructed by using multi-output GP (or multiple GPs) and putting the outputs through a softmax function (Williams & Barber 1998) or multinomial probit link (Girolami &
Rogers 2006). This strategy can be summarized by calculating C' different functions fe(x)) GP (0, K (x x' for c 1,.... C, where C' is the number of classes, and then calculating the class probabilitics through a link function, p(y[=) = softmax([fi (x), f2(x), fc(æ))).
This same logic can be used to construct a multiple classification model from ICKy. Succinetly, let re- (x))) C x where fNN,e denotes a neural network specific to the th class and (2)) represents the Nyström approximation specific to the kernel for the ea class. We note that often in a multi-output case the kernel parameters are shared, and so (2)) would be an identical vector for each class. Then, the output probabilities for a data sample as p(y]æ)) = (oltmax([n,....el)))) This framework is learned with a cross-entropy loss.
To provide proof-of-concey of this multiple classification strategy, we implemented this model on a version of Rotating MNIST. In this task, dataset was created by rotating each image in the dataset by uniform
14

---

Published in Transactions on Machine Learning Research (02/2024)
random value & [0,2 thus creating dataset with 60,000 images each with associated rotation covariate Φ. We implemented the above multiple classification model with periodic kernel over the rotation angle. This strategy yielded an accuracy of 92.3% on the validation data. This is lower than methods such as spatial transformers (Jaderberg et al. 2015) that report accuracy greater than 99%. However, those models explicitly use the fact. that the information is simply rotated, whereas ICK is modeling smooth transformation the prediction function as function of angle. This ICK classification model is much closer in concept the way Rotating MNIST is used to evaluate unsupervised domain adaptation. While the evaluation strategy is different than random validation set, the state-of-the-art accuracy unsupervised domain adaption is 87.1% (Wang et al. 2020). Due to the lack of complete and fair comparisons, we are not claiming that ICKy is state-of-the-ar for classification, but ICKy's classification model does seem reasonable and viable based upon this result.
6 Discussion
Computational Complexity and Flexibility of ICK Compared to exact composite GP models which scale O(N%), the training process of our ICK framework is more elficient as it leverages standard backpropagation to learn both the paramters of NN and the kernel function. Specifically, let Ng be the number of data points in one mini-batch and p be the number of inducing points for Nyström approximation. The computational complexity for g(-) is Opp + + The computational complexity for the forward pass of (-x(.) is O (Nn(k ... -k))) where hi,,... kr. are the widths of each layer for neural network of depth L. In addition, our ICK framework is more flexible compared to other joint models (i.e. BNNs and CNN-RF). Specifically, the BNNs of Pearce et al. 2020) cannot simulate complicated kernels such as the spectral mixture kernel we use in Section 5.1
Limitations As described in Section 5.1.3 ICKy starts to yield higher prediction error when we have more than 3 different sources of information. This phenomenon is likely to be caused by vanishing gradient may need to multiply small numbers logether due to the nature of inner product. Furthermore, in some cases, the properties of predictive posterior given by ICK ensemble are dominated by the neural network as shown in Appendix H It can sometimes be challenging to choose an appropriate NN architecture to make sure does not interfere with other specified kernels. In addition, as highlighted in Section 8.3.2 Williams & Rasmussen (2006), it is important to note that the predictive variance using the Nyström method is not guaranteed to be positive. This can result in the kernel matrix no longer being positive-semidefnite,, making the Cholesky decomposition infeasible. In such cases, we recommend either reducing the rank of the Nyström approximation or experimenting with alternative kernel functions.
Broader Impacts We believe our framework is extensively applicable to regression problems many fields of study involving high-dimensional data and multiple sources of information with perceptible trends, such as remote sensing, spatial statistics, clinical diagnosis. Also, we are not aware of any negative societal impacts of our work.
7 Conclusion
This paper presents novel yet surprisingly simple Implicit Composite Kernel (ICK) framework to learn from hybrid data containing both high-dimensional information and low-dimensional information with prior knowledge. We first analytically show the resemblance between ICK and composite GPR models and then conduct experiments using both synthetic and real-world data. It appears that ICK outperforms various benchmark models in our experiments with lowest prediction errors and highest correlations even with very limited data. Overall, we show that our ICK framework is exceptionally powerful when learning from hybrid data with prior knowledge incorporated, and we hope our work can inspire more future research on joint machine learning models, enhancing their performance, efficiency, flexibility, and generalization capability.
15

---

Published in Transactions on Machine Learning Research (02/2024)
References
Ben Adlam, Jachoon Lee, Lechao Xiao, Jeffrey Pennington, and Jasper Snock. Exploring the uncertainty properties of neural networks' implicit priors in the infinite-width limit. arXiv preprint ar 07355, 2020.
Ben Adlam, Jachoon Lee, Shreyas Padhy, Zachary Nado, and Jasper Snoek. Kernel regression with infinite- width neural networks on millions of examples. arXiv preprint arXiv:2303.054 2023.
Abdullah Al Imran, Md Nur Amin, Md Rifatul Islam Rifat, and Shamprikta Mehreen. Deep neural network approach for predicting the productivity of garment employees. In 2019 6th International Conference Control, Decision and Information Technologies (CoDIT), pp. 1402 1407. TEEE, 2019.
Christopher M Bishop and Nasser M Nasrabadi. Pullern recognition and machine learning, volume 4. Springer, 2006.
Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for largo-scale machine learning. Siam Review, 60(2):223 311, 2018.
Mohamed A Bouhlel and Joaquim RRA Martins. Gradient-enhancee kriging for high-dimensional problems. Engineering with Computers, 35(1):157-173 2019.
Mohamed Amine Bouhlel, Nathalie Bartoli, Abdelkader Otsmane, and Joseph Morlier. Improving kriging surrogates of high-dimensional design models by partial least squares dimension reduction. Structural and Multidisciplinary Optimization, 53(5):933-955 952, 2016.
Nidhan Choudhuri, Subhashis Ghosal, and Anindya Roy. Nonparametric binary regression using gaussian process prior. Statistical Methodology, 4(2):227-243) 2007.
Abhirup Datta, Sudipto Banerjee, Andrew O Finley, and Alan E Gelfand. Hierarchical nearest-neighhor gaussian process models for large geostatistical datasets. Journal of the American Statistical Association, 111(514):800 812, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.111 11929, 2020.
Petros Drineas, Michael W Mahoney, and Nello Cristianini. On the nyström method for approximating a gram matrix for improved kernel-based learning. journal of machine learning research, 6(12), 2005.
David Duvenaud. Automotic model construction with Gaussion processes. PhD thesis, University of Cambridge, 2014.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Tch. Neural processes. arXiv preprint arXiv: (ah:::887.00000 01622, 2018.
Adrià Garriga-Alonso, Carl Edward Rasmusson, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. orXiv preprint Xix::11808. (55::888800577, 2018.
Alan Gelfand and Erin M Schliep. Spatial statistics and gaussian processes: A beautiful marriage. Spatial Statistics, 18:86-104 2016.
Mark Girolami and Simon Rogers. Variational bayesian multinomial probit regression with gaussian process priors. Neural Computation, 18(8):1790 1817, 2006.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian deep ensembles via the neural tangent kernel. Advances in neural information processing systems, 33:1010-1022, 2020.
16

---

Published in Transactions on Machine Learning Research (02/2024)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEER/CVE Conference on Compuler Vision and Pallern Recognition, pp. 16000 16009, 2022.
James Hensman, Nicolo Fusi, and Neil D Lawrence. Gaussian processes for big data. ar preprint arXiv:1309.6es 1305.6855, 2013.
Geoffrey E Hinton and Russ Rt Salakhutdinov. Using deep belief nets to learn covariance kernels for gaussian processes. Advances in neurul information processing systems, 20, 2007.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural langent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
Ziyang Jiang, Tongshu Zheng, Mike Bergin, and David Carlson. Improving spatial variation of ground-lvvel pm2. 5 prediction with contrastive learning from satellite imagery. Science of Remote Sensing, pp. 100052, 2022.
Andre G Journel and Charles Huijbregts. Mining geostalislics. The Blackburn Press, 1976.
Hyoung-Moon Kim, Bani K Mallick, and Chris C Holmes. Analyzing nonstationary spatial data using
piecewise gaussian processes. Journal of the American Statistical Association, 100(470):633-6) -688, 2005.
Daniel G Krige. A statistical approach to some basic mine valuation problems on the witwatersrand. Journal of the Southern African Institute of Mining and Metallurgy, 52(6):119 139, 1951.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on newral networks, 9(5):987 1000, 1998.
Yann LeCun, Yoshua Bengio, and Gooffrey Hinton. Deep learning. nature, (7153))) 2015.
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schocnholz, Jeffrey Pennington, and Jascha Sohl- Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXi::1711.00165 2017.
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Dick-Dicks,,in, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient. descent. Advances in neurol information processing systems, 32, 2019.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continnous relaxation of discrete random variables. arXiv preprint ar orxi::6611.007 2016.
Gary Marcus. Deep learning: A critical appraisal. arXiv preprint arXiv:1801.0033/ 2018.
Takuo Matsubara, Chris Oates, and François-Xavier Briol. The ridgelet prior: covariance function approach to prior specification for bayesian neural networks. arXiv preprint arXiv:2010.08.0 2020.
Alexander G de G Matthews, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Sample-then-optmmize posterior sampling for bayesian linear models. In NewrIPS Workshop on Advances in Approcimale Bayesian Inference, 2017.
Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E, Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint (xXi::1004.1120 2018.
Brian McFee, Gert Lanckrict, and Tony Jebara. Learning multi-modal similarity. Journal of machine learning research, 12(2), 2011.
Ben Moseley, Andrew Markham, and Tarje Nissen-Meyer Solving the wave equation with physics-informed deep learning. Xi preprint orXi::006. 11894, 2020.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Radford M Neal. Priors for infinite networks. In Bagesion Learning for Neural Networks, pp. 29-53. Springer, 1996.
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. arXiv preprint 1rXiv:1810.0511 2018.
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schocnholz. Neural tangents: Fast and casy infinite neural networks in python. In International Conference on Learning Regresenlalions, 2020. URL
Tim Pearce, Russell Tsuchida, Mohamed Zaki, Alexandra Brintrup, and Andy Neely. Expressive priors in bayesian neural networks: Kernel combinations and periodic functions. In Uncertainty in artificial intelligence, pp. 134 144. PMLR, 2020.
Dejan Petelin, Alexandra Grancharova, and Juš Kocijan. Evolving gaussian process models for prediction of Ozone concentration in the air. Simulation modelling practice and theory, 33:68 80, 2013.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances newral information processing systems, 20, 2007.
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in newral information processing systems, 21, 2008.
Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. Advances in neural information processing systems, 18, 2005.
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics, pp. 567-574. PMLR, 2009.
Mark Van der Wilk, Carl Edward Rasmussen, and James Hensman. Convolutional gaussian processes. Advances in Neural Information Processing Systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. International Conference Machine Learning, 2020.
Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gordon Wilson. Exact gaussian processes on a million data points. Advances in neural information processing systems, 32, 2019.
Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. Advances in neural information processing systems, 13, 2000.
Christopher K Williams and Carl Edward Rasmusson. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
Christopher KI Williams and David Barber. Bayesian classification with gaussian processes. IEEE Transactions on pallern analysis and machine intelligence, 20(12):1342 1351, 1998.
Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapolation. In International conference on machine learning, pp. 1067-1075. PMLR, 2013.
Andrew G Wilson, Zhiting Hu, Russ R Salakhutdinov, and Eric P Xing. Stochastic variational deep kernel learning. Advances in newral information processing systems, 29, 2016a.
Andrew Gordon Wilson, David A Knowles, and Zoubin Ghahramani. Gaussian process regression networks. Xix preprint or Xiv: 1110.4411, 2011.
18

---

Published in Transactions on Machine Learning Research (02/2024)
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016b.
Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Hua Zhou. Nyström method random fourier features: A theoretical and empirical comparison. Advances in neural information processing systems, 25, 2012.
Daoqiang Zhang, Yaping Wang, Luping Zhou, Hong Yuan, Dinggang Shen, Alzheimer's Disease Neuroimaging Initiative, al. Multimodal classification of alzheimer's discase and mild cognitive impairment. Neuroimage, 55(3):856 867, 2011.
Tongshu Zheng, Michael H Bergin, Shijia Hu, Joshua Miller, and David E Carlson. Estimating ground- level pm2. using micro-satellite images by a convolutional neural network and random forest approach. Atmospheric Environment, 230:117451, 2020.
Tongshu Zheng, Michael Bergin, Guoyin Wang, and David Carlson. Local pm2. 5 hotspot detector at 300 m resolution: A random forest convolutional neural network joint model jointly trained on satellite images and meteorology. Remote Sensing, 13(1513356, 2021.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:103, 1 11866, 2021.
19


Reviews:
[{"review_id": "BfihXb8rV4", "reviewer": "Reviewer_b4Hq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is written clearly and is easy to follow.\n\n\n**Weaknesses**\n\n[Connection between ICK and GPR]\n\nThe paper does not deliver fully on its stated contributions. The equivalence to Gaussian process regression is not handled correctly. The posterior of a finite-width NN trained with SGD is not a GP in general. Algorithm 3 makes no assumptions about the scale of the weights at initialization, the learning rate, the number of training steps, or the loss function. To be in the lazy training regime, assumptions are needed on all these. \n\n[Experiments]\n\nI'm quite confused by Section 5.3. Why were these datasets selected and not other UCI datasets, since they seem to contain no high-dimensional component? I also do not understand why the methods in Tables 2 and 3 were selected to compare against. Are they the current SotA? Are the numbers taken directly from the papers cited or were the methods reimplemented? I am also not able to find the numbers reported for DKL in the citation given. Even the comparison to Wilson in Table 3 seems to report a different number of datapoints (1,311,539). \n\n\n\n**Minor issues**\n- Novelty. ICK is a straightforward combination of existing methods, namely, featurizing inputs with a NN, multiplying kernels, and Nystrom/RFF approximation. \n- Note that when using a Nyström approximation to the kernel, there is no guarantee that the variances will be positive. This is a known issue and should be discussed. See chapter 8 of [1].\n\n\n1. Gaussian Processes for Machine Learning. Carl Edward Rasmussen and Christopher K. I. Williams. The MIT Press, 2006. ISBN 0-262-18253-X.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes the implicit composite kernel method. This method is used to model datasets that contain both low- and high-dimensional data, where prior information is available. The authors attempt to draw a connection between their approximate method and Gaussian process regression. They evaluate the method on a synthetic task, showing that it can capture specific patterns. They also evaluate on a remote sensing dataset and two UCI regression tasks–demonstrating competitive performance compared to their benchmarks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There are a number of missing citations that should be included in future versions of the paper. \n- [1] should be cited on pages 2 and 3 for the connection between CNNs and GPs at infinite width.\n- [2], [3] and [4] are other examples of combining NNs and GPs.\n- It might be worth mentioning the line of work using iterative solvers to scale GPs to large datasets in the related work section. For example [5], even though this is cited in other places. See [6] also.\n\n1. Novak, Roman, et al. \"Bayesian deep convolutional networks with many channels are gaussian processes.\" arXiv preprint arXiv:1810.05148 (2018).\n2. Hinton, Geoffrey E., and Russ R. Salakhutdinov. \"Using deep belief nets to learn covariance kernels for Gaussian processes.\" Advances in neural information processing systems 20 (2007).\n3. Wilson, Andrew G., et al. \"Stochastic variational deep kernel learning.\" Advances in neural information processing systems 29 (2016).\n4. Adlam, Ben, et al. \"Exploring the uncertainty properties of neural networks' implicit priors in the infinite-width limit.\" arXiv preprint arXiv:2010.07355 (2020).\n5. Wang, Ke, et al. \"Exact Gaussian processes on a million data points.\" Advances in neural information processing systems 32 (2019).\n6. Adlam, Ben, et al. \"Kernel Regression with Infinite-Width Neural Networks on Millions of Examples.\" arXiv preprint arXiv:2303.05420 (2023).", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zHbPRI5IfF", "reviewer": "Reviewer_dxRk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is well-written, and the core ICK architecture (along with its several variations) is clearly presented with the aid of neat figures and derivations. The problem statement is compelling, and I appreciated the sections describing how this paper relates to existing work on composite GPs and NNs incorporating prior knowledge.\n- I appreciated that the authors took the time to show how ICK can be adapted to classification tasks, although it also come across as rushed - statements such as *\"ICK’s classification model does seem reasonable and viable”* are somewhat inconclusive, and I interpreted this as an indication of inconclusive results.\n- In the Limitations section, the authors indicate that ICK may not perform as well when there are a large number of information source. This claim alludes to an experiment that “stress tests” the model’s performance under this condition, but this isn’t explored in the current set of experiments.\n- The paper’s motivation centres around datasets characterised by *\"hybrid information”*, but the paper only contains one proper real-world example that demonstrates how this is suitable. The claim that *\"ICK will be particularly useful for remote sensing applications”* is commendable, but limited in scope. I would appreciate more insight, and preferably additional experiments, that showcase other real-world domains where the use of ICK is expected to be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors develop a composite kernel framework that allows for combining the expressive nature of standard GP kernels (that can be used to model properties such as periodicity) with the flexibility of learning from large dimensions that is typically better suited to neural network architectures. Under this set-up, different subsets of the input domain can be treated independently by different kernels - these latent spaces are then combined via a chained inner product of the extracted representations. The full posterior distribution can then be approximated using either ensembling or analytically computing the covariance matrix of the posterior distribution. The suitability of the model is assessed on a variety of synthetic and real-world regression problems, where the proposed ICK framework outperforms competing models. An illustrative example of how the framework can be adapted to classification problems is also included in the paper, where an experiment is performed on the Rotating MNIST dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I highlighted the weaknesses of this work in the previous section. Although the paper is well-written overall, I find that the experiments and real-word examples are currently very limited in light of existing methods that already enable fairly similar composition of kernels. More ablation studies would also be beneficial for showcasing the extent to which the method is extendable to multiple dimensions.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VzjAqE5EsX", "reviewer": "Reviewer_tUmv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- In general, the paper is well-written. Background information and relevant literature are adequately covered. Discussions about the strengths and limitations of the approaches are clearly outlined.\n- The technical developments of the work are rigorous, intuitive, and well-founded from existing works. Proofs are rigorous and correct, and algorithms and experiments are well-thought-out (several approaches to compute the kernel-to-latent-space transformation and uncertainty estimations; various neural network models and learning contexts).\n- The paper introduces several major technical contributions to the problem of interest, including:\n    - the introduction of the Implicit Composite Kernels (ICK) to approximate GP kernels\n    - a theoretical proof (Theorem 1) that the ICK framework, under reasonable assumptions, is approximately equivalent to sampling from a Gaussian process regression (GPR) model with a composite kernel *a priori*\n    - compared to exact composite GP models, the training process of the ICK framework is more efficient as it leverages standard back-propagation to learn both the parameters of NN and the kernel function\n    \n    The proposed approaches would be of broad interest in the machine-learning community from both theoretical and algorithmic viewpoints.\n    \n- The efficacy of the approaches is validated across several synthetic (2) and (4) real-world datasets and learning contexts (both regression and classification). The experiments show that the new approaches yield better performance on prediction and forecasting tasks, even with limited data.\n\n\n### Weaknesses\n\n- No weakness was noted.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of incorporating prior knowledge from multiple sources of information through implicitly defined Gaussian Process composite kernels. Instead of implementing the composite kernels explicitly, the paper proposes to use the inner product between two representations, one generated by a neural network and one by an efficient mapping function based on either Nyström approximation or random Fourier features. The manuscript then adopts a sample-then-optimize approach to approximate the full GP posterior distribution and investigates the efficacy of the approach on several synthetic and real-world datasets and learning contexts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "n/a", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces the Implicit Composite Kernel method, combining neural networks for high-dimensional data modeling with kernels for encoding prior knowledge. The subject of the paper is of importance to the ML community and is an area worthy of further investigation. However, as reviewers suggest, the paper would benefit from more comprehensive experiments and ablation studies.
Decision: Accept (poster)