Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel
Authors: Ziyang Jiang
Tongshu Zheng
David Carlson
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Neural network
Gaussian process
Composite kernel
Nystrom method

Abstract: It is challenging to guide neural network (NN) learning with prior knowledge. In contrast, many known properties, such as spatial smoothness or seasonality, are straightforward to model by choosing an appropriate kernel in a Gaussian process (GP). Many deep learning applications could be enhanced by modeling such known properties. For example, convolutional neural networks (CNNs) are frequently used in remote sensing, which is subject to strong seasonal effects. We propose to blend the strengths of deep learning and the clear modeling capabilities of GPs by using a composite kernel that combines a kernel implicitly defined by a neural network with a second kernel function chosen to model known properties (e.g., seasonality). Then, we approximate the resultant GP by combining a deep network and an efficient mapping based on the Nystrom approximation, which we call Implicit Composite Kernel (ICK). ICK is flexible and can be used to include prior information in neural networks in many applications. We demonstrate the strength of our framework by showing its superior performance and flexibility on both synthetic and real-world data sets. The code is available at: https://anonymous.4open.science/r/ICK_NNGP-17C5/. 

Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel
Anonymous Author(s)
Affiliation
Address
email
Abstract
It is challenging to guide neural network (NN) learning with prior knowledge. In contrast, many known properties, such as spatial smoothness or seasonality, are straightforward to model by choosing appropriate kernel in Gaussian process (GP). Many deep learning applications could be enhanced by modeling such known properties. For example, convolutional neural networks (CNNs) are frequently used in remote sensing, which is subject to strong seasonal effects. We propose to blend the strengths of deep learning and the clear modeling capabilities of GPs by using a composite kernel that combines a kernel implicitly defined by a neural network with second kernel function chosen to model known properties (e.g., seasonality). Then, we approximate the resultant GP by combining a deep network and an efficient mapping based on the Nyström approximation, which we call Implicit Composite Kernel (ICK). ICK is flexible and can be used to include prior information in neural networks in many applications. We demonstrate the strength of our framework by showing its superior performance and flexibility on both synthetic and real-world data sets. The code is available at: |htpps; //anonymoos 4open. science/ r/ICK IItCee/IIIIINNNGN1755555
2
4
6
10
11
13
14
15
Introduction
18 In complex regression tasks, input data often contains multiple sources of information. These sources
19 can be presented both nigh-dimensional (e.g. images, audios, texts, etc.) and low-dimensional
20 (e.g. timestamps, spatial locations, etc.) forms. A common approach to learn from high-dimensional information is to use neural networks (NNs) 33) as NNs are powerful enough capture the relationship between complex high-dimensional data and target variables of interest. In many areas,
21
23 NNs are standard practice, such as the dominance of Convolutional Neural Networks (CNNs) for
24 image analysis [26 61 62) In contrast, for low-dimensional information, we usually have some prior knowledge on how the information relates to the predictions. As a concrete example, consider remote sensing problem where predict ground measurements from satellite imagery with associated timestamps. priori, we expect the ground measurements to vary periodically with CNN
25
27
respect time between summer and winter due to seasonal effects. We would typically 29 to capture the complex relationship between the imagery and the ground measurements. In this case, want to guide the learning of the CNN with our prior knowledge about the seasonality. This is challenging because knowledge represented in NNs pertains mainly to correlation between network
31
32
units instead of quantifiable statements (66]
33 Conversely, Gaussian processes (GPs) have been used historically to incorporate relevant prior beliefs
by specifying the appropriate form of its kernel function (or covariance function) 2 [4]. One 35 approach to modeling multiple sources information to assign a relevant kernel function to each source of information respectively and combine them through addition or multiplication, resulting in
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

37 composite function [14. This formulation means that specifying kernel to match prior beliefs
38 on one source of information straightforward. Such composite kernel learning techniques are
39 extensively used in many application areas such as multi-media data [4),, neuroimaging [60], spatial
40 data analysis, and environmental data analysis [2 44 In view of the clear modeling capabilities of
41
GP, it is desirable to examine how a NN could be imbued with the same modeling ease.
42 In recent years, researchers have come up with a variety of methods to incorporate prior knowledge
43
into NNs. These efforts can be broken into many categories, such as those that add prior information 41 Here, we focus on the major category of those
44
through loss terms like physics-informed NNs 32 methods that build integrated models of NNs and GPs with various structures [50 57 58] Related
45
46
to our proposed methodology, Pearce al. 43 exploited the fact that a Bayesian neural network (BNN) approximates a GP to construct additive and multiplicative kernels, but they were limited to specific predefined kernels. Matsubara et al. [38 then resolved this limitation by constructing
47
48
49 priors of BNN parameters based on the ridgelet transform and its dual, but they did not explicitly show how their approach works for data with multiple sources of information. To our knowledge, none of these existing approaches allows a modeler to choose any appropriate kernel of known prior information from multiple sources. We address this limitation by presenting a simple yet novel Implicit Composite Kernel (ICK) framework, which processes high-dimensional information using kernel implicitly defined by a neural network and low-dimensional information using a chosen kernel function. The low- dineensiiioall kernels are mapped into the neural network framework to create straightforward and simple-to-leamd implementation. Our key results and contributions are:
50
51
52
53
54
56
57
We analytically show our ICK framework, under reasonable assumptions, is approximately equivalent to a Gaussian process regression (GPR) model with a composite kernel a priori.
58
59
We demonstrate that our ICK framework yields better performance on both prediction and forecasting tasks even with very limited data.
60
61
We compare to joint deep learning models, such as a neural network-random forest joint model, to show that ICK can flexibly capture the patterns of the low-dimensional information without deliberately designing pre-processing procedure or complex NN structures.
62
63
64 Based on these contributions, we believe ICK will be useful in the context of learning from complex hybrid data with prior knowledge, especially in the field of remote sensing and spatial statistics.
2 Related Work
66
Equivalence between NNs and GPs The equivalence between GPs and randomly initialized single- 68 layer NNs with infinite width was first shown by Neal [2). With the development of modern deep learning, researchers further extended this relationship to deep networks [34 3D and convolutional
67
69
70
neural networks (CNNs) 17 This relationship is crucial for proving the resemblance between GPR and our ICK framework, which will be discussed in Section 4.1
71
72 NNs with prior knowledge As mentioned before, one approach to equip NNs with prior knowledge
73 to modify the loss function. For example, Lagaris al. (32) solved differential equations (DEs)
74
using NNs by setting the loss to be function whose derivative satisfies the DE conditions. Another approach is to build integrated models of NNs and GPs. For example, Wilson et al. (58) implemented a regression network with GP priors over latent variables and made inference by approximating
75
76
77
the posterior using Variational Bayes or sampling from the posterior using Gibbs sampling scheme.
78 Gamelo et al. [16] introduced a class of neural latent variable models called Neural Processes (NPs)
79 which are capable of learning efficiently from the data and adapting rapidly to new observations. Zhu
80 et al. 100 proposed NeuralEF which can accurately approximately kernel functions by using series
81
of objective functions parameterized by NNs under the principle of gegen-dcoompoiiionn
82
GP with composite kernels Composite kernel GPs are widely used in both machine learning
83
14
and geostatistical modeling 18] GPR in geostatistical modeling is also known as kriging
84
27
31,
which serves as surrogate model to replace expensive function evaluations. The inputs for
85
a composite GP are usually low-dimensional (e.g. spatial distance) as GPs do not scale well with the
86
number of samples for high-dimensional inputs 4 5. To overcome this issue, Pearce et al. [3 and
87 Matsubara et al. (38) developed BNN analogue for composite GPs. Similar to these studies, our ICK
framework can also be viewed as simulation for composite GPs.
2

---

89
Approximation methods for GP For large data sets, approximation methods are needed as exact
kernel learning and inference scales ( Nyström low-rank matrix approximation [2] [33 and
90
91
Random Fourier Features 45 46 are two of the most commonly used methods. A common technique
92 choose inducing points as pseudo-inputs to efficiently approximate the full kernel matrix [49 23
93 Our work is inspired by these approximation techniques and we use them as transformation
funtirittiiooonnniiiii functions
94
to map the kernel matrix into latent space representations in Section 4.2
Background
95
96
Before elaborating on the details of our ICK framework, we introduce our notation, briefly go over the concepts of composite GPs, and describe the relationship between GPs and NNs.
97
3.1 Problem Setup
99
To formalize the problem, we have a training data set which contains N data points X le [21, 22 ...., ÆNl" and the corresponding labels of these data points are y [
(1), 1/2 ... uN] where ys E IR. Each data point æ; - 2 (2) Li (M) is composed of informa- tion from M different sources where the m source of information of the 2th data point is denoted as 103 J. (m)) E RP' Our goal is to learn function Ú; = =(xx)) {IR D2 RDw} } + R which takes
in data point æ; and outputs a predicted value t/.
105
3.2
Composite GPs
106
Gaussian process (GP) describes distribution over functions (54] A key property of GP is that it can be completely defined by mean function p(x) and a kernel function K (æx x'). The mean function u(x) often assumed to be zero for simplicity. In that case, the outcome function
107
108
f(x) ~ GP (0, K (xxx)))
109
Any finite subset of these random variables has multivariate Gaussian distribution with mean and kernel matrix K whose entries can be calculated as K K (æ; 22). ). In many situations, the full kernel function is built by a composite kernel by combining simple kernels through addition
110
111
112
K Keomp( (x, x²) = K2 æ²) + K2(x, x²) or multiplication Koomp(z: (2,x')
K1 (x, x K2(x, x') 14). useful property that ICK exploits is that K₁ and K₂ can take different subparts of as
113
their inputs. For example, Keomp xx, K₃
+ Kg ((()),,())))
Keomp(z: x²)
K1 (1()) 2,(1))) K2 (z(2), x2))) Other methods such as functional mapping are also valid if the
115
116
resulting kernel matrix K is positive semidefinite (PSD) for all possible choices of data set X 47
117
3.3
Correspondence between GPs and NNs
proved that single-hidden layer network with infinite width is exactly equivalent to GP over data indices = 1,2,..., N under the assumption that the weight and bias parameters of the hidden layer are i.i.d. Gaussian with zero mean. Lee et al. 34 and Garriga al. [I7] then extended this statement to deep neural networks and convolutional neural networks (CNNs), respectively. However, if the width (or the number of channels) of a network is finite, then these results state that the network approximately converges to GP with zero mean as in the following lemma.
Neal
42
119
121
122
123
Lemma =et fNN (z(1)) Rt - HR be the latent representation extracted from z(1) where is the dimension of the extracted representation and fww is neural network with finite width and zero-mean i..dd parameters. The with entry of this representation will approximately follow NN-implied GP
124
125
126
127
2k = fNN (a()) ~ GP Papppno (0. (2t), (0)))))))
(2)
That is say, the 4 component 2y of the representation extracted by the network has zero mean (1) 0 for all 1,2, N where 0 represents the network parameters. The co- (1) (1) variance between 2 and 2jk for different data indices i, = 1, ..,, can be approximated as
128
129
E
(((a)))
3

---

xx1 R
x
x(11 8
x
Given data contain- ing 2 sources of information 2(1) and x²²2, we can process the data using either (Left) a com- posite Gaussian process regres- sion (GPR) model or (Right) ICK framework where x(1) is processed with a neural net- work [NN(-) and x(22 pro- cessed with g(-) where g(-) con- sists of a kernel function K₂ and some transformation which maps the kernel matrix K₂ into the latent space.
Figure 1:
Kg0r.a7)
K
K
Ng(oxx))
K2
Combite
Keon
200 R2
200 3P
9 9- ((100)2)0) (
[ (1) (1)}
Pik 3sk = KNN (1) ())
(1))
(1) (23)) where x; and zj are the correspond-
Cov
Fike
3jk
= E (((O)))
132
ing inputs for the network and KNN is the kernel function implied by the network.
Implicit Composite Kernel (ICK) Framework
135
134
We show the structure of a composite GPR model and our ICK framework in Figure | To make the illustration clear, we limit ourselves to data with information from 2 different sources
135
(1),, (2) where (1)) is high-dimensional and x2 is low-dimensional (i.c. Dj >> D2) with some known relationship with the target y. We are inspired by composite GPR, which computes 2 different kernel matrices K1 and K₂ and then combines them into a single composite kernel matrix K Kaump- However, as discussed before, it is more suitable to use a NN learn from the high dimensional information (²(2), In our ICK framework, we process
136
137
138
139
(2()) with a NN (NN(.) Ri Rp and (2) with a mapping g(-) D= + R' which consists of a kernel function K₂ followed by a kemel- to- loeennett-epace transformation (described in Section 4.2). resulting in two latent representations (1), 22)) R Then, make prediction ú by doing inner product between these two representations I SNN (z!41) 9 (z!²)). Finally, the parameters of both the NN and the kernel function are learned via gradient-bas based optimization methods 3 30
140
142
145
148 In the sections below, we first analytically show that our ICK framework is approximately equivalent to a composite GPR model a priori using multiplicative kernel between the kernel implicitly defined by the NN on ({1 and the chosen kernel on 22(2) This theory is used motivate the model form. The model will deviate from the GP solution after learning, but we note that recent work suggests that the predictions from trained NN may not vary too much from its GP equivalent B4 We then
147
148
149
150
151
show how we implement the kernel-to wote-tto-llatt-eeeeee space transformation in detail. Here, we note that we
152
apply ICK for multiplicative kernels but note that an additive kernel may be constructed using the methods of Pearce et al. [43]
153
154
4.1
Resemblance between Composite GPR and ICK
We will analytically prove the following theorem for data with information from 2 different sources (2) (2) for clarity, and we note this theorem can be straightforwardly extended to M > 2.
157
Let JNN: DR - TR be a NN function with random weights and D2 - Rp be mapping function, and define an inner product between the representations (1) fNN (I (1) and
Theorem
158
(2) = g (2). Then this inner product approximately follows a composite GPR model
159
fICK (1()),
SNN (x
(z(2) GP approx (0. K K(2)
(3)
180
if includes the following deterministic -orel-tt--ltee--ppcc-e transformation
122)
2 , 2 (2)
(2) (2)
_12)
2
xx
(2)
T
9
x (2)
(4)

---

where KNN is a NN-implied kernel and K(2) is any valid kernel of our choice.
162
To prove Theorem 1, we first make the following assumption.
Assumption For latent representations z; and ; ((m) extracted from different data points æ; and 163
2j where i + j and m 21, the interactions between different entries of =f z((") and mm))) can be reasonably ignored. In other words, let ((m) be the parameters of the network or the kernel function which takes mm)) and ouiputs z ((m), we huve R. pp(cm)))
184
185
(m)) (m)) 0 for all + L.
186
ik
js
With Assumption I and Lemma 1, let Q 1001) represent the parameters of the ICK frame- work, we can calculate the covariance between úJ and új for different data indices i # j as follows
188
cov(ý 92)
pp)) [[,ū (ū: [ر,]]
(5)
p))))
(Ek=1
Fik Lik
(1) (2)
(Ei- 22002
(6)
Ep(()
Ehen Ei
(1) (2) (2) Pik Pjl Rik "jt
Ep(()
Ek=n Like Pjk "ik "jk
(1) (1) (2) (2)
(8)
= Eb=n E
P2(O(L)
(1) (1)
Fike 3jk
(((2)))
(2) (2) Fik 3j
(9)
2 KNN
(1)
k=
E ((022)
Bik 2sh
(10)
Here, from Equation 5 to Equation 6 use the statement B ((()))
2ik
(1)
0 from Lemma 1 and the
independence between ((1) and 8(2), which leads to E, (0: (e) أ From Equation to
170
Equation 8 we get rid of all the cross terms under Assumption 1. From Equation 8 to Equation we
again make use of the independence between ((1) and 0(2), From Equation to Equation 10 we use
172
173
the statement E (((cn))
Fik jk
(L) (1)
KNN [r " ) from Lemma 1. If the kermel-to-latent-spacc
(1)
174
transformation in g(-) is deterministic, we can remove the expectation sign from the summation term in Equation 10 and the covariance can be further expressed as
175
cov U5) = KW (a:"), (0."
(2) (2) =
K NN 2
(1) ( (1)
(1(2)
(2)
(11)
176
which means that 9 approximately follows GP with composite kernel Keomp (2 x,) KW o ()) x") k(2) x xx)²) a priori. This completes our proof of Theorem
=
177
178
4.2 Kermee-to-latent-ssace Transformation
We now show how we can construct appropriate mapping g(-) that approximately satisfies the assumed form of 4 and used in the derivation of ICK from 10 to (IT). Here we adopt two methods, Nyström approximation and Random Fourier Features (RFF), to map the kernel matrix into the latent space. Below, we give the formulations and results for the Nyström method, and give the methods and results for RFF in Appendix B According Yang al. [59]. the Nyström method
180
182
184
will yield much better performance than RFF there exists a large gap in the eigen-spectrum of the kernel matrix. In our applications,
185
also observe large eigen-gap (see details in Appendix C and Nyström method does generalize much better than RFF. We name our framework with Nyström method and random Fourier Features ICKy and ICKr, respectively.
186
187
188
4.2.1
Nyström Method
The main idea of Nyström method 5 is to approximate the kernel matrix K E N with a much smaller low-rank matrix K2 R284 where q < N so both the computational and space complexity of kernel learning can be significantly reduced
189
190
181
K ~ K
K ng K K ng.
(12)
5

---

192
The entries of Kg and K ng can be calculated as (Kg) - ((,,, E f1. ..... 9 and (K ng) ( fg), (1,2,...,NT, N).j (1, respectively. x represents the original
198
194
data points and å represents pre-defined inducing points (or pseudo-inputs 4.). In our study, these inducing points are chosen by defining an evenly spaced vector over the range of original data points.
185
196
By performing Cholesky decomposition
K² =U where U E =2xx K is
K = Kg K
ny
K UKK (UKT.) T
n4)
(UKT4) n4
(13)
197 Therefore, if we set the number of inducing points to be q = p. then we can use 2; - U KT
mp
198
as a kernel-to-lateee totttto-et-sssaeeeeeee transformation because each element in K approximately satisfies as stated in Theorem 1: K ((i,z,) = Kiz R K,; 2 zj. Conveniently, modern deep learning frameworks can propagate gradients through the Cholesky operation, making it straightforward to
199
200
201
update the kernel parameters with gradient methods. Note that as we increase the number of inducing
points p, the approximation error between K and K decreases. However, it is not recommended to set p very large as updating the Cholesky decomposition requires O(p"). The empirical impact of
202
203
204
p on computational time and performance is shown in Appendix [ In our experiments, only mild
205
values of p are necessary and the impact on computational is relatively small.
5
Experimental Results
208
207
We evaluate ICKy on 3 different data sets: a synthetic data set, a remote sensing data set, and data set obtained from UCI Machine Learning Repository 13. Note that in all the 3 experiments, our ICKy framework only consists of 2 kernels (i.e. M = 2), one NN-implied kernel and one chosen kernel function with trainable parameters. To verify that ICKy can work with more than 2 kernels, we create another synthetic data set with 3 kernels and show the corresponding results in Appendix In addition, the experimental results for ICKr is provided in Appendix [B] All experiments are conducted on a computer cluster equipped with a GeForce RTX 2080 Ti GPU, The implementation details of all the experiments in this section are provided in Appendix G
209
211
A
213
215
5.1
Synthetic Data
216
To verify that ICKy can simulate a multiplicative kernel, we create a synthetic data set y GO K2 (2) containing 3000 data points where x11) E [0, is the input for the linear kernel Ki and (2) E [0, 2] is the input for the spectral mixture kernel [56] K2 with 2 components. With ICKy, we process (1) with single-hidden-be layer NN and (2 with spectral mixture kernel function. We evaluate ICKy on both a prediction task (where we first randomly shuffle the data points and do a 50:50 train-test split) and a forecasting task (where we use only the data points with (2) < 0.6 for
220
221
training and the rest for testing).
223
We then compare ICKy with two models: a plain multi-layer perceptron (MLP) applied to the concatenated features and a novel multi-layer perceptron-random forest (MLP-RF) joint model
224
employed by Zheng et al. 61 where MI.P learns from (:(1) and RF learns from 2:))) We believe MIP-RF serves good benchmark model it is a joint model with similar architecture to our
225
226
ICKy framework. To see how ICKy simulates the spectral mixture kernel, we plot only z22) against the predicted value of y as shown in Figure 2 As can be seen from the figure, in the prediction task (top row), plain MLP only captures the linear trend. MLP-RF only captures the mean of the spectral mixture components. In contrast, our ICKy framework captures both the mean and the variance of the spectral mixture kernel. In the forecasting task (bottom row), ICKy also outperforms plain MLP and MLP-RF as it approximately captures the rising trend in the range of (22) E [0.6, 1]. When zz(2) > 1, ICKy is unable to confidently extrapolate, so it starts to "fail gracefully," by which we mean that the prediction reverts to the mean of the prior (e.g., no observed information case.), just as would be expected in a GP. However, we do not evaluate the posterior distribution to get a full sense of the posterior uncertainty.
228
230
231
233
235
236
237
We also test plain MLP, MLP-RF, and ICKy on the prediction task using different number of training samples. As displayed in Figure 3 ICKy yields the smallest error among all the 3 frameworks even with very limited data. addition, to test the robustness of ICKy, we conduct the same experiments on another synthetic data set in Appendix D to confirm that ICKy can simulate an additive kernel.
239

---

Predicioon otty pain
Prdiciion diy with NLPA!
1.
Prodiction oity with ICKy
15-
0.
0.5
20
0.
011
B1
-0.6
-06
-66
-13
-13
-13
-20
2.0
15
1/
-20
30
0.5
1.0
1.5
70
-2.
80
955
1.0
1.5
20
50
0.5
25
20
Forecasting dry plan
Poreaating MUP-ay
Forecasting ofyy with ICly
vahass of
-06
-06
-04
-3.3
-13
-3.
-2.0 0.60
0.35
1.39
3.85
2.00
-2.0
0.50
0.55
1.30
2.90
2.00
-2.0
3.60
0.55
1.66
1.85
270
Figure 2: Prediction (top row) and forecasting (bottom row) of y ~ GP(O, K2 where (1)) is input to linear kernel K1 and ,z(2) input to spectral mixture kernel Kg. We plot (2)) against the predicted We show results from a plain MLP (left column), MLP-RF (middle left column), and ICKy framework (middle right column), and we compare to the true values of y (right column).
5.2
Remote Sensing Data
242
We believe ICKy will be particularly useful for re- mote sensing applications. In this experiment, we collect remote sensing data from 51 air quality moni- toring (AQM) stations located in the National Capital Territory (NCT) of Delhi and its satellite citics over the period from January 1, 2018 June 30, 2020 (see Appendix For notes on data availability). Each data point = {x, t} contains 2 sources of information: a three-band natural color (red-bi bludbllu--rren) satellite image as the high-dimensional information and the corresponding timestamp as the low-dimensional information. Note that we convert the timestamps into numerical values (where the day 201 8-01-01 corresponds
0.6
Plain MLP -
MLP-RF
244
0.5
ICKy
246
0.4
RMSE
248
0.3
250
0.2
252
253
0.1
300
620
900
1200
1500
254
Number of training samples
255
0) before feeding them into the models. Our goal is to predict the ground-level PM2.s concentration ý f(x, t) using both sources of in-
Figure 3: Prediction error of plain MLP, MLP- RF, and ICKy with different amount of train- ing data generated by y = GP(O, K2
256
257
258
formation.
We split the train and test data set based on Specifically, testing and the rest for training. As PM2.5 varies with time on a yearly basis, we use an exponential- sine-squared kernel with period of T 365 (days) process the low-dimensional information
all the data points with > 500 for
280
262
The satellite images are processed with a CNN. Figure 4 shows the true versus the forecasted
PM2.5 values by both ICKy and 2 benchmarks: Convolutional Neural Network-Randoos Forest 263
254
(CNN-RF) joint model [6] 62] (similar to the MLP-RF model in Section 5. where RF learns the
285
temporal variation of PM and CNN captures the spatial variation of PM2.5 from satellite images)
266
and a carefully designed CNN-RF model that maps t into two new features, ain22tt/360 and
267
cos /365/ to explicitly model seasonality. As can be seen, ICKy outperforms both benchmarks
268
with the highest correlation coefficients and the lowest errors on the forecasting task. Specifically,
269
regular CNN-RF joint model fails forecast PM2.5 as shown in Figure 4a After including seasonality,
CNN-RF performs significantly better shown in Figure 4b but the forecasted PM2.5 values are still
271
less smooth than those from ICKy (Figure 4c) due to discontinuous nature of the RF regressor
6 19..
We also visualize these results in the form of time series in Appendix
7

---

Table Correlation and error statistics of ICKy and other joint deep models with both convolutional
and attention- basee architectures on the 2M2.5 forecasting task
Spearman R
Pearson R t
RMSE 4
MAE
CNN-RF [61
0.48
0.32
70.09
54.28
ViT-RF III
0,42
0.32
70.58
55.15
Seasonal CNN-RF
0.65
0.73
52.60
39.25
Seasonal ViT-RF
0,66
0.74
49.92
36.22
Seasonal Deep VİT-RF
63
0.68
0.76
48.87
35.32
Seasonal MAE- VIT-RF
12
0.68
0.76
48.43
34.92
CNN-ICKy
0.70
0.77
47.15
32.84
ViT-ICKy
0.66
0.77
47.17
34.00
DeepViT-ICKy
0.62
0.73
48.68
34.10
We note that the inner product operation in ICK is similar in mathematical structure to attention-based mechanisms [51] popular in many deep learning frameworks. Therefore, we compare ICKy with 4 attention-basee benchmarks that constructured based off of Vision Transformer (ViT) [II] architecture, including VIT-RF, Scasonal VIT-RF, Seasonal DiepViT-RE [6]] and Seasonal MAE-
274
276
VIT-RF where VIT is pre-trained by Masked Autoencoder
2 as displayed in Table 1 These the same RF and sinsusoidal mappings as described previously to input the temporal information into the model. We note that we unaware of Vision Transformers being used in this manner, and that all these models represent novel formulations. It can be observed that standard ViT-RF model fails to forecast PM2.5 without seasonality incorporated, just in CNN-RE. After introducing seasonality
278
280
281
282
by mapping into sinusoidal features, ViT-based joint models yield higher correlation and smaller error than CNN- RF but still underperform CNN- based ICKy. We also considered using a ViT-based architecture for the CNN ICKy, and observed similar performance in these ICKy variants.
283
284
285
5.3
UCI Machine Learning Repository Data
To see if our ICKy framework generalizes to other domains, acquire another data set containing the normalized productivity and corresponding features of garment workers from the UCI Machine Learning Repository. Imran al. []] employed dense MLP with hidden layers to predict the worker productivity with collected features such as date, team number, targeted productivity, etc. To test our ICKy framework, we separate out the date and use it the low-dimessional information. The rest of the features (excluding the temporal information) are then concatenated together to serve the nigh-dimensional information. Observing that the daily averaged worker productivity kernel. The network architecture of ICKy is the same as that of the wwohidden-layer MLP benchmark. To demonstrate the strength of ICKY compared to other methods that equip NNs with GPs, we also add 2 additional benchmarks here: a Gaussian Neural Process (GNP) 7 and an Attentive Gaussian Neural Process (AGNP) 29 Based on the results shown in Table 2 ICKy has the best performance when the period parameter of the kernel is set to be benchmarks by almost one order of magnitude. When we set T 2 or T = 7, this improvement is less significant, which aligns with our initial observation that the daily averaged productivity has monthly seasonal trend. It is also worth noting that the GNP benchmarks here yield larger errors than MLPs. possible explanation that GNP does not allow explicit assignment of stationary kernel (as the kernel models a posterior covariance) so it is hard for GNP to identify specific patterns in the data such as seasonality without being given the pattern a priori.
287
289
281
283
has an approximate monthly trend, we again use an
294
295
296
297
298
30 (days) and it outperforms both MLP and NP
299
300
302
304
Table 2: Prediction error of actual worker productivity on the test data set with ICKy and other benchmark models (MLPs and NPs)
MSE [ (*10-3)
MAE | (*10-
MAPE $
MLP 1
20.16 + 1.26
9.93 + 0.36
17.30 + 0.82
Cyclic MLP
20.97 = 1.98
10.16
17.48 = 1.37
GNP 77 37
57.25 = 4.31
19.39 + 0.94
29.58 + 1.63
AGNP [29]
43.11 + 5.95
14.38 + 0.88
22.59 + 1.42
ICKy, T 2
3.43 + 1.42
4.85 + 1.00
6.74 + 1.37
ICKy, T
0.44 + 0.13
1.43 + 0.15
2.22 + 0.21
ICKy, T= 30
0.31 + 0.09
1.17 + 0.14
1.79 + 0.22
8

---

Spcarman
Spearman
500
Pearson - 0.32
Forecastod PMG (ur m--)
300
Pearson - 0.73
52.6
Fore Forecaasted PMG, (uo m-3)
Pearson 0.77
47.15
4cu.
MAE = 54.28
Roo
MAE 39.25
MAE 32.84
Forecarted PRt
300
300
200
250
200
0
100
200
300
460
500
600
100
200
300
400
500
600
0
100
200
300
460
500
605
True 2M (ag m-]
True PM lug m-))
True PMG log m-0)
(a) CNN-RF
(b) Seasonal CNN-RF
(c) ICKy
Figure 4: Density plots of the true PM2.5 concentrations against the forecasted PM2.5 concentrations for 2 500 using (a) a CNN-RF joint model [6]] (b) a CNN-RF joint model with seasonality incorporated, and (c) our ICKy framework
Discussion
305
306
Efficiency, Flexibility, and Generalization Compared exact composite GP models which scale (NN)) the training process of our ICK framework is more efficient as it leverages standard back- propagation to learn both the paramters of NN and the kernel function. In addition, the network architecture of ICK can be very simple, can be seen in all 3 experiments of ours, which further reduces its time and space complexity. Besides efficiency, our ICK framework more flexible compared to other joint models (i.e. BNNs and CNN-RF). To be specific, the BNNs implemented by Pearce et al. [43 cannot simulate complicated kernels such as the spectral mixture kernel we use in Section 5.1 The CNN-RF joint model implemented by Zheng et al. [6] requires to carefully design the input pre-processing procedure. Also, ICK generalizes well to unseen data even with very limited training samples. There is a potential concern that ICKy may into computational challenges when a large number of inducing points are required. This was not a problem in our experiments, but in large scale models this could be tackled by considering conjugate gradient methods, which have heen recently popular in GP inference
307
308
310
312
314
316
318
15
Limitations A major limitation of ICK lies in method of combining latent representations the nature of inner product (i.e. the effect of multiplying small numbers) may cause vanishing gradient problems when have a large number of sources of information (i.e. M is large). Furthermore, this paper only discusses the theoretical relationship between ICK and composite GPR a priori. This relationship will not exactly hold true posteriori, although empirical results (34) and theoretical
320
322
323
324
results (24) in slightly different contexts suggest that they may be close. Future work will evaluate
325
this gap by exploring Bayesian neural networks and a posteriori properties.
328
Broader Impacts We believe our framework is extensively applicable to regression problems in many fields of study involving high dibeeeinnnionll data and multiple sources of information with perceptible trends, such as remote sensing, spatial statistics, clinical diagnosis.
327
328
7
Conclusion
330
This paper presents a novel yet surprisingly simple Implicit Composite Kernel (ICK) framework to learn from hybrid data containing both high-dimensional information and low-dimensional informa- tion with prior knowledge. We first analytically show the resemblance between ICK and composite
331
332
333
GPR models and then conduct experiments using both synthetic and real-world data. appears
334
that ICK outperforms various benchmark models in our experiments with lowest prediction errors and highest correlations even with very limited data. Overall, we show that our ICK framework
335
exceptionally powerful when learning from hybrid data with our prior knowledge incorporated, and
337
we hope our work can inspire more future research on joint machine learning models, enhancing their performance, efficiency, flexibility, and generalization capability.

---

References
339
340
[]] Abdullah AI Imran, Md Nur Amin, Md Rifatul Islam Rifat, and Shamprikta Mehreen. Deep neural network approach for predicting the productivity of garment employees. In 2019 6th International Conference on Control, Decision and Information Technologies (CoDIT), pages 1402-1407. IEEE, 2019.
341
342
343
344
[2]
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[3] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-31 2018.
347
348
[4] Mohamed A Bouhlel and Joaquim RRA Martins. Gradient-enhanced kriging for high- dimensional problems. Engineering with Computers, 35(1):157-173, 2019.
349
350
[5] Mohamed Amine Bouhlel. Nathalic Bartoli, Abdelkader Otsmane, and Joseph Morlier. Improv- ing kriging surrogates of nigh-dimensionne design models by partial least squares dimension reduction. Structural and Multidisciplinary Optimization, 53(5):935-952, 2016.
352
353
[6] Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
354
[7] Wessel Bruinsma, James Requeima. Andrew YK Foong, Jonathan Gordon, and Richard Tumnr. The gaussian neural process. arXiv preprint 101.:22000.03666, 2021.
355
101.
356
[8] Nidhan Choudhuri, Subhashis Ghosal, and Anindya Roy. Nonparametrie binary regression using a gaussian process prior. Stafistical Methodology. 4(2):227-243, 2007.
357
99]
Abhirup Datta, Sudipto Banerjee, Andrew O Finley, and Alan E Gelfand. Hierarchical nearest- neighbor gaussian process models for large geostatistical datasets. Journal of the American Statistical Association, 111(5144 (((551)):000880) 812, 2016.
359
361
[10]
Zhijie Deng. Jiaxin Shi, and Jun Zhu. Neuralef: Deconstructing kernels by deep neural networks.
382
Xiviv preprint arXiv:2205.0016 2022.
363
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint S.Kiv:2010.1192: 2020.
364
385
386
387
[12] Petros Drineas, Michael W Mahoney, and Nello Cristianini. On the nyström method for approximating gram matrix for improved bemel-based learning. journal of machine learning research, 6(12), 2005.
369
370
[13] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
[14] David Duvenaud. Automatic model construction with Gaussian processes. PhD) thesis, Univer- sity of Cambridge, 2014.
372
373
[15] Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G Wilson. Gpytorch: Blackbox matrix- matrix gaussian process inference with gpu acceleration. Advances in neural information processing systems, 31, 2018
375
376
116] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv: /807.0/60 2018
377
378
[17] Adria Garriga-Alonso Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. arXiv preprint arXiv: 05587, 2018.
379
[18] Alan E Gelfand and Erin M Schliep. Spatial statistics and gaussian processes: A beautiful marriage. Spatial Statistics, 18:86-104, 2016.
381
382
[19] Pierre Geurts, Damien Ernst, and Louis Wehenkel. Extremely randomized trees. Machine learning, <3(1):3-42, 2006.
384
[00] Mark Girolami and Simon Rogers. Variational bayesian multinomial probit regression with gaussian process priors. Neural Computation, 1798)_199011877, 2006.
385
386
21]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[22] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
388
10

---

390
[23] James Hensman, Nicolo Fusi, and Neil D Lawrence. Gaussian processes for big data. arXiv preprint arXiv:1309.6833 2013.
391
392
[24] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
394
385
[25]
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances neural information processing systems, 28, 2015.
396
397
[26]
Ziyang Jiang, Tongshu Zheng, Mike Bergin, and David Carlson. Improving spatial variation of ground-level pm2. 5 prediction with contrastive learning from satellite imagery. Science of Remote Sensing, page 100052, 2022.
399
400
[27] Andre G JJurnel and Charles J Huijbregts. Mining geostatistics. The Blackburn Press, 1976.
[28] Hyoung-Moon Kim, Bani K Mallick, and Chris C Holmes. Analyzing nonstationary spatial data using piecewise gaussian processes. Journal of the American Statistical Association, 100(470);653-668, 2005.
402
404
[29] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint 1rXiv:1901.05761, 2019.
405
408
407
[30] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv 2014.
[31] Daniel G Krige. A statistical approach to some basic mine valuation problems the witwa- tersrand. Journal of the Southern African Institute of Mining and Metallurgy, 52(6): 19-139, 1951.
410
411
412
[32] Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions neural networks, 95):9877 1000, 1998.
414
415
[33]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015.
416
417
[34] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv: 2017.
419
420
[35] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: (tt::://1.007/// 2016.
422
[36] Gary Mareus. Deep learning: critical appraisal. arXiv preprint wwxiiv/80/.00633: 2018.
423
[37] Stratis Markou, James Requeima, Wessel Bruinsma, and Richard Turner. Efficient gaussian neural processes for regression. arXiv preprint arXiv:2108.09670 2021.
425
[38] Takuo Matsubara, Chris J Oates, and Fravçois-Xavier Briol. The ridgelet prior: covari- ance function approach to prior specification for bayesian neural networks. arXiv preprint arXiv:2010.08488, 2020.
426
427
428
[39] Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint 1177, 2018
430
431
[40] Brian McFee, Gert Lanckriet, and Tony Jebara. Learning multi-modal similarity. Journal of machine learning research, 12(2), 2011.
432
433
[41] Ben Moseley, Andrew Markham, and Tarje Nissen-Meyer. Solving the wave equation with physics-informed deep learning. arXiv preprint 1xXi::0006.11944 2020.
435
[42]
Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pages 29-53. Springer, 1996.
436
[43] Tim Pearce, Russell Tsuchida, Mohamed Zaki, Alexandra Brintrup, and Andy Neely. Expressive priors in bayesian neural networks: Kernel combinations and periodic functions. In Uncertainty in artificial intelligence, pages 134-144. PMLR, 2020.
438
11

---

440
[44]
Dejan Petelin, Alexandra Grancharova, and Juš Kocijan. Evolving gaussian process models for prediction of ozone concentration in the air. Simulation modelling practice and theory, 33:68- 80, 2013.
441
443
[45]
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
444
[46] Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimiza- tion with randomization in learning. Advances in neural information processing systems, 21, 2008.
446
448
[77
John Shawe-Tay Nello Cristianini, et al. Kernel methods for pattern analysis. Cambridge university press, 2004.
449
450
[48]
Tao Shi and Steve Horvath. Unsupervised learning with random forest predictors. Journal of Computational and Graphical Statistics, 15(1):118-138, 2006.
451
452
[49]
Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs, Advances in neural information processing systems, 18, 2005.
[50] Mark Van der Wilk, Carl Edward Rasmussen, and James Hensman. Convolutional gaussian processes. Advances in Neural Information Processing Systems, 30, 2017.
455
456
[11] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
457
458
[52] Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. International Conference on Machine Learning, 2020.
460
461
[53] Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. Advances in newral information processing systems, 13, 2000.
462
483
[54] Christopher K Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. volume 2. MIT press Cambridge, MA, 2006.
464
485
[55] Christopher KI Williams and David Barber. Bayesian classification with gaussian processes. IEEE Transactions on pattern analysis and machine intelligence, (0(12):1342-1311 1998.
466
467
[56]
Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapo- lation. In International conference on machine learning, pages 1067-1075. PMLR, 2013.
489
[57] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378. PMLR, 2016.
470
[58] Andrew Gordon Wilson, David A Knowles, and Zoubin Ghahramani. Gaussian process regression networks. arXiv preprint / 2011.
472
473
[59]
Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Zhi-Hua Zhou. Nyström method vs random fourier features: A theoretical and empirical comparison. Advances in neural information processing systems, 25, 2012.
475
478
[60]
Daoqiang Zhang, Yaping Wang, Luping Zhou, Hong Yuan, Dinggang Shen, Alzheimer's Disease Neuroimaging Initiative, et al. Multimodal classification of alzheimer's disease and mild cognitive impairment. Neuroimage, 35(3):856-867 2011.
477
478
661]
Tongshu Zheng, Michael Bergin, Guoyin Wang, and David Carlson. Local pm2. 5 hotspot detector at 300 m resolution: A random forest-convolutionaal neural network joint model jointly trained on satellite images and meteorology. Remote Sensing, 13(7):1356, 2021.
480
481
482
[62]
Tongshu Zheng, Michael H Bergin, Shijia Hu, Joshua Miller, and David E Carlson. Estimating ground-level pm2. 5 using micro-satellite images by a convolutional neural network and random forest approach. Amoopheric Environment, 230: 1 17451, 2020.
483
484
485
[63]
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.1 1186,
486
1021.
12

---

Checklist
488
489
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
491
contributions and scope? [Yes] See Section T
482
(b) Did you descrihe the limitations of your work? [Yes] See Section 6
493
(c) Did you discuss any potential negative societal impacts of your work? [No] We are unaware of direct negative societal impacts.
494
495
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
487
2. If you are including theoretical results...
488
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 4.1
489
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 4.1
500
3. If you ran experiments..
501
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] The code, data, and instructions to reproduce the experimental results are provided via the anonymous repository URL in the abstract.
503
505
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 5
508
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Section 5.1 5.3
508
509
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See section
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
512
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Appendix |
(c) Did you include any assets either in the supplemental material as URL? [Yes] We have included code, with the link in the abstract to an anonymous repository. This will be switched
515
maintaindd repository if accepted.
517
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A] We do not release data.
518
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] None of our datasets contain personally identifiable information or offensive content.
520
522
5. If you used crowdsourcing or conducted research with human subjects...
523
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
524
525
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
526
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
528
13

---

50 Speaman
40
Spearman
60
Spearman
Pearson 0.76
Pearson
30
40
20
10
Yved
Yeed
0
10
-2
-30
-40
-50
-40
-60
-50
-30
-10
10
50
-40
-20
20
40
-60
-40
-20
0
20
40
60
Youe
Yive
Youe
(a) 1
JuN (20))
(b) ú = fIcKs (i.0),
(c) 1 = fIcks (x
(2)
Figure Al: Scatter plots of the true values of y against the predicted values of y using our ICKy framework with (a) one source, (b) 2 sources, and (c) sources of information
20 R2
(M)
PM
Ky(xx'))
K₂
KM
200
200 RP
2 E R
20 IR
Inner product: Ex 11
Figure A2:
Given data containing M sources of information æ -1x(1), (M) }. we can
process the data using our ICK framework where high-dimensional information (e.g. (1) in the
figure) is processed using neural network and low-dimensional information (e.g. xx() in the figure) is processed using kerne function followed by Nyström or RFF transformation.
A ICK with More Than Two Kernels
530
Besides the visualization presented in Figure we also show our ICK framework for processing data
(22), (2)) represent different types of kernels with trainable parameters. The final prediction is calculated by
(1((1)
.MM)) } with M > 2 sources of information in Figure A2 Here K(2), .... (())
531
532
533
chained inner product of all extracted representations
Σ [m=12 (m)
534
To confirm that ICKy can work with more than kernels, we construct another synthetic data set containing 3000 data points in total. Each input æ = { (2) _x(3) has sources of information.
535
The output is generated by y = (3) tanh (2x(1) cos (Tx /50)) + where e is a Gaussian noise We process z()) with small single-hidden-Hee hiddee--ddeeer NN, (2) with exponential sine squared kernel, and (3) with radial-basis function (RBF) kernel. Figure Al shows the prediction results
536
term.
539
as we progressively add more sources information into our ICKy framework with corresponding kernel functions. It can be observed that ICKy yields both smallest error and highest correlation
14

---

50 Speaman
40
Spearman
60
Spearman
Pearson 0.76
Pearson
7.58
30
5.48
40
20
10
Yped
Yped
0
10
-2
-30
-40
-50
-40
-60
-50
-30
-10
10
50
-40
-20
20
40
-60
-40
-20
0
20
40
60
Youe
Yive
Youe
(a) 0 = JuN (20))
(b) 1 = ficK (2)))
(c) ÿ = fickr (2
Figure B1: Scatter plots of the true values of y against the predicted values of y using our ICKr framework with (a) one source of information, (b) 2 sources of information, and (c) 3 sources of information. Note that here
RFF for kernel-to-latent-spar transformation.
with information from all 3 different sources. Hence, ICKy works well with the M = 3 case and the regression performance improved as we add in more information related to the target.
542
543
B
Random Fourier Features
B.1
Methodology
545
Random Fourier Features (RFF) is another popular approximation method used for kernel learning
45
Unlike the Nyström method which approximates the entire kernel matrix, RFF directly ap-
proximates the kernel function K using some randomized feature mapping Φ RDm - p2dm such
547
(m) that K A Jry (m))
(sr(m))
T
2 (m)
To obtain the feature mapping d based Bochner's
549 theorem, we first compute the Fourier transform p of kernel K
p(w)
(m /
1
e e- J (())dō
(14)
550 where 8 = _,(mm) x the feature mapping @ as follows
(m) Then we draw dm i.i.d. samples w1, W2, Witm from p(w) and construct
(z(m))
(15)
1/2 1/2 cos (wFz((m)) cos (WT slm)) sin (mff(()) ,..,, sin (wT m
Since @ (x((m) E R2dm, we need to set dm p/2 when using RFF as a kermel-to-lateet-ssppce transformation. In addition, since RFF involves sampling from distribution, the kernel parameters
552
553
554
are thus not directly differentiable and we need to apply reparameterizztion trick (35) to learn those parameters.
555
B.2
Experimental Results
557
B.2.1
Synthetic Data
We use the same toy data set where each data point x = {1((1, _,(2),,)))))))) ,,,3] contains 3 sources of information described in Appendix A Also, we use the same types of kernels as those in ICKy as discussed in Appendix [A] The only difference here is that we use RFF instead of Nyström method to transform the kernel matrix into the latent space in ICKr framework.
558
559
580
562
The results are displayed in Figure B1 information (2) along with the exponential sine squared kernel, both the correlation and the predictive performance are improved (though not good as the results from ICKy shown in Figure Al).
can be observed that when we add in only the side
563
15

---

1000- Spearman - 0.7
1000
Spearman
Pearson - 0.77
Pearson -0.11
70
Forecasted PM2s (uig 7
800
MAE 32.84
Prodicted PM2s lug m--4
RMSE 104.32
73.87
60
600
400
200
200
20
-200
200
400
600
800
1000
400
600
True PM (ug m +)
True PMg,: (pg m=))
ICKy
(b) ICKr
Figure B2: Density plots of the true PM2.5 concentrations against the forecasted PM2.5 concentrations for 2 500 using our ICK framework with (a) ICKy and (b) ICKr
However, after we further include x²(3) with the RBF kernel, we realize that the parameters ICKr become very hard to optimize and it fails to make valid predictions and starts to guess randomly around zero.
565
567
568
B.2.2
Remote Sensing Data
We also try ICKr the forecasting task using the remote sensing data (see Section 5.2) and compare the results with those from ICKy. Each data point x {ar, contains a satellite image ær as the the low-dimensional information.
570
nigh-dimensional information and its corresponding timestamp The satellite images are processed with a two-layer CNN and the timestamps are processed with an pponential-sine-sqqueee kernel with period of T = 365 (days). As can be observed from Figure ICKr yields much higher error compared to ICKy.
572
574
C
Estimated Kernel Matrix and its Eigen-spectrum
575
1.25
1.25
1.25
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
(a) Kim
(b) KICKy
(c) K
Kickyl
Figure Cl: Visualization of (a) True matrix (b) estimated matrix by our ICKy framework, and (c)
absolute difference between the true and estimated matrix for the spectral mixture kernel
576
We first examine whether ICKy can retrieve the spectral mixture kernel in the prediction task. After fitting the parameters the spectral mixture kernel ICKy, we compute the kernel matrix KICK, using these learned parameters and compare it with the true kernel matrix K by calculating the absolute difference between them as displayed in Figure CI As can be observed, K iCK, and K tu
577
578
579
are similar and their absolute difference is relatively small, indicating that ICKy can approximately retrieve the spectral mixture kernel.
581
16

---

582
Yang et al. [59 studied the fundamental difference between Nyström method and Random Fourier Fea- tures (RFF). They conclude that Nyström-method- based approaches can yield much better generaliza- tion error bound than RFF-based approaches if there exists a large gap in the eigen-speetrum of the kernel matrix. This phenomenon is mainly caused by how these two methods construct their basis functions. In particular, the basis functions used by RFF are sam- pled from a Gaussian distribution that is independent from the training examples, while the basis functions used by the Nyström method are sampled from the training samples they are data-dependent. In our synthetic data experiments, we train our ICK frame- work using batch size of 50. The eigenvalues of the kernel matrices computed from the first 4 batches of the synthetic data are displayed in Figure C2
0.6
Batch
583
0.5
Batch
585
Batch
Eigenvalues/N
0.4
587
0.3
589
691
0.1
583
0.0
Rank
585
Figure C2: Eigenvalues of the kernel matrix
computed from the first 4 batches of training data
587
598
589
can be observed that the first few eigenvalues of the
600
kernel matrix are much larger than the remaining eigenvalues. Namely, there exists large gap the
eigen spenttppe of the kernel matrix, which helps explain why ICKy has much better performance
602
than ICKr.
17

---

Procktion ay ML
Swddiies
Prodeson dl
Trus dy
03
-14
x-0.0
-L-7
-2.5
-300
23
03
1.
20
60
05
20
30
35
15
2.0
35
13
15
Eo0cadiin of' Jain VLP
Fo000giig of with H.P.ap
Ecoccstingg of
dfy
-15
2.53
3.5
-15
x-133
+1.75
-00
-15
-25
1.25
1.25
220
Figure DI: Prediction (top row) and forecasting (bottom row) of GP(O, K1 K2), where 2(1) input linear kernel K₁ and z²(2) is input to spectral mixture kernel K2. Here we only plot (²(2) against the predicted We implement 3 types of models: plain MLP (left column), MLP-RF (middle left column), and our ICKy framework (middle right column), and we compare the results with the true values of y (right column).
D Simulation of an Additive Kernel
604
While ICK is designed to capture multiplicative ker- nels, we evaluated how well could capture addi- tive kernels. We conduct experiments using another synthetic data set generated by an additive kernel y (P(O, K₃ + K2) with the same training settings. As shown in Figure DI ICKy again outperforms plain MLP and MLP-RF in both the prediction and
Plain MLP-RF
605
1.0
ICKY
607
0.8
609
RMSE
0.6
611 the forecasting tasks. Moreover, we again test plain MLP, MIP-RF, and on the prediction task using different number of training samples. As displayed in Figure D2 ICKy yields the smallest error among
0.4
613
615 all the 3 frameworks. Also, the performance gap between ICKy and the other benchmark models shrink as we feed in more training data. Therefore, conclude ICKy is robust enough simulate both additive and multiplicative kernels.
0.2
600
1500
Number training samples
617
Figure D2:
Prediction error of plain MLP, MLP-RF, and ICKy with different amount of training data generated by y ~ GP(O, K1 K2)
E Number of Inducing Points
620
as increase the number of inducing points p. we expect the approximation error between the true kernel matrix K and the approximated kernel matrix I to decrease. Here, we empirically show how the value of p impacts our predictions. In Figure Ela we plot the prediction error of ÿ fick, (1) (()) against the number of inducing points using the synthetic data generated Appendix A As can be observed, the prediction error drops sharply as we raise p from small value (e.g. 2). When is relatively large, increasing yields smaller improvement on the predictions. Additionally, in Figure ETb] we plot the total training time against The total training time is dependent on how long single iteration takes and the total number of epochs required. We note that once p > 80 training time relatively flat, which due to the
As discussed in Section
622
624
625
626
G27
628
629
fact that the total computation in the Cholesky is less than the computation in the neural network. Interestingly, it appears that when p is very small, ICKy takes longer to converge due to the need for many more epochs. As increase the training time goes down and then goes up again due the
631
18

---

60
50
Training time (sec)
6.0
40
RMSE
5.5
30
5.0
4.5
4.0
10
100
Number integrating points
Number of integrating points p
(a) Prediction error
(b) Training time
Figure E1: Plots of (a) prediction error and (b) training time of ÿ = -ficky (z(1)x (8)) against the number of inducing points p
computational complexity, i.e. O(p"), of the Cholesky decomposition. Based on these observations, we are overly concerned about the computational complexity for reasonable values of p.
634
625
F
Visualization of Remote Sensing Data as Time Series
636
To better illustrate the results Section 5.2 we visualize those results in the form of time series. As shown in Figure FT] the plain CNN-RF model does not work it tends to forecast constant PM2.s values. In contrast, both the seasonal CNN-RF model and our ICKy framework captures the overall trend of the true daily averaged PM2.5 values, but the forecasted values by ICKy are smoother and yield smaller error.
638
640
500
j04ts 68.05
forecasted Daily Averaged Mil data lev plus Des
504
p0400 $3
Tran ferecareet Daily Averoped
200
RMSS - 47-33
Forccsted ged
-20
H425 $9.29
452
H42
-00 HAX
De swcrgeed aro0cds
5
200
300
110 (61.
Thaasrr
(a) CNN-RF
(b) Seasonal CNN-RF
(c) ICKy
Figure Fl: Time series visualization of the true against the forecasted daily averaged PM2.s con- centrations for f 2 500 using (a) CNN-RF joint model 6 62]. (b) a CNN-RF joint model with seasonality incorporated, and (c) our ICKy framework
G Experimental Details
641
642
G.1
Synthetic Data
643
We use the GPytorch package 15
to generate the synthetic data. Before feeding (11) into MLP, we first map (1)) into higher dimension using an unsupervised algorithm called Totally Random Trees Embedding 48 All the MLP structures in this experiment (including those in MLP-RF and ICKy) contain one single fully connected (FC) layer of width 1000, which serves as a simple benchmark since mee-hidden-layee MLP can only capture linear relationship between the input and output. For model training, we optimize Mean Squared Error (MSE) objective using Adam optimizer 30 with weight decay of O.I.
644
645
646
648
19

---

Table Model architecture and training details for remote sensing data experiment in Section 5.2
Backbone architecture details
Output FC layers dimension
Optimizer
Adam
# Conv blocks 1, Channels 16,
CNN-RF
1000, 512, 512,
81 = 0.9
Kernel size = 3, Stride
82 0.999
# Transformer blocks = 6,
Adam
ViT-RF
# Attention heads 8,
1000, 512, 512,
31 = 0.9
Dropout ratio 0.1
82 0.999
Transformer blocks = 6.
Adam
ViT-VET-RF
# Attention heads 8,
1000, 512,
3 = 0.9
Dropout ratio = 0.1
8 = 0.999
Transformer blocks 6,
Adam 31 = 0.9
# Attention heads 8,
MAE-ViT-RF
1000, 512, 512,
Dropout ratio 0.1,
0.999
Masking ratio = 0.75
# Conv blocks 1, Channels 16,
SGD momentum = 0.9
CNN-ICKY
1000, 512,
Kernel size = 3, Stride =
Transformer blocks 6,
SGD momentum = 0.9
ViT-ICKy
# Attention heads 8,
1000, 512,
Dropout ratio 0.1
# Transformer blocks 6.
SGD momentum 0.9
ViT-VICy
# Attention heads = 8,
1000, 512,
Dropout ratio 0.1
G.2
Remote Sensing Data
651
The model architecture and training details are listed Table Here p denotes the length of latent representations discussed in Section 4 Note that we use stochastic gradient descent (SGD) optimizer with momentum of 0.9 for ICKy as we realize that SGD helps ICKy find a local minimum on the objective more efficiently. We use MSE objective for ICKy and all benchmark models in this experiment.
653
655
656
G.3
UCI Machine Learning Repository Data
657
The MLPs (including the MLP part in ICKy) in this experiment share the same structure as the one used (T, which consist of hidden layers of width 128, 32, and 32, respectively. For plain MLP, cyclic MIP, and ICKy, we use the mean absolute error (MAB) objective put less weight on the GNP and AGNP, we maximize biased Monte Carlo estimate of the log- likelihood objective as discussed in 37 All these objectives are optimized
658
660
outliers and thus enhance the model performance.
662
by an Adam optimizer with 31 = 0.9 and Bg = 0.999.
H Adapting ICK for Classification
663
While regression tasks are the primary motivation for this paper, there are many ways adapt GPR for classification tasks. For example, a binary classification model can be created by using a sigmoid
685
55
or probit link [ on the output of the GP. Succinetly, given function f(z) GP (0, K K(x, x x²)).
687
the binary outcome probability is be given as p(y 1]f(x)) = off(x). Likewise, a multiple
688
classification model can be constructed by using multi-output GP (or multiple GPs) and putting
689
the outputs through a softmax function (55) or multinomial probit link (20). This strategy can be
670
summarized by calculating C' different functions fe(z) GP (0. (xx x²)) for 1, ..... C, where C' is the number of classes, and then calculating the class probabilities through a link function,
671
672
ply|z) x) softmax([fi ([f.(z).. fz(z), fo(x))).
This same logic can be used to construct a multiple classification model from ICKy, Succinetly, let 673
T. (x()) @ (22), where SNN,e denotes a neural network specific to the 2 class and (2) represents the Nyström approximation specific to the kernel for the eh class. We note that often in
20

---

multi-output case the kernel parameters are shared, and so (22) would be an identical vector for each class. Then, the output probabilities for a data sample as p(yla) (oft.ax((i......... This framework is learned with cross-entropy loss.
677
679
To provide proof-of-concey of this multiple classification strategy, we implemented this model on version of Rotating MNIST. In this task, a dataset was created by rotating each image in the dataset by uniform random value
680
[0, 2x), thus creating a dataset with 60,000 images each with an associated rotation covariate Φ. We implemented the above multiple classification model with
681
682
685
periodic kernel over the rotation angle. This strategy yielded an accuracy of 92.3% on the validation
data. This is lower than methods such spatial transformers [25] that report accuracy greater than 99%. However, those models explicitly use the fact that the information simply rotated, whereas function of angle. This ICK classification model is much closer in concept to the way Rotating MNIST is used to evaluate unsupervised domain adaptation. While the evaluation strategy is different than our random validation set, the sate-of-the-ass art accuracy on unsupervised domain adaption is 87.1 % [52]. Due to the lack of complete and fair comparisons, are not claiming that ICKy is state-of-the-arr for classification, but ICKy's s classification model does seem reasonable and viable based upon this result.
685
ICK is modeling smooth transformation in the prediction function
687
689
691
I
Generation, Accessibility, and Restrictions of the Data
892
The synthetic data y ( in Section 5.1 and y GP(O, + K2) in Appendix D are generated using the GPyTorch package. The remote sensing data in Section 3.2 is downloaded using PlanetScope API whose content protected by copyright and/or other intellectual property laws. To access the data on PlanetScope, the purchase of an end-user license is required. When this 696
693
694
695
697
manuscript is accepted, we will provide the codes we used to acquire the data. The UCI machine
learning repository data we use in Section 5.3 has an open access license, meaning that the data is freely available online.
689
21


Reviews:
[{"review_id": "sNcjo6L7ECd", "reviewer": "Reviewer_pXCj", "summary": "The paper proposes to blend the strengths of deep learning and the clear modeling capabilities of GPs by using a composite kernel that combines a kernel implicitly defined by a neural network with a second kernel function chosen to model known properties. Technically, the authors approximate the resultant GP by combining a deep network and an efficient mapping based on the Nyström approximation. Results on both synthetic and real-world data sets show the superior performance and flexibility of the proposal.", "questions": "See Weaknesses", "limitations": "the authors have adequately addressed the limitations", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper proposes a practical method to incorporate prior knowledge into NNs. It offers an effective way for processing data with multiple sources of information where some explicit priors can be introduced in the form of GPs.\n- The paper establishes a way to integrate NNs and GPs efficiently and train them jointly.\n- The empirical results are interesting and insightful.\n\n\nWeaknesses\n- Assumption 1 seems to apply to NN-GP kernels but does not apply to the approximate NN-GP kernels corresponding to finite-width NNs.\nIt does not apply to typical kernels like RBF/polynomial ones as well. Can you clarify its validity theoretically/empirically?\n\n- In the case of an NN (whose parameters are $\\theta^{(1)}$) plus a classic kernel (whose parameters are $\\theta^{(2)}$), the learning is to find a good distribution $p(\\theta^{(1)})$ and a point-estimate $\\theta^{(2)}$? If so, can you clarify how you parameterize $p(\\theta^{(1)})$? (E.g., a variational BNN with Gaussian variational whose mean is fixed as 0?) If not, and if you just learning a point-estimate $\\theta^{(1)}$, Lemma 1 would not apply and Theorem 2 does not hold.\n\n- The loss function for training is just the MSE error? Can the framework be extended to handle classification problems? It seems that it is non-trivial as the output should only be 1-D. \n\n- How to pre-define the integrating points? Can they be updated by the training signals?\n\n- How does the proposal compare to the recent work NeuralEF?\n\nNeuralEF: Deconstructing Kernels by Deep Neural Networks. ICML 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lXiq4OAkSCI", "reviewer": "Reviewer_qDxM", "summary": "The authors introduce a neural network construction that forms predictions by computing inner products between a hidden representation generated by a neural network and a kernel-derived basis of functions.  This construction is motivated by a desire to combine the flexibility of neural networks with the domain-specific knowledge that can be encoded in kernels (e.g. seasonality). The properties of the resulting method are explored on three data sets.", "questions": "- For M>2 the authors suggest using a \"chained inner product\" to form predictions (Appendix A). The authors also state (line 295ff) that \"A major limitation of ICK lies in our method of combining latent representations as the nature of inner product (i.e. the effect of multiplying small numbers) may cause vanishing gradient problems when we have a large number of sources of information (i.e. M is large).\" Have you tried M>3? When do you expect ICK to break down?\n- What happens when Assumption 1 (line 158ff) does not hold? Can you test this experimentally with an artificial dataset that breaks this assumption?", "limitations": "The authors do not discuss any potential negative societal impacts of their work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this submission, as I see it, is that one of the empirical comparisons includes a remote sensing data set, which would seem to be a rich test bed for these kinds of methods---though it should be noted that accessing this data set requires a PlanetScope license.\n\nUnfortunately, the weaknesses of this submission outweigh the strengths by a good margin. The novelty and empirical validation presented would seem to be more in line with a workshop submission than a full paper.\n1. The suggested method is more trivial than not and I am somewhat skeptical that very similar and/or identical methods do not already exist in the literature. However, even if we take it for granted that nothing like this has been done before, the empirical comparisons are too shallow, see below.\n2. The method isn't specified in sufficient detail. We are told (line 143ff) that \"the parameters of both the NN and the kernel function are learned via gradient-based optimization methods.\" OK but with what objective function? Mean squared error for regression tasks? Something else? Does the objective function matter? \n3. The number/variety of benchmark algorithms used in experiments is inadequate. What about a MLP that is given access to cyclical/fourier features in the time domain in the experiments in Sec. 5.3? What about various neural processes, e.g. reference [1] below? What about attention-based mechanisms? The omission of the latter is particularly troubling given the empirical success of attention-based methods and the similarity of attention-based mechanisms with your inner product construction. Given the simplicity of the proposed construction, a thorough empirical assessment (using a variety of datasets and benchmark methods) is absolutely crucial to convince the reader that the proposed method is useful in practice. In general the appendix reveals very little information about the MLPs used in the experiments. How many hidden units? How were these trained? etc. The proposed method may be interesting, but these toy experiments do not convince. \n\nReferences\n1. \"Convolutional Conditional Neural Processes,\"\nJonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, Richard E. Turner\n2. \"Attention Is All You Need,\"\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "klSjXjlfBHw", "reviewer": "Reviewer_4n3e", "summary": "The paper proposes a way to combine prior knowledge inside deep networks. It assumes data comes from different sources (at least one low-dimensional source and one high-dimensional source), and prior knowledge is known about the low-dimensional one (e.g., seasonality). They build a composite kernel where the low-dimensional data is processed with a user-defined kernel, while the high-dimensional one with a neural network. The former is then processed though kernel approximation methods to obtain a latent representation which is multiplied by the embedding of the neural network to get the final prediction. They perform several experiments ranging from a synthetic dataset to a realistic use case in remote sensing.", "questions": "- I think computational time should be discussed more. The Cholesky decomposition is cubic in the number of pseudo-inputs, and the overhead required by differentiating through it is unclear to me.\n- \"Incorporating Prior Knowledge\" is a pretty strong title and motivation for the paper, since the technique is feasible only in the very specific setting described in the summary. It is impossible (as far as I understand) to include generic prior information if it is relative to the high-dimensional data (e.g., an image). I think the claims in the paper should be downsized accordingly.\n- Since the kernel is included in the autodiff process, I wonder whether it is also possible to optimize some of its parameters (e.g., the seasonality) automatically. Have the authors considered this point?", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea is simple but (as far as I know, as I am not an expert in neural Gaussian Processes) novel. The paper is easy to read, the technique sound, and the experiments are interesting and relatively clear in the benefit of the method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XZ3ARXKB-UX", "reviewer": "Reviewer_UfZP", "summary": "The authors propose ICK, a regressor that can naturally incorporate priors over low-dimensional subsets of the input. A high-dimensional input is processed with a neural network to produce a latent vector, and the low-dimensional input produces a latent vector of the same size via a Gaussian process (GP), where the kernel function incorporates the known priors. The GP is efficiently integrated using the Nystrom method and Cholesky decomposition.", "questions": "As mentioned in related work, there are at least 1-2 other methods that equip NNs with priors using GPs. Can you compare with them or at least explain when/why they would be less effective than your approach?\nIt's not entirely clear to me what integral is being approximated with Nystrom quadrature. Is it the expectation in (10)? Might be helpful to briefly mention somewhere in 4.2.1 when you talk about integrating points.\nConsider rescaling the numbers in Table 1 so there aren't so many decimal places.\nPerhaps the conjugate gradient method is a feasible alternative to Cholesky decomposition if p needs to be larger? For future consideration.\nLine 90 chose -> choose", "limitations": "Yes. Assumption 1 seems reasonable to me, but the authors should try to characterize the cases where it doesn't hold. Perhaps a rigorous condition is hard to derive, but is there any intuitive argument that the cases where it doesn't hold are ill-behaved?", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe approach is an elegant combination of GPs and neural networks for incorporating useful priors, supported by a number of experiments.\nThe paper is well-written and the figures are very clear.\nRe: vanishing gradients as a limitation. I actually think your approach improves on the naive approach wrt vanishing gradients! Normally a network may ignore low-dimensional inputs even if they are informative, but your projection of both high-dimensional and low-dimensional data sources into a common latent space encourages them to be treated more equally.\n\nWeaknesses\nOne of the nice properties of GPs is that they can capture uncertainty in the prediction. However, the toy experiment suggests that this uncertainty is perhaps lost or not well calibrated. For example, I'm not sure that reverting to 0 (Fig. 3 bottom) counts as \"failing gracefully\" when the standard deviation is so small. Why is it confident in its bad prediction, and is there a way to mitigate this? It would be nice to see an experiment/metric that also demonstrates that your method matches distributions well.\nIt may be helpful to discuss what kinds of priors are amenable to being represented as the kernel function of a GP. It seems that this work focuses on continuous scalar variables where the prior represents periodicity, and the experiments are a little limited in this respect. Discuss some other types of priors and their corresponding kernels, and if possible add an experiment along this line.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject