Convolutional Deep Kernel Machines
Authors: Edward Milsom
Ben Anson
Laurence Aitchison
Venue: ICLR 2024 poster (2024)
Keywords: Gaussian process
infinite-width neural network
NNGP
Bayesian deep learning
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We develop a convolutional version of the recently introduced deep kernel machine algorithm, using a novel inducing point scheme and NN inspired techniques to obtain a practical algorithm for CIFAR-10 size datasets.

Abstract: Standard infinite-width limits of neural networks sacrifice the ability for intermediate layers to learn representations from data. Recent work (“A theory of representation learning gives a deep generalisation of kernel methods”, Yang et al. 2023) modified the Neural Network Gaussian Process (NNGP) limit of Bayesian neural networks so that representation learning is retained. Furthermore, they found that applying this modified limit to a deep Gaussian process gives a practical learning algorithm which they dubbed the “deep kernel machine” (DKM). However, they only considered the simplest possible setting: regression in small, fully connected networks with e.g. 10 input features. Here, we introduce convolutional deep kernel machines. This required us to develop a novel inter-domain inducing point approximation, as well as introducing and experimentally assessing a number of techniques not previously seen in DKMs, including analogues to batch normalisation, different likelihoods, and different types of top-layer. The resulting model trains in roughly 77 GPU hours, achieving around 99\% test accuracy on MNIST, 72\% on CIFAR-100, and 92.7\% on CIFAR-10, which is SOTA for kernel methods.

Published as a conference paper at ICLR 2024
CONVOLUTIONAL DEEP KERNEL MACHINES
Edward Milsom
Ben Anson School of Mathematics University of Bristol ben www.nebristooo.ocoooooooo. uk
School of Mathematics
University of Bristol
edward. mi mwlssm@bristol.co
Laurence Aitchison
Department of Computer Science
University of Bristol
laurence aitchison com
ABSTRACT
Standard indtnite-width limits of neural networks sacrifice the ability for inter- mediate layers to Iearn representations from data. Recent work ("A theory of representation learning gives deep generalisation of kernel methods", Yang et al. 2023) modified the Neural Network Gaussian Process (NNGP) limit of Bayesian neural networks so that representation learning is retained. Furthermore, they found that applying this modified limit to deep Gaussian process gives practical learning algorithm which they dubbed the "deep kernel machine" (DKM). How- ever, they only considered the simplest possible setting: regression in small, fully connected networks with e.g. 10 input features. Here, we introduce convolutional deep kernel machines. This required us to develop novel inter-domain inducing point approximation, well as introducing and experimentally assessing a number of techniques not previously seen in DKMs, including analogues to batch normali- sation, different likeliboods, and different types of top-layer. The resulting model trains in roughly 77 GPU hours, achieving around 99% test accuracy on MNIST, 72% on CIFAR- 100, and 92.7% on CIFAR- 10, which is SOTA for kernel methods.
INTRODUCTION
A key theoretical approach to studying neural networks is taking the infinite-width limit of a randomly
initialised network. In this limit, the outputs become Gaussian process (GP) distributed, and the
resulting GP is known as an NNGP (Neal 1995 Lee et al. 2017 Matthews et al.] 2018). However,
despite a large body of work attempting to improve the predictive performance
NNGPs (Novak
| Shankar cct al. 2020: Adlam et al., 2023). they still empirically underperform finite NNs. Understanding why understanding of neural networks. One hypothesis (Aitchison, 2020 MacKay 1998) is that the NNGP lacks representation learning, as the NNGP kernel is a fixed and
et al. 2018 Garriga-Alonso et al. 2018 Arora et al. 2019: Lee et al. 2020 Li et al.
this gap exists will improve
deterministic function of the inputs, while the top-layer representation in finite neural network is
learned; representation learning is believed to be critical to the success of modern deep learning (Bengio et al.) 2013) LeCun et al., 2015). One way to confirm this hypothesis would be to find an NNGP-like approach that somehow incorporates representation learning, and gives improved performance over the plain NNGP. While some theoretical approaches to representation learning have been developed (Antogmimi] 2019; & Sur-Arri | Hanin & Nical 2019 Aitchison 2020
Li & Sompolinsky |0200] 2020: Naveh et [l][202]]/avatone
Zavatone- Veth et al. 2021 Zavatone- Veth
& Pehlevan| [222]] Roberts et al. 2021 Naveh & Ringel][002]]]]]
et al. 2021 Seroussi et al.
2023), they are not scalable enough to apply to many common datasets (e.g. Seroussi et al. 2023 could only consider a subset of 2048 points from CIFAR-10). Here, we develop the convolutional deep kernel machine (DKM), an NNGP-like theoretical approach which captures representation learning and be scaled to datasets like CIFAR-10 10. Convolutional DKMs reduce (but still do not fully close) the gap between theoretical approaches and real neural networks, giving further evidence that representation learning the critical component missing from approaches like the NNGP.

---

Published as a conference paper at ICLR 2024
Our theoretical approach is based on DKMs Yang et al. 2023). DKMs are fundamentally deep Gaussian process (DGP), where the infinite- limit has been taken carefully, to ensure that representation learning is retained. While DGP is not NN, they are very similar: both DGPs and NNs are deep, nonlinear function approximators that admit representation learning, and many formal connections between NNs and DGPs have been drawn in the literature (Dutordoir et al.) 202T] Agrawal et al.) 2020, Pleiss & Cunningham 2021). Furthermore, DGP-based theoretical approaches (i.e. DKMs) offer considerable simplifications over NN-based approaches. These mostly arise from the fact that the posterior over features in an infinite-wide DGP is exactly multivariate Gaussian, even when that infinite-width limit retains representation learning. This is an important factor in the scalability of DKMs.
Additionally, Yang et al. 2023) developed inducing- point methods for DKMs inspired by the DGP literature, making them particularly scalable. Naive, full-rank DKMs have cubic time-complexity in the number of training points, which is intractable in all but the smallest datasets. By contrast, inducing- point DKMs require linear compute in the number of datapoints (and cubic compute in the far smaller number of inducing points). However, Yang et al. (2023) only considered regression in small connecco architectures with e.g. 10 input features. To compare the performance of DKMs to CNNs on datasets such as CIFAR- 10, we must develop convolutional DKMs.
However, developing efficient, scalable DKM inducing point scheme for the convolutional setting is highly non-trivial. Naively applying the inducing point inference scheme from Yang et al. (2023)
implies taking images inducing points, resulting prohibitive O ((P;) W H3) (P operations, where P
is the number of inducing points, and W, // are the width and height of the images. One alternative
inducing patch schemes from the GP literature (van der Wilk et al.) 2017 Blomqvist al. 2018
Dutordoir et al. 2020); however they cannot be applied to DKMs, as they rely on intermediate features, whereas DKMs only propagate kernel/Gram matrices through the layers, not features. We were thus forced to develop an entirely new domain (Lázaro- -aeaalllrae & Figueiras- Vidal [2009] Hensman et al. 2017 Rudner et al. 2020) convolutional inducing point scheme, designed specifically for the DKM setting. To summarise, our contributions are:
Introducing convolutional DKMs, which achieve SOTA performance for kernel methods on CIFAR-IO with accuracy of 92.7%.
Introducing an efficient inter-domain inducing point scheme for convolutional DKMs, which allows our largest model train in 77 GPU hours, 1-2 orders of magnitude faster than full NNGP/ / NTK / Myrtle kernels.
Developing a number of different model variants, empirically investigating their perfor- mance, including various normalisation schemes (Appendix C) two likelihoods (Ap-
pendix D.2)
and two different top-layers (Appendix F3
2 RELATED WORK
DKMs were developed in Yang et al. (023), ), using a modified NNGP limit that retains representation learning. However, they only considered regression in small fully-connected networks.
The NTK setting (Jacot et al. 2018) is a theoretical approach to understanding neural networks that is also based on widhh limits. However, the NTK deals with very different phenomena from the NNGP/DKM setting, in that it describes the dynamics of gradient descent applied to a loss function. By contrast, the NNGP/DKM (Yang et al. 2023) describes Bayesian networks under the posterior, which does not, in principle, involve any dynamics. A problem arises in the NTK setting, whereby the NTK, describing the dynamics of gradient descent, is constant through training
particular infinite-width limits. Recent work on the mu-P scaling has resolved this issue, by finding alternative limits in which the NTK is able to evolve through training (Yang & Hu 2021 Yang et al. 2022).
There is body of literature on infinite-width convolutional neural networks (CNNs) NNvaa et al. 2018 Garriga-A Alonso et al. 2018). specifically in the NNGP setting (Neal 1995 Lee et al. 2017 7 Matthews al.) 2018). As mentioned before, NNGPs have kernel that fixed, deterministic function of the inputs, and therefore do not allow for flexibility in the learned kernels/representation. unlike real, finite CNNs and unlike convolutional DKMs (Aitchison) 2020 Yang et al.) 2023).
Convolutional DGPs use inducing patches for scalability (van der Wilk et al., 2017 Blomqvist et al. 2018. Dutordoir et al. 2020). However, we cannot their scheme here, requires features
2

---

Published as a conference paper at ICLR 2024
at each layer, while we work solely with Gram matrices. Therefore we were forced develop new scheme. Our DKM inducing point scheme is inter-domain (Lázaro-Gredilla & Figueiras- Vidal 2009; Hensmmnn et al. 2017 Rudner et al. 2020), in the sense that the inducing points do not mirror datapoints (images), but instead store information about the function in a different domain. Existing work on inter-domain inducing points does not address the DKM convolutional settings.
An alternative approach to getting a flexible kernel is to take the inputs, transform them through a NN (e.g. 10-40 layer CNN), then to use the outputs of the NN as inputs a standard kernel. This approach is known as deep kernel learning (DKL; Calandra et al. 2016; Wilson et al.. 2016b a Bradshaw et al., 2017] Bohn al. 2019 Ober et al., 2021). DKMs are very different to DKL that there is no underlying neural network. Instead the DKM directly transforms and allows flexibility in the kernels.
There are a small number of other "deep kernel" approaches such as convolutional kernel machines, (Mairal et al. [2044] Mairal 2016, [Suykens, 2017 Chen et al., 2019 TTnnn et al., 2023] Achten et al.) 2023). These differ from approach in that they lack the theoretical connection to representation learning in NNGPs, and thus cannot answer the questions raised in the Introduction. Moreover, many of these methods use a neural-network like architecture to give a flexible, feature-based approximation to an underlying fixed kernel, whereas our approach fundamentally allows for flexibility in the
underlying kermel/representation;; we believe this explains our performance advantage (Table 3} Nonetheless, there are fruitful connections to be drawn between NNGPs, physics-based theory (e.g. Seroussi et al., 2023) and e.g. convolutional kernel networks (Mairal et al. [014 Mairal 2016). and we hope that our work contributes to fostering dialogue between these areas.
3 BACKGROUND
Deep Kernel Machines. detailed derivation of DKMs can be found in Appendix ^ or Yang et al. (2023). In a nutshell, DKMs are derived from deep Gaussian processes where the intermediate layers F RPxN, with P datapoints and N₂ features, have been made infinitely wide (N₂ oo). However, in the traditional wiith limit, representations become fixed (e.g. [itchison 2020
Yang et al. 2023). The DKM (Yang et al., 2023) modifies the likelihood function so that, in the infinite limit, representations remain flexible and adapt to data. While the full derivation is somewhat involved, the key idea of the resulting algorithm is to consider the Gram matrices (which are P x P matrices like kernels) each layer of the model as parameters, and optimise them according an objective. Here we give short, practical introduction to DKMs.
As DKMs are ultimately infinite-width DGPs, and we cannot practically work with indtnite-width features, we must instead work in terms of Gram matrices:
(1) Note that from this point on we will always refer to Go in Eq. 11) as a "Gram matrix", whilst the term "kernel matrix" will carry the usual meaning of matrix of similarities computed using a kernel function. Ordinarily, at each layer, DGP computes the kernel matrix Kreatres (Fé-1) E IR PxP from the previous layer's features. Since we no longer have access Fé-1, but instead propagate G computing Kreaures (F--1) is not possible in general. However, it turns out that for many kernels of interest, we can compute K from G For example, isotropic kernels (such as the squared exponential) only depend on F through the normalised squared distance Rij between points, which can be computed from the Gram matrices,
e Nr F (F+)" E PXP.
R1j N.. Ei F22)² N, EX (F3 2F + F)A) = G - -2G + G
(2)
and hence we can instead compute the kernel matrix as
K(Gf-1)- Kyeatures (F* (F--1)
(3)
Here, K(-) and Kreatues (.) are functions that compute the same kernel, but K(-) takes a Gram matrix as input, whereas Kfeaues (-) takes features as input. Notice that (GG e RPx has the same shape as G g- and so it is possible to recursively apply the kernel function in this parametrisation.
taking GO i X where X E R.. Px the input data:
No
G² = K(G
K(K K(G))) )))
(4)
times
3

---

Published as a conference paper at ICLR 2024
It turns out Eq. exactly describes the traditional infinite-widd limit of DGP (i.e. the NNGP limit). Notice that G" is a fixed function of G and thus cannot adapt to data (perhaps with the exception of small number of tunable kernel hyperparameters). This is major disadvantage compared to finite NNs and DGPs, which flexibly learn representations at each layer from the data.
The DKM solves this problem by taking modified limit, the "Bayesian representation learning limit" (Yang et al., 2023). Under this alternative limit, the Gram matrices have some flexibility, and are no longer fixed and equal to those given by Eq. 4 Instead, the L Gram matrices, G G...... G4, become parameters which are optimised by maximising the DKM objective,
(5) The first term encourages good predictions by maximising the log-likelihood of the training data Y, whilst the other KL-divergence terms form a regulariser that encourages G to be close to K (G The model reduces to the standard infinite-width limit (Eq. 4 when no data is observed, since the KL- divergence is minimised by setting G' = K(G Formally, the DKM objective is the evidence lower bound (EIBO) for an indt DGP in the Bayesian representation learning limit. After optimising the training Gram matrices, prediction of unseen test points (which requires us to predict the test-test and test-train Gram matrix blocks for each layer progressively before predicting the final layer output) proceeds using algorithm inspired by DGPs (see Yang et al. 2023 for details).
G = log (YIG*) Ei DRL (N (o, (")
Sparse DKMs. As with all naive kernel methods, directly computing the DKM objective for all data is OPP)). where P, is the number of datapoints, which is typically infeasible. As a DKM is ultimately an infinite-wide width DGP parameterised by Gram matrices rather than features, Yang et al. (2023) developed an inducing point scheme inspired by the DGP literature.
In the GP context, inducing point schemes reduce computational complexity by using variational inference with an approximate posterior that replaces the training set (of size P)) with set of P, pseudo-inputs, called inducing points, where P, is usually much smaller than P.. The inducing points can be learned as variational parameters. In DGPs, each layer is GP, we learn one of inducing points for each layer. When using inducing points, the features at each layer are partitioned into inducing points F and training points FÇ. Likewise, the kernel matrix is partitioned into blocks corresponding to inducing and training points. In DKM, we partition Go similarly,
G = (G
( (.())) Kr G K.,
G
K(G')
(6)
with G. E RXX G E RAXB, and G E R At each layer L, G is learnt, whilst G and Gf are predicted using G and K ((G(=1) (see Yang et al. 2023). The sparse DKM objective is
Lima G GP--LLL (Y: G G Gb:)
Σ Us DEL (N (0, G (O,
where GP-ELBO (Y; GY G G.)) is the ELBO for sparse GP with data Y, and kernel K (G)) and contains a likelihood term which measures the performance of the model on training data. We consider two different likelihoods: categorical and Gaussian (see Appendix D for further details). This scheme is efficient, with time complexity O(L(P, + P)). both because the KL terms in the sparse DKM objective depend only G.,, and because obtaining predictions and computing the performance term in the DKM objective requires only the diagonal of G (which mirrors the requirements for the GP ELBO; see Yang et al. 2023 for details).
METHODS
The overall approach to defining convolutional DKM is to build convolutional structure into kernel, K(-), by taking inspiration from the wndinite-width convolutional neural network literature (Novak et al. 2018: Garriga-Alonso et al. 2018). Specifically, will define the kernel to be the covariance between features induced by single convolutional BNN layer.
Take convolutional BNN layer, with inputs H PSe 1xN, and outputs F R N,, Here, P is the number of images, N₂ is the number of output channels at layer E, and S; is the number of output spatial locations in the feature map (e.g. S² W₁ H2 for a 2D image). The convolutional
4

---

Published as a conference paper at ICLR 2024
weights W P and D is the number of spatial locations in filter (e.g. a x 3 convolutional patch would have D = 9). The BNN convolution operation (i.e. Conv2d in PyTorch) can be written as
F+r ir.A En
Ne- H ((r+d), dux
W
(8)
Here, indexes the image, r the spatial location in the image, the location in the patch, and μ and the input and output channels, respectively. This is natural for ID convolutions, e.g. E {1,..., St......) and d E D = {-1, O, 1} (with = 3 in this example), but also generalises to higher dimensions, e.g. using vectors. In a typical CNN, would apply nonlinearity é, e.g. relu, to the output of the previous layer to obtain the input to the next layer, i.e. H ((F(11)
We will define the kernel the covariance between the outputs of the convolutional layer (Eq. 8). I fr = E
F P.P A
H
where we have placed an IID Gaussian prior on the weights:
ja.2
P (WWax) N
-O-WNWWa (WauxiO, DN....
Note that this implies all features (indexed by A) are IID, so we consider an arbitrary feature X. This
is valid even if N₂ + oo. Since the weights have zero mean, the distribution over features (Eq. 8 has zero mean, so ir.jj is just the conditional covariance of the output features. To compute this covariance it will be useful define a Gram- like matrix f R PSc PXPS 1 from the input H',
Ω
Nr-1 H (H"-))² "H''""
PS
IXPS:
No M=1
Hjs
(10)
No-1
For the matrix multiplication to make sense, we have interpreted H RR with PS,-1 rows and Ne-1 columns. The covariance I irrjs can be computed (Appendix B.3 as,
1 as matrix
Tings (2 (0- = E [PL FF H
1 Df ((++d))(++d)
(11)
This is not yet a complete kernel function for our convolutional DKM since it does not take the previous layer Gram matrix G (Eq. 1) as input. To complete the kernel, recall that H is defined by applying e.g. a ReLU nonlinearity to F Taking inspiration from infinite NNs, that would suggest we can compute 2 from G G using an arecos kernel (Cho & Saul 2009
Ω ((
(12)
Here, 12(-) is function that applies e.g. an arccos kernel to Gram matrix, while f is the specific value at this layer. Combining this with Eq. (11 we obtain the full convolutional DKM kernel,
Koom 2eamm(G (G--1)= = F
T( E IR PS
(13)
where in Eq. (3) we have written the equation for the entire matrix instead of a specific element as in Eq. 11 Here, f R PSc denotes single feature vector, i.e. the X" column of F*. From this perspective, r(-) (Eq.
11 can be viewed as the kernelised version of the convolution operation.
Sparse convolutional DKMs. Computing the full kernel for all training images is intractable in the
full-rank case. For example, CIFAR- 10 has 50 000 training images of size 32 x 32, so we would need invert a kernel matrix with 50 000 32 32 51 200 000 rows and columns. Hence, we need a more efficient scheme. Following Yang et al. 2023). we consider a sparse inducing point DKM. However, inducing points usually live in the same space as datapoints. If we take inducing points as images, we end up with G E Se P SPP. This is still intractable, even for a small number of images. For CIFAR-10, S = 32 - = 1024, so with as few as P. = 10 inducing points, G is 10, 240 X 10, 240 matrix, which is impractical invert. Hence we resort to inter-domain inducing points. However, we cannot use the usual inducing patch scheme (van der Wilk et al. (017)), as it requires access to test/train feature patches intermediate layers, but we only have Gram matrices.
Therefore we propose a new scheme where the inducing points do not have image-like spatial structure. In particular, the Gram matrix blocks have sizes Gf E R' G Pf xS.PP and G with one row/column for
P. so that the full Gram matrices are G E ()) each inducing point, and additional row/column for each location in each of the train/test images.
5

---

Published as a conference paper at ICLR 2024
Note that Pf will be able vary by layer in our cestucing-poinn scheme. All kernel/Gram matrix like objects (specifically, the outputs of [() and S2(-)) have this same structure.
We now show how define these inducing points and compute the relevant covariances, forming the primary technical contribution of this paper. Previously, we considered only spatially structured H and F Now, we consider H E (R + PP) formed by concatenating non- spatial H R. - (subscript "1" for inducing points) with spatially structured H
15x-1 (subscript "t" for test/train points). Likewise, we have F' + formed by combining non-spatial F/ R x with spatially structured F CR
H
F F
H f-1
F
(14)
As in Eq. 10 we define 22 as the normalised product of the activations with themselves. Breaking this is
this expression into blocks (with the bottom right block 22 matching Eq.
10p. H H
12 - Ω
H
H H
T
H
n -
2 2
Ne-1
H
H
(15)
Note that while the features in Eq. 14 can only be realised for finite Ng. Eq. 15 valid as N₂ co. As in Eq. 13 the kernel matrix Koom (G (G--)) is formed as the covariance of features, with f E
R +S,R) single feature vector column of F Broken into the inducing and test/train blocks,
T
Conv K Conv
KConn) KCone
(u((2 -1 [ # -1)
T ( T. (2
pice
pise
(ES) E (PA)
#
H
f
pizl
(16)
At high-level, this approach natural extension of the sparse DKM procedure (Yang et al. 2023). particular, the test/train block (u remains the same as Eq. (13). However, to get concrete forms for Fn and Fú we need to choose r/e For this pie to work well, it needs to have useful, informative correlations with Fic One way achieve this is to define F in terms of the same weights W as F, (from Eq. 8). thereby making them correlated. However, it is not obvious how to do this, as W in Eq. 8 is a convolutional filter, with spatially- while F E R P X has no spatial structure. Our solution is create artificial patches from linear combinations of the inducing points using a new learned parameter, C² E DD x
inputs and outputs,
F = ΣΣΣ Waper
(17)
dED
Intuitively, Ev Ca. H.. H takes non-spatial inducing inputs, H and converts them into tensor with shape DP x Nr-1, which can be understood as patches. Those patches can then be convolved with W to get single output for each patch, F As neat side effect, note that changing Pf in C E DDD x lets us vary the number of inducing points at each layer. The parameters C are learned by optimising Eq. 77). Using F., we can then compute the covariances (Appendix B) to obtain Ia (1 and Tú ( (we again emphasise that all features are IID, so the covariance is defined for an arbitrary feature A):
P
p-11
TH (2
E FiR
H
C Ca S
Ca -it-11
(18)
dED
P=1 y'=1
pe-1 Cde
1. (S)
E
pill
F Ja.A
H
01::-11 Di j(s+dd)
(19)
DED
This fully defines our inducing point scheme. At the final layer, we collapse the spatial dimension (either using linear layer global average pooling, see Appendix and then perform classification using the final Gram matrix G as the covariance of a GP. For efficiency, we only store the diagonal of the test/train "n"" block all layers, which is sufficient for IID likelihoods (Appendix E.2).

---

Published as a conference paper at ICLR 2024
Table Test accuracies (%) for model selection experiments. Base model used Ny Batch Batch normalisation, Batch Batch rescaling, Global Average Pooling, and categorical likelihood. Bold shows values statistically similar the maximum each category, according a one-tailed Welch test.
Category
Setting
MNIST
CIFAR-10
CIFAR-100
00 (inducing NNGP)
72.19 0.11
53.74 0.38
30.94 0.54
10*
80.79 0.49
52.26 0.27
28.54
0.11
102
96.96 0.10
71.06 0.11
48.23
0.11
Regularisation Strength ve
100
99.20 = 0.02
87.86 = 0.09
64.08 = 0.24
10 10-2
99.16 = 0.05
89.48 = 0.15
65.64 0.20
10-4
99.14 + 0.05
89.68 + 0.03
65.91 + 0.20
0
99.16 0.05
89.75 = 0.09
65.93 0.19
Batch / Batch
99.20 = 0.02
87.86 = 0.09
64.08 + 0.24
Normalisation (Inducing / Train-Test)
Batch / Location
99.05 + 0.03
87.91 + 0.17
64.05 + 0.27
Local Image
99.14 0.01
87.73 0.12
64.37 0.28
Local / Local
99.13 0.05
87.54 = 0.09
63.75 0.11
None / None
99.16 + 0.04
87.50 = 0.19
64.23 + 0.08
Batch Batch
99.20 0.02
87.86 0.09
64.08 0.24
Batch / Location
99.07 0.02
87.98 0.16
62.72 0.15
Rescaling
Local / Batch
99.19 + 0.02
87.56 1 0.05
63.92 1 0.13
(Inducing / Train-Test Test)
Local / Location
99.03 + 0.03
87.80 - 0.04
63,30 + 0.43
Local None
99.13 0.04
87.75 0.04
63.88 0.20
None / None
99.13 + 0.04
87.77 0.11
64.05 + 0.14
Global Average Pooling
99.20 0.02
87.86
0.09
64.08 0.24
Lincar
99.13 0.03
87.89 0.15
64.06 0.19
Top Layer
GAP + Mixup
99.15 = 0.01
87.69 - 0.10
64.42 - 0.34
Linear + Mixup
99.12 + 0.02
87.66 1 0.05
63.01 + 0.08
Gaussian
99.05 0.01
85.96 + 0.29
26.93 + 0.69
Likelihood
Categorical
99.20 + 0.02
87.86 + 0.09
64.08 + 0.24
Table 2: Test accuracies (%) using different numbers of inducing points in the ResNet blocks, selecting the best hyperparameters from Table T Bold shows values statistically similar to the maximum in each category, according to a one-tailed Welch test. Also included is walltime per epoch.
Ind. Points
MNIST
CIFAR-10
CIFAR-100
Time per epoch (s)
16/32 64
98.99 0.06
83.15 0.26
53.89 0.25
45
32/ / 64 / 128
99.09 + 0.05
86.75 + 0.14
61.10 + 0.22
54
64 128 / 256
99.12 1 0.01
88.99 + 0.05
64.31 0.19
82
128 256 512
99.10 0.04
91.00
67.14
155
256 512/ / 1024
99.16 0.03
92.17 +.10
70.33
0.23
420
512 / 1024 2048
99.26 0.02
92.69 + 0.06
72.05 + 0.23
1380
EXPERIMENTS
Model variants To fully evaluate our algorithm, it was necessary to introduce and test variety of model variants and new techniques. particular, we consider hyyorparaaeterr//acchhtcuuuu choices: the regularisation strength constant in the objective function (Eq. 7) the normalisation and rescaling schemes (Appendix[C C), the top layer, iee. linear (Appendix E.I vs. global average pooling (Appendix E.3), and the likelihood (Appendix D.2). We vary each hyperparameter in turn, starting from a reasonable "default" model, which uses v/ = 1 (we use the same v, for all layers), "Batch / Batch" normalisation and "Batch/ Batch" scaling (we apply different variants the inducing and
7

---

Published as a conference paper at ICLR 2024
Table
Comparison of test accuracy against other kernel methods with and without learned parame-
ters.
Paper
Method
CIFAR-10
This paper
DKM-DA-GAP
92.69 %
Novak et al. 20118
NNGP-GAP
77.43%
Kernel methods without
Arora et al.
2019
NNGP-GAP
83.75%
Lee al. 2020)
ANGP-GAP-DA
84.8%
Li (2019)
NNGP-LAP-fip
88. 88.92%
parameters
Shankar
(2020)
Myrtle10
89.80%
Adlam et al.
2023
Tuned Myrtle 10 DA CG
91.2%
(Shi et al.
2020)
SOVI Conv GP
68.19%
Kernel methods with
(Blomqvist et al.
2018
Conv DGP
75.89%
Dutordoir al. 2020)
Conv DGP
76.17%
parameters
Mairal cl al. 2014)
CKN Sup CKN
82.18%
Maira 2016
89.8%
test-train blocks the Gram matrices, see Appendix C global average pooling, and a categorical likelihood.
We used a model architecture that mirrors the ResNet20 architecture (He et al. 2016), meaning use the the same number of convolution layers with the same strides, we normalisation in the same places, and we use skip connections, which compute a convex combination of the Gram matrix outputted by block with the imputted Gram matrix, thereby allowing information "skip" over blocks. To mirror the ReLU, we used a first-order arccos kernel (Cho & Saul (009) for ()--) everywhere. Where the original ResNet20 architecture uses [16,32,64) channels its blocks, initially used (128,256,512) inducing points (i.e. we set P. = 128 for layers e contained in the first block, P = our inducing point scheme allows P to vary across layers, which is not true of the original sparse DKM. { 128.256.512) was chosen balance model capacity and computational resources for the model selection experiments. The model was implemented in PyTorch (Paszke et al. 2019). using double precision floating points. We optimised all parameters using Adam, with 31 = 0.8, B2 0.9, training for 100 epochs and dividing the learning rate by 10 at epochs 40 and 80, with an initial learning rate of 0. More careful optimisation (e.g. more epochs and less aggressive scheduling) could potentially give better results, but in the interest of total experiment time we chose this heuristic. used data augmentation (random cropping and horizontal flips), and batch size 256. also used ZCA (zero-phase component analysis) whitening, which is commonly used in kernel methods (e.g. Shankar et al. 2020 Lee et al. 2020) (for ZCA regularisation parameter, e, we used 0.
256 in the second block, and P = 512 in the third block). This is only possible since
We initialise the inducing inputs by randomly selecting patches from the training set, and then initialise the inducing Gram blocks G at the NNGP case by propagating the inducing inputs through the model and setting G at each layer equal to the incoming kernel matrix block K. The inducing output parameters y and A are initialised randomly from Gaussian distribution (for A we initialise L as Gaussian and then compute A LL to ensure A is positive-definite. We initialise the mixup parameters C randomly.
We tested on the MNIST (eCCun et al. 1998), CIFAR-10, and CIFAR- 100 KKrizhessky & Hinton 2009) image classification datasets. We report mean test accuracy with standard errors (over random seeds) in Table see Appendix for other metrics. For each hyperparameter and dataset, we bold all settings whose performance was statistically similar to the best, using a one-tailed Welch's t-test 5% significance level. For the normalisation and rescaling schemes, test certain combinations for the inducing and test-train blocks that we feel are natural; see Appendix F2 for more details.
Code: http:////..t.tccccc/////aaaaaaat https///ww/wvvokwpacceecem.
'The w do for MNIST used an initial learning rate of 0.001 due numerical issues caused by 0.01.
https://yanne www.lleecnnccccceeeeeeeee hicp://
Licence: CC BY-SA 3.0
html Licence: Unknown
8

---

Published as a conference paper at ICLR 2024
Setting oo causes the KL term in the objective to dominate, and forces G a standard fhiiite-wwdthh CNN (Novak et al. 2018 Aloniga-Aoonso et al. 2018). As v controls the strength of the regularisation, performance decreases for larger values, as this causes the model to underfit. On the other hand, we expect the model to overfit if we set y too small, and we indeed observe degradation in test L.I. (Table 4 though not accuracy when we set
K(G as in
O. We find p=10-2 is the strongest regularisation statistically indistinguishable from optimal accuracy. We did not observe meaningful differences in test accuracy between any of the normalisation or rescaling schemes. though not using any normalisation or rescaling can negatively impact train performance (see Table 5). Hence we selected the next simplest scheme, "Batch Batch", for the benchmark experiments later on. This scheme divides the Gram matrix by the mean of its diagonal elements for the normalisation, and multiplies each block by a learned scalar for the rescaling. We saw little difference between the top layers, so we settled on global average pooling for later experiments, reflecting common practice in deep learning and NNGPs (e.g. Novak et al., 2018). We also tried adding inducing point "mixups" similar to the mechanism in our convolutional layers, but this seemed to have no benefit. Using a categorical likelihood provides moderate advantages for CIFAR-10 10 and MNIST, which have 10 classes, and dramatic improvements for CIFAR-100, which has 100 classes.
Benchmark performance. Using what we learned from the model selection experiments, we ran set of experiments with us 10-2, "Batch / Batch" normalisation and rescaling, global average pooling, and a categorical likelihood. In these experiments, we varied the number of inducing points in each block from {16, 32, 64) (in the first, middle, and last blocks) up to (512, 1024, 2048). We trained for 200 epochs, reducing the learning rate by a factor of 10 at epochs 80 and 160, starting from an initial learning rate of 0.01. Mean test accuracies are given in Table 2 with other metrics in Appendix F As expected, increasing the number of inducing points increases observed test accuracy, with diminishing returns for MNIST, moderate improvements for CIFAR-10, and considerable improvements for CIFAR-100. Our largest model, with 2048 inducing points in the last block, achieved 99.3% test accuracy on MNIST, 92.7% on CIFAR- 10, and 72% on CIFAR-100. Training this model on CIFAR- DIFAR-10//100 took around 77 hours on single NVIDIA A100, but we emphasise that we used double precision floating points for simplicity of implementation, and with some care taken to preserve numerical stability, this could be dramatically sped up using single precision arithmetic.
we compare our best model against other pure kemel methods, including those without parameters like NNGP and NTK, and those with parameters like convolutional kernel networks. Our model outperforms even the best pure kernel methods (Adlam) al. 2023 Shankar al., 2020 Li et al. 2019: Mairal (016). These models rable to ours, as we developed an efficient inducing point scheme, whereas NNGP/NTK methods usually compute the full training kernel matrix. Benchmarks from Google's Neural Tangents library found that the most efficient Myrtle-5/7/1 kernels from Shankar et al. (2020) took 3 316/330/508 GPU hours for the full 60 000 CIFAR-10 dataset on an Nvidia VI00, which is around one order of magnitude more than our training time. Overall, we have narrowed (but not entirely eliminated) the gap to NNs by improving the performance of kernel methods.
Comparisons against other models. In Table 3
directly compa-
6 CONCLUSION AND LIMITATIONS
We have developed convolutional DKMs, and efficient inter-domain inducing point approximation scheme. We performed a comprehensive set of experiments to investigate the effect of different architectural choices, before selecting the final architecture to compare against existing methods. Our best models obtained 99% test accuracy on MNIST, 92.7% on CIFAR-10 and 72% on CIFAR-100. This is -tate-of-the-art for kernel-methods infinite NNs, bringing theoretical approaches closer in performance to neural networks, and there is considerable room for development to close the gap even further. One limitation of our work is that we did not use high resolution image datasets like ImageNet (Deng et al., (209) as we have not yet developed the efficient tricks (such as lower/mixed precision floating point arithmetic, more efficient memory utilization, and numerical methods for e.g. inverting matrices) to enable these larger datasets. It may be possible to adapt existing ideas from the kernel literature (Adlam et al. (2023) and Maddox et al. (2022), for example) to address some of these issues, in addition to leveraging multi-GPU setups, but we leave this to future work.
g wttbb.ccm//ooole//eerrrr -tangents

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
Edward Milsom and Ben Anson are funded by the Engineering and Physical Sciences Research Council via the COMPASS Centre for Doctoral Training at the University of Bristol. This work was carried out using the computational facilities of the Advanced Computing Research Centre, University of Bristol hhtt:: //ww/. bris ac. wk/acrc// We would like thank Dr Stewart for funding compute resources used in this project.
8 REPRODUCIBILITY STATEMENT
The experiments in this paper may be reproduced using the code and instructions given in the following publicly available GitHub repo: https: com edwawdddddmmmm wwammilsmmmmoimmmmmmmmmmooonmeeo cow.ckmppccccpp
REFERENCES
Sonny Achten, Francesco Tonin, Panagiotis Patrinos, and Johan AK Suykens. Semi-supervised classification with graph convolutional kernel machines. arXiv preprint arXiv:2301. 13764, 2023.
Ben Adlam, Jaehoon Lee, Shreyas Padhy, Zachary Nado, and Jasper Snoek. Kernel regression with infinite-width neural networks on millions of examples, 2023.
Devanshu Agrawal, Theodore Papamarkou, and Jacob Hinkle. Wide neural networks with bottlenecks are deep gaussian processes. J. Mach. Learn. Res., (1), jan 2020. ISSN 1532-4435.
Laurence Aitchison. Why bigger is not always better: finite and infinite neural networks. In ICML., 2020.
Laurence Aitchison, Adam X. Yang, and Sebastian W. Ober. Deep kernel processes. In ICML, 2021.
Joseph M Antognini. Finite size corrections for neural network gaussian processes. In ICML Workshop on Theoretical Physics for Deep Learning, 2019.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1788-1825 2013.
Kenneth Blomqvist, Samuel Kaski, and Markus Heinonen. Deep convolutional gaussian processes, 2018. UKL/:ttpss//rrxiv.orr//bbbb/
03052
Bastian Bohn, Christian Rieger, and Michael Griebel. A representer theorem for deep kernel learning. The Journal of Machine Learning Research, 2031):2302-3333 2019.
John Bradshaw, Alexander G de G Matthews, and Zoubin Ghahramani. Adversarial examples, uncertainty, and transfer testing robustness in gaussian process hybrid deep networks. arXiv 2017.
preprint urXiv:/707.000
Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Marc Peter Deisenroth. Manifold gaussian processes for regression. In 2016 International Joint Conference on Neural Networks (IJCNN), pp. 3338-3345. IEEE, 2016.
Dexiong Chen, Laurent Jacob, and Julien Mairal. Recurrent kernel networks. Advances in Neural Information Processing Systems, 32, 2019.
Youngmin Cho and Lawrence K Saul. Kernel methods for deep learning, In NeurIPS, 2009.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1
10

---

Published as a conference paper at ICLR 2024
Vincent Dutordoir, Mark van der Wilk, Artem Artemev, and James Hensman. Bayesian image classification with deep convolutional gaussian processes. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (AISTATS), volume 108 of Proceedings of Machine Learning Research. PMLR, 2020.
Vincent Dutordoir, James Hensman, Mark van der Wilk, Carl Henrik Ek, Zoubin Ghahramani, and Nicolas Durrande. Deep neural networks as point estimates for deep gaussian processes. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 9443-9455. Curran Asso- cc/paper files/paper) pdf
ciates, Inc., 2021. URL htt.:////..//teer//pooeeoeooooe.
Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feynman diagrams. arXiv preprint lrXiv:1909./1304, 2019.
Adrià Garriga-Nlonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks shallow gaussian processes. arXiv preprint arXiv:1808.0m 05587, 2018.
James Halverson, Anindita Maiti, and Keegan Stoner. Neural networks and quantum field theory. Machine Learning: Science and Technology, 2(3):03500,, 2021.
Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. arXiv preprint arXiv: (et::0999909999 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recog- nition. In Proceedings of 2016 IEEE Conference Computer Vision and Pattern Recognition. CVPR 16. IEEE, 2016.
James Hensman, Nicolas Durrande, Amo Solin, al. Variational fourier features for gaussian processes. J. Mach. Learn. Res., 18(1)55337-5588 2017.
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kerne:: Convergence and generalization in neural networks. In NeurlPS, pp. 8580-8589, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
Miguel Lázaro-Gredilla and Anibal Figueiras-Vidal. Inter-domain gaussian processes for sparse inference using inducing features. Advances in Neural Information Processing Systems, 22, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-227) 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint 17100171 2017.
Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. Advances in Neural Information Processing Systems, 13:15156-11172, 2020.
Qianyi Li and Haim Sompolinsky. Statistical mechanics of deep linear neural networks: The back-propagating renormalization group. arXiv preprint arXiv:2012.04030, 2020.
Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon S Du, Wei Hu, Ruslan Salakhutdinov, and Sanjeev Arora. Enhanced convolutional neural tangent kernels. arXiv preprint arXiv: 1.00000 2019.
David John Cameron MacKay, Introduction to gaussian processes. 1998. URL /ttpss://ap.
semant cscho ar. 1
11

---

Published as a conference paper at ICLR 2024
Wesley J. Maddox, Andres Potapcynski, and Andrew Gordon Wilson. Low-precision arithmetic for fast gaussian processes. In James Cussens and Kun Zhang (eds.), Proceedings of the Eighh- Conference Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pp. 1 1306-1316. PMLR, 01-05 Aug 2022. URL htt.:::ts////t. httttp:://proceeddnng
mlr. wttppppsssw/lmmmaaaaeaammo
htmll
Julien Mairal. End-to-end kernel learning with supervised convolutional kernel networks, Advances in neural information processing systems, 29, 2016.
Julien Mairal, Piotr Koniusz, Zaid Harchaoui, and Cordelia Schmid. Convolutional kernel networks.
Advances in neural information processing systems, 27, 2014.
Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv: 1804. 1/271, 2018.
Gadi Naveh and Zohar Ringel. A self consistent theory of gaussian processes captures feature learning effects in finite cnns. arXiv preprint arXiv:2 106 106.04/ 10, 2021.
Gadi Naveh, Oded Ben-David, Haim Sompolinsky, and Zohar Ringel. Predicting the outputs of finite networks trained with noisy gradients. arXiv preprint arXiv:2004.01190 2020.
Radford M Neal. BAYESIAN LEARNING FOR NEURAL NETWORKS. PhD thesis, University of Toronto, 1995.
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels gaussian processes. arXiv preprint arXiv:1810.055 1810.05188. 2018.
Sebastian W Ober, Carl E Rasmussen, and Mark der Wilk. The promises and pitfalls of deep kernel learning. arXiv preprint 210.::2002.12000. 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- -er--performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Geoff Pleiss and John P Cunningham. The limitations of large width in neural networks: A deep gaussian process perspective. Advances in Neural Information Processing Systems, 34:3349-3366 2021.
Daniel A Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory. arXiv preprint arXiv:2106. 10165, 2021.
Tim GJ Rudner, Dino Sejdinovic, and Yarin Gal. Inter-domain deep gaussian processes. In Interna- lional Conference on Machine Learning, pp. 8286-8294. PMLR, 2020.
Hugh Salimbeni, Stefanos Eleftheriadis. and James Hensman. Natural gradients in practice: Non- conjugate variational inference in gaussian process models. In AISTATS, 2018
Inbar Seroussi, Gadi Naveh, and Zohar Ringel. Separation of scales and a thermodynamic description of feature learning in some cnns. Nature Communications, 14(1):908, 2023.
Vaishaal Shankar, Alex Fang, Wenshuo Guo, Sara Fridovich-Ked Jonathan Ragan-Kelley, Ludwig Schmidt, and Benjamin Recht. Neural kernels without tangents. In International Conference on Machine Learning, pp. 8614-8623 PMLR, 2020.
Jiaxin Shi, Michalis Titsias, and Andriy Mnih. Sparse orthogonal variational inference for gaussian processes. In International Conference Artificial Intelligence and Statistics, pp. 1942-1942. PMLR, 2020.
Johan AK Suykens. Deep restricted kernel machines using conjugate feature duality. Neural computation, 29(8):2123-2163, 2017.
12

---

Published as a conference paper at ICLR 2024
Francesco Tonin, Qinghua Tao, Panagiotis Patrinos, and Johan AK Suykens. Deep kernel principal component analysis for multi-level feature learning. arXiv preprint arXiv:2332.11220, 2023.
Mark van der Wilk, Carl Edward Rasmussen, and James Hensman. Convolutional gaussian processes, 2017. httta:://axxv..or///aa....
Andrew G Wilson, Zhiting Hu, Russ R Salakhutdinov, and Eric P Xing. Stochastic variational deep kernel learning. Advances in Neural Information Processing Systems, 29, 2016a.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016b.
Sho Yaida. Non-gaussian processes and neural networks at finite widths. In Mathematical and Scientific Machine Learning. pp. 165-192. PMLR, 2020.
Adam X. Yang. Maxime Robeyns, Edward Milsom, Ben Anson, Nandi Schoots, and Laurence Aitchison. A theory of representation learning gives deep generalisation of kernel methods. ICML, 2023.
Greg Yang and Edward Hu. Feature learning in indinite-width neural networks. In International Conference on Machine Learning, 2021.
Greg Yang, Michael Santacroce, and Edward Hu. Efficient computation of deep nonlinear infinite- width neural networks that learn features. In International Conference on Learning Representations, 2022.
Jacob Zavatone- Veth and Cengiz Pehlevan. Exact marginal prior distributions of finite bayesian neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Jacob Zavatone- Veth, Abdulkadir Canatar, Ben Ruben, and Cengiz Pehlevan. Asymptotics of repre- sentation learning in finite bayesian neural networks. Advances in neural information processing systems, 34:24765-2477 2021.
A DEEP KERNEL MACHINES
In this appendix we provide a more detailed introduction to deep kernel machines. Deep kernel machines (DKMs) are deep generalisation of kernel methods that arise when considering a particular infinite limit of deep Gaussian processes (DGPs), called the Bayesian representation learning limit Yang et al. (203). The original motivation for this limit was study representation learning deep models e.g. neural networks, but it turned out that when applied to DGPs, the resulting posteriors are exactly multivariate Gaussians, and hence we obtain practical algorithm that can used for supervised learning. It is a deep generalisation of kernel methods in the sense that the model has Gram matrices rather than features at the intermediate layers. These Gram matrices are optimised by maximising the DKM objective; the derivation of the objective and reasoning behind needing to optimise Gram matrices will be elucidated in the rest of this appendix.
The simplest way explain DKMs is to walk through the steps Yang et al. (0033) took that led to the Bayesian representation learning limit, and more specifically how it applies to deep Gaussian processes.
A.1 DEEP GAUSSIAN PROCESSES
We start by defining deep Gaussian process (DGP). A DGP maps inputs X E PxNo outputs Y E P using L intermediate layers of features F E TR. PxN: Here P is the number of input points, and N₂ is the number of features at layer & so that No is the number of input features, and NL is the number of output features. Writing F and Y as stacks of vectors
F
(ff
f5
(20a)
Y
(y1
Y2
Nruts):
(20b)
13

---

Published as a conference paper at ICLR 2024
where f IRP gives the values of particular feature for all P input points and yı E R gives the value of a particular output for all P input points. The features Fi....,Fp are sampled from a Gaussian process whose covariance depends on the previous layer's features,
P (F - (Ex: O, K(G(( Here, we compute the kernel by first computing the Gram matrix,
(21)
G' = G²= N. ( N, F'("""
(22)
then use a kernel that can be computed directly from the Gram matrix (Eq. (3) also see Aitchison al. 202T). To see why this works for isotropic kernels that depend only on distance, see Eq. (2) For regression the outputs are also sampled from Gaussian process,
P (Y = I N (yx: 0, K (G((F)) + +
(23)
Other likelihoods could be used. For example, in this paper we do classification using a categorical likelihood with softmax probabilities.
A.2 DGPs WRITTEN IN TERMS OF GRAM MATRICES
We wish to consider the DGP analogue of the standard neural network Gaussian process (NNGP) limit, which takes the intermediate layers & {1,..., I.} to have infinitely-man features (number of features here is an analogue of layer width in neural network). Of course, we cannot work with matrices F PxN,, with N₂ co, inner product between features (Eq, 22). We can then rewrite the DGP as
will instead work with Gram matrices that represent the
P (F'GG") NV v P (G = 5 (G'
(24a)
(24b)
PYY]'") = I (Yx,"""'",
(24c)
We consider regression likelihood for concreteness, but, again, other likelihoods be used. Here, we have written (G' [²)) using the Dirac-delta, as G is deterministic function of F (Eq. 22).
All that remains to prepare the model for taking the infinite limit is to integrate out F from the intermediate layers (note that the likelihood already depends on G alone):
P P = p P (F dF
(25)
It turns out that in the case of the DGP model, Eq. 25 is actually Wishart distribution (one can see this by noting that 22 is the outer product of IID multivariate Gaussian vectors, which matches the definition of the Wishart),
P P )) = Wishart GG 1 KGG " N²)
(26a)
No
log P (G' = N P-1 log G
Ni log K [KG
Nr Tr (K (G + (26b)
where,
at N,P log 2 + N,P log N₂ - Ip (M
(27)
constant w.r.t. all G and Pp is the multivariate Gamma function. Note that the distribution (Eq. (6a) is valid for any N₂ 2 1, but the log density in Eq. (26b) is valid only in the full-rank case where N, 2 (though this restriction does not cause problems because we will take N₂ + 20).
A.3 STANDARD INFINITE-WE WIITITE-WIDTII LIMIT
We now consider the DGP analogue of the standard NNGP infinite limit, that is we will take the intermediate layers to have infinite width infinitely-mann -any features:
N₂ Nue for EE (1....,.) L.). with N co.
(28)
14

---

Published as a conference paper at ICLR 2024
The log log-posterior for the DGP written in terms of Gram matrices is
log P PG' G Y) = log P (Y/G4) + Eí log (G + const,
(29)
where we emphasise that G G are P x P matrix arguments chosen by us to specify where we wish evaluate the log-posterior, and so are not affected by any infinite-width limits we take. That said, the actual value of log P (G'IG"-) is affected by the limit. In particular, in the infinite
limit specified in Eq. (28). the log-prior term log P (G) (Eq, 26 scales with N₂ and therefore N, so we must divide by to obtain a finite limit:
lim N log P (G' " (log K (G G9 Tr (K-K (K
lim ay
-M DKL. (W (0, G I| (o, (o,K(G + const.
(30)
where limn- /z/N is given in Yang et al. (002)). Note that this limit has been written KL-divergence between two Gaussians. By contrast, the log-likelihood term log P (Y|G²) (Eq.
24c is constant w.r.t. N, so will vanish if we divide by N in the limit. Putting this all together, the limiting log-posterior (divided by N remain finite) is
lim A log P (G' ( X,Y) -E DEL (N (N + const.
(O,K(G
(31)
Now, as this log- pe--ostterirr scales with N, the posterior itself converges to a point distribution at its global maximum convergence)
(Gh)* (see) Yang et al. (2023) for more rigorous discussion of weak
lim P (G'...., x) N
(32)
From Eq. (3I), it is clear that the unique global maximum is obtained when (G')² (() since this minimises the KL divergence. In other words, can compute the maximum recursively by applying the kernel repeatedly. Importantly, this means that the limiting posterior has no dependence on Y, training targets, and it not possible for representation learning occur. This is big problem, since the success of modern deep learning hinges on its ability to flexibly learn good representations from data.
A.4 THE BAYESIAN REPRESENTATION LEARNING LIMIT AND DKMS
Representation learning was lost because the likelihood term in the normalised posterior vanished in the infinite limit, whilst the prior term did not. The Bayesian representation learning limit (Yang) et al. 2023) retains representation learning by sending the number of output features to infinity, in addition to the intermediate features:
N₂ Nug for LE
with
oo.
(33)
Note that the only difference between this limit and Eq. 33 is that we include layer I, + 1 (the output layer) in limit. This defines valid probabilistic model with well-defined posterior using the same prior as before (Eq. 25), and a likelihood assuming each output channel is IID:
PY = 11 N (5x: K (G*)) ) +
(34)
where, to distinguish from the usual likelihood (Eq, 24c Ý R Px infinite width (Eq. 33
whereas the usual DGP data, Y E Px is finite width.
In practice, width data does not usually exist. To apply the DKM to finite-width data Y e Px we let Y be formed from N copies of Y concatenated together, i.c
Y Y
Y).
(35)
Since we assumed the output channels to IID (Eq, 34/, the log-likelihood now scales with N:
log P (YIG4) N log P (Y
(36)
15

---

Published as a conference paper at ICLR 2024
Computing the log- posterostror before, therefore find that the likelihood no longer vanishes in the infinite limit, thus allowing representation learning:
L(G',... ...., G lim t log P (G',...., G Y)= const,
N
Pog (Y|GH) - (N (0. (O.
(37)
The limiting normalised log-posterior, C(G',...., ,,),, forms the "DKM objective". As before, assuming the global maximum of this expression unique, then the posterior is point distribution around that maximum (Eq. b2). However, unlike before, we cannot simply compute the maximum by recursively applying the kernel function to data, since the maximum now also depends the likelihood term. Instead, we must optimise the Gram matrices using e.g. gradient ascent, and there no guarantee anymore that the maximum is unique.
Yang et al. (2023) use the DKM objective in practice in fully-connected settings, and call the resulting approach a "deep kernel machine". As already mentioned, the limiting log-posterior of the Gram matrices in this context is dubbed the "DKM objective" as we maximise it to fit the Gram matrices in our model. Once the Gram matrices have been optimised according to the DKM objective using the training data, one can perform prediction on new test points using method similar to that in DGPs (see Yang et al. 2023 for details).
B FULL DERIVATIONS FOR THE KERNEL CONVOLUTION
Here we give the full derivations for the covariances used to define the kernel convolution in the main text. For the full-rank (no sparse inducing point scheme) case, reader can skip straight to Section Sections B.1 and B.2 Bnno concern the inducing point scheme.
B.3
B. INDUCING BLOCK
For the inducing th component,
r ij
(o
Fli pize 3.
(38)
Substituting the definition of Fil (Eq.
E
Σ
We
Hi EWan C2s vd''g'
Hie-
(39)
udd
The only random variables are the weights,
Σ
Hid-! Hee Cos.. HiC E WE dy. W. W
(40)
As the weights are IID with mean and variance 1/DN,-1 (Eq. 9
Σ Cas Hish-
C C
Ho Hyp'' DÑ Sunda,d
(41)
where 6p.p is the Kronecker-delta which is when μ v and zero otherwise. The Kronecker-deltas pick out elements the sum,
Σ
Cdyyy H Hhe-1
(42)
DNE
p.d.y'
Rearrange to bring together the multiplication of previous- layer activations, H
= Σ
Cla
Cl. 1 1 Ne
Σ
(43)
d.
P
16

---

Published as a conference paper at ICLR 2024
And remembering the definition of Day 0i
(Eq. 10
D
Σ
Cax C
(44)
In practice, we implement this by averaging the result of batched matrix multiplications,
( E (Ca)
T
(45)
B.2 INDUCING-TEST/TRAIN BLOCK
We follow the same process for the inducing-test/train test/train it'th component,
I ritzé (S2
F die ast J5.2
(46)
E
Σ
W ann W 1l vd
hit1 Σ Euer
1oo Wand
(47)
duui
E
Caes Hi Hhit-1 7.H
[ E [Wanx War
(48)
de' uui
E
Ha
H
f.lotar DN Snndan
(49)
dd' uuil
Cos
Hit Hiil-
(50)
DNI-1 dip.e
Vi slotdip
1 Σ
Cas 5(x+d) ((std)
(51)
d.²'
In practice, this can implemented by standard convolution operation, treating Cdi,y' as the "weights", and 1
((++d)) as the "inputs".
B.3 TEST/TRAIN BLOCK
Again, follow the same process for the test/train tt'th component. Note that does require superscript e here since the function is the same no matter what layer it is operating on, as there are no weights C when we are dealing with just test/train points.
ru in ( 1) = E
(52)
J.sA
E
Htl-1
tir+d)p du, Σ Hhe-1 to W
we
(53)
du
d'v H E [Wh W W H 3,0'v
Σ
(54)
= Σ
Hiit-1 Hlil-1
air+d)y (( DNC I Sppdda
(55)
de' pu
(56)
DN
Qud u 1 Σ
(57)
Remember that we end up only needing the diagonal,
Ilt on (2 = 1 Σ
((r+)))
(58)
d
and this operation be implemented efficiently as average-pooling.
17

---

Published as a conference paper at ICLR 2024
C NORMALISATION AND SCALING FOR GRAM MATRICES
We develop various analogues to batch/layer normalisation (using the terms to loosely describe any technique for normalising activations) that be used on Gram matrices in order improve the performance of our model. Since we do not have explicit features, there is some flexibility in how to do this. Here we describe the variants we tested and comment on them.
We think of batchnorm as consisting of two components: normalisation and scaling. Normalisation is parameter-free, and is analogous to dividing by feature variances in standard batchnorm. Scaling has parameters, and is analogous to multiplying the features by a learned scalar in standard batchnorm. In our setting, we can apply different normalisation and scaling schemes the inducing and test/train points: so we end up with four choices: inducing normalisation, test/train normalisation, inducing scaling and test/train scaling.
All of our batchnorm- like schemes have the following form.
v/ as Gn (G))
NT)(() G
(59)
V.
Ni Nyje
(G) Gijje agl Gn Aja (Gn)
(60)
V/2
ar (G) Grrrr A (G,)
(61)
where N(G) is normalized Gram matrix. Here, 1/; is the learned parameter that represents the inducing scale and Vr the learned, potentially location-dependent parameter that represents the test/train scaling. Likewise, a, (G.s) and A (Gu) represent the normalisation terms for inducing and test/train points respectively. In the subsequent sections, we explore potential choices for a, (G;;)) and Air( (G.) corresponding different normalisation schemes, and choices for th and V,, corresponding to different scaling schemes.
C.1 INDUCING NORMALISATION
No inducing normalisation. We can turn off inducing normalisation by choosing,
a, (G.)
(62a)
Kernel Batch inducing normalisation is, a, (G = numn (Gi) = V 1 E G3 P.
(62b)
Kernel Local inducing normalisation is,
a. (Gi) = a. norm (Gi) = G
(62c)
C.2 TEST/TRAIN NORMALISATION
There are a much richer set of choices for how to normalize the test/train points, as they are spatially structured.
No test/train normalisation. We can turn off normalisation by choosing,
Air(G.)) =
(63a)
Kernel Batch normalisation averages the diagonal of G over spatial locations in images to get single scalar normalizer,
(G, Ahatch((G,)
G"
(63b)
3T4
Kernel Image normalisation averages the diagonal of G. over all spatial locations for separate images to get a different normalizer for each image,
Agr Aar((r, (G₂) iimage (Ga) V + Giaiis
(63c)
18

---

Published as a conference paper at ICLR 2024
Kernel Location normalisation, averages the diagonal of Ga over all images for separate spatial locations to get different normalizer for each spatial location,
Air (G, Alocation (Gu) V W
(63d)
Kernel Local normalisation does not average at all, so the normalizer is just the square root of the diagonal of Gu.
Aar(Gn) V
(63e)
C.3 INDUCING SCALING
Now, there are three choices for the inducing scaling. No inducing scaling. We can turn off inducing scaling by choosing,
(64a)
Kernel Batch inducing scaling, in which there is single scale applied to all inducing points,
(64b)
Kernel Local inducing scaling, in which there is a different scale applied to each inducing point,
t/: m/
ind
(64c)
C.4 TEST/TRAIN SCALING
There are three choices for the test/train scaling.
No test/train scaling. We can turn off test/train scaling by choosing,
v 1.
(65a)
Kernel Batch test/train scaling We can choose the test/train scaling to be a constant across spatial locations,
V const
(65b)
Kernel Location scaling We can choose the test/train scaling to be different across spatial locations, in which case have,
V
1p location
(65c)
Note that we never apply scaling image-wise, since the batch changes every iteration, so this would be nonsensical. On the other hand, spatial location carries meaning across all images, so scaling by location makes sense.
D COMPUTING THE GP-ELBO TERM
D.1 AMENDED DKM OBJECTIVE
In the original deep kernel machine paper (Yang et al. 2023). the authors proposed a sparse variational inducing point scheme for DKMs, roughly based on similar work in the DGP literature by Salimbeni jt al. (2018). However, rather than taking the usual approach of defining an approximate posterior over the inducing outputs FL+1, they simply treat them as variational parameters. In this paper, we take the more usual approach of defining an approximate posterior over our inducing outputs, which we take to be Gaussian:
NZ+1
Q( = II A(C)
14x, AA)-
(66)
X=1
Here, Ni+1 is the (finite) dimension of the top layer, and μx AA are variational parameters for each channel of the output. This modifies the resulting objective function slightly, but believe the
19

---

Published as a conference paper at ICLR 2024
resulting formula is actually more natural. Following the derivation of the inducing point scheme exactly as in Appendix of Yang et al. (202)), we find that the new sparse objective is
Lind (F
G ........ G lim EF
G G..... G
N +00
No+1
= =BQ [log (Y
Σ
DEL (M AN ((G)()
x=1
Sparse GP-ELBO
ve DxL (N (o, G (N
(67)
DKM Regularisation Terms
We can see that with the full approximate posterior over the inducing outputs, the inducing objec- tive be neatly split up into the ELBO for sparse shallow GP (the top layer) and chain of regularisation terms for the hidden layers of the DKM.
In practice, cost. We also use minibatching, so we normalise the objective by the total number of datapoints, meaning that computing our objective value the minibatch provides estimate for the fullbatch update.
the same A A for all output features À to memory and computational
Therefore, the mean minibatch objective we actually use in our implementation
Cbutch (F.
1---1 Gl....,, GF
Σ
Eq llog P [y)(F(+1)]
Batch size
pEhatch
Nu+1
1 (N
Σ
DKL (O)A
Σ ve DxL (N (l, 1=1
Trainset Size
(68)
D.2 LIKELIHOOD FUNCTIONS
We considered different approaches to the likelihood: Gaussian (mirroring the approach the early NNGP work, including Lee et al. 2017 and Matthews et al. 2018). and a Monte-Cart estimate of categorical likelihood, which more natural for classification tasks.
Computing the GP-ELBO with these different likelihoods is not trivial especially with inducing points. In this section, we detail precisely how these different likelihood functions can be incorporated into the GP-ELBO, including showing that the GP-ELBO with Gaussian likelihood has a closed form, and that the categorical likelihood can be approximated using Monte-Carlo estimation. Before we do this, recall that the inducing DKM objective in Eq. 67 contains the term
EQ [lg [Y([
(69)
which the expected log-likelihood under our variational joint distribution Q Q(E(+((F/ (B over top-layer features F given the previous layer G
Concretely, we choose Q(E+++)) as in Eq. 66 and 1(F G = P(F F G i.e. we use the same conditional Gaussian formula for test points given the inducing points the true distribution.
We need only consider the predictive term (Eq. 69 since other terms in the objective (Eq. 67 depend on our choice likelihood function P(Y, P(Y,F)) F.). With this in mind, we now derive the expected log-likelihood term for the two likelihood functions consider.
20

---

Published as a conference paper at ICLR 2024
D.2.1 GAUSSIAN LIKELIHOOD
This is the same likelihood function used in the original DKM paper Yang (023)) though they did not provide detailed derivation of the closed-form expressions used to compute it. Since some readers will find this useful, we provide one here. order compute the DKM objective function, we must compute the predictive term
Q 1/ (*)Q((/+1)) [log P(Y,)
(70)
where, for a Gaussian likelihood,
NL+1 = II A=1
FL+,)
M ff I
(71)
Substituting the definition of the multivariate normal log probability density and ignoring additive constants, we have
E QQUE " (0)00(t)) [log P(Y.|F(+))
(72)
No+1
1
1 o²
(x
2y [fA] + Eq [[]]]
(73)
20
where we use Bęg mean I. QF! " and to avoid unruly notation we drop the layer
and train/test labels from piL+1 just write fx. This be further decomposed into separate
terms for each data point (which allows minibatching):
N241
-0 log 0²
2vix LGSN + IQQ [f31]
(74)
20
j=1
A=1
P
NL11
=-0.5N++1 ²²
ViA -uyyuu [faal Varq IfAAl + EQ [taN?] (75)
20
j=1
A=1
NL+1
(UJA- Eq - lf + Varg [fial
(76)
20
Hence to compute the predictive term we will need to know the mean and variance of fJx (the test points f under the joint distribution Q (joint over both training and inducing). We assumed earlier that our inducing outputs fj have variational distribution
((f)) = (x.
(77)
Since the conditional mean and covariance of fī given the inducing f) just the conditional Gaussian formulae, this is very simple to compute via the laws of total expectation and total variance:
Eq | = E E
[
KK fll
K.KT mx
(78)
Var[ft]
= [aar (arff [+St] + Var [E[
= -Ki K,K, Kk + Var KK f.] K₁
Kn. KiK, Kr K, Var [F]] K
-K,, 1.i-KAn K,,KK,K K, K,K A -KK
(79)
Combining this with Eq. 76 gives us concrete formula. Alternatively, one can use these quantities
to generate samples of f which is multivariate Gaussian, and then compute the Gaussian likelihood via Konte-Carlo estimation. We used the latter strategy in our experiments using Gaussian likelihoods, since forced sample for the categorical likelihood anyway.
21

---

Published as a conference paper at ICLR 2024
D.2.2 CATEGORICAL LIKELIHOOD
We want use categorical distribution our likelihood function, where the probabilities for each class are given by the softmax of our final layer features:
P (((IF!+)) = II
(F)tt U3
(80)
j=1
11 exp j.y
(81)
A=1 exp fyx
N441
st:L+1
Hence the expected log- ligllikelihood would be
NL+1 exp
Eq loog P (Y
Σ
EQ
poit1] 1905
EQ
log
piib+1 fA
(82)
The log-exp term is problematic, since we cannot compute its expectation in closed-form. However, since phL+1 is just multivariate Gaussian distributed, with mean and covariance given by Eq. 78 and
Eg. 79 respectively, we can easily generate samples and just compute a Monte-Carlo estimate Eq.
82
E FLATTENING THE SPATIAL DIMENSION AT THE FINAL LAYER
E.1 LINEAR
As in neural networks, we must collapse the spatial structure at the final layer, converting a S P x StP. matrix to P. Pt, that may perform the final classification. Convolution with zero- padding, and = Sy is equivalent to a linear top-layer. It returns feature map with only a single spatial location. Hence we obtain similar expressions our usual convolutional kernel. The tt'th part of these expressions was known to (Novak et al. 2018 Garriga-Alonso et al. 2018), but the inducing components (specifically A and Aa) are specific to our scheme.
p
P.
( E pigb+1
H
OL+1 CL+
(83)
SL
1'1'==1'=
pt
(G
E
11: pplL+1
J.A
H
ST.
1
CL+
Gj'jj
(84)
P=1
EG = E
p1L++1
p+L+1
H4
Σ
GLL
(85)
j.
S1
18
E.2 ONLY DIAGONALS OF TEST/TRAIN BLOCKS ARE NEEDED
Naively applying these equations is very expensive, we still need work with P.S. P.Sp covariance matrices in the tt' th block. The key trick is that we only need to represent the diagonal of the tt'th matrices. In particular, note that Eq. (83) indicates that we only need the P.S elements Gts.j corresponding to the covariance across images at the same spatial locations. Furthermore, the standard DKM derivations imply that (for IID likelihoods) we only need the diagonal of Gn, meaning that in this case we only need the P.S. elements G corresponding to the variance of each image at each spatial location. This is only at the output layer, but as in infinite CNNs (Novak et al. 2018 Garriga-Alon et al. 2018). propagates backwards: if we only need GilL only need $8.00
is,is at one layer, then we
- at the previous layer.
1 Σ Q (o+t))((o+d))
Tis,ss Tis,s((2, = D
(86)
dED
22

---

Published as a conference paper at ICLR 2024
E.3 GLOBAL AVERAGE POOLING
It is common practice in convolutional neural network top-layers we take the mean over the spatial dimension, and only learn weights on the channels (in contrast to a linear final layer where we would have a separate weight for each pixel). In this section we explain how this can be done for the convolutional DKM. Note that here we use P. to denote the input convolution layer, but the reader should focus on the operation of global average pooling rather the specific letters we choose use here. For notational convenience, we let S St the following derivations.
global average pooling, where
the global average pooling operation, which would usually be the result fo
E.3.1 FULL-RANK
For features, we define global average pooling as
FGPP
F Σ T/
GAP. H.d
(87)
where, WGAP N are IID Gaussian such that
E [WG 3-5
Nr 6pw-
(88)
Now, computing the covariance Eiy (again, do not attribute too much meaning to the choice of letters), we have
Ei = E GAP ECAP
(89)
(CCA)) )
]
5o E
Σ
FL for
Jise W-SAP
(90)
Rearranging the sums, we obtain
E
3d
Σ
Fl IE [W..AWWCC
(91)
which, by substituting the covariance of the weights (Eq. 88
becomes
Eg
Σ
P Jev Ne ppv-
(92)
3Z
Finally, combining the delta-function and sum, obtain
Eij Nr
Σ
J.s.
(93)
Ed
Gh.
(94)
52
i.e. we simply take the mean over both spatial indices (remember there are two because our covariance comparing pixels to pixels).
E.3.2 INDUCING
Since global average pooling eliminates the spatial dimensions (or equivalently shares the same weights across spatial locations within the same image), the inducing points can be directly multiplied by the weights, and we do not need any C weights to "map" them up to spatial structure:
SGAP A
Σ #
W GAP piL u.d i.p
(95)
23

---

Published as a conference paper at ICLR 2024
The inducing- induuiing block A is straightforward:
+ = E GAP
(96)
E
Woi Eip Σ
w WeAe pill Fip
(97)
Flil i.v E [WTAPWWATT v.
(98)
Flil pit 1 Nr. 8p
(99)
-
L
(100)
i.c. it is just the same as the input. Since our inducing points have no spatial structure, it makes sense that global average pooling would do nothing to them.
The teducing-test/rrann block An just averages over the single spatial index:
E
GAP GAP
(101)
E
WGA FiLL 1 s
Σ aV
FliL s,w
(102)
5
E
W.CAPWOA piuk puik 2,50
(103)
1
E
Fli Fii Gi
(104)
SNL
Ms
I
(105)
s
i.js
And the diagonal of Au (recall we need only store the diagonal for IID likelihoods) averages over both inputted spatial indices:
E = E
PAPP G.P
(106)
E
Cosp WGAP
Eich vid
(107)
g
E
WOAAP pith FisL 2,op
A P.A
(108)
g2
LST
E
Σ
pil p&L E,3 i.r
(109)
No
Larp
3z
ss,er
(110)
ST
However, there a problem. This does not depend only on the P. x S diagonal of the input, but also on the covariance of all pixels within test image, which means actually require an object of size P. x S x S versus the full matrix Gu of size P. x P. x S x S which is still bad since S is usually much larger than either P, or P.
To solve this problem, we use a corrected Nyström approximation on the input:
G R G G
GG + DD. G
(111)
where D is a diagonal matrix that corrects the diagonal to match the true diagonal of G which we know.
24

---

Published as a conference paper at ICLR 2024
However, this still seems like somewhat of problem, since the Nyström approximation of Gu is still of size P, x P. x S. Luckily, turns out that, since we only need the diagonal of Kn. we don't t need to compute the entire Nyström approximation:
ALLL lsjr
(112)
8P
Apply the Nyström approximation:
Σ
[G v
(.f))) + Dis,iodrjdor
(113)
80
Σ
G
G
Σ
G Jo'rr
+ 1 3 Σ Die,ioii
(114)
Vj
(115)
Σ
G
E + 52t Σ Dis.
(116)
Ki''
And because we only need the diagonal,
E E G
Elyy + 82 EDisiss
Dis,is
(117)
With that problem solved, we can now practically apply global average pooling to our Gram matrices at the top-layer.
F EXPERIMENTAL RESULTS FURTHER METRICS
In this appendix give further metrics (test log- likelihood, train accuracy, and train log- liliiiiiihood) for the experiments in the main text. All experiments were run for different random seeds, and we report the means with standard errors. See Section for further details of these experiments.
As in the main text, for all tables and for each dataset, we identify the best configuration, and then bold all those configurations whose performance was statistically similar to the best, using meitailed Welch's s t-test with confidence level 5%.
F.1 REGULARISATION STRENGTH
Test accuracy for regularisation strength were already given in the main text (Table L. We give the remaining metrics in Table
As we can see from performance metrics, test log-likelihoods are maximised by using v though this comes at slight performance hit to test accuracy (Table For smaller values, higher train performance over test performance suggests overfitting is occurring (though this seems to only negatively affect the ligeeiisisssss and not the accuracies). It may be interesting future direction to investigate alternative formulations for the regularisation term to see if this can be improved upon.
F.2 BATCH NORMALISATION TYPE
The test metrics for normalisation and rescaling were already given the main text (Table []. We give the remaining metrics for normalisation in Table 5 for rescaling in Table
We chose the combinations of variants by requiring that the inducing blocks and test-train blocks used the same type of normalisation and scaling. In cases where are using image specific variants of normalisation on the test-train blocks that don't 't exist for the inducing blocks, we use the following mapping: "image" normalisation in the test-train block corresponds to "local" normalisation in the inducing block, since it doesn't 't average across the datapoints in the batch, whilst "location" " normalisation corresponds to "Batch" normalisation in the inducing block since it does average across
25

---

Published as a conference paper at ICLR 2024
Table 4: Train accuracies (%) and log- liceliiiss using different regularisation strengths.
Metric
MNIST
CIFAR-10
CIFAR-100
00
-0.9026 0.0039
-1,3546 + 0.0055
-2.8909 0.0105
104
-0.6393 = 0.0158
-1.3830 + 0.0063
-2,9914 = 0.0030
10a
-0.2488
+ 0.0014
-1.1972 + 0.0018
-2.6351 + 0.0206
10a
-0,0942 + 0.0029
-0,8322 + 0.0003
-2.0077 + 0.0058
10
-0.0380 = 0.0013
-0.5070 = 0.0058
-1.5307 = 0.0133
Test LL
100
-0.0263 + 0.0006
-0.3885 = 0.0022
-1,638 = 0.0093
10-1
-0,0294
+ 0.0011
-0.4652 = 0.0051
-1,7877 = 0.0309
10-2
-0.0344 0.0013
-050048 = 0.0102
-1.9923 0.0310
10-3
-0,0356
0.0011
-0,5295 = 0.0049
-2.0162 : 0.0144
-0.0371
0.0012
-0.5279 0.0017
-2,0488 = 0.0107
0
-0.0359 0.0010
-0,2280 0.0072
2.0552 0.012
73.10 0.09
52.99 + 0.22
30.19 + 0.25
104
82.03 + 0.55
51.30 + 0.31
27.75 + 0.13
10a
92.76 0.04
58.24 = 0.06
34.72 = 0.41
102
96.82 0.09
70.66 0.07
50.09 = 0.18
10a
98.78 0.03
83.84 + 0.23
65.20 + 0.57
Train Acc.
100
99.48 0.02
93.22 + 0.02
82.48 + 0.51
10-1
99.77 0.00
97.45 0.07
91.07 = 0.34
10-2
99.87 = 0.01
98.38 = 0.08
93.72 = 0.37
10-3
99.90 + 0.01
98.65 + 0.04
94.18 + 0.28
10
99.90 0.01
98.67 0.06
94.34 = 0.26
0
99.91 0.00
98.70 + 0.05
94.41 + 0.25
-0.8735 + 0.0021
-139939 = 0.0036
-2.9417 = 0.0073
104
-0.6149 + 0,0169
-1.4230 + 0.0067
-3.0430 + 0.0032
103
-0.2589
0.0006
-1.2351 = 0.0012
-2.6898 0.0188
10a
-0.1084 0.002
-0,8547 = 0.0003
-1.9135 = 0.0096
0.0096
10
-0.0382 + 0.0008
-0.4651 + 0.0053
-1.2463 = 0.0212
Train LL
10
-0.0164
0.0005
-0.1944 + 0.0012
-0.8835
0.0171
10-1
-0.0072 + 0.0001
-0.0729 + 0.0018
-0.2833 + 0.0102
10-2
-0.0041
0.0001
-0.0462 + 0.0019
-0.1980 - 0.0100
10-3
-0.0033
0.0000
-0.0387 + 0.0012
-0.1814 1 0.0083
10-1
-0.0032
0.0001
-0.0377
+ 0.0013
-0.1758
0.0081
0
-0.0029
0.0001
-0.0365 + 0.0011
-0.1736 + 0.0079
the batch. For rescaling, the are no longer "local" or "image" variants for the test-train block, since it does not make sense have scalar for individual datapoints in batch that will persist across arbitrary batches. Hence, when we are using "local" on the inducing block, we test all possible variants for the test-train block.
Our batchnorm schemes were originally introduced to provide smoother training dynamics, but the story the results tell mixed. Sometimes using normalisation scaling results in worse training performance, like on CIFAR-100, so we decided that seemed sensible to use the simplest "Batch" variants for the final experiments (Table 2 the main text, and Table (9 for other metries), which seemed to perform well across all the datasets.
F.3 FINAL LAYER AFTER CONVOLUTIONS
Test accuracies for the final layer type given in the main text (Table T). We give the rest of the metrics in Table
Our global average pooling layer requires the of Nyström approximation. To ensure this was not adversely affecting performance, we also test a final layer that performs a convolution with kernel size equal the feature map size (analogous flattening and using linear layer in CNN) which
26

---

Published as a conference paper at ICLR 2024
Table Train accuracies (%) and log-likelihoods using different types of normalisation (Inducing Train-Test).
Metric
Ind. / Train-Test
MNIST
CIFAR-10
CIFAR-100
Batch / Batch
-0.0263 + 0.0006
-0.3885 + 0.0022
-1.4638 + 0.0093
Batch Location
-0.0284
0.0009
-0.3850
0.0027
-1.4708 + 0.0129
Test LL
Local Image
-0.0277 = 0.0008
-0.3863 + 0.0011
-1.4868
0.0195
Local / Local
-0.0270 0.0009
-0.3952
0.0034
-1.5365
0.0118
None / None
-0.0276 + 0.0008
-0.4025 + 0.0046
-1.4316 + 0.0007
Batch / Batch
99.48 + 0.02
93.22 +: 0.02
82.48 0.51
Batch / Location
99.44 + 0.01
92.94 + 0.17
82.42 + 0.61
Train Acc.
Local Image
99.51 + 0.01
93.06 - 0.17
83.88 0.27
Local / Local
99.41 + 0.01
92.99 + 0.17
83.69 = 0.20
None None
99.49 + 0.01
92.48 0.13
79.68 0.16
Batch Batch
-0.0164 + 0.0005
-0.1944
0.0012
-0.5835 + 0.0171
Batch Location
-0.0175 + 0.0001
-0.2037 + 0.0047
-0.5855 + 0.0205
Train LL
Local Image
-0.0158 + 0.0004
-0.1998
0.0050
-0.5302 + 0.0088
Local Local
-0.0185 0.0002
-0.1995
0.0054
-0.5395 + 0.0071
None / None
-0.0157 + 0.0002
-0.2162 + 0.0036
-0.6821
+ 0.0067
Table 6: Train accuracies (%) and log-likelihoods using different types of rescaling after normalisa- tion.
Metric
Ind. Train-Test
MNIST
CIFAR-10
CIFAR-100
Batch Batch
-0.0263 + 0.0006
-0.3885
0.0022
-1.4638 + 0.0093
Batch / Location
-0.0276 + 0.0008
-0.3892 + 0.0045
-1.5375 + 0.0149
Local / Batch
-0.0257 + 0.0005
-0.3930
0.0025
-1.4708 + 0.0126
Test LL
Local / Location
-0.0287 + 0.0007
-0.3894 + 0.0030
-1.5050 0.017
Local / None
-0.0261 + 0.0010
-0.3894
+ 0.0023
-1.4298 0.0076
None / None
-0.0275 0.0008
-0.3904
0.0039
-1.3514 1 0.0017
Batch Batch
99.48 + 0.02
93.22 0.02
82.48 0.51
Batch / Location
99.43 + 0.01
93.43 + 0.09
81.69 + 0.68
Local / Batch
99.48 = 0.01
92.63 = 0.12
81.90 0.15
Train Acc.
Local / Location
99.41 = 0.02
92.82 0.24
81.83 = 0.24
Local None
99.44 = 0.01
93.26 - 0.08
81.31 0.59
None / None
99.46 + 0.02
93.07 + 0.07
78.99 + 0.34
Batch / Batch
-0.0164 0.0005
-0.1944 0.0012
-0.5835 1 0.0171
Batch / Location
-0.0179 + 0.0002
-0.1874 + 0.0035
-0.6116 + 0.0246
Local Batch
-0.0163 0.0005
-0.2099
0.0036
-0.5993 + 0.0051
Train LL
Local / Location
-0.0185 + 0.0003
-0.2042 + 0.0067
-0.6101 + 0.0080
Local None
-0.0178 0.0005
-0.1950
0.0011
-0.6251
+ 0.0212
None / None
-0.0174 + 0.0002
-0.1953 + 0.0027
-0.7135
+ 0.0135
does not require any extra approximations. We found that global average pooling either matched or outperformed this method across the board, in line with common deep learning practice, despite the use of the Nyström approximation.
We also experimented with learning weights to "mix up" the inducing points in convolutional layers, but results were mixed, and sometimes even made the model worse (presumably due to the increased complexity introduced the optimisation problem by these extra weights).
27

---

Published as a conference paper at ICLR 2024
Table 7: Train accuracies (%) and log- log-liielshoods using different final layers.
Metric
Final Layer Type
MNIST
CIFAR-10
CIFAR-100
Global Average Pooling
-0.0263 + 0.0006
-0.3885
0.0022
-1.4638
0.0093
Linear
-0.0275 + 0.0006
-0.3751 + 0.0040
-1,4053 + 0.0087
Test L.L
GAP + Mixup
-0.0275 + 0.0004
-0.3836
0.0008
-1.4861
0.0107
Linear Mixup
-0.0280 + 0.0007
-0.3875 = 0.0032
-1.3734 + 0.0065
Global Average Pooling
99.48 + 0.02
93.22 = 0.02
82.48 = 0.51
Linear
99.46 + 0.02
92.39 + 0.11
79.93 + 0.30
Train Acc
GAP Mixup
99.49 0.01
92.77 + 0.23
83.41 0.19
Linear Mixup
99.47 = 0.01
92.07 = 0.35
74.26 = 0.21
Global Average Pooling
-0.0164 + 0.0005
-0.1944 + 0.0012
-0.5835 + 0.0171
Linear
-0.0173 + 0.0007
-0.2197 0.0035
-0.6682 0.0111
Train LL
GAP Mixup
-0.0160 = 0.0003
-0.2075 = 0.0002
-0.5533 + 0.0063
Linear Mixup
-0.0169 + 0.0004
-0,2276 + 0.0108
-0.8785 0.0048
Table 8: Train accuracies (%) for different likelihood functions. (log-likelihoods are not comparable). Bold shows values statistically similar the maximum.
Metric
Likelihood
MNIST
CIFAR-10
CIFAR-100
Gaussian
99.12 0.01
88.85 0.59
27.17 + 0.67
Train Acc.
Categorical
99.48 + 0.02
93.22 + 0.02
82.48 + 0.51
F.4 LIKELIHOOD FUNCTION
Test accuracies for the choice of likelihood function given in the main text (Table We give the train accuracies in Table g Since we are modifying the likelihood function, does not make sense compare the log- likelihoods between these methods.
We tested two different likelihood functions: the Gaussian likelihood and the categorical likelihood with softmax probabilities. We estimated both the categorical likelihood and Gaussian likelihood via Monte-Carlo, though the Gaussian likelihood does permit a closed-fort objective function. Details of both likelihood functions can found in Appendix D.2 D.2 used 1000 samples from the top layer features.
For the Monte-Carlo approximation, we
As expected for classification, the categorical likelihoods performed better than the Gaussian likeli- hood. In the case of CIFAR-100, which has 100 classes compared to CIFAR- 10 and MNIST which have 10 classes, the Gaussian likelihood resulted in catastrophically bad training performance, which then carried over into test performance.
F.5 FINAL EXPERIMENTS NUMBER OF INDUCING POINTS
For the final experiments, using hyperparameters selected using the results from the previous experi- ments, test accuracies for different numbers of inducing points were given in the main text (Table 2). We give the other metrics in Table 9 Interestingly, whilst we observe good test accuracy on the largest models, the test log-likelihood actually degrades past certain size of model. Since DKMs are not considered truly Bayesian due to the model mismatch introduced by the modified likelihood layer (Yang al., (202)), we hypothesise that this the model overfitting the train likeliieldoodd and causing poor calibration.
28

---

Published as a conference paper at ICLR 2024
Table 9: Train accuracies (%) and log lieliiiiiioodss using different numbers of inducing points.
Metric
Ind. Points
MNIST
CIFAR-10
CIFAR-100
/ 32 / 64
-0.0312 + 0.0008
-0.4960 = 0.0047
-1.7118 + 0.0073
32 64 / 128
-0.0365 = 0.0025
-0.4548 = 0.0049
-1,5182 = 0.0085
64 128 / 256
-0.0443 + 0.0007
-0.6446 + 0.0146
-2.3413 + 0.0656
Test LL
128 256 512
-0.0603 0.0019
-0.7593 + 0.0214
-3.1398 + 0.0491
256 512 / 1024
-0.0659
= 0.0016
-0.6939 = 0.0154
-2.4611 = 0.0196
512 1024 2048
-0.0676 0.0036
-0.6502 = 0.0125
-2.0553
0.0207
/ 32/ / 64
99.45 0.02
86.27 = 0.21
58.22 = 0.27
32/ 64 / 128
99.76 + 0.00
93.75 0,04
75.15 = 0.41
64 128 / 256
99.94 + 0.00
98.83 0.11
94.76 = 0.38
Train Acc.
128 256 512
99.99 = 0.00
99.90 = 0.01
99.83 = 0.02
256 512 / 1024
100.00 0.00
99.98 0.00
99.98 = 0.00
512/ 1024 2048
100.00 0.00
99.99 0.00
99.98 0.00
32/3/// 64
-0.0166 0,0005
-0.3907 = 0.0061
-1.4938
0.0098
32 64 / 128
-0.0075 + 0.0003
-0.1743 + 0.0012
-0.8324 + 0.0144
64/ 128 256
-0.0020 0.0001
-0.0328 + 0.0028
-0.1631
0.0114
Train L.I
128 / 256 / 512
-0.0004 + 0.0000
-0.0033 + 0.0002
-0.0104 + 0.0007
256 512 / 1024
-0.0002
0.0000
-0.0010
+ 0.0000
-0.0026 + 0.0001
512/ / 1024 2048
-0.0001 + 0.0000
-0.0005
+ 0.0001
-0.0015 + 0.0000
29


Reviews:
[{"review_id": "2KML5wtZ7q", "reviewer": "Reviewer_5yKS", "summary": "This paper proposes an extension of deep kernel machines to leverage convolutions. It is based on the patch-based inter-domain inducing points in the convolutional GP literature. The main difference between convolutional DKMs and convolutional DGPs is that DKMs allow to perform representation learning--the Gram matrix is not fixed as in NNGPs but learnable, while regularized by a NNGP prior. Therefore, the model can have a large number of trainable parameters. Together with the convolutional structure, the model achieves 92% on CIFAR10, which outperforms previous convolutional GP methods and NNGP/NTKs.", "questions": "Please see above questions.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The proposed convolutional DKM model has strong performance on classification benchmarks. 92% accuracy on CIFAR10 is quite impressive if we classify DKM into the broad class of kernel methods. \n\n* The technical development is sound--the proposed model formulation and training algorithms are sensible and computationally efficient.", "weaknesses": "* The contribution seems incremental. The proposed method is a combination of DKMs and the inter-domain inducing points used in convolutional (D)GPs.\n\n* The objective for learning parametric gram matrices is regularized by a NNGP prior: $KL(N(0, G^\\ell) \\| N(0, K(G^{\\ell-1}))$ computed from the gram matrix of the previous layer. Although this seems a sensible objective function to enable representation learning, it is unclear what the first principle is behind it. It is mentioned briefly in the paper \"Formally, the DKM objective is the evidence lower bound (ELBO) for an infinite-width DGP in the Bayesian representation learning limit.\" It would be great to see a proof on this.\n\n* The results are quite impressive if we categorize the proposed method into the \"kernel method\" class. However, I feel more evidences are needed to justify that this is a fair comparison. For example, the number of parameters in DKMs is much larger than NNGPs and convolutional (D)GPs due to the trainable gram matrices. It would be good to report these numbers in experiments and compare to a ResNet with the same number of parameters. Would the regularization bring any advantages over a such a ResNet? \n\n* One big benefit of (D)GPs is that they can produce uncertainty estimates. Do convolutional DKMs exhibit similar capabilities? Are their predictions well-calibrated? Answering these questions could greatly strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "25OoToysDE", "reviewer": "Reviewer_3QEZ", "summary": "The paper extends the work of deep kernel machines with infinite-width Bayesian NN to convolutional networks. While the extension of the base model to convolutional networks is quite straightforward, making it efficient with inducing points is more challenging and is where the main technical novelty of this paper is.", "questions": "My main question is about Eq. 18 and the reason behind mixing all the previous inducing points.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The work is a significant contribution to the field of infinite-width NN\n- The extension to sparse convolutional DKM is novel\n- The method has good results on several standard benchmarks.", "weaknesses": "- I found the definition of $C^\\ell$ in Eq. 18 very confusing and unintuitive. This should be the inducing points equivalent of the BNN convolution operation in Eq. 8. However this involves summing over all of the inducing points in the previous layer. This is unusual and isn't part of any standard convolution.\n- Experiments only show accuracy and NLL, and while this is fine for standard machine learning models, the benefit of the Bayesian approach is mostly in the other benefits it adds besides plain accuracy. It would make the paper much better if you show that your VI approximation still holds these properties with additional evaluations on these benchmarks like calibration or out-of-distribution detection.\n- The writing needs improvement. English editing is needed to make the paper read like a more coherent text, instead of disjoint sentences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "padhhRa0ul", "reviewer": "Reviewer_hjfZ", "summary": "This paper formulates the convlutional deep kernel machines based on the framework of a very recent work (Yang et al. 2023), which presents a modified  Neural Network Gaussian Process (NNGP) for retaining a better representation learning. In this work, the authors formulate the convolutional operations in deep kernel machines built upon the convolutional kernel, apply the NNGP framework from (Yang et al. 2023) with an ELBO objective attained, further utilize the sparse GP technique with specific consideration to the convolutional kernel case for yielding a tracble training scheme with the induced points, and conduct experiments mainly mirroring ResNet20 with good results compared to other kernel techniques.", "questions": "### Major concerns:\n1. In the presented content, it does not clearly elaborate or reveal the classical benefits of using kernel method (e.g., more interpretation, primal-dual representation and optimization, analytical results from RKHS, or [1] etc), nor provide practical values of using this method (e.g., varying the architecture besides ResNet20, a possibility of using a simple architecture). \n\nBesides, Would it be possible  that the GP-relied framework in this deep kernel machine shall have advantages in dealing uncertainty in data/images with better robustness or generalization ability. I would suggest the authors to articulate this aspect.\n\n[1] Bohn, B., Rieger, C., & Griebel, M. (2019). A representer theorem for deep kernel learning. The Journal of Machine Learning Research, 20(1), 2302-2333.\n\n2. To my personal experience, the title of this paper looks too \"giant\", as there exist different lines of work which try to deepen kernel machines w.r.t. different principles, setups and goals. The DKM is basically relying on the NNGP framework (Yang et al. 2023) and the convolution kernel. Besides the cited work in the paper, parts of other work regarding deep kernel machines can involve (**not limited to**):\n \nMairal, Julien, et al. \"Convolutional kernel networks. Advances in neural information processing systems 27 (2014),\n\nMairal, Julien. \"End-to-end kernel learning with supervised convolutional kernel networks.\" Advances in neural information processing systems 29 (2016), \n\nChen, Dexiong, Laurent Jacob, and Julien Mairal. \"Recurrent kernel networks.\" Advances in Neural Information Processing Systems 32 (2019), and some other works of Julien Mairal https://lear.inrialpes.fr/people/mairal/resources/pdf/Ohrid_CKN.pdf, where neural networks are involved quite a lot towards a deep architecture.\n\nSuykens, Johan AK. \"Deep restricted kernel machines using conjugate feature duality.\" Neural computation 29.8 (2017), \n\nTonin, Francesco, et al. \"Deep Kernel Principal Component Analysis for Multi-level Feature Learning.\" arXiv preprint arXiv:2302.11220 (2023), \n\nAchten, Sonny, et al. \"Semi-Supervised Classification with Graph Convolutional Kernel Machines.\" arXiv preprint arXiv:2301.13764 (2023), \n\nand other works of Johan AK Suykens, where the primal-dual framework retaining feature maps (e.g., NNs) in the primal and kernel tricks in the dual is a central idea. I would suggest to have a more comprehensive review on deep kernel machines in related work, though they are designing the model differently. It might also help to distinguish the unique benefits of the proposed method.\n\n\n### Minor aspects:\n1. A configuration figure presenting the convolutional kernel, the deep  architecture and the flow of optimization with sparse GP can be helpful for illustration I suppose.\n\n2. Some typos and non-mathemcatical edits, e.g., the 2nd paragraph under Table 3 on page 8.\n\n3. In the provided codes, there are different variants of nonlinear kernels. But I didn't notice a clear descriptions on how these nonlinear units are chosen or considered.\n\n4. How about the empirical efficiency difference on varying the inducing points setups in Table 2?\n\n5. In Table 1, many factors are investigated, and in different sections mostly the performances did not changge drastically in the compared techniques and were similar to other sections. The evaluation is comprehensive, but the conlusion is less focused, as we would hope to quickly get which factors are the most important ones. I these results don't change much and all attain good results, and then a mild suggestion or a global defaulting setting can be a take-away message.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In general, the elements involved in the work and the used technique framework are clear. Despite of the \"a bit giant\" title, the content presented in the work reflect what the title conveys, so the focus of this work is  well presented.", "weaknesses": "However, for the work itself, I found a bit struggling to appreciate the (both theoretical and practical) benefits of applying the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qe1oh86SDh", "reviewer": "Reviewer_jw8S", "summary": "The paper proposes convolutional deep kernel machines (DKMs) which builds upon a previous work of DKM while considers only settings with small-scale and low-dimensional data. The authors first introduce how DKM parameterizes the intermediate kernel matrices in NNGP limit by modifying the objective function. The convolutional DKM is then constructed by incorporating convolutional structure into the kernel function with Gaussian prior on the filter weights. To reduce its computational complexity, the authors leverage the idea of inducing point approach by eliminating spatial structure in inducing points while keeping spatial structure in the remaining data points. The proposed model is tested on several benchmark computer vision datasets with different sets of structures and hyperparameters and compared to other kernel-based models.", "questions": "•\tThe authors state that DKMs are derived from deep GPs where the width of the intermediate layers goes to infinity, but it seems that the kernel function in Section 4 is calculated based on finite number of channels (i.e., $N_l$ is finite). Can authors provide some clarification?\n\n•\tIt is not clear whether the kernel matrix $\\textbf{K}\\_{\\text{Conv}}$ is positive-definite by defining $\\textbf{F}_{i}^{l}$ using learnable parameter $\\textbf{C}^{l}$, or at least a priori (i.e., before training). Can authors give some intuition on this?\n\n•\tWhy is the convolutional DKM compared with other models only on CIFAR-10 but not CIFAR-100 in Table 3?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "•\tTo my knowledge, the proposed inter-domain inducing point approach seems to be novel and work well in the given context.\n\n•\tThe authors conduct a comprehensive study of different architectures and hyperparameters of the proposed method.\n\n•\tThe paper is clearly written and well organized.", "weaknesses": "The scope and applications of convolutional DKM are somewhat limited. From my perspective, a main purpose of kernel-based methods (i.e., NNGP, NTK, DMK, etc.) is to help explain the learning mechanism of Bayesian NNs instead of giving state-of-the-art performance on regression or classification tasks (as can be observed in Table 3). It is thus desirable to briefly discuss how the proposed method contributes to this domain (i.e., explaining the learning scheme of Bayesian convolutional NNs).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)