Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machines
Authors: Edward Milsom
Ben Anson
Laurence Aitchison
Venue: NeurIPS 2024 poster (2024)
Keywords: gaussian process
deep gaussian process
kernel methods
representation learning
Primary area: probabilistic_methods
TL;DR: We improve an existing kernel method to achieve 94.5% test accuracy on CIFAR-10, a significant increase over the current SOTA for kernel methods.

Abstract: Recent work developed convolutional deep kernel machines, achieving 92.7% test accuracy on CIFAR-10 using a ResNet-inspired architecture, which is SOTA for kernel methods. However, this still lags behind neural networks, which easily achieve over 94% test accuracy with similar architectures. In this work we introduce several modifications to improve the convolutional deep kernel machine’s generalisation, including stochastic kernel regularisation, which adds noise to the learned Gram matrices during training. The resulting model achieves 94.5% test accuracy on CIFAR-10. This finding has important theoretical and practical implications, as it demonstrates that the ability to perform well on complex tasks like image classification is not unique to neural networks. Instead, other approaches including deep kernel methods can achieve excellent performance on such tasks, as long as they have the capacity to learn representations from data.

Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machines
Edward Milsom
Ben Anson School of Mathematics University Bristol
School of Mathematics
University Bristol
edward mi www.soombrrittt...e.. ac.
ben. anson@rristol. ac. uk
Laurence Aitchison
School of Engineering Mathematics and Technology University of Bristol
laurence aitchison@gmail.c com
Abstract
Recent work developed convolutional deep kernel machines, achieving 92.7% accuracy on CIFAR-10 10 using ResNet-inspired architecture, which is SOTA for kernel methods. However, this still lags behind neural networks, which easily achieve over 94% test accuracy with similar architectures. In this work we intro- duce several modifications improve the convolutional deep kernel machine's s generalisation, including stochastic kernel regularisation, which adds noise to the learned Gram matrices during training. The resulting model achieves 94.5% test accuracy CIFAR-IO. This finding has important theoretical and practical impli- cations, as demonstrates that the ability to perform well on complex tasks like image classification is not unique to neural networks. Instead, other approaches including deep kernel methods can achieve excellent performance on such tasks, as long as they have the capacity to learn representations from data.
Introduction
Neural network Gaussian Processes (NNGPs) (Lee et al. 2017) are a key theoretical tool in the study of neural networks. When a randomly initialised neural network taken in the infinite- width limit, becomes equivalent Gaussian process with the NNGP kernel. A large body of work has focused
on improving the predictive accuracy of NNGPs (Novak et al. [2018] Garriga-Alons et al.) 2018
Arora et al. 2019 Lee et al., 2020 Li al. 2019 Shankar al. 2020 Adlam et al.) (023). but they still fall short of finite neural networks (NNs). One hypothesis is that this is due to the absence of representation learning: the NNGP kernel is fixed and deterministic function of its inputs KMac 1998 |Atthhsson 2020), but finite neural networks fit their top-layer representations to the task; this aspect of learning considered vital for the success of contemporary deep learning (Bengio et al.) 2013 LeCun et al., 2015). Exploring this hypothesis through the development of NNGP-like models equipped with representation learning could deepen our understanding of neural networks. Although
there are some theoretical frameworks for representation learning (Dyer & Gur-Ari] [019 Hanin &
Nica 2019 Aitchison) 22200] Li Sompolinsky [2020] Antognini 2019 Yaida [2020] Naveh et
2020 Zavatone- Veth et al. [2021] Zavatone Veth & Pehlevan
et al. [avehh
Naveh &
Ringel 2021 Halverson al. 2021: Seroussi al. 2023). they are generally not scalable enough
handle important deep learning datasets like CIFAR-10.
One recent promising proposal in this area is deep kernel machines (DKMs Yang al. 2023 Milsom et al.) 2024). DKMs are an entirely kernel-based method (i.e. there are no weights and
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

no features) that is nonetheless able to learn representations from data. DKMs narrow the gap to DNNs, achieving 92.7% performance on CIFAR- 10 Milsom al.) 2024 vs 91.2% (Adlam al.) 2023) for traditional infinite-width networks. However, 92.7% is still far from standard DNNs like ResNets, which achieve around 95% performance on CIFAR-10. Here, we narrow this gap further, achieving 94.5% performance on CIFAR-10, by introducing two key modifications. First, we introduce a regularisation scheme inspired by dropout (Srivastava et al. 2014) where we randomly sample positive-definite matrices in place of our learned Gram matrices. We refer to this scheme as stochastic kernel regularisation (SKR). Second, we use single-precision floating point arithmetic to greatly accelerate training, allowing us to train for more epochs under fixed computational budget. This presents challenges concerning numerical stability, and we develop a number of mitigations to address these.
2 Background: Convolutional Deep Kernel Machines
Here, we provide brief overview of convolutional deep kernel machines. For a more in-depth introduction see Appendix A or Milsom et al. (2024). Deep kernel machines are a class of supervised learning algorithms that compute a kernel from inputs and transform this kernel through a series of learnable mappings over Gram matrices. We can then perform prediction with the top layer kernel representation using e.g. GP regressin/classifcatton to deep Gaussian process, with the key difference being that instead of features F E IR at each layer, where P is the number of datapoints and N₂ is the number of features at layer f, we work with Gram matrices G E PxP.. We define the Gram matrices as the (normalised) dot product between features:
deep kernel machine is similar in structure
1 1-11 F'(TTT
G¹ G'= Ne
(1)
Many common kernel functions K(.), such as the arccos (Cho & Saul 2009) and squared-exponentian depend on features only through their pairwise dot products, and thus can be computed in this reparametrised model as K(G') instead of the usual K (FF). To obtain a deep kernel machine, all layers are taken in the anfintt-wwddth limit, i.e. N₂ -> co, and the likelihood function is scaled to retain representation learning. Under this limit the Gram matrices, {G')/_) )/_1 which were previously random variables whose distribution we might approximate through variational inference (see "deep kernel processes" Aitchison et al. [ Ober & Aitchison 2021 Ober et al. 2023), become deterministic/ / point-distribute., and can therefore be treated as learnable parameters. We learn the Gram matrices by optimising the deep kernel machine objective (which is itself derived from the evidence lower-boun bound (ELBO) for variational inference in the infinite- width limit) using gradient ascent:
C(G G = log P (Y|G²) - -E DRL. (N (o. G¹) || (OG)(G))))
(2)
Since the computations involved are very similar Gaussian processes, DKMs naturally scale like ((P)) with the number of training points. Implementations of shallow Gaussian processes often compute the full kernel matrix using lazy evaluation (Novak al., 2019) Gardner et al. 2018 Charlier iet al. 2021) which avoids storing the entire kernel matrix in memory at once. This process is often very slow (Google's 's neural tangents library takes 508 GPU hours compute the Myrtle- 10 kernel on 1FFAR-10/Go02
2020 Novak et al., 2019). and therefore infeasible for our setting where the model is both deep, and requires thousands of iterations of gradient ascent. Instead, previous work on deep kernel machines utilises sparse variational inducing point schemes, inspired by similar work on deep Gaussian processes (Salimbeni Deisenroth, 2017). These schemes replace the P. training points with P. pseudo-datapoints, where P < P.. which leads to O(PS + P)), computations which are much cheaper. In deep Gaussian processes, separate set of Pf inducing points is learned for each layer f, approximating the input-output functions locally. Deep kernel machines typically use an analogous inducing point scheme, learning an inducing Gram matrix Gj E RR xP, at each layer (rather than the full Gram matrix for all training points, G E TR Rx/P) by optimising a similar objective Eq. (2) (see Eq. 27 in Appendix A for further details). Prediction of train/test points is described in A Algorithm|] but at a high level, it is similar to Gaussian process prediction, except instead of working with feature vector partitioned into inducing points and train/test points, have a Gram matrix partitioned into 4 blocks,
F F
F
T
N.
F
(F))"
N.
I (F
G C
N
F
Ne
(F(()))))
2

---

The goal in prediction is to compute the conditional expectation of GÚ E ER and G E RR conditioned on the learned inducing block G. E P
Convolutional deep kernel machines combine deep kernel machines with convolutional kernel functions from the infinite-width neural network literature / Gaussian process literature (van der Wilk et al. 2017; Dutordoir et al., 20200 Garriga-Alonso et al.) 2018: Novak et al. 2018). In these settings, kernel matrices are of size PW H PWH where P the number of training images, and W, H. are the width and height of the images, respectively. In particular, this implies that inducing Gram matrices G are of size P. H P W / which is too expensive even for CIFAR- 10 (c.g. 10 32 x 32 inducing CIFAR-I0 images results in a 10, 240 x 10, 240 kernel matrix). To avoid this exorbitant computational cost, Milsom et (2024) proposed an inducing point scheme where the learned inducing blocks G have no "spatial" dimensions, i.e they are of size P. x Pf whilst the predicted train / test blocks G E ///WH.,, spatial dimensions. They use linear maps C' E PPf to map the P(-1 non-spatial inducing points into Pf patches with D pixels, which can be used by the convolutional kemnl.
G R. XP.W G IR retain their
The full prediction algorithm is given in Algorithm 1 In practice, the IID likelihood functions used at the output layer mean only the diagonal of G is needed, dramatically reducing the computation and storage requirements to vector G := diag( (G E Rg/ and only minibatches of the data are required (Yang et al. 2023). The parameters are updated via gradient descent by computing the objective (Equation 27 in Appendix A using the predictions, and backpropagating.
3 Methods
We seek to improve the generalisation of the convolutional deep kernel machine via two main strategies. First, seek to reduce overfitting of representations by introducing randomness to the learned inducing Gram matrices at train time, replacing them with samples from the Wishart distribution. We refer to this process as "stochastic kernel regularisation" to avoid ambiguity with terms like "random sampling". Second, we improve the numerical stability of the algorithm enough to utilise pwwer-precision TF32 cores in modern NVIDIA GPUs, which are significantly faster but more prone to round-off errors. Using TF32 cores makes training roughly x faster than the implementation in Milsom al. (2024), which used double- prebie-preciion floating points, and therefore allows us train for significantly more epochs. Numerical stability is improved through a combination of stochastic kernel regularisation and Taylor approximation to the problematic log-determinant term in the objective function, which we show has no negative effect on predictive performance in our ablation experiments. We also observed that keeping the regularisation strength v in the DKM objective non-zero was crucial in preserving numerical stability.
(Equation 2
3.1 Stochastic kernel regularisation
Milsom et al. (2024) observed that convolutional deep kernel machines suffer from overfitting. Under the infinite-width limit, the distributions over Gram matrices become point distributions, and offer no stochastic regularising effect. Inspired by dropout in neural networks (Srivastava et al. 2014), we introduce random noise into the training process to reduce overfitting of representations. Specifically, replace the inducing Gram matrices G each layer with sample G from the Wishart distribution,
G W(G
which has expectation G. and variance inversely proportional to 7. Strietly speaking, the Wishart distribution has support over positive-definite matrices only. This positive-definiteness constraint corresponds to the requirement 2 Pf. which in turn upper-bounds the variance of the samples. In highly expressive models with many inducing points, it may be beneficial to have much higher variance samples than this, so we relax the constraint on y, leading to potentially singular matrices, and then apply jitter to the samples, i.e. G - G + XI, to ensure sesitive-defnnienness. Random sampling is disabled at test-time, though we still add the same jitter to eliminate bias between train and test predictions. We refer to this process as stochastic kernel regularisation (SKR).
3

---

Algorithm Convolutional deep kernel machine prediction. Changes from this paper are in red.
Input: Batch of datapoint inputs: X, IR. P......... XMo
Output: Distribution over predictions Y
Parameters: Inducing inputs X,, inducing Gram matrices M inducing output GP approxi-
mate posterior parameters Mi.....Muu E (shared across classes), inducing "mix-up" parameters
(C where C
RDP
Initialise full Gram matrix
G
GO
X X
X,X, X,X
G
for in (I...., L) do
Apply kernel non-linearity (-)) (e.g. arecos kernel)
(GG
G - (G
L
Apply convolution and "mix up" inducing points (see Appendix A or Milsom et al. 2024
Indexing d represents pixels within patch, and ri, denotes "image feature map 1, spatial location
Ki D Ex (H()(C
Ki(t.)) = D Ed Pu(t.rta) (C)) i.e. K; = conv2D Pb, filters C²)
Ku(Er))(ss) = Ed $ (f)++) similar to the avg avg_pool2D operation Apply stochastic kernel regularisation to inducing Gram matrix
G W(G + AI
Predict train/test components of Gram matrix conditioned on inducing component G
Kui = K K KT
G K G K + Kui
G K K
end for
Average over spatial dimension, forming an additive GP (van der Wilk et 2017 (r) and (s)
index spatial locations in feature map, and S is the total number of spatial locations.
G G Σ, Gu(r)
G
37 Ers (u(r)(s)
L
Final prediction using standard Gaussian process expressions
Kii
K
(af
(Gflar
(Gllay G
Ki
K,
Sample features Fx pliI++ conditioned on K and inducing outputs ((1) ~ N( Σ)
x pl:--1 ~ (KK 2x Kn K 'K, K2 ' K,)
Monte-Carlo average over softmax of features to obtain categorical distribution over classes
Y = E [sottmax (f% (t: f.:,++)
Training: Compute DKM objective (Eq. 27 with Taylor approximation (Eq. using true labels
Y, and backpropagate update parameters.
3.2 Enabling lower-precision floating point arithmetic
Previous implementations of deep kernel machines (Yang et al. 2023 Milsom al.) 2024) used doublc-precision floating point arithmetic, which is very slow. Modern GPUs are highly optimised for lower-preeision floating point operations. For example, the NVIDIA 1000 GPU marketing material (NVIDIA 2021) quotes 9.7 TFLOPS for FP64 operations, 19.5 TELOPS for FP32 operations, and 156 TFLOPS for TensorFloat-32 (TF32) operations, proprietary standard that has the 8-bit exponent (range) of FP32 but the 1i- mantissa (precision) of FP16, for a total of 19 bits including the sign bit (Kharya, 2020). Therefore, switching from FP64 to TF32 numbers suggests potential speedups of up to 8x, though in reality speedups will be more modest as not all operations support TF32. Working with kernel methods low precision arithmetic requires care to ensure numerical stability. For example, direct inversion of the kernel matrix K₁ should be avoided, and instead all operations

---

of the form K, B should instead be computed by factorising K and solving the system X B (Trefethen & Bau 1997).
Unfortunately, the convolutional deep kernel machine as presented in Milsom et al. 2024 is highly unstable with now-precision arithmetic. Subroutines for the exact computation of Cholesky de- compositions fail completely (halting any further progress in training) when large round-off errors accumulate. This problem particularly acute when dealing with large kernel matrices, which are typically very ill-conditioned. The usual solution is to add jitter to the kernel matrices, but we found this was insufficient when using such low-precision arithmetic (Table 3 Instead, we hypothesised that the problem lay not in the kernel matrices K, but rather in the learned inducing Gram matrices Gf. In particular, we observed that the condition number of G tended worsen over time (Fig. 4 suggesting that learning highly expressive representations led to ill-conditioned Gram matrices.
Though the stochastic kernel regularisation scheme proposed did result in improved condition numbers during training (Fig- [ we still observed occasional failures in our large-scale experiments on CIFAR-IO (see ablations T)ble
B). We suspected that the issue might be due the regularisation / KKL-divergence terms in Eq. (2). These KL-divergence terms can be written as
DKL (N (O, G)| (0, K)) = Tr(K GG) logdet(K G) + const.
(5)
This should be understood ()..2). we would set G = G and K = KG').. The KL divergence is problematic in terms of stability for two reasons. Firstly, the log-determinant term is highly unstable operation, particularly in the backward pass which involves inverting the kernel matrix (Petersen & Pedersen (012). Secondly, computing K- G for the trace requires forward and backward substitution using the cholesky of K, which is typically a very ill-conditioned kernel matrix.
function, with two arguments. G and K. To evaluate the objective
To reduce the number of unstable operations, replaced the log-determinant and trace terms with their second-order Taylor expansions. Since we expect the Gram representations to be close to those of the NNGP, i.e. G- x I, our Taylor expansions are taken around A, = 1, where X, is the ith eigenvalue of G GG In particular, the beg-determinant term can be approximated as,
-logdet(K --ieettK-GG)) G) logdet(G (GoattG-Gi GG
(6a)
Σ log A
(6b)
Σ (A: 1) [A; - 1)2
(6c)
(G-'K -
- Tr
(6d)
In the "~ " step we have taken the second order Taylor expansion of log(A:) around X; = 1, and in the final step we have used the fact that the trace of a matrix is equal to the sum of its eigenvalues. Similarly for the trace term we have,
Tr(K G) Σ
(7a)
(A. 11- (A: 11)
(7b)
--TrGG 1) + (G- (G- 1) + const.
(7c)
Putting these approximations together we obtain,
DKL (N (0, G)I ()) ~ Tr (G- 1)²] + const = 1 G - 1P + const,
(8)
where I| is the Frobenius norm. Computing G-lK K should be more stable than K - G since the inverse only backpropagated the learnt cholesky of G. rather than through K earlier parts of the model, avoiding further compounding of round-off errors.
Experiments
4.1 Image classification experiments
We evaluated our method on the CIFAR-10 dataset (Krizhevsky Hinton 2009 containing 60, 000 RGB images (50, 000 train, 10, 000 test) of size 32 x 32 divided into 10 classes. We use the same
5

---

Method
Test Accuracy (%)
Test Log- Likelihood
Conv. Deep Kernel Machine (This Paper)
94.52 + 0.0693
-0.3611 + 0.0073
Conv. Deep Kernel Machine Milsomm et al. 2024
92.69 + 0.0600
-0.6502 + 0.0125
Tuned 1yrtle10 Kernel DA CG AAdam et al. 2023
91.2
NNGP-LAP-fiip (Li et al. 2019)
88.92
Neural Network (Adam) Neural Network (SGD+ + Weight Decay)
94.55 + 0.0361
11.303 + 0.0226 -..2112 + 0.0037
95.36 + 0.0523
Table I: Test metrics on CIFAR-10 10 using a DKM and a neural network with the same architecture. We report means and 1 standard error over random seeds.
architecture as in Milsom et al. (024) for ease of comparison, i.e. a ResNet20-inspired architecture with an extra size-2 stride in the first block, so that the output feature map sizes of the blocks are {16, 8, respectively. Wherever the ResNer architecture contains a convolution layer, we use a convolutional deep kernel machine layer as described in the loop in Algorithm TT That is, we apply base kernel (in our case, the normalised Gaussian kernel described in (Shankar et al. (2020), a more efficient/ / numerically stable alternative to arccos kernels), perform the (kernel) convolution, and then predict the train/test Gram matrix blocks conditioned on the kernel and the inducing Gram matrix block. In place of batch norm we use the batch kernel normalisation approach suggested by Milsom et al. (2024). Skip connections compute convex combinations of the kernel before and after pairs of layers. At the final layer, we average over the remaining spatial locations, forming an additive GP kernel akin to convolutional GPs (van der Wilk et al. (017) that is used to make the final predictions. categorical likelihood function is used in the lop-layer GP. Since the number of inducing points can vary between layers, we use (512, 1024, 2048} inducing points in the three blocks of convolutions, respectively, giving more expressive power to the later layers (similar to how ResNets are wider in later layers). For the stochastic kemel regularisation, we used 7 P /4 and a jitter size of X = 0.1, and for the objective we used a regularisation strength of y = 0.001. We train all parameters by optimising the sparse DKM objective function (Equation 27 with Taylor approximated terms from Section 3.2) using Adam (Kingma & Ba 2017). with 31 = 0.8, 32 = 0.9 and with an initial learning rate of 0.01 which is divided by 10 at epochs 800 and 1 100, for total of 1200 epochs. The model is implemented in PyTorch (Paszke et al. 2019).
We also train neural network with the same architecture for comparison, using a modified version of the popular "pytorch-cifar" GitHub repository the interest of fair comparison, we use network widths of (512, 1024, 2048} in the three blocks so that the model has a comparable number of parameters to the convolutional deep kernel machine. The neural network was trained for 1200 epochs, and report results using two different optimisers. One model used Adam with initial learning rate of 0.001 and the same learning rate scheduling as the convolutional deep kernel machine, and the other used SGD with momentum term of 0.9, weight decay strength of 0.0005, initial learning rate of 0.1 and a cosine annealing learning rate scheduler. We ran all experiments with 4 random seeds, and report the results in Table I with standard error of the mean, assuming normally distributed errors for statistical tests. On an NVIDIA A100 with TF32 matmuls and convolutions enabled, the Adam-trained neural network takes -45s per epoch, whilst our model takes ~260s per epoch. We estimate (very roughly) a total time, including the ablations and 1IFAR-100 experiments detailed later, of around 2000 GPU hours for all experiments in this paper, and around 2-3 times that number when including preliminary and failed experiments during the entire project.
The deep kernel machine matches the Adam-trained neural network with mean test accuracy of 94.52% compared to 94.55% for the neural network (two-tailed t-test with unequal variances gives p-value of 0.7634, suggesting significant difference). Furthermore, the deep kernel machine provides better uncertainty quantification as measured by (mean) log ligeleillooodd on the test data (higher is better), with average of -0.3611 compared to the
neural network's ' -1 -1.3003. Our model also far surpasses the convolutional deep kernel machine presented in Milsom al. 2024).. However, all these models still lag behind the SGD-trained network, which achieves a higher test accuracy of 95.36% (p-value of 0.0001 when compared to our model) and higher test
I 'Code at /tt.d hatrps///gthaao.c.ccccmnaa/l///t

---

Method
Test Accuracy (%)
Test LikeLikeld
Cony. Deep Kernel Machine (This Paper)
75.31 + 0.0814
-1.4652 + 0.0183
Conv. Deep Kernel Machine Miilsom al. 2024)
72.05 = 0.2300
-2.0553 = 0.0207
Neural Network (AdamW) Neural Network (SGD+ + Weight Decay)
74.13 + 0.0442
-1.9183 = 0.0070 -0.8890 + 0.0021
79.42 + 0.0380
Table 2: Test metrics CIFAR-100 using DKM and neural network with the same architecture We report means and 1 standard error over 4 random seeds.
Ablation
Test Accuracy
Test I. Log-Likelihood
Failures
No ablation/ Our full method
94.52 + 0.0693
0.3611 + 0.0073
0/4
No Taylor approximation to objective
94.46 = 0.0406
-0.3951 = 0.0081
1/4
No SKR
93.71 = 0.0150
-0.4512 = 0.0168
2/4
No Taylor+ + No SKR
93.25 (1 run)
-0.5113 (1 run)
3/4
No SKR but keep À = 0.1 jitter
93.46 (1 run)
-0.4762 (1 run)
3/4
V = 0 (Eq. 2
Fail
Fail
4/4
200 epochs
93.45 + 0.0225
-0 -0.2607 + 0.0016
0/4
Table 3: Test metrics on CIFAR-I0 with different ablations applied to our headline model (Table
D We report means and I standard error over the random seeds that ran to completion. Failures
indicates how many of the 4 random seed runs for each setting resulted in numerical error.
log-likelihood of 102211 (p-value 0.00002 when compared to our model). SGD is well known to train neural networks with better generalisation properties, and in particular for ResNets (Zhou et al. 202T; Keskar & Socher 2017 Gupta al. (02T), so this is perhaps not too surprising. We briefly experimented with using SGD to optimise the deep kernel machine but found it generally less stable than Adam. We hypothesise this is because the deep kernel machine has many different "types" of parameters to optimise, as seen in Algorithm I which may benefit from different optmisation strategies, whilst the neural network only has weights and few batchnorm parameters to optimise.
We further evaluated our method on the CIFAR-100 dataset (Krizhevsky & Hinton 2009), with results being presented in Table As in CIFAR-10, we found significant improvements over previous deep kernel machine work (Milsom et al. 2024), and we found our method is competitive with a ResNet trained with Adam, but still lags behind a ResNet trained with SGD, which is known to perform excellently on these tasks (Zhou et al.) 2021 Keskar & Socher 2017: Gupta et al., (02T). Note that we additionally had to use weight decay and a cosine annealing learning rate schedule with the Adam-trained ResNet to obtain acceptable performance on CIFAR-100.
To further investigate the effects of our changes, ran series of ablation experiments that are presented in Table 3 We report test accuracies and test log-likelihoods, but also the number of times each ablation failed out of the 4 random seeds as proxy for numerical stability. Our experiments verified that stochastic kernel regularisation (SKR) did yield a statistically significant improvement in test accuracy (p-value 0.0009). To verify that the improvement was in fact coming from the random sampling of matrices and not an implicit regularising effect of the large amount of jitter, we tested the model with SKR disabled but still applying the jitter X. We found that performance was still far worse than with SKR enabled; only seed ran to completion without a numerical error for this setting, so we cannot compute the standard deviation necessary for the t-test, but based on the other experiments it is very unlikely the variance would be high enough for this not to be statistically significantly lower than our headline number. Furthermore, our Taylor approximation in the objective function did not harm performance. In fact, on log- lieelhoosss we obtain p-value of 0.02953, suggesting a statistically significant improvement when using our Taylor approximated objective, but we believe this would require further investigation to verify. We also tested training with only 200 epochs, scheduling down the learning rate at epochs 160 and 180, and found that training for a 1200 epochs did indeed give a substantial boost to test accuracy. We found no single trick was enough to ensure stability over all our training runs, but rather a combination of our proposed modifications was necessary. We provide some brief analysis of the stability of the learned Gram matrices in the next section.
7

---

1040
10.0
Coailtie Number
10-1
10
10
25/100
Eipech
Ejuch
Ejods
Figure 1: Effects of different regularisation methods on Gram matrix condition number, in the toy binary classification problem trained for 2000 epochs. The left plot shows the condition numbers when different amounts of stochastic kernel regularisation (7) are applied. The middle and right plots show the condition numbers when the coefficient v of the KL regularisation terms are varied, with and without a Taylor approximation, respectively.
4.2 Effects of regularisation on Gram matrix condition number
To further investigate the numerical stability of our model, we ran a I layer deep kernel machine with the squared exponential kernel and 100 inducing points on a toy binary classífication problem. We show the condition number of the learned Gram matrix at each epoch for various values of the SKR parameter y (left, Fig. [T). the DKM objective regularisation strength when using the Taylor KL divergence terms (middle, Fig. D. and y when using the true KL divergence (right, Fig. D For the plot varying 7, we used irrelevant. Note that 7 = oc refers to no SKR. We ran these experiments in double precision to avoid numerical errors, and used the Adam optimiser with learning rate fixed 0.01. These experiments took about CPU hour in total.
that the effect of the Taylor approximation
the KI. terms is
We observe that more variance (smaller 7) in stochastic kernel regularisation slows down the rate at which the condition number G worsens. This makes sense, as the noise we add to the Gram matrix makes it more difficult to learn the "optimal" Gram matrix for the data, which would likely be highly overfitted, leading to extreme eigenvalues. However, running the experiment for long enough eventually leads to the same condition number for all settings of (see Fig in Appendix B We may expect this behaviour since the expected value of the SKR samples matches the true Gram matrix, but it's not clear how effectively the optimiser could achieve this outside of simple toy examples.
clear that, when using our proposed Taylor approximation to the KL divergence terms in the objective, even tiny values for the strength of these terms result in learned Gram matrices with condition numbers orders of magnitude better than without, and this effect grows proportionally with We also see an improvement in condition number when not using the Taylor approximation to the KL divergence, but only for large v. Setting / too large tends to harm generalisation (Milsom et al.) 2024), so it is beneficial to the Taylor approximation which reduces condition numbers even for small v. We also observed that the minimum condition number achieved across all settings of v was few orders of magnitude lower when using the Taylor approximation vs. when using the true KI. divergence. Furthermore, the behaviour in the plot using the true KI. divergence is rather erratic. For example, the 10 curve (right, Fig. T initially rises very poor condition numbers, but after few hundred epochs rapidly drops to a much smaller condition number. This leads us to believe that the difference between these two schemes can be explained by optimisation.
Our Taylor approximated terms penalise the Frobenius norm |GK - 1| much simpler operation than the true KI. divergence terms which penalise Tr (G- logdet (G- This complex penalty may result in a difficult optimisation landscape in practice.

---

5
Related Work
There is a substantial body of literature attempting to push the performance of kernel methods to new heights. These methods can broadly be split into "kernel learning" and "fixed kernel" methods.
Deep kernel machines, already extensively discussed in this paper, fall into the former category, as does the area of "deep kernel learning" (e.g. /Wilson et al., 2016) (Achituve et al.) 2023 Ober et al. 202T] to name a few). In deep kernel learning, a neural network is used to produce rich features which then passed as inputs into traditional "shallow" kernel machine, aiming give the best of both deep learning and kernel methods, Another "kernel learning" method is the "convolutional kernel machine" (Mairal et al. 2014 Mairal (016)). which draw theoretical connections between kernel methods and neural networks, though the resulting model is fundamentally a neural-network-llin architecture based on features, which distinguishes from deep kernel machines. [Song et al. (2017) also utilised deep neural networks to generate task- representations in a more complex model involving an ensemble of RKHS subspace projections. The main difference between deep kernel machines and these other methods is that deep kernel machines do not involve neural networks at any stage; the representations are learned directly as Gram matrices, not features.
By contrast, "fixed kernel" methods do not perform any representation learning during training, instead fixing their feature space before data is seen via the choice of kernel function. Though this could cover practically the entire field of kernel methods, the best performing methods on image tasks typically utilise kernels derived from the infinite-width neural network literature (Lee al.
2017 Jacot et al. 2018 Lee et al. 2020), sometimes called "neural kernels" (Shankar et al. (0200) In particular, Adlam et al. (2023) pushed the state of the art for CIFAR-10 test accuracy with "fixed kernels" to 91.2%, using Myrtle kernels (Shankar et al. 2020), type of neural kernel, by massively scaling up their method with distributed preconditioned conjugate gradient methods. Apart from the obvious lack of representation learning in this work, another key difference from our work is that they focus on computing large full-rank kernel matrices and finding approximate solutions using iterative solvers, whilst we use sparse inducing point approximations resulting in smaller kernel matrices, which we then solve exactly.
Deep kernel machines can be viewed as an infinite-width limit of deep kernel processes YYng et al. 2023) or deep Gaussian processes (Damianou & Lawrence 2013) with modified likelihood function, which results in the Gram matrices having point distributions. This can lead to overfitting. In deep Gaussian processes and deep kernel processes, the representations (Gram matrices in the case of deep kernel processes) have continuous distributions with broad support. thereby offering regularising effect. Our stochastic kernel regularisation scheme can be seen as analogous to sampling the inducing Gram matrix G in deep kernel process. Unlike deep kernel process, the other blocks G and G in our model remain deterministic, simplifying the model implementation. Other approaches to regularising kernel methods include "kernel dropout" proposed by (Song et al. al.] (2017). though in their context dropout refers to randomly removing latent representations from their ensemble during training. This therefore very different to our setting. In the neural kernel literature, Lee et al. (2020) identified correspondence between diagonal regularisation of kernels (jitter) and early stopping in neural networks, and found this usually improved generalisation. In this paper, we focused on regularising the learned intermediate representations/ / Gram matrices, rather than the final kernel, and found that diagonal regularisation had little effect on generalisation when applied to these matrices.
Previous work has attempted to improve numerical stability in kernel methods, though using different approaches. For example, Maddox et al. (2022) developed strategies to ensure numerical stability when using conjugate gradient solvers for GPs with low-precision arithmetic, but we do not use numerical solvers in this paper. (van der Wilk et al. (2020) circumvent the issue of computing inverses entirely using an approach based on reparametrisation of the variational parameters, but applying such an approach to the deep kernel machine domain would be a substantial undertaking, which we leave to future work.
6 Limitations
Though we have considerably advanced the state-of-the-art for kernel methods, from 92.7% MMilsom ett al. 2024) to 94.5% in CIFAR-10, there still remains gap to the best performing neural networks, both in terms of accuracy, and in terms of runtime. Nonetheless, given that we have shown that
9

---

representation learning in kernel methods has dramatically improved performance in kernel methods, from 91.2% (Adlam al. 2023) 94.5%. becoming increasinly likely that representation learning really is the key reason that NNGPs underperform DNNs. We leave further narrowing or even closing the remaining gap to DNNs for future work.
Constraints on computational resources meant that we could only run a limited number of experiments, so we focused on providing concrete insights single dataset with a series of ablations, rather than performance metrics for multiple datasets with no further analysis. Nevertheless, we provide all the code necessary to run these experiments on other datasets.
Conclusion
In this paper have increased the kernel SOTA for CIFAR-10 94.5% test accuracy using deep kernel machines, considerably higher than the previous record of 92.7% (Milsom et al. 22024). and significantly higher than NNGP-based approaches, such the 91.2% achieved by Adlam et al. (2023). We achieved this by developing a novel regularisation method, stochastic kernel regularisation, and by exploiting modern GPU hardware with lower- pewerrprecisoon arithmetic, which required us improve the numerical stability of the algorithm via a multi-faceted approach. We have highlighted the important role that representation learning plays in deep learning, which unfortunately absent from NNGP-based theory. We hope this work will encourage more research into theoretical models with representation learning.
Acknowledgements
Edward Milsom and Ben Anson funded by the Engineering and Physical Sciences Research Council via the COMPASS Centre for Doctoral Training at the University of Bristol. This work was carried out using the computational facilities of the Advanced Computing Research Centre, University of Bristol lhtt::// www. bris. iiissaac ac ac.uk/eckeeeuuu hak/acrc// We would like to thank Dr. Stewart for GPU compute resources.
References
Achituve, I.. Chechik, G., and Fetaya, Guided deep kemnel learning. In Evans, R. J. and Shpitser, I. (eds.). Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 11-21. PMLR, Jul-04 Aug 2023. URL.https:.. RtLL:ttts///prroceddnng
mlr watsss/v11//achii wwc11//cchitue tuve2aa. hhmll html
Adlam, B., Lee, Padhy, S., Nado, Z., and Snock, J. Kernel regression with infinite-width neural networks on millions of examples, 2023.
Aitchison, L. Why bigger is not always better: on finite and infinite neural networks. In JCML, 2020.
Aitchison, L., Yang, A. X., and Ober, Deep kernel processes. ICML, 2021.
Antognini. J. M. Finite size corrections for neural network gaussian processes. In ICML Workshop on Theoretical Physics for Deep Learning, 2019.
Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., and Wang. R. On exact computation with infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 3588) 2013.
Charlier, B., Feydy, J., Glaunès, A., Collin, F.-D., and Durif, G. Kernel operations on the gpu, with autodiff, without memory overflows, 2021.
Cho, Y. and Saul, L. K. Kernel methods for deep learning. In NewrIPS, 2009.
Damianou, and Lawrence, N. Deep gaussian processes. In Artificial Intelligence and Statistics, pp- 207-215, 2013.
10

---

Dutordoir, V., van der Wilk, M., Artemev, A., and Hensman, J. Bayesian image classification with deep convolutional gaussian processes. In Chiappa, and Calandra. (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (AISTATS), volume 108 of Proceedings of Machine Learning Research. PMIR, 2020.
Dyer, E. and Gur-Ari, G. Asymptotics of wide networks from feynman diagrams. arXiv preprint arXiv: 190..9999.1144.. 2019.
Gardner, J., Pleiss. G., Weinberger, K. Q.. Bindel, D., and Wilson, A. G. Gpytorch: Blackbox matrix- matrix gaussian process inference with gpu acceleration, In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. Utt/hctps://prroeediiii neur ips. cc/ sopper.p.f...
paper_ httpg///..aaeee///f1////////2271
Garriga-Alonso, A., Rasmussen, C. E., and Aitchison, L. Deep convolutional networks as shallow gaussian processes, arXiv preprint arXiv:1808.00558/ 2018.
Google. Neural tangents. 0.2.1.
2020. Version
Gupta, A., Ramanath, R., Shi, J., and Keerthi, S. S. Adam vs. SGD: Closing the generalization gap on image classification. OPT2021: 13th Annual Workshop on Optimization for Machine Learning. Sunnyvale, CA, 2021. Linkedln. https://opt-nnt ottp//looooopppppee///2222 paper53. pdf
Halverson, J., Maiti, A., and Stoner, K. Neural networks and quantum field theory. Machine Learning: Science and Technology, 2(3):035002, 2021.
Hanin, B. and Nica, M. Finite depth and width corrections to the neural tangent kernel. arXiv preprint 1rX 2019.
Jacot, A., Hongler, C., and Gabriel, F. Neural tangent kernel: Convergence and generalization in neural networks. In NeurIPS, pp. 8580-8589, 2018.
Keskar, N. S. and Socher, R. Improving generalization performance by switching from adam sgd, 2017.
Kharya, P. Nvidia blogs:
Tensorfloat-33 32 accelerates training hpc upto 20x, May 2020. URL https://blogs iwwiii..i..................mii.m
Kingma, D. and Ba, J. Adam: A method for stochastic optimization, 2017.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521(7553):436 -4:44436444440 2015.
Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., and Sohl-Dickstein, Deep neural networks gaussian processes. arXiv preprint arXiv:1711.00165, 2017.
Lee, J., Schoenholz, S., Pennington, J., Adlam, B., Xiao, L., Novak, R., and Sohl-Dickstein, J. Finite infinite neural networks: an empirical study. Advances in Neural Information Processing Systems, 33:151 15172, 2020.
Li, Q. and Sompolinsky, H. Statistical mechanics of deep linear neural networks: The back- propagating renormalization group. arXiv preprint arXivv2012.040000, 2020.
Li, 7.,, Wang, R., Yu, D., Du, S. S., Hu, W., Salakhutdinov, R., and Arora, Enhanced convolutional neural tangent kernels. arXiv preprint /91 191.19991,,,,,,, 2019.
MacKay, D. J. C. Introduction to gaussian processes. In Introduction Gaussian processes, 1998. URL https: //api. semanticscholarr
http://iw.....ac.oooooooooooaaia :tt./OorpuSIID:1122..0.

---

Maddox, W. J., Potapeynski, A., and Wilson, A. G. Low-pre precision arithmetic for fast gaussian processes. Cussens, J. and Zhang, K. (eds.), Proceedings of the Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pp. 1306-1316. PMIR, 01-05 Aug 2022. URL https: https://prooeeednngs .ml wwwpiiinss.l.......eeee. wwwss/eessvv maddox22a.htan
Conference on
Mairal, J. End-to-end kernel learning with supervised convolutional kernel networks. Advances in neural information processing systems, 29, 2016.
Mairal, J., Komiusz, P., Harchaoui, Z., and Schmid, C. Convolutional kernel networks. Advances in neural information processing systems, 27, 2014.
Milsom, E., Anson, B., and Aitchison, L. Convolutional deep kernel machines, 2024.
Naveh, G. and Ringel, Z. A self consistent theory of gaussian processes captures feature learning effects in finite cnns. arXiv preprint arXiv:2 04/ 10, 2021.
Naveh, G., Ben-David, O.. Sompolinsky, H., and Ringel, Z. Predicting the outputs of finite networks trained with noisy gradients. arXiv preprint arXiv:2004.01 190, 2020.
Novak, R., Xiao, L., Lee, J., Bahri, Y., Yang, G., Hron, J., Abolafia, D. A., Pennington, J., and Sohl- Dickstein, J. Bayesian deep convolutional networks with many channels are gaussian processes. arXiv preprint arXhh:1810.05116. 2018.
Novak, R., Xiao, L., Hron, J., Lee, J., Alemi, A. A., Sohl-Dickstein, J., and Schoenholz, S. S. Neural tangents: Fast and easy infinite neural networks in python. arXiv preprint arXiv:1912.028D0 2019.
NVIDIA. Nvidia a100 tensor core gpu. 2021.
URL
https://wwy
nvidia. httm://ontettttmmaeeeaaaaoaaaaaa hatpa////DDaaaeeteeeeaa//////// nvidia Sondia-aal---dataahee----eees wwwdia- 17590000--1199000-----0-- pdf
Ober, S. and Aitchison, L. A variational approximate posterior for the deep wishart process. Confer- ence on Neural Information Processing Systems, 2021.
Ober, S., Anson, B., Milsom, E., and Aitchison, L. An improved variational approximate posterior for the deep wishart process. Conference on Uncertainty Artificial Intelligence, 2023. In press.
Ober, S. W., Rasmussen, C. E., and van der Wilk, M. The promises and pitfalls of deep kernel learning. arXiv preprint arXiv:2102.12108, 2021.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.., Desmaison, A., Kopf, A., Yang, E.,, Vetooo, Z.,, Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, nigh-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035.
Curran Associates, Inc., 2019.
Petersen, K. B. and Pedersen, M. S. The matrix cookbook, nov 2012. wtt.httpp//wwn..commpp dtu. dk/pubdb/ hwwpud//wbbs////pppppppppppbbppt /aabs/3274-fuu full. html Version 20121115.
Roberts, D. A., Yaida, S., and Hanin, B. The principles of deep learning theory. arXiv preprint arXiv:2106. 10165, 2021.
Salimbeni, H. and Deisenroth, M. Doubly stochastic variational inference for deep gaussian processes. In Advances Neural Information Processing Systems, pp. 4588 4599, 2017.
Seroussi, I., Navch, G., and Ringel, Z. Separation of scales and a thermodynamic description of feature learning in some cnns. Nature Communications, 14(1):908, 2023.
Shankar, V., Fang, A., Guo, W., Fridovich-Keii S., Ragan- Kelley, J., Schmidt, L., and Recht, B. Neural kernels without tangents. In International Conference on Machine Learning, pp. 8614-8623, PMLR, 2020.
Song, H., Thiagarajan, J. J., Sattigeri, P., and Spanias, A. Optimizing kernel machines using deep learning, 2017.
12

---

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting, Jourmal of Machine Learning Research, 15(56): 1929-1958, 2014. URI. htth:httpp:...re/ /t/
html
Trefethen, L. N. and Bau, D. Numerical Linear Algebra. SIAM, 1997. ISBN 0898713617.
van der Wilk, M., Rasmussen, C. E., and Hensman, J. Convolutional gaussian processes, 2017. URL https://arxiv org/ www.///............ 01894
van der Wilk, M., John, S., Artemev, A., and Hensman, Variational gaussian process models without matrix inverses. In Zhang, C., Ruiz, F., Bui, T., Dieng, A. B., and Liang, D. (eds.), Proceedings of The 2nd Symposium on Advances in Approximate Bayesian Inference, volume 118 of Proceedings of Machine Leurning Research, pp. 1-9. PMLR, 08 Dec 2020. URL https 7/ //proceedings mlr, pttps//l1//wwleeeeeeee
html
Wilson, A. G., Hu, Z., Salakhutdinov, R., and Xing, E. P. Deep keree learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Yaida, S. Non- non-gaussian processes and neural networks at finite widths. In Mathematical and Scientific Machine Learning, pp. 165-192. PMLR, 2020.
Yang. A. X., Robeyns, M., Milsom, E., Anson, B., Schoots, N., and Aitchison, L. A theory of representation learning gives deep generalisation of kernel methods. ICML, 2023.
Zavatone- Veth, and Pehlevan, C. Exact marginal prior distributions of finite bayesian neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Zavatone- Vetth, J., Canatar, A., Ruben, B., and Pehlevan, C. Asymptoties of representation learning in finite bayesian neural networks. Advances in neural information processing systems, 34: 24765-24777, 2021.
Zhou, P., Feng. J., Ma, C., Xiong, C., Hoi, S., and E, W. Towards theoretically understanding why sgd generalizes better than adam in deep learning, 2021.
A Introduction to (Convolutional) Deep Kernel Machines
An introduction to both fully-connected and convolutional deep kernel machines can be found in (Milsom et al., (2024), but for completeness we provide an overview here. In particular, we show how sampling a large number of features at each layer of deep Gaussian process gives rise to a deep kernel method.
A.1 Fully-connected deep kernel machines
A DKM can been seen as a wide deep Gaussian process (DGP), optimised using a tempered ELBO objective. To see why, we first define DGP where subsequent layers of features are conditionally multivariate Gaussian. Assume we have P input data points X E IR Px No, and corresponding label categories y E (1....,C) C.. where C is the number of categories. Then we can place the following DGP prior on the data,
F =x X,
(9a)
P(F
[F-11)
II
N(C O, K Keaaurrs (F(-!)),
(9b)
A=1
P(y pb+11
II Categorical(y;; softmax ( (FL+!) )s:))
99c)
Here Fe E PxNe denotes the N₂ features at layer e, and F5 is the Xth feature at layer &. K Keatures (.) a kernel function that takes in features (as kernel functions usually do), written with the subscript "features" to different it later from kerne functions that take Gram matrices as input. Notice that the
13

---

final layer features F are logits for a categorical distribution over labels, though the likelihood distribution can be easily changed for the regression setting.
To derive an ELBO, we perform variational inference by defining the following approximate posterior the features at intermediate and final layers,
No P(F X, Q(F*) II
N Q(1) = N(C G
(10a)
x=1
x=1
No+1 N(f Pxx Σ).
P(F X, Q(FL=(()))))) II Q(Ex A=1
II A=1
(10b)
We learn the mean and covariance at the final layer, but only the covariances G E PxP at the intermediate layers. This choice is justified by the fact that after taking the limit No oc, this approximate posterior family (Eq. [O).[TO) contains the true posterior (see Yang et al. (023 for more details). The ELBO of the DGP with respect to the variational parameters is,
CELBO (G.,.... GI..!...... MNL+1 Σ)
(11)
No+1
EQ(E++1) [lg Ply [(+1)]
Σ
(K [F4))
(F4))
A=1
Σ (NeDKL(((F")
II P(P 1)
(12)
Here 3 parameter that tempers the prior. Before proceeding, make the following assumption that the kernel can be calculated using only the sample feature covariance G
Kreatures (F*) = K(G'),
(13a)
where G No FF()
(13b)
Assumption actually not very restrictive - satisfied by common kernels (such as RBF. Matern), and indeed any isotropic kernel. It is also satisfied in the limit N₂ + oo when K feature (X) (elJU(X)ReTTU(X
by the arccosine kernel (Cho Saul 2009).
We are now ready to recover a DKM. We set N₂- = NN; for each intermediate layer f = 1,... L. and temper with 3 N-1, In the limit N cc, is,
Yang al. (2023) showed that the limiting ELBO
Nu+1 E) E) [I
CELBO L := EEqro+))
log P(y F
DD G.) (IN(O.K((")))) 1
(14)
We can evaluate the expected log-likelihood in the DKM objective (Eq. (14) using the reparameterisation trick, and the KL divergence terms can be calculated in closed form. By optimizing L w.r.t. the variational parameters we 'train' the DKM. Optimisation not possible in closed form general (it is possible in the linear kernel case for regression problems, see Yang et al. (2023)), but we can train the parameters using gradient descent. The number of parameters is ( and the time complexity for evaluating the objective is O(P3), therefore can only optimise Eq. (14) directly for small datasets. We later discuss an inducing point method that enables linear scaling in the number datapoints, but first we introduce DKMs with convolutions.
where K(.): RPP + PxP a kernel function satisfying Assumption 13
A.2 Convolutional deep kernel machines
Above we outlined how a fully-connected DKM DGP with wide intermediate layers trained by tempering the ELBO. We establish a DKM for the convolutional setting by introducing convolution
14

---

into the DGP in Eq. 9
Fo X,
(15a)
Mr
P(H F*) II
Nh O, K Keaaures (F")),
(15b)
A=1
dud mia (0, (ID) -
(15c)
Me-1
F
Σ
H i(r+d)),
W
(15d)
duA
dED
for I,...,, as well as a spatial pooling layer before final classification,
SpatialPool(F"))
(15e)
dat
NLI1
P(F F
II
N 0, features (F (aa)),
(15t)
P
P(y F = II
Catogorical softmax (F
(15g)
2=1
Here, we consider datapoints to be spatial locations (a.k.a. patches) across range images, and index these with (image) and (location). By concatenating all patches (i.e. every patch in every image) together, we can represent an entire dataset of images with a single feature matrix F' E x where S is the set of patches in an image and is the number of images. For us, y a set of image-level labels rather than patch-level labels, Nç. The convolution (Eq. 15d uses convolutional weights
also include spatial pooling layer.
The pooled features Fi.. have size w' E P|x M, No, where D is the set of spatial locations in the filter. In our context, we only consider 2-dimensional images, therefore D will contain [D] 2-dimensional locations of patches.
We proceed by deriving a convolutional kernel. The conditional covariance of the features F has closed form,
B[F F ir js.n
H H = E
Σ
H H dyp
We dn,
Σ H a We Wa'
(16)
dj
d'u
Σ
He-1 HC-1
T.. [We Wan We W
(17)
d(r+dyy i(s+d y
ddl pa
1 DM
H ddr+d) j((oddjp
H-1
(18)
p=1
1
(19)
D
((r+d)s(oodd)
where Ω² is the sample covariance of H'. Under the layer-wise, infinite-width limit M oo. the sample covariance becomes the true covariance, s² K (F*). This means that we can compute the covariance of F$ conditioned only on the previous layer,
1 Σ K features (a(r))))))))tttt D dED
BIF Ff
F Fe-1
(20)
We can view Eq. 20 as kernel convolution operation, and we introduce the following notation for (I(K)) LEE ((++)))
Equipped with convolutional kernel r, we can recover convolutional deep kernel machine by taking the limit N₂ + oc. We again use the approximate posterior defined in Eq. 10 and temper the
15

---

prior. This gives us the convolutional DKM objective,
NL
C: :=:oppatt) log P(y
[F++))]
Σ DRL ( ((p[a(((paaPPPoolGG))
DN GO) |I
(21)
where again used Assumption 13 The spatial pool operation used in this paper is mean pooling (see Algorithm[]] [). This convolutional DKM objective implies a fixed convolutional kernel, but allows flexibility at intermediate layers via the variational parameters G
A.3 Inducing point approximations
Due to O(P2) parameters, and O(P)) computational cost, it is not feasible optimise the DKM objectives (Eq. 14 and Eq. (T) directly. To resolve this scaling problem, we appeal to inducing point methods. The idea is to approximate the full train/test dataset with smaller set of datapoints. We demonstrate an inducing point convolutional DKM here. See Appendix M Yang et al. (2023) for the fully-connected case.
We define the train/test datapoint features F. RRX,, and the inducing datapoints F X Pt is the number of train/test datapoints and Pe is the number of inducing points at layer L. We take the inducing points and train/test points to be jointly distributed according to the deep Gaussian process in Eq. 15 In other words, the concatenation of features
F
F F
E (P(+))))
(22)
satisfies the prior in Eq. 15 with the exception that the convolution for the inducing points is slightly modified so that,
Mr
(F)
Wa din
Σ
(H
(23)
dED
Milsom et al. (2024) motivated the extra 'mixup' parameter C2 E RIDIP"'PP as allowing infor- mative correlations between the inducing points and the train/test points to be learned. We can view the matrix multiplication C' H in Eq. 23 as having the effect of taking non-spatial inducing points H and mapping to them to inducing patches. This allows them to correlate meaningfully with the train/test patches. The covariances among inducing and train/test features can then be calculated,
((FF))AAF)) H = ΣEΣ
1 D dED &
Ca C Ch Dey = (T ( Dis, (24a)
B((T) [H- 1 D
Cte ))) = (ff Ma,g
(24b)
DED
B((F))aAAF)) H
1
M = D e-1 "
Dingo-
(24c)
D
dED
Here, st = (H") is the sample covariance for combined inducing and train/test samples H' [HE H² Suffices ii refer to inducing-inducing correlations, ti to train/testindducing correlations, and tt refer to train/test-train/tss correlations; though they may better be understood as referring different blocks of covariance matrix. Eq. [4 gives us learnable convolutional kernel, which we call (().. When we take the convolutional widths Mt be large, have,
N₂ N O, r (K features (FtiunssFFFF.) (Ft-1))).
P(F FC-1)= = II
(25)
A=1
16

---

As in the non-inducing case, we will compute an ELBO. To do so, we place an approximate postenior on the inducing points, similar Eq. I0
N₂
N₂
Q(F = II
Q
= II O
(26a)
A=1
N
Q(F++) II +(
II
N(C M2x E) E).
(26b)
A=1
A=1
Taking the layer-wise -ndinite-width limit Ne xo (again tempering the prior as in Eq. 14) we recover the following limit of the ELBO,
Linducing = EQ(FE-1) log P(y [F(-1)]
(27a)
11
(KL( Σ) (1, ((SpataaaPPPo))(
Σ
(N G)) [I
(27b)
However, it still remains perform inference on the train/test points. To this end, we 'connect' the train/test points to the inducing points by assuming the approximate posterior over all features decomposes like so,
Q(F Fー P(F; FFF
(28)
Due to the Gaussian structure of the DGP prior, the first term in Eq. 28 is Gaussian and can be written down using standard conditional Gaussian expressions,
N₂ P(F F F = II A=1
N S
( TEA F. - -
(29)
where I the result after applying the non-linearity kernel and then the convolutional kernel, i.e. I' geature (F*)). In other words, we can write F terms of standard multivariate Gaussian
noise E E ERAXN
Ff (T FF + LKK
(30a)
where r ua = rú. -
(30b)
In the fffnite-width limit N₂ oo, the sample feature covariance must converge the true covariance by the law of large numbers.
G i
FF -(F())) = =
(31)
N
G
G
We already know the true inducing point covariance matrices, G because they are parameters in our approximate posterior. However we can write down the remaining blocks the covariance using Eq. 30. We identify G and Gf, as,
G lim I F((FF) r -
(32a)
No N:
G lim
Ni+000 Nr (F()) - ' T
(32b)
Equations 32 and 24 as seen in Algorithm alongside leamned inducing points.
I allow us to propagate train/test points through the model
In the above, we treat datapoints as independent. This allows us perform minibatch training, thus greatly improving the scaling of the method over the full-rank version. Alongside the reduction learnable parameters from the inducing point scheme, we get linear scaling with the size of the dataset.
17

---

10a3
1090
10a.
Condition Number
1044
104
10
7 P
109
7 /1/10
7 21/100
5000
10000
Epoch
Figure 2: Effects of stochastic kernel regularisation Gram matrix condition number strength, in the toy binary classification problem trained for 10000 epochs. See Section 4.2
B Extra Figures
C Licenses
ResNets from hhoppp//..ggthhh.cccmcmmmiiiiiip MIT licensed.
CIFAR-10 is from https://www cs. Coronto.edu. html and has no li-
cense evident.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The abstract succinctly outlines the primary objectives and achievements of the study, aligning well with the detailed descriptions provided the main body of the paper. Similarly, the introduction effectively sets the stage by contextualiz- ing the research within the existing literature, clearly stating the problem addressed, and summarizing the approach taken. Both sections are consistent with the detailed findings and conclusions presented, ensuring that readers have clear and accurate preview the paper's content and scope right from the beginning.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
See Limitations section (Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
19

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There no formal theorems.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Sec. 4 which outlines all relevant details. We also provide the code reproduce our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
Depending on the contribution, reproducibility
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
20

---

Justification: We give error bars for the headline results of the paper (see Table [ as well for ablations (see Table Statistical significance tests are given for certain claims in Section 4 where we state the assumption of normally distributed errors. We only used one 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code provided in the supplementary materials, and at the anonymous link https: hate:///nnoommuss wop.n. scieeee//cinnn///rss.... http:///skksmmmmmmmee B10e-/BEE1//ER...... md
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co.
wtw.lmmmssioooooiiio.oooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. wwt.cc/ppccpcccc..eeeeeeeooooeee gwtp:///ieess/ee//ee www.eoomiiiiioonnooo.ooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Sec. for these details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
random seed the plots in Section 4.2 (Figures and ( avoid cluttering them, but this section is exploratory, and is not part of the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialisation, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
in addition to specifying what type of GPU used. We also estimate the total compute used during the project.
Justification: We report individual and aggregate runtimes in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't ' make it into the paper).
paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics htttps://neurrre http:////../////ccceiiieeeeeeeee
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics, and the research conducted in the paper conforms in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
22

---

Justification:
The paper is foundational research not tied to a particular application, analo- gous to improvements in optimisation algorithm. As such, the guidelines note that is not necessary to speculate about potential unforeseen societal implications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Licenses in Appendix
C
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
23

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not provide new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
24

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "skGGbAdrrT", "reviewer": "Reviewer_V8UE", "summary": "The paper reports numerical results in which the authors have achieved state-of-the-art performances for an image classification task, namely the CIFAR-10 dataset, with a ''convolutional deep kernel machine''. The performance of this kernel-based model is close to the ones of state-of-the-art neural network architectures on the same dataset. To achieve this result, the authors introduce a ''stochastic kernel regularization'' procedure consisting of adding random noise to the parameter during training. They also use an approximation of the training objective which improves the numerical stability of training and allows to leverage the performances of modern GPUs to speed up the training procedure.", "questions": "- Did you try to perform experiments on other datasets such as Imagenet ? Should we expect deep convolutional kernel machines with stochastic kernel regularization to have performances similar to ResNets or would they perform way worse ?\n\n- One of the encountered problem during training is that the condition number of the Gram matrices $G^\\ell_{ii}$ tends to worsen over time. Isn't that behavior to be expected when the number of inducing points is larger than the number of classes ? In my understanding, if the features are sufficiently expressive, the rank of the Gram matrices should be equal to the number of classes. Do you agree with this intuition ? Did you try to compute the rank (or eigenvalue distribution) of the Gram matrices ?\n \n- It is written that ''we expect the Gram representations to be close to those\n  of the NNGP'', could you provide a justification of this statement ?\n\nThe rest of my questions are concerned with some incoherences which might simply be typos but hinder my comprehension of the paper. Please correct me if I simply misunderstood those equations.\n\n- At the end of section 4.2 you write several times $G^{-1} K G^{-1}$, shouldn't it be $G^{-1} K$ ?\n  \n- In eq. 9b you write $K_{features}(F^\\ell)$, shouldn't it be $K_{features}(F^{\\ell-1})$ ? Also in this equation it is not really clear for me what the function $K_{features}$ is.\n  \n- In eq. 14 shouldn't the divergence term be $D_{KL}(N(0, G^\\ell)||N(0, K(G^{\\ell-1}))$ ?\n  \n- In eq. 15b shouldn't it be $K_{features}(F^{\\ell-1})$ ?\n  \n- In eq. 16 shouldn't it be $\\mathbb{E} \\left[ F^\\ell_{ir,\\mu} F^\\ell_{js,\\mu'} | H^\\ell \\right]$ ?\n  \n- In eq. 21 shouldnt' it be $\\Gamma(K(G^{\\ell-1}))$ ?\n  \n- In eq. 25 it should be $F^{\\ell-1}$.\n  \n- I do not understand eq. 29. The r.h.s. uses $\\Gamma^\\ell_{tt}$ which, if I understand correctly, is defined using the whole feature matrix $F^\\ell$ whose probability density we are trying to compute. Shouldn't it be $\\Gamma^{\\ell-1}$ in eq. 29 ?", "limitations": "Limitations of this work are discussed in the dedicated section. On top of that, I think that an important limitation is to only provide results for the CIFAR10 dataset. It could be discussed if one should expect convolutional deep kernel machines to have performances similar to the ones of neural networks for more involved image classification problems.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, clearly introduces the ''convolutional deep kernel machine'' model it is studying, and clearly reports its methodology and its result. While I am not necessarily very familiar with the literature on deep kernel machines, the improvement in generalization with respect to previous works is significant and certainly contributes to closing a gap between kernel-based models and neural networks. Also the introduction of a stochastic regularization procedure for those kinds of kernel-based models seems like an interesting contribution.", "weaknesses": "In my opinion, the main weakness of the paper is to only provide results for the CIFAR10 dataset. It would be very enlightening to perform experiment for more involved image classification tasks, for example on Imagenet.\n\nIn addition, I think the paper sometimes lacks clarity in the way the training procedure is exposed. For example, in eq. 2 the kernel function $K$ is not defined. Also, the stochastic regularization term does not appear in eq. 2 while it is presented as the objective function used during training. To address this issue, the authors could add a precise description of the training algorithm (in the same spirit as algorithm 1 but in my understanding algorithm 1 describes the algorithm for inference on the training/test data, and the last line about training is quite elusive).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review.\n\n> \"...main weakness of the paper is to only provide results for the CIFAR10 dataset\"\n\nWe have managed to secure enough compute resources to also benchmark our method on the CIFAR-100 dataset. We improve the CIFAR-100 test accuracy from 72.1\\% in earlier CDKM literature [1] to 75.3\\% with our methods.\n\n> \"I think the paper sometimes lacks clarity in the way the training procedure is exposed ... To address this issue, the authors could add a precise description of the training algorithm\"\n\nThank you for this feedback. Condensing complex background material down so there is still space for novel contributions can be tricky, so hopefully incorporating your suggestions in our working draft will improve readability.\n\nTo address some specific questions:\n\n> \"the stochastic regularization term does not appear in eq. 2 while it is presented as the objective function used during training\"\n\nThe objective function consists of 2 terms: the log-likelihood term, which encourages good performance on the training data, and the sum of KL divergences, which encourages the representation of each layer to be similar to the previous layer (a form of regularisation). We use the samples from \"stochastic kernel regularisation\" (SKR) during the forward pass, as described in the algorithm, which affect the log-likelihood term, but do not modify the KL divergence terms, meaning the KL terms still act directly on the original parameters $\\mathbf G^\\ell_\\text{ii}$, i.e. the means of the SKR samples. This is because only the log-likelihood term can cause overfitting to data, so there is no good reason to use the random samples in the regularising terms.\n\n> \"the last line about training is quite elusive\"\n\nWe have updated this to be easier to understand. Training is similar to any other pytorch model, in that we input data into our model, compute a loss function (in our case a rather complex DKM objective function) and then update the parameters of the model by backpropagation.\n\n> \"One of the encountered problem during training is that the condition number of the Gram matrices tends to worsen over time. Isn't that behavior to be expected when the number of inducing points is larger than the number of classes?\"\n\nAgreed, this is expected, at least in later layers.\nHowever, we found empirically that good performance required a number of inducing points far beyond the number of classes, so we only asked the simpler question \"how can we encourage these Gram matrices to be better conditioned?\".\n\n> \"It is written that \"we expect the Gram representations to be close to those of the NNGP\", could you provide a justification of this statement?\"\n\nThis is based on the intuition that we regularise our model to be close to the NNGP via the KL divergence terms in the objective (Eq. 2), combined with the fact that we initialise at the NNGP (i.e. set $\\mathbf G^\\ell = \\mathbf K^{\\ell-1}$).\n\n>\"At the end of section 4.2...\"\n\nThis is a confusion between matrix multiplication and function composition; we will simplify the notation here to make it clearer.\n\nRegarding typos in the appendix (we especially appreciate the reviewer's efforts in reading the appendix in addition to the main text):\n\n>\"In eq. 9b...\"\n\nFixed. $K_\\text{features}$ is just a traditional kernel like arccos or sqexp, we have now added a clarifying comment.\n\n>\"In eq. 14 shouldn't the divergence term...\"\n\n>\"In eq. 15b shouldn't it...\"\n\n>\"In eq. 16...\"\n\n>\"In eq. 21...\"\n\n>\"In eq. 25...\"\n\nAll fixed. Many thanks.\n\n>\"I do not understand eq. 29...\"\n\nThis is again a mixup between $\\ell$ and $\\ell -1$. We have thoroughly proofread the appendix and made sure these issues are now fixed / consistent.\n\nReferences:\n\n[1] Milsom, E., Anson, B., and Aitchison, L. Convolutional deep kernel machines, 2024, International Conference on Learning Representations (ICLR).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CAMLxX1OWO", "reviewer": "Reviewer_44ig", "summary": "This paper proposes a new method for Deep Kernel Machine, which achieves state-of-the-art results on kernel methods by using methods including regularization.", "questions": "Is there a comparison between results trained with double precision and single precision?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Compared to previous kernel methods, the proposed method achieves better results in CIFAR-10 test accuracy.\n\nThe use of stochastic regularization is straightforward and reasonable.\n\nSome ablation studies are provided, including results with different hyper-parameters.", "weaknesses": "Though achieving state-of-the-art results for kernel methods, the proposed method is still relative time consuming and requires much resource.\n\nAccording to the experiments, the proposed method might be sensitive to hyper-parameters and can lead to failure. Given the fact that it is already a time-consuming method, obtain a successfully trained model with good performance on new dataset might be very expensive.\n\nCIFAR-10 is a relatively easy dataset, whether the proposed method works well on complicated cases are not investigated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review.\n\n> \"Though achieving state-of-the-art results for kernel methods, the proposed method is still relative time consuming and requires much resource... sensitive to hyper-parameters\"\n\nWe agree that the method is time-consuming and sensitive to hyperparameters relative to SOTA neural network methods, and that these are important avenues for future work.  However, we do not believe that these are reasonable grounds for rejection, as our method is still SOTA for kernel methods by a considerable margin. Indeed, there are no kernel methods which are able to achieve close to this level of performance without facing similar issues.\n\n> \"CIFAR-10 is a relatively easy dataset, whether the proposed method works well on complicated cases are not investigated.\"\n\nWe have now additionally obtained performance-metrics on a different dataset, CIFAR-100. Our changes improve the test accuracy from 72.1\\% in previous CDKM work [1] to 75.3\\% in this work.\n\n> \"Is there a comparison between results trained with double precision and single precision?\"\n\nWe do not provide direct comparisons between single and double precision due to the expense of running these experiments in double precision (modern GPUs are far better optimised for single precision), but one can compare our work to previous work on CDKMs [1] which utilised double precision, and observe that even in our ablations, there is no noticable degradation in predictive performance when using single precision arithmetic.\n\nReferences:\n\n[1] Milsom, E., Anson, B., and Aitchison, L. Convolutional deep kernel machines, 2024, International Conference on Learning Representations (ICLR).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TChNgrJUy8", "reviewer": "Reviewer_cbyd", "summary": "The paper explores how to improve the convolution deep nuclear machines (DKMs)to improve their generalization ability, especially on the CIFAR-10 dataset. The authors introduced several modifications, especially random kernel regularization (SKR), which involves adding noise to the learned Gram matrix during training. The technique is inspired by discarding in neural networks and aims to reduce overfitting. Moreover, they use single-precision floating-point arithmetic to accelerate the training, thus allowing for more training cycles with a fixed computational budget.", "questions": "How does the stability of the single precision calculation results ensure, and whether the article has additional requirements on the data set", "limitations": "yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Improving performance: The proposed modification greatly improves the test accuracy of DKMs on the CIFAR-10, from 92.7% to 94.5%.\nNew regularization techniques: Introducing stochastic kernel regularization is an innovative way to reduce overfitting in DKM.", "weaknesses": "Modifications involve complex changes to the DKM framework, which may be difficult to implement and understand for developers unfamiliar with these methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n> \"Modifications involve complex changes to the DKM framework, which may be difficult to implement and understand for developers unfamiliar with these methods.\"\n\nWe agree that the methods themselves are very novel and hence somewhat unfamiliar.  To address these issues, we have explained extensively the background (Section 2), methods (Section 3), and included a concrete algorithm with the changes from our work highlighted in red (Algorithm 1).\nMoreover, we have provided code with the paper, and we are currently in the processes of developing an easy-to-use PyTorch like library that incorporates these innovations.\n\n> \"How does the stability of the single precision calculation results ensure, and whether the article has additional requirements on the data set\"\n\nWe include \"number of failures\" as a proxy for numerical stability in our ablation experiments. Section 4.2 provides further insight into the effects of our changes on the numerical stability of the model.\n\nWe make no assumptions on the dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O6MlGKqB64", "reviewer": "Reviewer_YneB", "summary": "The authors present an improved convolutional deep kernel machine (CDKM) that achieves state-of-the-art performance for kernel methods on the CIFAR-10 image classification task. They introduce a novel regularization technique where the learned inducing Gram matrices are randomly sampled from a Wishart distribution during training. This helps reduce overfitting and improves generalization.", "questions": "Can the authors describe how this is more similar to a kernel method than a type of neural network since the Gram Matrices and mixup parameters are parameterized and trained with backpropagation?", "limitations": "Societal impact not applicable.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The experiments evaluating the performance of their methods are thorough and convincing in providing an improvement over previous work. \n\nThey achieve 94.52% test accuracy on CIFAR-10, which is a significant improvement over the previous best kernel method result of 92.69% and comes close to matching comparable neural network performance (94.55% for an Adam-trained neural network with the same architecture).\n\nAs shown in Table 2, their improvements in numerical stability allow for the use of TF32 arithmetic, making training about 5 times faster than previous implementations that used double-precision floating point arithmetic.\n\nIn Table 1, the test-log likelihood improvements over previous work and traditional neural networks (without weight decay) is compelling as a result. But also surprised to see models with weight decay outperforming the other methods, if I understand correctly, CDKMs naturally provide uncertainty estimates for their predictions. Why would traditional models (with weight decay) be better at this?", "weaknesses": "Disclaimer: I am not familiar with the space of CDKMs. It's unclear to me what the main scientific insight is provided in this work. The results appear to be state-of-the-art in this space. But aside from performance, it's unclear to me that this isn't just a variation of traditional neural networks with some aspects of Kernel Machines added.\n\nBut this might be an issue with the general space of CDKMs and not this work specifically. Since the CDKM method is using some form of learning with backpropagation, what is the main point of showing this particularly models works as compared to conventional neural networks (like convolutional neural networks)?\n\nWith the Gram Matrix itself being learnable and also patch-based, this CKDM architecture is reminiscent of an MLP-Mixer style architecture  which under certain circumstances (particularly reasonably large dataset) also performs competitively with Convolutional Networks. Would it be unreasonable to consider the CDKM itself as just yet another neural network architecture?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n> \"In Table 1, the test-log likelihood improvements over previous work and traditional neural networks (without weight decay) is compelling as a result. But also surprised to see models with weight decay outperforming the other methods, if I understand correctly, CDKMs naturally provide uncertainty estimates for their predictions. Why would traditional models (with weight decay) be better at this?\"\n\nIt has been noted many times in the past that SGD leads to exceptionally good generalisation for ResNets [1], though the exact reason remains unclear. Although CDKMs are derived from Bayesian deep Gaussian processes, in order to retain representation learning, the precise infinite-limit taken means that we cannot expect all the benefits of fully-Bayesian methods to carry over when using DKMs. This point is discussed in more detail in previous DKM literature [2,3]. A better choice for uncertainty quantification would be a Deep Kernel Process or a Deep Gaussian Process. Nonetheless, uncertainty quantification was not our primary objective in this paper, but rather we sought to investigate representation learning in models other than neural networks, specifically kernel methods.\n\n> \"Would it be unreasonable to consider the CDKM itself as just yet another neural network architecture\"\n\n> \"Can the authors describe how this is more similar to a kernel method than a type of neural network since the Gram Matrices and mixup parameters are parameterized and trained with backpropagation?\"\n\n**A deep kernel machine (DKM) is not a neural network with a kernel component shoehorned in somewhere; a deep kernel machine never uses neural network features or weights anywhere in the architecture.**\nInstead, it works only with kernels, $\\mathbf K$, or Gram matrices, $\\mathbf G$ (which are very similar to kernels, they just have a slightly different use in the algorithm).\n\nTo see quickly why a DKM is a kernel method and not a neural network, notice that the core \"prediction\" steps in Algorithm 1, just below the text \"Predict train/test components of Gram matrix conditioned on inducing component\" are\n\n$$\\mathbf G_\\text{ti} = \\mathbf K_\\text{ti} \\mathbf K_\\text{ii}^{-1} \\mathbf G_{\\text{ii}}$$\n\n$$\\mathbf G_\\text{tt} = \\mathbf K_\\text{ti} \\mathbf K_\\text{ii}^{-1} \\mathbf G_\\text{ii} \\mathbf K_\\text{ii}^{-1} \\mathbf K_\\text{ti}^T + \\mathbf K_\\text{tt} - \\mathbf K_\\text{ti} \\mathbf K_\\text{ii}^{-1} \\mathbf K_\\text{it}.$$\n\nThese steps use the similarity of datapoints (as given by the kernel $\\mathbf K$) to make predictions on the unseen datapoints, which is precisely what kernel methods like kernel ridge regression and Gaussian processes do. In other words, this is a function-space model. This is very different from neural networks / weight-space models, where we simply multiply our features by a set of weights.\n\nThe confusion potentially lies in the fact that this is a kernel method with a **learned kernel**. Since the Gram matrices (along with other parameters of the variational approximate posterior) are learnable parameters, and we optimise them according to an objective function, it is convenient to use backpropagation for this purpose. Backpropagation is often associated with neural networks, but it is in fact used in a wide assortment of machine learning / statistical models. Also note that some other \"learned kernel\" approaches such as Deep Kernel Learning [4] actually use a neural network to map features before passing them to a traditional kernel function. Deep kernel machines don't use neural networks at any point, as discussed above. This therefore makes our empirical results quite significant.\n\n\nReferences:\n\n[1] Keskar, N.S. and Socher, R., 2017. Improving generalization performance by switching from adam to sgd. arXiv preprint arXiv:1712.07628.\n\n[2] Yang, A. X., Robeyns, M., Milsom, E., Anson, B., Schoots, N., and Aitchison, L. A theory of representation learning gives a deep generalisation of kernel methods. ICML, 2023.\n\n[3] Milsom, E., Anson, B., and Aitchison, L. Convolutional deep kernel machines, 2024, International Conference on Learning Representations (ICLR).\n\n[4] Wilson, A. G., Hu, Z., Salakhutdinov, R., and Xing, E. P. Deep kernel learning. In Artificial intelligence and statistics, pp. 370–378. PMLR, 2016.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their helpful comments. All reviewers recognised that our work presents state-of-the-art results for kernel methods on CIFAR-10. 

> "the improvement in generalization with respect to previous works is significant and certainly contributes to closing a gap between kernel-based models and neural networks" - Reviewer V8UE

> "Compared to previous kernel methods, the proposed method achieves better results in CIFAR-10 test accuracy." - Reviewer 44ig

> "The proposed modification greatly improves the test accuracy of DKMs on the CIFAR-10, from 92.7\% to 94.5\%." - Reviewer cybd

> "They achieve 94.52\% test accuracy on CIFAR-10, which is a significant improvement over the previous best kernel method result of 92.69\% and comes close to matching comparable neural network performance" - YneB

This was made possible through our novel methods to the CDKM model:

> "Introducing stochastic kernel regularization is an innovative way to reduce overfitting in DKM." - Reviewer cybd

> "the introduction of a stochastic regularization procedure for those kinds of kernel-based models seems like an interesting contribution" - Reviewer V8UE

> "their improvements in numerical stability allow for the use of TF32 arithmetic, making training about 5 times faster than previous implementations" - Reviewer YneB

> "They introduce a novel regularization technique where the learned inducing Gram matrices are randomly sampled from a Wishart distribution during training." - Reviewer YneB


Some reviewers would have liked to see more empirical evaluation of our methods. To address this, we managed to secure enough compute to evaluate our method on an additional dataset, CIFAR-100. With our new techniques, CDKMs achieve 75.31\% test accuracy on CIFAR-100, which represents a \~3\% improvement over the previous best result from the DKM literature, which was 72.05\% [1].

We have also made some minor changes to our working draft in order to improve readability, following the helpful suggestions of the reviewers, but no serious issues about presentation were raised:

> "The paper is well-written, clearly introduces the ''convolutional deep kernel machine'' model it is studying, and clearly reports its methodology and its result." - Reviewer V8UE

References:

[1] Milsom, E., Anson, B., and Aitchison, L. Convolutional deep kernel machines, 2024, International Conference on Learning Representations (ICLR).

Meta-review: Deep Kernel Machines are an interesting theoretical tools: this paper aims at boosting the performance of those theoretical models, by introducing several regularization (notably, noise)

The main concern of the reviewers is that this paper should have considered also challenging datasets like imagenet. Yet,  given the significant improvement on cifar10, both from the accuracy perspective and speed, I recommend accepting this paper.
Decision: Accept (poster)