Model-Efficient Deep Learning with Kernelized Classification
Authors: Sadeep Jayasumana
Srikumar Ramalingam
Sanjiv Kumar
Venue: ICLR 2022 Submitted (2022)
Keywords: deep networks
kernels on the sphere
nonlinear classification

Abstract: We investigate the possibility of using the embeddings produced by a lightweight network more effectively with a nonlinear classification layer. Although conventional deep networks use an abundance of nonlinearity for representation (embedding) learning, they almost universally use a linear classifier on the learned embeddings. This is suboptimal since better nonlinear classifiers could exist in the same embedding vector space. We advocate a nonlinear kernelized classification layer for deep networks to tackle this problem. We theoretically show that our classification layer optimizes over all possible kernel functions on the space of embeddings to learn an optimal nonlinear classifier. We then demonstrate the usefulness of this layer in learning more model-efficient classifiers in a number of computer vision and natural language processing tasks.

Under review as a conference paper at ICLR 2022
MODEL-EFFICIENT DEEP LEARNING WITH KERNEL- IZED CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
We investigate the possibility of using the embeddings produced by lightweight network more effectively with a nonlinear classification layer. Although conven- tional deep networks use an abundance of nonlinearity for representation (embed- ding) learning. they almost universally use linear classifier on the learned embed- dings. This could suboptimal for network with a limited-capacity backbone since better nonlinear classifiers could exist in the same embedding vector space. We advocate a nonlinear kerneiized classification layer for deep networks to tackle this problem. We theoretically show that our classification layer optimizes over all possible radial kernel functions on the space of embeddings to learn an optimal nonlinear classifier. We then demonstrate the usefulness of this layer in learn- ing more model-efficient classifiers a number of computer vision and natural language processing tasks.
INTRODUCTION
traditional classification deep network consists of two parts: a representation learner that maps an input to a vector-valued representation called the embedding, and classifier that classifies this embedding into the correct class. For example, in the text classification setting, the input text may be sent through a transformer encoder network with CLS CLS-pooling (the representation learner) to obtain an embedding vector for the text (Devlin et al. (018)). A fully-connected layer (the classifier) usual fully connected layer with the
then operated on this embedding. A classifier learned softmax loss is linear in the space of embeddings. Therefore, the representation learner has to learn sinearly-separable embeddings to well the classification task.
When the representation learner backbone is unable to learn linearly separable embeddings to satis- factorily solve given classification task, the usual fix is to use a deeper and/or wider backbone to generate better embeddings (Ture al. 2019; He et al. 2016). However, bigger backbones demand more resources in terms of compute, memory. and model storage during both training and infer- ence. Therefore, it is natural to ask whether it is instead possible capacity-limited network more effectively by using a more sophisticated classification layer.
embeddings produced by
In this work we address this problem by proposing nonlinear, kernelized classification layer. This classification layer finds an optimal nonlinear classifier for the embeddings by mapping them into a Reproducing Kernel Hilbert Space (RKHS) that optimally separates them into different classes. We borrow the key idea from kernel methods in the classic machine learning literature (Cortes & Vapnik, 1995 Schölkopf & Smola 2002): instead of running classifier directly on the embeddings, they are first mapped to much higher dimensional vectors in an RKHS using a positive definite kernel function. A linear classifier then run on this high-dimensional RKHS. Since the dimensionality of the embeddings is dramatically increased via this mapping, a linear classifier in the RKHS cor- responds to a powerful nonlinear classifier in the original embedding space. Such a classifier can therefore utilize even linearly inseparable embeddings satisfactorily solve classification task as demonstrated in Figure [
A common issue with traditional kernel methods is the choice of the kernel function used to obtain the RKHS mapping. While a collection of well-known kernels such as the linear kernel, polynomial kernels, and the Gaussian RBF kernel is available, it is often unclear which kernel would work best for a given problem. We tackle this issue by sweeping over all possible kernel functions within the deep network itself to find the optimal one via backpropagation and stochastic gradient descent.

---

Under review as a conference paper at ICLR 2022
(a) Training data
(b) Softmax classifier (b) Kernelized classifier
Figure I: Benefits of kernelized classification. the second and third images, regions identified by each classifier are shown in blue and orange colors. Note that the usual softmax classifier can only separate cap-like "linear" regions on the sphere, whereas our kernelized classifier can do more complex nonlinear classification thanks to the higher dimensional RKHS embedding of the sphere. See 6.. for the experiment details.
There is wealth of literature making deep learning models more efficient by using techniques such as pruning, quantization, low-rank factorization, and distillation (Cheng et al. 2017: Deng et al., (020). We approach the model efficiency from an orthogonal angle by asking whether it is possible to utilize embeddings produced by a given representation learner more effectively by doing nonlinear classification. Therefore, our approach is complementary to these existing model compression techniques.
In summary, we propose a method for model-efficient classification in a deep network with three contributions: (i) we introduce a kernelized classification layer with built-in kernel learning that can utilize a given representation learning model more effectively. (ii) we theoretically establish that is possible to optimize, within the deep network itself, over all possible kernel functions we care about to find the best RKHS that optimally separates embeddings. (iii) we empirically show that the kernelized classification layer is a viable alternative to using a larger backbone to improve the classification accuracy in a number of computer vision and natural language processing tasks.
2 RELATED WORK
There have been several explorations of loss functions other than the usual softmax crossentropy loss in CNN settings, specially in the open-set classification setting (Wen et al. 2019) Cevikalp & Saglamlar 2021 Deng al. 2019: Wang et al.) 2018). An example the Center Loss (Wen al.) 2019). which encourages low intra-class variance in the feature vectors. Other methods such as Deng al. (2019) achieve higher performance by leaving additional margins in the softmax loss. Our work differs from these since we work in the closed-set classification setting and employ an automatically learned kernel to obtain nonlinear classification boundaries.
Second order pooling methods (Lin et al. 2015 Li et al. 2017 Wang et al. 2020) propose way to perform nonlinear classification in the embedding space. has been shown that second order pooling is equivalent to using second-degree polynomial kernel (Gao et al. 2016 Cai al. 2017). Cui et al. (0177) extended second-order pooling to higher orders while learning the coefficients of higher order interactions. Their method requires explicit calculation of feature maps, which they tackle using Fast Fourier Transforms. Mahmoudi et al. (2020), use the kernel trick in the dense layer along with the polynomial kernel. Our work differs from the above works in that never calculate explicit feature maps and we theoretically show that our method learns over the space of all possible positive definite kernels on the hyper-sphere, which includes all polynomial kernels.
Some methods focus on extending the linear convolution in CNNs to a nonlinear operation. Con- volutional kernel networks (Mairal al. 2014) provide kernel approximation lutions. Zoumpourlis et al. (2017) used Volterra series approximations to extend convolutions to a nonlinear operation. WWang et al. (2019b) proposed a kernelized version of the convolution opera- tion and demonstrated that it can learn more complicated features than the usual convolution. Our work differs from theirs in a number of ways: Some kernels used in their work, such as the T.P-norm kernels, are not positive definite (Berg et al. 1984) and therefore do not represent a valid RKHS map- ping (Aronszajn) 1950). In contrast, we strictly work with positive definite kernels, which represent valid mappings to RKHS. Furthermore, learning of hyperparameters of pre-defined kernels advo- cated in their work is principally different from the kernel learning method presented in this paper- theoretically show that our method optimizes over the space of all radial positive definite kernels on the unit sphere to find the best kernel, instead of limiting the optimization to the hyperparameters of pre-defined kernels.
interpret convo-
2

---

Under review as a conference paper at ICLR 2022
Prior to the dominance of deep learning methods, picking the right kernel for given problem has been studied extensively in works such as Howley & Madden (2005); Ali & Smith-Miles (2006);
Jayasumana et al. (2014); Gönen et al. (2011). In particular, Multiple Kernel Learning (MKL) approaches (Gönen et al. fortunately, these methods scale poorly with the size of the training dataset. In this work, lomatically learn the kernel within a deep network. This not only allows autornatc representation learning, but also scales well for large training sets. Kernels have also been considered for deep learning to reduce the memory footprint of CNNs. This was accomplished by achieving an end- to-end training of a Fastfood kernel layer (Yang et al. (013). which uses approximations of kernel functions using Fastfood transforms (Le et al. 2013). Other related methods involving both kernels and deep learning include stochastic kernel machines (Dai et al. (0033), deep SimNets (Cohen al.) 2013), scalable deep kernels (Wilson et al., 2015). KerNET (L et al. 2020), and deep belief network based work Le et al. (2016).
Varma & Ray, 2007) were popular in conjunction with SVM. Un-
3 NONLINEAR SOFTMAX CLASSIFICATION
In this section we discuss the usual classification inside a deep network and its nonlinear extension. Let us consider a classification problem with a training set {( 1:), where each x; E X, each w E [Z] = 21,2,.... L,, X' is nonempty set, L is the number of labels, and N is the number of training examples. For instance, each training datum (Iz, 31) can be an image with its class label.
A deep neural network that solves this task has two components: a representation learner and classifier. In the case of image classification, the representation learner consists of modules such as convolution layers, max-pooling layers, and fully-connected layers. The classifier is the last fully- connected layer operating on the learned representations (embeddings). This layer is endowed with a loss function during training.
Let (0) X Rdd denote the representation learner, where d is the dimensionality of the embed- dings and θ represents all the parameters in this part of the network. The classifier is characterized by a function g( : Rd + [L]. where f denotes all the parameters in the last layer of the net- work. Usually, Q consists of weight vectors W1, w....., .., WL with each w Rd, and bias terms b,, br with each by E R. The functtion (0 then takes the form:
by
g (00 (z) = argmax w/
where z = (((0 (z)) Rd is the embedding for input the x. Note that we have dropped the additive bias term by. with no loss of generality, to keep the notation uncluttered. During inference, the deep network's class prediction ij" for an input x² the composite of these two functions: ÿ (g(50) (()) (x*).
Although conceptually there are two components of the deep network, their parameters O and Ω learned jointly during training. The de facto standard way of training a classification network is minimizing the sofimax loss applied to the classification layer. The softmax loss is the combination of the softmax function and the cross-entropy loss. More specifically, for single training example (ar, y) with the embedding z = (x), the softmax loss calculated as:
exp(w (apww22)
I(y z) = - log E exp(w
(2)
Note that the classifier g (00) trained is this manner completely linear in IRd, the space of the embeddings 2s, as is evident from
From the classic knowledge in kernel methods, we are aware that more powerful nonlinear classifiers on dd can be obtained using the kernel trick. The key idea here is to first map the embeddings zs into a high-dimensional RKHS H and perform classification there. Although the classification is linear in the nigh-dimensional H. it is nonlinear in the original embedding space Rd Let ø Rd + H. represent this RKHS embedding. Performing classification in H. is then equivalent to training the neural network with the following modified version of the softmax loss:
exp ((d(wy); (22)) Ln 2) = -log
(3)
ΣJ exp ((0(w;), 0(())) ((o(w,))((()))))
3

---

Under review as a conference paper at ICLR 2022
where (. denotes the inner product in the Hilbert space H. The key difference between (2) and (3) is that the dot products between w,s and z have been replaced with the inner products between D(w,)s and o(z). The more general notion of inner product is used instead of dot product because the Hilbert space 't can be infinite dimensional.
For a network trained with this nonlinear softmax function, predictions can be obtained using modified version of the predictor:
Jal (z) argmax (Ø(w;), ((2))
(
Note that the Hilbert space embeddings (()s can be very-high, even infinite, dimensional. There- fore, computing and storing them can be problematic. We can use the kernel trick from classic machine learning (Schölkopf & Smola 2002; Shawe-Taylor & Cristianini 2004) to overcome this
problem: explicit computation of ((.)s can be avoided by directly evaluating the inner product be- tween them using a kernel function k: TRdd x IR + RR. That is:
(o(w): (z)) H = k(w,
(5)
Intuitively, mapping d-dimensional embeddings into a much higher dimensional RKHS using kernel would help in finding complex, nonlinear patterns in the embeddings that the usual softmax classification is unable to find due its linear nature. We therefore expect kernelized classification to use embeddings provided by a given representation learner more effectively.
4 KERNELS ON THE UNIT SPHERE
was shown in the previous section that, given a kernel function on the embedding space, we can obtain nonlinear classifier in the last layer of deep network by modifying the softmax loss function during training and the predictor during inference. However, only positive definite kernels allow this trick (Aronszajn. 1950 Berg et al. 1984). There are various choices for kernel functions in the classic machine learning literature. Some popular choices include the polynomial kernel, the Gaussian RBF kernel (squared exponential kernel), and the Laplacian kernel. Nevertheless, it is often difficult to decide the optimal kernel for a given problem. Furthermore, many of the kernels have hyperparameters that need to be manually tuned. The generally accepted solution to this problem in classic kernel methods is the MKL framework (Gönen et al. (211). where a better kernel is learned as a linear combination of some pre-defined kernels. Unfortunately, like SVM, MKL. methods do not scale well with the train set size. Furthermore, usual MKL provides no guarantee to explore all possible kernel functions to find the optimal one.
In this section, we present some theoretical results that will pave the way to define neural network layer that can automatically learn the optimal kernel from data. By formulating kernel learning as a neural network layer, we inherit the desirable properties of deep learning, including scalability and automatic representation learning. Importantly, we show that our method can sweep over the entire space of positive definite kernels applicable to our problem setting to find the best kernel. We start the discussion with the following definition of positive definite kernels (Berg et al. 1984).
Definition 4.1. Let U be a nonempty set. A function ls (u U) + R called a positive definite kernel if k(u, v) = k((e, u) for all u, v E U and Σ/ EN Ge,k(u,, uz) 2 O, for all N e N, (ui,..., 3 C li and (CI....,cN' C IR.
Properties of positive definite kernels have been studied extensively in mathematics (Begg et al., 1984). The following summarizes some important closure properties of this class of functions. Proposition 4.2. The family of all positive definite kernels a given nonempty set forms a convex cone that is closed under pointwise multiplication and pointwise convergence.
Proof. To intuitively understand this result, it is helpful to recall that the geometry of the family of the all positive definite kernels on given nonempty set is closely related to that of the space of the d x d symmetric positive definite matrices, which forms convex cone. The formal proof of this proposition can be found in Remark 3.1.1I and Theorem 3.1.12 of Berg et a (1984).
To simplify the problem setting, we assume that both the embeddings zs and the weight vectors wys are n.2-normaiized Not only this simplifies the mathematics, but also it is practice in use
4

---

Under review as a conference paper at ICLR 2022
for stabilizing the training of neural networks (Yi et al. 2019 Liu et al. 2017 Hoffer et al. 2018 Due to this assumption, we are interested in positive definite kernels on the unit sphere in IR" From now on, we radial kernels on S". Radial kernels, kernels that only depend on the distance between the two input points, have the desirable property of translation invariance. Furthermore, all the commonly used kernels on S", such as the linear kernel, the polynomial kernel, the Gaussian RBF kernel, and the Laplacian kernel are radial kernels. The following theorem, origins of which can be traced back to Schoenberg (1942). fully characterizes radial kemels on S"
S", where
d - to denote this space. We also restrict our discussion to
Theorem 4.3. A radial kernel k S 5 - TR is positive definite for any n if and only if admits unique series representation of the form
k(u.v)= = a-2l(u, v) E [[1,11]++ a-1([(u,
v) =-]]+
Σ m=0
am (u. v) m
(6)
where each Om 2 0, _m==2 Om < 00, and [-] depicts the Iversion bracket.
Proof. The kernel h1 S x 8 + |-1,1] (u (u, v) is positive definite on 8 for any n since Ej Ei GG3 (u,, u₁) = Σ, Gu; 12 2200 0. Therefore, from the closure properties in Propo- sition 4.2 the kernel km (u. + (u, v) " is also positive definite for any m E N. Furthermore, km is positive definite for m = 0 since E, Σ, Gi2y (u: n = 11, O.
Let us now consider the two sequences of kernels Sodd k1, k......... k2m+1 and Seven k2, ka..... kmm..... Since -1 S (u.v) S 1, is clear that Sodd and Seven converge pointwise to koda v) = [(u, = 1] [u,, v) = -1] and lievon (u, = [(u, {-1, respec- tively. From the last closure property of Proposition 4.2 both kodd and keven are positive definite on S"h Invoking Proposition 4.2 again, we conclude that any finite conic combination of the kernels keven, kodd, ku, ki.... is positive definite on S" for any n. This completes the forward direction of the proof. The proof of the converse is found in Chapter 5 of Berg al. (1984).
Equipped with a complete characterization of the positive definite radial kernels on S"h we now dis- cuss how we can combine this result with the nonlinear softmax formulation in 3 to automatically learn the best kernel classifier within deep network.
5 THE KERNELIZED CLASSIFICATION LAYER
We now introduce a kernelized classification layer that acts as drop-in replacement for the usual softmax classification layer in a deep network. This new layer classifies embeddings in a high- dimensional RKHS while automatically choosing the optimal positive definite kernel that enables the mapping into the RKHS. As a result, we do not have to hand-pick a kernel or its hyperparameters. Our nonlinear classification layer utilize embeddings more efficiently than softmax classifier, which is linear in the embedding space. We also show that the kernelized classification layers comes with negligible added cost during both training and inference.
5.1 MECHANICS OF THE LAYER
Our classification layer is parameterized by the usual weight vectors: WI, W2,..., WT. and some additional learnable coefficients: a-2. a....... R.M. where M E N and each am 2 0. During training. this classifier maps embeddings zs into a high- nigh-dimensional RKHS Hopt that optimally separates embeddings belonging to different classes, and learns linear classifier in Hape- During inference, the classifier maps embeddings of previously unseen inputs to the RKHS it learned during training and performs classification in that space. This is achieved by using the nonlinear softmax loss defined in (3) during training and the nonlinear predictor defined in (4) during testing, with the inner product in H given (ow(), 6(2)) = (o(w), d(z)) Rapa kope (w, z), where kope(
is the reproducing kernel of Hopt- The optimal RKHS Hape for given classification problem is learned by finding the optimal kernel hipp during training as discussed in the following.
Theorem 4.3 states that any positive definite radial kernel on S" admits the series representation shown in 66) Therefore, the optimal kernel hopt must also have such series representation. We approximate this series with a finite summation by cutting off the terms beyond the order M:
Kope. (W, z) 2 a 2kiveen (w, z) + a 1 (w,22 2) +
al kink (0mm(w: (w, z).
(7)
5

---

Under review as a conference paper at ICLR 2022
where, Keven. Rodd, Ro, .i....,,........................ KAs have meanings defined and a-2, a-1,...... ................................ O.M 2 0. Using Proposition 4.2 and the discussion in the proof of Theorem 4.3 one can easily verify that this ap- proximation does not violate the positive definiteness of kopt. A rigorous analysis of the accuracy of this approximation is provided in Appendix F.
With this, kopt is learned automatically from data by making the coefficients at- 01-1--.----- O learnable parameters of the classification layer. Let a [a-22, Q-11...... aMr]". The gradient of the loss function with respect to a can be calculated via the backpropagation algorithm using (3) and 7 Therefore, it can be optimized along with W1, W w.. during the gradient descent based optimization of the network. This procedure is equivalent to automatically finding the RKHS that optimally separates the embeddings belonging to different classes.
The constraint 2--2, a-1...., QM 2 0 in (7) can be imposed with a ReLU function (see Ap- pendix D.5 for more discussion). As shown later & 7.3 the exact value of M not critical as long as it is sufficiently large. This is because, as discussed in the proof of Theorem 4.3 the higher order terms that are truncated approach either kodd or kiveen both of which are already included in the finite summation. On the other hand, if the terms beyond some order M' < M are not important, the network can automatically learn to make the corresponding a coefficients vanish. We observed that M = 10 works well enough in practice and stick to this number in all our experiments.
Importantly, the kernelized classification layer described above can pass on the loss gradients to its inputs. Therefore, the kernelized classification layer is fully compatible with end-to-end training and can act as a drop-in replacement for an existing softmax classification layer.
5.2 ADDITIONAL COMPLEXITY
Since we propose kernelized classification as a replacement for the usual softmax classification to improve model efficiency, one might wonder about the added cost of the kernelized classification layer. It uses (M + 3) extra learnable parameters. During both training and inference, the added computational complexity is O(M d) per datum, assuming a commonly-available constant-time operation for taking powers. Additional memory footprint is (M) during training to account for cached gradients. Note that M = 10 and d ranges from 64 to 768 in our experiments. Therefore, the kernelized classification comes with negligible added cost over the softmax classification in terms of compute, memory, and trained model storage.
EXPERIMENTS
We now present experimental evaluation of our method. For all experiments, the main baseline is the standard softmax classifier. Where appropriate, we show three additional baseline results based on the linear kernel, second order pooling (Lin et al. (0155), and kervolutional networks (Wang et al. 2019b). Note that the focus of our experiments is to demonstrate the benefits of kernelized classification in efficiently utilizing embeddings learned with various representation learners, not to claim state-of-the-ee art results on already well-explored benchmark datasets. Details about our experimental setup is in Appendix A
6.1 SYNTHETIC DATA
We first evaluate the kernelized classification layer as an isolated unit by demonstrating its capabil- ities to learn nonlinear patterns on St. We train a softmax classifier and our kernelized classifier on the synthetic dataset described in Appendix B Results on the test set are shown in Table [ We also report the theoretical maximum accuracy, the accuracy of the Bayes optimal classifier. The accuracy of our kernelized classification layer significantly outperforms the baseline and gets close to theoretical best. This can be attributed to the layer's 's capabilities learn nonlinear patterns on the sphere by embedding the data into an RKHS that optimally separates the classes.
We visualize the outcomes of the classifiers in Figure [ separate like regions on S2, this is result of its being a linear classifier with respect to the em- beddings. Our kernelized classifier, on the other hand, can do a more complex nonlinear separation of the embeddings.
Note that the softmax classifier can only

---

Under review as a conference paper at ICLR 2022
Method
Accuracy
Accuracy
Dataset
Softmax classifier (baseline)
85.51
Baseline
Ours
Kernelized classifier (ours)
94.20
CIFAR- -IAAR-IO
76.06
79.85
Bayes optimal classifier
95.06
CIFAR 100
44.38
46.48
Table I: Classification results on the synthetic dataset.
Table 2: Results in the distillation setting.
6.2 IMAGE CLASSIFICATION
We now report results CIFAR- 10 and CIFAR- 100 real world image benchmark Krizhevsky
2009). To demonstrate better model-efficiency with the kernelized classification, we experiment with several CIFAR- architectures (He al., 2016) with increasing model capacity. We consider four different baselines: (1) Softmax: the standard softmax loss, (2) LIN: normalized embeddings and weights with only the linear kernel along with learnable coefficient. This is similar to the approach discussed in Hoffer et al. (2018), but with additional freedom to learn the weight vectors, (3) SOP: second order pooling (Lin et al. 2015), which is also equivalent to Mahmoudi ct al. (2200) with a second degree polynomial, and (4) KERVO: kervolutional networks (Wang
et al. 2019b) with the best out of the Gaussian RBF kernel and the polynomial kernel.
Increased model-efficiency obtained with the kemeliied classifier is evident from the accuracies summarized in Table For example, the same accuracy of ResNet-56 (540K parameters) with the standard softmax classifier can be obtained with much smaller ResNet-32 (300K parameters) when the kernelized classifier is used. Our method significantly outperforms the other baselines as well. This shows the benefits of optimizing over the entire space of positive definite kernels instead of restricting ourselves to linear methods or pre-defined kernels.
Backbone
# params
Softmax
LIN
SOP
KERVO
Ours
ResNet-8
61K
83.73 53.82
82.45 54.00
84.03 55.80
85.15 / 56.92
86.93 58.27
ResNet-14
121K
89.87 63.85
90.16/63.66
90.47 / 63.53
90.43 / 64.14
91.48 / 66.67
ResNet-20
181K
91.14/65.99
91.01 /65.79
91.75 67.97
91.34 /7.31
92.88 / 69.33
ResNet-32
300K
92.22/68.96 68.96
92.21/69.
92.31 / 70.39
92.42 / 69.40
93.70 / 71.30
ResNet-44
420K
92.10/ 70.16
93.10/ 70.54
92.42 / 71.13
92.88 771.09
94.05 / 73.20
ResNet-56
S40K
93.01 71.23
93.13 / 72.11
93.33 / 73.12
93.10 / 72.39
94.15 / 74.10
Table Results on the 1AAAR-O0CIIFA-IOO datasets.
6.3 NATURAL LANGUAGE UNDERSTANDING
In this section, we show the benefits of the kernelized classification layer in solving four different text classification tasks in the GLUE benchmark (Wang et al. 2019a). We use mask-LM pretrained BERT models of different capacities (Ture et al.] (019)) and finetune them on each classification task. Note that we do not use distillation and directly finetune the models with the dataset labels. Since detailed analyses GLUE test datasets not allowed (Wang et al. 2019a). we tune hyperparam- eters on subsets of train sets and report results on the validation sets in Table More details about the experiment setup are provided in Appendix A
Table provides evidence that kernelized classification layers helps in extracting more gains out of given representation learner. For example, across all the datasets, BERT-Mini (11.3M parameters) with kernelized classification can get similar results as the BERT-Small (29. I M parameters) with softmax classification. Therefore, using the nonlinear, kernelized classifier an effective alternative to using bigger backbone for increasing classification performance.
6.4 KNOWLEDGE DISTILLATION
We now evaluate our method in the distillation setting to show that it is complementary to existing model compression techniques. We used the CIFAR-10 10 and CIFAR-100 datasets, the softmax CI- FAR ResNet-56 models from TablesBaas as the teacher models, and the LeNet-5 network (Lecun et al., utilize the kernelized classification layer in the distillation setting described in Hinton et al. (2015) by replacing usual logits with their
1998) the student model. Note that it is straightforward
We use the standard data augmentation in CIFAR-10/100 (He al. 2016). Some authors refer these datasets CIFAR-10+H 10+/100+ when data augmentation is used. We omit the keep the text uncluttered.

---

Under review as a conference paper at ICLR 2022
Dataset
Method
BERI-Timy
BERT-Mini
BERI-Smal
BERI-Medium
BERT-Bas
params
4.4M
11.3M
29.1M
41.7M
110.1M
Softmax
82.77/74.50
05,20/79.22
88,41 / 83.75
89.72 85.50
92.58 / 89.50
LIN
82.00/73.75
84.99/79.18
88.51 83.79
89.88
85.62
92.34 89.48
MPRC (F//Acc)
SOP
83,81 77.21
89.21 80.42
84.38
89.99 85.78
92.68 89.73
Ours
/
88.89 / 84.50
90.12 85.75
91.60 87.75
93.24 / 90.50
Softmax
$1.1786.22
84.09 88.10
85.18/ / 88.87
86.12 89.73
87.18 / 90.30
LIN.
81.07 86.11
84.05 88.09
85. 88.87
86.05 89.69
87 $7.20 / 90.31
QQP (Fi/Acc)
SOP
82.01/87.00
84,50 88,56
85,99 90.00
86.15 89.99
87.21 90.34
Ours
82.78 87.13
85.00
88.84
85.83 89.46
86.83 90.17
87.97 / 91.04
Softmax
63.53
65.74
66.01
66.84
71.46
LIN
62.32
65.69
65.97
66.02
71.50
RTB (Acc)
SOP
63.59
65.78
68.02
72.96
Ours.
65.96
66.93
67.72
67.94
73.28
Softmax
82.38
87.11
87.19
89.86
91.97
LIN.
82.04
86.92
87.29
89.50
91.72
SST-2 (Acc)
SOP
82.89
87.19
88.52
90.12
91.98
Ours
84.07
87.14
89.42
90.83
92.69
Table 4: Results on several natural language understanding tasks in the GLUE benchmark.
kernelized counterparts. We use the cross-entropy loss with the teacher scores with the temperature set to 20 in all cases. Results are shown in T Significant gains are observed with the kemel-- ized classification layer. This can be attributed the layer's capabilities to approximate complex teacher probabilities even with weak embeddings due to the nonlinear classifier.
6.5 ACTIVE LEARNING
Active learning focuses on reducing human annotation costs by selecting a subset of images to label that are more likely to yield the best model (Settles (2099). We used different sampling methods such as random, margin (Lewis & Gale 1994 Scheffer et al. 2001), and k-center (Sener & Savarese 2017 Wolf 2 1) to generate subsets of various sizes. The setup is detailed in Appendix C As shown in Table 5 our results on random subsets outperform the softmax ResNet-56 models on margin and k-center based subsets, and we achieve even better results using improved sampling methods. This demonstrates that the kernelized classification layer can produce better models in limited-dail settings as well.
ABLATION STUDIES
We now present ablation experiments using CIFAR-100 and the ResNet-56 backbone.
7.1 KERNEL LEARNING
To investigate the benefits of automatic kernel learning compared using pre-defined kernel in the kernelized classification layer, we compare our kernel learning method with two pre-defined kernels in the kemeliied classification layer: the polynomial kernel of order 10 and the Gaussian RBF kernel. We also show results with an MKL baseline with linear, 2nd order, and Gaussian
Baseline
Ours
%
rDd
mgn
k-etr
rnd
mgn
k-ctr
74
30
58.03
58,88
58.41
61.66
61.80
63.08
40
61.05
61.81
62.02
65.25
66.35
50
64.81
65.36
65.47
67.17
68.14 14
69.41
60
66,26
67.03
68.25
69.17
70.61
70.10
71
70
67.47
69.16
69.84
70.06
70.90
71.50
70
80
69.59
69.47
71.25
71.66
72.21
72.64
12
90
70.25
71.41
71.11
72.60
73.90
73.14
Order the Approximation
Terms md, mgn, and k-ctr refer to random, margin, and k-center, respectively.
Table 5:
Active learning CIFAR- 100.
Figure 2: Accuracy versus the order of the approximation (M) CIFAR-100.
8

---

Under review as a conference paper at ICLR 2022
kernels. Results are shown in Table of Theorem 4.3 to find the best kernel yields significant practical benefits. This is not surprising because the polynomial and Gaussian kernels are members of the large search space of kemells that we optimize over in our method.
6 It is evident that sweeping all possible kernels in light
Method
Acc
Method
Acc
Gaussian RBF with fixed a
73.21
No normalization
71.16
Gaussian RBF with learned o
73.23
Embeddings normalized
71.23
Polynomial kernel
73.16
Embeddings & weights normalized with fixed 'T'
71.19
MKL Ours
73.25
Embeddings weights normalized with learned
72.11
74.10
Ours
74.10
Table Benefits of learning the best kernel.
Table 7: Effects of normalization.
7.2 EMBEDDING NORMALIZATION
As discussed previously, we L2 n'-oomlali both embeddings and weights in the classification layer. We study the effect of this normalization for the baseline softmax loss in Table 7 Note the setting where both emheddings and weights are normalized with an appropriate temperature is equivalent to the cosine softmax loss (Liu et al., 2017 Chen al. 2019 Wang et al., 2020). Note also that the LIN baseline considered in Tables 3 and uses normalized embeddings and weights with an auto- matically learned T, same as the fourth row in Table 7 In our softmax baselines we use normalized
embeddings since it works consistently better than the unnormalized version.
7.3 SENSITIVITY TO THE ORDER OF THE APPROXIMATION
As discussed 5.1 intuitively, the order M of the approximation should not matter as long as is large enough. We verify this in Figure 2 with the CIFAR- 100 dataset, where show the changes in accuracy with increasing M. We use 10 in all our experiments.
7.4 DO MORE TULLY-CONNECHE LAYERS HELP?
One could wonder whether more fully-connected layers at the end of the network would improve the classification. To address this, we added an additional fully-connected layer with d units to ResNet-56. The accuracy improved only marginally from 71.23 to 71.29, opposed 74.10 with our method. This observation is in-line with the discussion in Rendle et al. (2020): MLP scorers are somewhat difficult train. An explanation for this could be the common observation that, although MLPs can theoretically approximate any function, learning one from data is difficult. This has motivated inductive-bias based models such as CNNs and Transformers. Kernelized classification can also be viewed as a way of presenting an inductive bias motivated by the classic kernel method theory. Note also that, unlike the kernelized classification layer, added MLP layers come with a significant increase in the model's computational and memory complexity.
7.5 DIFFERENT LOSS FUNCTIONS, ACTIVATION FOR CORFFICIENT, FTC.
We also provide a number of other ablations studies on different loss functions, effects of embedding rectification, kernel coefficient activation, and other settings in Appendix D
CONCLUSION
We presented a kernelized classification layer for deep neural networks aiming to extract the best possible classifier with embeddings produced by a given representation learner. This classification layer classifies embeddings in a high dimensional RKHS while automatically learning the optimal kernel that enables this high-dimensional mapping. We showed that a classification network with a lightweight representation learning backbone can be made more effective by replacing the usual softmax classifier with the kernelized classifier. We showed consistent and substantial accuracy im- provements in image classification, natural language understanding, distillation, and active learning settings. These accuracy improvements strongly support the usefulness of kernelized classification layer in finding nonlinear patterns in the embeddings.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This work concerns mathematical and empirical analysis of deep learning based classification tech- niques with applications in image recognition and natural language understanding. We do not fore- see our work having undue societal effects. Our work does not explicitly consider issues of fairness in classification, which an important yet under-studied dimension. We do not foresee our tech- niques as unduly amplifying biases in existing algorithms.
REFERENCES
Shawkat Ali and Kate A. Smith-Miles. A meta-learning approach to automatic kernel selection for support vector machines. Neurocomputingg. 2006. Neural Networks.
Nachman Aronszajn. Theory of Reproducing Kernels. Transactions of the American Mathematical Society, 1950.
Tensorflow Authors. TensorFlow Datasets: a collection ready-to-use datasets. URL https:
7/ www.teesorffor.. http://///gataaaooeeeooeee [Online; accessed Feb-04-2021].
C. Berg, J. P. Christensen, and P. Ressel. Harmonic Analysis on Semigroups. Springer, 1984.
Cai, Zuo, and I.. Zhang. Higher-order integration of hierarchical convolutional activations for fine-grained visual categorization. In ICCV, 2017.
H. Cevikalp and H. Saglamlar. Polyhedral Conic Classifiers for Computer Vision Applications and Open Set Recognition. TPAMI, 2021.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. CoRR, abs/1904.04232, 2019. URL htttp://axxiv.oo org/
abs 2bs/1904, 04232
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. CoRR, abs/1710.09282,, 2017. URL http:://axxv...rrrgg/////////
1710.09288 09282
Nadav Cohen, Or Sharir, and Amnon Shashua. Deep simnets. CoRR, tbb//150.03359, 2015.
C. Cortes and V. Vapnik. Support Vector Networks. Machine Learning, 1995.
Y. Cui, F. Zhou, J. Wang, X. Liu, Y. Lin, and S. Belongie. Kernel Pooling for Convolutional Neural Networks. CVPR, 2017.
Bo Dai, Bo Xie, Niao He, Yingyu Liang, Anant Raj. Maria-Florina Balcan, and Le Song. Scalable kernel methods via doubly stochastic gradients, 2015.
Deng. Dong, R. Socher, L.-J. Li. Li, and L. Fei-Fei. ImageNet: Large-Scale Scale Hierarchical Image Database. In CVPR09, 2009.
J. Deng, J. Guo, N. Xue, and S. Zafeiriou. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In CVPR, 2019.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020. doi: 10. 0..1000PROC.2220.227777777
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
Mei
2018.
Y. Gao, O. Beijbom, N. Zhang, and T. Darrell. Compact Bilinear Pooling. In CVPR, 2016.
Priya Goyal, Piotr Dollár, Ross B. Girshick, P. Noordhuis, L. Wesolowski. Aapo Kyrola, Andrew Tulloch, Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet Hour. ArXiv, abs/1700 0001170060677777 2017.
10

---

Under review as a conference paper at ICLR 2022
R. Guo, P. Sun, E. Lindgren, Q. Geng, D. Simcha, F. Chern, and S. Kumar. Accelerating large-scale inference with anisotropic vector quantization. In ICML, 2020. URL htttp:://axxi...tttt abs/1908. 10396
Mehmet Gönen, Ethem Alpaydın, and Francis Bach. Multiple kernel learning algorithms. JMLR, 2011.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series Statistics. Springer New York Inc., New York, NY, USA, 2001.
K. He, Zhang. S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the Knowledge in a Neural Network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL httt::tppaaax:tt
org/ abs/ 1503. 02531
Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last weight layer. In ICLR, 2018.
Tom Howley and Michael G. Madden. The genetic kernel support vector machine: Description and evaluation. Artif. Intell. Rev., 2005.
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss cross- entropy in classification tasks. 2020.
Sadeep Jayasumana, Richard Hartley, Mathieu Salzmann, Hongdong Li, and Mehrtash Harandi. Optimizing Over Radial Kernels on Compact Manifolds. Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-graing grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), 13), Sydney, Australia, 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Ivano Lauriola, Claudio Gallicchio, and Fabio Aiolli. Enhancing deep neural networks via multiple kernel learning. Pattern Recognition, 2020.
Linh Le, Jie Hao, Ying Xie, and Jennifer Priestley. Deep kernel: Learning kemeel function from data using deep neural network. In 2016 IEEF/ACM 3rd International Conference on Big Data Computing Applications and Technologies (BDCAT), 2016.
Quoc Le, Tamás Sarlós, and Alex Smola. Fastfood: Approximating kernel expansions in loglinear time. In JCML, 2013.
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pp. 2278 2278-2324, 1998.
D. D. Lewis and W. A. Gale. A sequential algorithm for training text classifiers. In SIGIR, 1994.
Li, J. Xie, Q Wang, and W. Zuo. Second-Order Information Helpful for Large-Scale Visual Recognition? In ICCV, 2017.
Lin, A. RoyChowdhury, and S. Maji. Bilinear CNN Models for Fine-Grained Visual Recognition. In ICCV, 2015.
W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song. SphereFace: Deep Hypersphere Embedding for Face Recognition. In CVPR, 2017.
Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep Hyperspherical Learning. In Advances in Neural Information Processing Systems, 2017.
I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference Learning Representations (ICLR) 2017 Conference Track, April 2017.
11

---

Under review as a conference paper at ICLR 2022
M. A. Mahmoudi, A. Chetouani, F. Boufera, and H. Tabia. Kernelized Dense Layers For Facial Expression Recognition. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 2226-2230, 2020. doi: 10.1 ww..1//1//17777...... 10.1099/CI14007722200000999999 011.778.222009996699
Julien Mairal, Piotr Koniusz, Zaïd Harchaoui, and Cordelia Schmid. Convolutional kernel networks. CoRR, abs/ 130..3323,, 2014.
O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Steffen Rendle, Walid Krichene, Li Zhang, and John R. Anderson. Neural Collaborative Filtering vs. Matrix Factorization Revisited. Fourteenth ACM Conference on Recommender Systems, 2020.
T. Scheffer, C. Decomain, and S. Wrobel. Active hidden markov models for information extraction. In Advances in Intelligent Data Analysis, 2001.
I. J. Schoenberg. Positive Definite Functions Spheres. Duke Mathematical Journal, 1942.
Bernhard Schölkopf and Alexander J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2002.
O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach, 2017.
B. Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison, 2009.
John Shalor and Nello Cristianini. Kernel Methods for Pattern Analysis. Cambridge Univer- sity Press, 2004.
Ingo Steinwart. Sparseness of Support Vector Machines. JMLR, 2003.
Iulia Ture, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better:
On the importance of pre-training compact models. arXiv preprint arXiv: /908/v9908.... 099662,,, 2019.
Manik Varma and D. Ray. Learning the discriminative power-invanee power-invariance trade-off. In IN ICCV, 2007.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t. ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural In- formation Processing Systems, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019a. In the Proceedings of ICLR.
Chen Wang, Jianfei Yang, Lihua Xie, and Junsong Yuan. Kervolutional neural networks. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 31-40, 2019b.
H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong. J. Zhou, Z. Li, and W. Liu. CosFace: Large Margin Cosine Loss for Deep Face Recognition. In CVPR, 2018.
Q. Wang, J. Xie, W. Zuo, L. Zhang, and P. Li. Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization. TPAMI, 2020.
Xin Wang, Thomas E. Huang, Trevor Darrell, Joseph E. Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection, 2020.
Y. Wen, K. Zhang, Z. Li, and Y. Qiao. A Comprehensive Study on Center Loss for Deep Face Recognition. IJC,, 2019.
Andrew Gordon Wilson and Hannes Nickisch. Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP). In ICML, 2015.
12

---

Under review as a conference paper at ICLR 2022
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. CoRR, abs/1511.02222, 2015.
Gert W. Wolf. Facility location: Concepts, models, algorithms and case studies. series; Contribu- tions to management science. 2011.
Z. Yang. M. Moczulski, Denil, N. De Freitas, L. Song, and Z. Wang. Deep fried convnets. In 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
Xinyang Yi. Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. Sampling-Bitn Bias-Corrected Neural Modeling for Large Corpus Item Recommendations. In Proceedings of the 13th ACM Conference on Recommender Systems, New York, NY, USA, 2019. Association for Computing Machinery.
Georgios Zoumpourlis, Alexandros Doumanoglou, Nicholas Vretos, and Petros Daras. Non-linear convolution filters for enn-based learning. CoRR, abs/1708.07033. 2017.
13

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL SETUP
In all experiments, we use M 10 for the kernelized classification layer. As discussed in g 5.1 and Appendix D.3]
ReLU activation and weight decay of 0.0001 on the parameter vector. This is the same amount of weight decay used the other parts of the network, when applicable. The parameter vector is initialized with all ones.
For all image classification experiments, we use SGD with 0.9 momentum, linear learning rate warmup (Goyal al. 2017), cosine learning rate decay (Loshchilov & Hutter (017), and decide the base learning by cross validation. When a better learning rate schedule is available for the baseline (e.g. the CIFAR schedule in He et al. 2016). we experimented with both that and our schedule and report the best accuracy of the two. The maximum number of epochs was 450 in all cases, Mini- batch size was 128 for the synthetic and CIFAR datasets and 64 other datasets with larger images used in Appendix D.2 We used the standard CIFAR data augmentation method in He et al. 2016 for CIFAR-IO and CIFAR-IO0 datasets, and the Imagenet data augmentation in the same paper for other image datasets. Some ResNet models use ReLU activation on embeddings. We also remove this activation to utilize the full surface of S The same is done to the baseline models to enable a fair comparison (see AppendixD.4ffor more details).
For the natural language understanding tasks in & 6.3 we use the publicly-available MLM- pretrained BERT/SmalI-BER' models from TensorFlow Hub (Turc et al. 2019). Usual CLS pool- ing is done at the end of the Transformer encoder to obtain an embedding for each input sen- tence/sentence pair. We use the AdamW optimizer with linear ramp-up and decay as is standard for BERT model finetuning. Note that we do not distill the final task from a bigger model and di- rectly finetune with the one-hot labels in the original datasets. Hyperparameter search followed the procedure described in Ture et al. (2019). Since detailed analysis of different methods on GLUE test tests are not allowed, we tune hyperparameters on a 10% subset of the train set and report results on the validation sets. Once the hyperparamers are decided, the full train set is used to train the final model. Since GLUE validation sets are small, we report the average accuracy over different runs for each method. We do not however notice significant variance in accuracy across different runs.
B SYNTHETIC DATA GENERATION
We generated the binary classification dataset used in $6.1 using a mixture of Gaussians, inspired by the blue-orange dataset in Hastie et al. (2001). More specifically, we first generated 10 cluster centers for each class by sampling from an isorropic Gaussian distribution with covariance 0.5 / and mean [1, 0, 0]" for the blue class and [o. 0) for the orange class. We then generated 5,000 train observations for each class using the following method: for each observation, we uniform-randomly picked a cluster center of the corresponding class and then generated sample from an isotropic Gaussian distribution centered at that cluster center with covariance 0.02 T3- All the observations were projected on to S2 by L n'=morzaaiziing them. The test set was generated in the same manner using the same cluster centers as the train set.
C ACTIVE LEARNING
The goal of the active learning experiment is to show that the kernelized classification layer can produce accurate models even with less data. In order study this, we produce subsets of datasets under various budgets using several sampling techniques, and evaluate the models trained on this. The simplest one is random sampling, where images are selected randomly under a given budget. Other methods include margin (Lewis & Gale, 1994] [Shheferr et al. 2001). and k-center (Sener & Savarese 2017 Wolf 201 T) where class prediction scores and embeddings from the images are used in the subset selection.
We do not rely on the actual labels of the images in the dataset during the subset selection, since active learning is driven toward reducing label annotation costs. We used 10% random subset of the original CIFAR-100 dataset with labels to first learn an initial seed model, which was then used to generate predictions and embeddings. Note that only embeddings and class prediction scores from this initial seed model are used in subset selection, and we do not access the original class labels of the images. We batch setting where we do not incrementally update the model after selecting
14

---

Under review as a conference paper at ICLR 2022
every image, and we directly select entire subsets under given budget. In all experiments, we used the CIFAR ResNet-56 model. The learning rate, batch size and the number of epochs are provided in Appendix A The embeddings are of dimension 64. For the k-center method, need distances between the embeddings, and we used cosine distances computed using fast similarity search of (Guo al. 2020).
D ADDITIONAL EXPERIMENTS
We report number of additional experimental results in this section.
D.1 ADDITIONAL. BACKBONES
Image classification results with VGG-16 and DenseNet-40-15 12 backbones are reported in Table g Since the original VGG-16 designed for 224 x 224 images, we used modified CIFAR version with 256 dimensional hidden size at the end.
Backbone
Softmax
LIN
SOP
KER 93.21/72.1 94.92 / 75.08
Ours 94.39 / 73.32
VGG-16
92.58 1.58/71.44 71.48
71.52 92.59 /7..52
93,20/ 72.10
DenseNet
94.76/ 75.58
94.73 / 75.67
94.98 / 75,00
95.31 / 76.87
Table 8: Results on the CIFAR-IOCCIAAR 100 datasets with clifferent backbones.
D.2 IMAGE CLASSIFICATION TRANSFER LEARNING
Here, we evaluate our method in image classification transfer learning setting. To this end, take a ResNet-50 network pre- pre-trained trained on the Imagenet ILSVRC 2012 classification dataset (Deng et al. 2009) and finetune it on Oxford-IIIT Pets (Parkhi al. 2012) and Stanford Cars (Krause al. 2013 datasets. For each dataset, we use the train/test splits provided by the standard Tensorflow Datasets implementation (Authors). Results are summarized in Table Note that the KERVO baseline not possible in this setting as it involves changes to the backbone network. On both datasets, kernelized classification layer results in significant gains over the baselines. This is intuitive to understand since the embeddings learned from the source task (Imagenet) might not linearly separate the new classes the target task. We can therefore benefit from a nonlinear classifier the transfer learning setting.
Accuracy
Dataset
LIN
SOP
Ours
Oxford-IITT Pets
92.06 90.78
91.99
92.28
93.56
Stanford Cars
90.83
91.04
92.60
Table 9: Results on the transfer learning datasets.
D.3 DIFFERENT LOSS FUNCTIONS
To evaluate the kernelized classification layer under loss function other than the cross-entory loss, we report CIFAR-10/100 results with the squared loss (Hui & Belkin 2020) in Table [0] Note that the application of squared loss to the kernelized classification layer's outputs straighttorward since it outputs logits in the usual sense.
D.4 EFFECT OF EMBEDDING RECTIFICATION
As discussed previously, different to the usual image classification networks in He et al. (016)), we remove the ReLU activation from the embeddings. This is to utilize the full surface of S without restricting ourselves to only the nonnegative orthant. As is evident from Table IT, removing ReLU has only a marginal effect on the standard softmax baseline. is however an important factor for our method. We consistently used embeddings without the ReLU activation in all our experiments in the previous sections.
15

---

Under review as a conference paper at ICLR 2022
CIFAR-TÓ
CIFAR-TOD
Backbone
Sq. Loss
Ours+Sqq Loss
Sq. Loss
Ours+Sq. Loss
ResNet-8
83,70
86.58
51.55
55,99
Re
89.86
91.67
62.94
65.06
ResNet-20
91.20
92.75
64.00
68.61
ResNet-32
92.19
93.65
68.10
71.33
ResNct-44
92.16
94.12
69.48
72.25
ResNet-56
93.19
94.22
70.34
73.13
Table 10: Results on the CIFAR-10/101 datasets with the square loss.
Method
Accuracy
Softmax classifier with: rectified embeddings unrectified embeddings
70.96
71.23
Our classifier with:
rectified embeddings
71.61
unrectified embeddings
74.10
Table 11: Effect of rectification of the embeddings.
D.5 EFFECT OF ACTIVATION ON THE KERNEL COEFFICIENTS
Following the discussion in & 5.1 the constraint a- 02 ..M 2 is important to preserve 0-2,00-1--- the positive definiteness of Kopp- This can be imposed by using a ReLUU(')) where a' is the learnable parameter vector. However, ReLU has no upper-bound and allowing the scale of a to grow unboundedly causes issues in optimization: Assume we have an instantiation Qn of the vector a. By replacing Qu with Ayy, where À > 1, we scale all the inner product terms in (3) and (4) by the same X. As a result, we improve the loss of the already correctly classified training examples, but without making any effective change the predictor. Therefore, under this setting, once the majority of the training examples are correctly classified, the neural network can easily improve the loss just by increasing the norm of a. which is not useful. We therefore advocate L (2-regularization term a when ReL activation is used.
Alternatively, one could also use guarantee Q- 0-1,...,, aM 2 0, but also produce bounded a. Therefore, no regularization on a needed for these options. The softmax activation here should not be confused with the softmax loss discussed in [ The usage of the softmax activation in this context is similar to that in the self-attention literature (Vaswani al., 2017), where it is used to normalize the coefficients of linear combination. We experimented with these different activations on a' and summerized the results in [2]
sigmoid(a')
softmax(a'), both of which not only
We used a weight decay of 0.0001 on the coefficient vector whenever ReLU activation is used. Although sigmoid and softmax activations eliminate the need for weight decay, they put hard constraint on (ó( (o(w), o(f))
To overcome this limitation, it is helpful to use a temperature hy- perparameter in 3 where each inner product is divided by T' before taking the exponential. We used a temperature of 0.1 and 0.005, with sigmoid and softmax, respectively. Although sigmoid gives the best performance in Table we occasionally observed optimization issues with it, which could be due the vanishing gradient issue associated with this activation function. We therefore stick to ReLU in all other experiments. We however note that, in most cases, competitive results can be obtained with softmax as well, when used with temperature of 0.005.
also interesting to note that using no activation function on a' causes frequent divergence in training. This is consistent with the theory: The summation in (7) is not guaranteed to be posi- tive definite when a, are allowed to be negative (see Proposition 4.2) Therefore, the theory of kernelized classification is valid in this case.
16

---

Under review as a conference paper at ICLR 2022
Activation function
Accuracy
sigmoid
74,96
softmax
73,69
ReLU
74.10
None (lincar)
unstable
Table 12: Different activation functions the cocfficient vector. Note that the kernelized classifier unstable when no activation function is used, this agrees with the theoretical analysis.
E APPROXIMATION ERROR BOUNDS
In this section, we analyze error bounds for the approximation in Eq. 7 We start by proving the following theorem, which establishes a rigorous upper bound for the approximation error.
Theorem E.1. Let k S" 5th + R be any positive definite radial kernel on S with the series expansion k(u, v) = a-2keven (u, v) a-1koda ( v) _m= am {u, v)"", and KM : S" x S" + be its th partial sum. Define approximation error bound for the partial sum of the kernel is given by
[-1,1] + IR as (xxx) = Em=o am x Then the
[k(u, (u, (u, v)l
1
max
(x)+
(M + 1)! #E(-1.1)
for all (u,v) E S x S
Proof. Since k is positive definite, from Theorem 4.3 it has series expansion of the form:
k(u, v) = a- 2 (u, v) E 11, + a-1(((u, v) 1] [uu, v) =-1]) 11])= +
Omm (u, v)
=--2l(u, v) {-1,1]| + a-1((uu v) = ---1[luuv)) =-1]) + a6 (u,
Note that v is an analytic function. Furthermore,
M
has v) = a [-1,1]] a- v) 1] [(u, v) = -1] + Σ am (u, v)"
m=U
(-111)) a ([ v) = =1] [(u, va= = -1] + + v)),
where u (I) = Em=o am Therefore, e'a (x) is the Mh order Maclaurin polynomial approx- imation of (x)). Since (xx) is analytic, and therefore infinitely differentiable, we can obtain the Lagrange form of the approximation error as:
(()+))(E)
(xx) (M))) (M + 1)!
x. (M+1),
for some & E (-x, x), for all a E [-1, 1], where 2f(M+1) is the (M + 1)th order derivative of sb. It follows that, for all (u, E S x Sm
|k(u,, v) - - v)
max (vxx)) wM (x))
1,11
1
(x)(+))()))
max
(M + 1)! xE(-1,1)
The above theorem states that the absolute error made by cutting off the terms beyond order M is less than or equal to the maximum absolute value of the (M + 1)th order derivative of k, attenuated 1)!. To put this into context, since use M 10. the attenuation factor is
by factor of 1/(M Therefore, to make the error significant, the 11th order derivative of the kernel around 2.5 x 10 8
would have to be very high, suggesting kernel function with abrupt changes looking almost like discontinuities. Since such functions are unlikely to useful to learn a generalizable model, we believe that the error caused by this approximation is indeed negligible. Note also that, set kas
17

---

Under review as a conference paper at ICLR 2022
be the Maclaurin polynomial in the proof of Theorem E.1 make the derivations easier. However, since we have the freedom to learn the coefficients the summation, approximate & even better. In particular, even when l has abrupt changes causing a non-nelligible emoor in the Maclaurin approximation. will be possible to capture some of the residuals using the kernels kidd and Keven in kM expansion. This because the higher order kernels reach one of these kernels in the limit as discussed in the proof of Theorem 4.3
theoretically possible to
F COMPARISON TO NON-PARAMETRIC KERNEL METHODS
We use a parametric model with kernels in our method. This is in contrast to the more popu- lar usage of kernels with non-parametric models, such as support vector machines and Gaussian processes. Non-parametric models are usually more flexible. They are also more interpretable than deep srtwork-baseee parametric models. However, unfortunately, non-parametric models scale poorly with the train set size. For example, the number of support vectors grow linearly with the train set size (Steinwart 2003), and Gaussian processes methods scale with the cube of the train set size, or linearly after some optimizations (Wilson & Nickisch 2015).
In contrast, parametric models, such as the one proposed in this work, scale well with the training set size since they use a constant number of parameters regardless of the number of training examples. fact, this is one of the main reasons why deep learning methods have become extremely popular in the recent years. non-parametric models also allow faster inference, making them suitable for developing models for compute- limited scenarios, which the primary focus of this work.
18


Reviews:
[{"review_id": "dSgW0FU0sS", "reviewer": "Reviewer_pWF3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally well written with the ideas explained\nclearly. However, the main problem I have is the generality of this\napproximation used to define a functional for evaluating the\nkernel. The paper truncates higher order terms in the kernel expansion\nand fixes the number of terms (M) to 10 in all the experiments. The\nmath however shows that the kernel is indeed sensitive to M, which\nisn't surprising. The experiment in Figure 2 doesn't establish\nanything significant in this regard. The paper will benefit immensely\nfrom a more rigorous treatment of the kernel approximation. \n", "summary_of_the_paper": "This work considers the problem of using a non-linear classification\nfunction as a final layer of a neural network instead of the\nconventional softmax function. To this end, the authors proposed a new\nkernelized classification layer which is approximately learned using a\nmodified version of the standard softmax classification\nloss. Experimental evidence shows that the proposed kernelized\nclassification layer is able to outperform baselines, especially\nstandard softmax on synthetic and real world datasets.\n", "summary_of_the_review": "The proposed kernel classification layer is a promising idea, however\nI believe it needs more theoretical and empirical analysis to make it\ngenerally applicable.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Umr4cwXhBpH", "reviewer": "Reviewer_WDU4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n- The paper is well-written and well-motivated. \n- The problem the paper is concerned with is interesting. Instead of widening or deepening the model, the authors determine to use a more powerful classification layer for parameter efficiency.\n- The empirical studies are thorough and the ablation is insightful.\n\n\n# Weaknesses & concerns\n- Your assumption \"this is suboptimal since better nonlinear classifiers could exist in the same embedding vector space\" perhaps should only apply to the backbones with limited expressiveness, e.g., the used ResNet-CIFAR architectures. As you have not tried more powerful architectures like the wide-ResNet family or the ResNet-ImageNet family, I hold concerns about the correctness of this assumption.\n\n- You claim \"we theoretically show that our classification layer optimizes over all possible kernel functions on the space of embeddings to learn an optimal nonlinear classifier\". But, as detailed in Sec 4, you confine the kernel family as the radial kernels. You can clarify this in the original statement to avoid over-claim.\n\n- When first hearing the idea of combining DNNs and kernelized classifiers, I thought that the non-parametric kernelized classifiers are leveraged to make predictions. But, in fact, you insist on the parameterized softmax classifier (Eq 3). Can you provide a discussion on the pros and cons of the two kinds of model choices? I realize the primary con of non-parametric kernelized classifiers is that they are not so scalable in the test phase, although they enable analytical inference.\n\n- If you write the feature extractor as a mapping f and combine f with the radial kernel layers, you are basically defining a deep kernel. The main difference between this work and Wilson et al., 2016 is that you normalize the output of f and learn with parameterized modeling as well as SGD?", "summary_of_the_paper": "The paper considers replacing the linear softmax layer at the end of a classifier with a kernelized softmax layer to boost the expressiveness of the whole model, especially when using a lightweight feature extractor.\nThe authors have provided theoretical justifications for their layer design, and claim that the designed kernelized softmax layer optimizes over all possible kernel functions on the space of embeddings. \nThe authors evidence the usefulness of this layer in learning more model-efficient classifiers in a number of computer vision and natural language processing tasks.", "summary_of_the_review": "Given the identified weaknesses, I recommend a weak acceptance for this submission. I am glad to increase my score if the authors are able to resolve my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MZbORRlohlK", "reviewer": "Reviewer_PtK6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nThe paper is overall well-presented with clear description of the technical methods. The authors conducted extensive experiments with different tasks, the proposed method is effective by outperforming the compared baselines and other methods. Moreover, several ablation study setting are explored and thus strengthen the convincingness.\n\nCons:\n1. The authors conducted the experiment on various ResNet backbone. Instead, it could be more convincing if the author conduct experiment on more types of backbones as in Wang et al. 2019. Besides, the authors have emphasized that comparing with MKL, the proposed method scales better in large dataset setting. I suggest the author conduct experiment on ImageNet like Wang et al. 2019 to show empirical novelty from this perspective. \n2. The idea of this work is not quite original. Clearly, this work is inspired by the MKL work and Kervolution neural network. Though the authors have included these work in experiment and show the empirical superiority, the originality and novelty of the idea itself is not very strong.", "summary_of_the_paper": "The paper presents a kernelized classification layer as an alternative to the linear classification layer that is widely used in deep networks. The kernel classification layer maps learnt feature vector and weight vector to higher dimensional space to conduct scorer function. The key idea is to represent a kernel into a linear combination of predefined kernels and the combination coefficients are jointly learned with the backbone network.The author showed experiment results on multiple tasks to demonstrated the superiority of the proposed layer.", "summary_of_the_review": "Overall, this paper is well written and the idea is clear. Great amount of experiments are conducted and empirical novelty is shown. Though the originality of the idea is not very strong, I would conclude my current review for this work as: merits over flaws, and marginally above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-cYMz8gZyu9", "reviewer": "Reviewer_ZEPc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper the authors propose a method that uses nonlinear classification stage at the end of the deep neural network classifiers. To this end, they use kernel trick and radial kernel functions. The strengths and the weaknesses of the proposed method can be summarized as follows:\nStrengths:\ni) A new method is proposed using radial kernel functions.\nii) The proposed method outperforms the deep neural network methods using softmax loss function and some other methods using kernelized classifiers.\nWeaknesses:\ni) The main weakness of the paper is that the proposed method is not compared to the related methods well. The authors briefly mention some of the deep neural network classifiers using kernelized classifiers yet there is not any satisfactory argument why the proposed method must be an alternative to these methods. The authors state that the main difference is the use of positive definite kernels (more precisely, they argue that the related methods do not use positive definite kernels whereas they use them). This is unacceptable, in fact majority of the related methods also use positive definite kernels and there should be basic arguments clearly describing the differences and advantages of their proposed method over these ones.\nii) Experimental study is also very weak. The authors only compare their method to Kervo on asingle dataset. But, comparisons to the other deep neural networks using  kernelized classifiers (such as KerNet [R1],  Convolutional Kernel Networks of Marial et al., Deep Kernel [R2], etc.) are missing.\niii) There is a recent study [R3] demonstrating that every deep neural network classifier using standard gradient descent algorithms is in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel). This somewhat shows that deep neural networks are in fact kernel machines and there is not a need for using nonlinear classifier stage.\niv) I also strongly believe that the proposed method has major shortcomings compared to the kernelized classifiers implemented by SVM type algorithms. Please note that nonlinear SVMs use support vectors that are selected from training samples that lie in critical zones where the class samples approach to each other. Therefore, when evaluating a decision function one has to use multiple kernel function valuation against the support vectors. However, in the proposed method, there is no concept of support vectors. To assign a test sample, we make evaluation only a single vector, w (All m+2 kernel use the same w). This already restricts to return complex decision boundaries. In fact, the same effect can be obtained by using successive fully connected layers.\nv) for synthetic experiment, it would be better if the authors show the decision boundaries obtained on a 2d or 3d embedding space returned by deep neural network classifiers as in Center loss paper of Wen et al.\n\nReferences\n[R1] Lauriola et al. “enhancing deep neural networks via multiple kernel learning” Pattern Recognition, 101, 2020.\n[R2] Let et al. “Deep Kernel: Learning kernel function from data using deep neural network,” International conference on big Data Computing, Applications and Technologies, 2016.\n[R3] Pedro Domingos, “Every model learned by Gradient Descent Is Approximately\na Kernel Machine,” arXiv:2012.00152, 2020.\n", "summary_of_the_paper": "In this paper the authors propose a method that uses nonlinear classification stage at the end of the deep neural network classifiers. The authors argue that deep neural networks learn nonlinear representations by using convolutional layers and activation functions yet a linear classifier is used on the learned embeddings which is suboptimal. To overcome this problem, the authors integrate a kernelized classifier that uses a radial kernel function to the last stage of the deep neural network architecture. Then the proposed method is compared to a baseline deep neural network classifier using softmax loss function. Some improvements are obtained compared to the baseline network.", "summary_of_the_review": "The advantages of the proposed method over the related methods are not discussed and demonstrated well. There is also limitations of the proposed method as described in may main review in the sense that its capacity for creating highly nonlinear decision boundaries is limited. Therefore, I believe the contributions are not enough for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes the replacement of the softmax layer in a neural network with one parametrized by a kernel. The kernel itself is learned during training from the space of radial basis kernels. The resulting models are compared against identical networks with softmax, linear kernels, second order pooling and kervolutions on several datasets, encompassing vision and NLP tasks.

First, the reviewers raised questions about the novelty of the work. Theorem 4.3, based on which the method is derived, has existed in the literature and seems to be related to the uniqueness of the power series expansions for kernels. There is novelty in using this theoretical result to write an approximation of a positive definite kernel in a way which can be learned. Specifically, it is written as a finite weighted sum of existing kernels, where the coefficients are learned. Reviewer pWF3 posed a valid question about the quality of the approximation, to which the authors responded with an equally valid, and comprehensive, appendix on the error bounds of the approximation. Still, it is worth tempering the statement that the search is 'exhaustive' over the space of radial kernels or that the kernel is optimal (instead, the search appears over a large class of radial kernels, and the kernel is approximately optimal with an extremely low distance from the actual optimum).
Along the same lines of rephrasing claims, reviewer WDU4 also pointed out several statements and claims which were not entirely accurate, which the authors then proceeded to resolve, resulting in notable changes from the initial version of the paper. Specifically, there was mention of a "non-parametric kernelized classifier". This has been fixed, but it did seem to have initially confused other reviewers, who suggested related work that, it turns out, are not necessarily suitable contenders. The changes made definitely improved the paper, and resolved most of the reviewer's concerns.
Nevertheless, the appendix added comparing the method to non-parametric models could be improved. For instance the authors stated "Wilson et al. use Gaussian RBF and spectral mixture kernels. Our method has the capability to automatically learn any positive definite radial kernel. Note that Gaussian RBF and spectral kernels are all radial kernels." - is there any intuition, or proof, of a case when the method introduced here learns a network + classifier that the method by Wilson et al. cannot learn? Or for which deep kernel learning requires considerably more resources? (DKL has been optimized and made considerably faster since the initial paper in 2016).  https://proceedings.neurips.cc/paper/2016/hash/bcc0d400288793e8bdcd7c19a8ac0c2b-Abstract.html
Also, while the present work is backed by 4.3, DKL also has a theoretical grounding.
https://www.jmlr.org/papers/volume20/17-621/17-621.pdf

There was some discussion on the exhaustiveness of the experiments, and it was concluded that the datasets are sufficient, while the reviewers were not in agreement as to whether the authors considered sufficient contenders. A comparison against DKL, at least, appears to be warranted.

Overall, the paper brings a contribution in terms of improving the performance of backbones with limited expressiveness through the use of a kernel-parametrized classifier, learned by optimizing an approximation of a formulation that spans the entire space of radial basis kernels. The paper was updated considerably during the reviewer process, to its betterment, however, an experimental comparison against deep learning with non-parametric kernelized classifiers is still missing.
Decision: Reject