Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work considers the problem of using a non-linear classification
function as a final layer of a neural network instead of the
conventional softmax function. To this end, the authors proposed a new
kernelized classification layer which is approximately learned using a
modified version of the standard softmax classification
loss. Experimental evidence shows that the proposed kernelized
classification layer is able to outperform baselines, especially
standard softmax on synthetic and real world datasets.

main_review: The paper is generally well written with the ideas explained
clearly. However, the main problem I have is the generality of this
approximation used to define a functional for evaluating the
kernel. The paper truncates higher order terms in the kernel expansion
and fixes the number of terms (M) to 10 in all the experiments. The
math however shows that the kernel is indeed sensitive to M, which
isn't surprising. The experiment in Figure 2 doesn't establish
anything significant in this regard. The paper will benefit immensely
from a more rigorous treatment of the kernel approximation. 

summary_of_the_review: The proposed kernel classification layer is a promising idea, however
I believe it needs more theoretical and empirical analysis to make it
generally applicable.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers replacing the linear softmax layer at the end of a classifier with a kernelized softmax layer to boost the expressiveness of the whole model, especially when using a lightweight feature extractor.
The authors have provided theoretical justifications for their layer design, and claim that the designed kernelized softmax layer optimizes over all possible kernel functions on the space of embeddings. 
The authors evidence the usefulness of this layer in learning more model-efficient classifiers in a number of computer vision and natural language processing tasks.
main_review: # Strengths
- The paper is well-written and well-motivated. 
- The problem the paper is concerned with is interesting. Instead of widening or deepening the model, the authors determine to use a more powerful classification layer for parameter efficiency.
- The empirical studies are thorough and the ablation is insightful.


# Weaknesses & concerns
- Your assumption "this is suboptimal since better nonlinear classifiers could exist in the same embedding vector space" perhaps should only apply to the backbones with limited expressiveness, e.g., the used ResNet-CIFAR architectures. As you have not tried more powerful architectures like the wide-ResNet family or the ResNet-ImageNet family, I hold concerns about the correctness of this assumption.

- You claim "we theoretically show that our classification layer optimizes over all possible kernel functions on the space of embeddings to learn an optimal nonlinear classifier". But, as detailed in Sec 4, you confine the kernel family as the radial kernels. You can clarify this in the original statement to avoid over-claim.

- When first hearing the idea of combining DNNs and kernelized classifiers, I thought that the non-parametric kernelized classifiers are leveraged to make predictions. But, in fact, you insist on the parameterized softmax classifier (Eq 3). Can you provide a discussion on the pros and cons of the two kinds of model choices? I realize the primary con of non-parametric kernelized classifiers is that they are not so scalable in the test phase, although they enable analytical inference.

- If you write the feature extractor as a mapping f and combine f with the radial kernel layers, you are basically defining a deep kernel. The main difference between this work and Wilson et al., 2016 is that you normalize the output of f and learn with parameterized modeling as well as SGD?
summary_of_the_review: Given the identified weaknesses, I recommend a weak acceptance for this submission. I am glad to increase my score if the authors are able to resolve my concerns.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper the authors propose a method that uses nonlinear classification stage at the end of the deep neural network classifiers. The authors argue that deep neural networks learn nonlinear representations by using convolutional layers and activation functions yet a linear classifier is used on the learned embeddings which is suboptimal. To overcome this problem, the authors integrate a kernelized classifier that uses a radial kernel function to the last stage of the deep neural network architecture. Then the proposed method is compared to a baseline deep neural network classifier using softmax loss function. Some improvements are obtained compared to the baseline network.
main_review: In this paper the authors propose a method that uses nonlinear classification stage at the end of the deep neural network classifiers. To this end, they use kernel trick and radial kernel functions. The strengths and the weaknesses of the proposed method can be summarized as follows:
Strengths:
i) A new method is proposed using radial kernel functions.
ii) The proposed method outperforms the deep neural network methods using softmax loss function and some other methods using kernelized classifiers.
Weaknesses:
i) The main weakness of the paper is that the proposed method is not compared to the related methods well. The authors briefly mention some of the deep neural network classifiers using kernelized classifiers yet there is not any satisfactory argument why the proposed method must be an alternative to these methods. The authors state that the main difference is the use of positive definite kernels (more precisely, they argue that the related methods do not use positive definite kernels whereas they use them). This is unacceptable, in fact majority of the related methods also use positive definite kernels and there should be basic arguments clearly describing the differences and advantages of their proposed method over these ones.
ii) Experimental study is also very weak. The authors only compare their method to Kervo on asingle dataset. But, comparisons to the other deep neural networks using  kernelized classifiers (such as KerNet [R1],  Convolutional Kernel Networks of Marial et al., Deep Kernel [R2], etc.) are missing.
iii) There is a recent study [R3] demonstrating that every deep neural network classifier using standard gradient descent algorithms is in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel). This somewhat shows that deep neural networks are in fact kernel machines and there is not a need for using nonlinear classifier stage.
iv) I also strongly believe that the proposed method has major shortcomings compared to the kernelized classifiers implemented by SVM type algorithms. Please note that nonlinear SVMs use support vectors that are selected from training samples that lie in critical zones where the class samples approach to each other. Therefore, when evaluating a decision function one has to use multiple kernel function valuation against the support vectors. However, in the proposed method, there is no concept of support vectors. To assign a test sample, we make evaluation only a single vector, w (All m+2 kernel use the same w). This already restricts to return complex decision boundaries. In fact, the same effect can be obtained by using successive fully connected layers.
v) for synthetic experiment, it would be better if the authors show the decision boundaries obtained on a 2d or 3d embedding space returned by deep neural network classifiers as in Center loss paper of Wen et al.

References
[R1] Lauriola et al. “enhancing deep neural networks via multiple kernel learning” Pattern Recognition, 101, 2020.
[R2] Let et al. “Deep Kernel: Learning kernel function from data using deep neural network,” International conference on big Data Computing, Applications and Technologies, 2016.
[R3] Pedro Domingos, “Every model learned by Gradient Descent Is Approximately
a Kernel Machine,” arXiv:2012.00152, 2020.

summary_of_the_review: The advantages of the proposed method over the related methods are not discussed and demonstrated well. There is also limitations of the proposed method as described in may main review in the sense that its capacity for creating highly nonlinear decision boundaries is limited. Therefore, I believe the contributions are not enough for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a kernelized classification layer as an alternative to the linear classification layer that is widely used in deep networks. The kernel classification layer maps learnt feature vector and weight vector to higher dimensional space to conduct scorer function. The key idea is to represent a kernel into a linear combination of predefined kernels and the combination coefficients are jointly learned with the backbone network.The author showed experiment results on multiple tasks to demonstrated the superiority of the proposed layer.
main_review: Pros:

The paper is overall well-presented with clear description of the technical methods. The authors conducted extensive experiments with different tasks, the proposed method is effective by outperforming the compared baselines and other methods. Moreover, several ablation study setting are explored and thus strengthen the convincingness.

Cons:
1. The authors conducted the experiment on various ResNet backbone. Instead, it could be more convincing if the author conduct experiment on more types of backbones as in Wang et al. 2019. Besides, the authors have emphasized that comparing with MKL, the proposed method scales better in large dataset setting. I suggest the author conduct experiment on ImageNet like Wang et al. 2019 to show empirical novelty from this perspective. 
2. The idea of this work is not quite original. Clearly, this work is inspired by the MKL work and Kervolution neural network. Though the authors have included these work in experiment and show the empirical superiority, the originality and novelty of the idea itself is not very strong.
summary_of_the_review: Overall, this paper is well written and the idea is clear. Great amount of experiments are conducted and empirical novelty is shown. Though the originality of the idea is not very strong, I would conclude my current review for this work as: merits over flaws, and marginally above the acceptance threshold.
review_title: null

Meta-review: The paper proposes the replacement of the softmax layer in a neural network with one parametrized by a kernel. The kernel itself is learned during training from the space of radial basis kernels. The resulting models are compared against identical networks with softmax, linear kernels, second order pooling and kervolutions on several datasets, encompassing vision and NLP tasks.

First, the reviewers raised questions about the novelty of the work. Theorem 4.3, based on which the method is derived, has existed in the literature and seems to be related to the uniqueness of the power series expansions for kernels. There is novelty in using this theoretical result to write an approximation of a positive definite kernel in a way which can be learned. Specifically, it is written as a finite weighted sum of existing kernels, where the coefficients are learned. Reviewer pWF3 posed a valid question about the quality of the approximation, to which the authors responded with an equally valid, and comprehensive, appendix on the error bounds of the approximation. Still, it is worth tempering the statement that the search is 'exhaustive' over the space of radial kernels or that the kernel is optimal (instead, the search appears over a large class of radial kernels, and the kernel is approximately optimal with an extremely low distance from the actual optimum).
Along the same lines of rephrasing claims, reviewer WDU4 also pointed out several statements and claims which were not entirely accurate, which the authors then proceeded to resolve, resulting in notable changes from the initial version of the paper. Specifically, there was mention of a "non-parametric kernelized classifier". This has been fixed, but it did seem to have initially confused other reviewers, who suggested related work that, it turns out, are not necessarily suitable contenders. The changes made definitely improved the paper, and resolved most of the reviewer's concerns.
Nevertheless, the appendix added comparing the method to non-parametric models could be improved. For instance the authors stated "Wilson et al. use Gaussian RBF and spectral mixture kernels. Our method has the capability to automatically learn any positive definite radial kernel. Note that Gaussian RBF and spectral kernels are all radial kernels." - is there any intuition, or proof, of a case when the method introduced here learns a network + classifier that the method by Wilson et al. cannot learn? Or for which deep kernel learning requires considerably more resources? (DKL has been optimized and made considerably faster since the initial paper in 2016).  https://proceedings.neurips.cc/paper/2016/hash/bcc0d400288793e8bdcd7c19a8ac0c2b-Abstract.html
Also, while the present work is backed by 4.3, DKL also has a theoretical grounding.
https://www.jmlr.org/papers/volume20/17-621/17-621.pdf

There was some discussion on the exhaustiveness of the experiments, and it was concluded that the datasets are sufficient, while the reviewers were not in agreement as to whether the authors considered sufficient contenders. A comparison against DKL, at least, appears to be warranted.

Overall, the paper brings a contribution in terms of improving the performance of backbones with limited expressiveness through the use of a kernel-parametrized classifier, learned by optimizing an approximation of a formulation that spans the entire space of radial basis kernels. The paper was updated considerably during the reviewer process, to its betterment, however, an experimental comparison against deep learning with non-parametric kernelized classifiers is still missing.
Decision: Reject