Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors consider the infinite-width limit of neural networks and show the equivalence of NN trained by soft margin loss and SVM with neural tangent kernel. They proceed to show that any finite-width neural networks trained by GD is approximately a kernel machine with a certain data dependent kernel. They further show that one can get a robustness certificate from NTK.
main_review: This work belongs to the recent push in the deep learning theory community that showed that in some large-width limit, NNs effectively behave as kernel methods, with the so called neural neural tangent kernel. However I have several concerns:

- While it is true that a lot of attention has been given to the linear regime in the past three years, the limitations of this approach have been studied extensively since then. Even though there was a surge of interest for kernel methods, which were proven more efficient than previously thought, it is now clear that the NTK regime provides only limited insights into the behavior of neural networks.

- While I am not sure whether the hinge-loss, and non-smooth loss have been explicitly studied before, plenty of work have already shown quite generally that large width network behave as kernel methods, for example Chizat, Oyallon and Bach 2018 (especially, here no rates of convergence are provided so one can take a sequence of smooth approximation).

- I am not sure I understand section 4.1: while it is true that you can write it as a solution of a kernel method, the kernel is now data dependent. It seems to be a trivial statement. For example, it is always true that one can train a neural network and then say that this is the solution of a kernel methods by retraining the last layer while keeping the other layers fixed with the learned weights. This does not give any particular insight, except that gradient descent is a way to learn a good kernel.

- I understand the big improvement in robustness compared to IBP, but is it not a weakness of IBP? Will the bound of Theorem 4.2 work well when using the kernel obtained at the end of the learning trajectory? 

For all the reasons presented above, I don’t think this paper as currently written is suitable for NeurIPS.


--- after rebuttal ---
Changed rating to 5.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper shows the equivalence between the infinitely wide NN trained by soft margin loss and the standard 1-norm soft margin SVM with NTK. The authors further consider general loss functions with l2 penalization and show the equivalence between NN and regularized kernel machines. They also provide two practical applications of the developed theory.
main_review: The paper establishes the equivalence between neural network and support vector machine and provides several applications according to the developed theory. 

My main concerns are as follows:
Some related literature are missing. To name a few, Andras(2002), Ren and Yang (2010), Nong (2012). Without careful comparison and discussion, I cannot see solid contribution based on the existing results.

The paper lacks theoretical novelty. The main results follow from some existing literature and simple analysis.

The application part from the theory lacks further detailed discussion and explanation. The authors need to explain more on Theorem 4.1 and 4.2.
limitations_and_societal_impact: The paper lacks theoretical novelty and the application part is not sufficient.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper theoretically analyzes the connection between Neural Network (NN) and Support Vector Machine (SVM). It shows the equivalence between SVM with Neural tangent Kernel (NTK) and infinite-width NN trained by soft margin loss with subgradient descent.  Besides, they extend their theory to general $\mathit{l}_2$ regularized loss functions and show finite-width NN trained by a $\mathit{l}_2$ regularized loss function is approximately a kernel machine.  It provides robustness certificates for infinite-width NN. 
main_review: Strength:
1. This paper provides a connection between the SVM and NN  with solid theoretical derivations. It shows that the SVM with NTK and infinite-width NN trained by soft margin loss with subgradient will have the same dynamics and converge to the same solution. 
2. The claims of this paper are clear and easy to read.  

Weakness:
1. This paper theoretically analyzes the equivalence between SVM with NTK and infinite-width NN trained by soft margin loss with subgradient descent. However, 1-norm soft margin loss is not widely used in NN.
 
2. The robust analysis is new but not surprising as regularization is widely used to improve the robustness in machine learning. For example, "Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network" has discussed the ReLU network with $l_2$ regularizer and its connection to the kernel ridge regression.

Questions:
1. In the experiment part, the training dynamics of the network and SVM are similar while the step < $10^2$, but when the number of steps is larger than $10^3$, the dynamic of SVM seems not to agree with NN.

2. I think this paper: "An analytic theory of shallow networks dynamics for hinge loss classification, NeurIPS 2020" might be relevant here. It would be worth discussing the connections.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies ANNs for classification from an NTK perspective. The authors rely on earlier results about the NTK to derive convergence results about ANNs trained with hinge loss (and other loss functions) with an l^2 regularization, showing that these essentially fall in the NTK regime and thus connecting them with SVM. Most of the results are theoretical, but some numerics are also given.
main_review: This is an interesting paper, which shows (if the results are correct, of which I am reasonably confident, though not absolutely certain) that neural network classification with hinge loss and with l^2 loss functions fall within the NTK regime, thus allowing us to connect them with SVM. I think the authors should do a better job at emphasizing that it is not trivial that ANNs for classification fall in the NTK regime and connecting with empirical results which seem to show a different behavior (although without an l^2 regularization). The techniques are not particularly new or exciting, but the appendix looks clean. 
limitations_and_societal_impact: No societal impact.
review_title: null

Meta-review: This paper establishes the equivalence between neural network and support vector machines, following the recent trend in identifying equivalence between infinite-width neural networks and kernel machines. In particular, neural network classification trained by soft margin loss with subgradient descent and extension to general regularized loss functions (including L2 regularization) and show infinite-width limit falls in the NTK regime. As applications of the developed theory, authors obtain non-vacuous generalization bound of NN via the corresponding kernel machines as well as robustness certificates.

After the discussion, the reviewer’s final recommendations are 2 weak accepts, 1 weak reject  and 1 reject. Given that the reject recommendation comes with low confidence and was disengaged despite the author's significant attempt to clarify, the AC is inclined to significantly downweight the suggestion. Even so the paper remains borderline.

In the end, the majority of reviewers agreed there’s merit in the paper (new equivalence, L2 regularization, application to robustness and generalization bound) and should be published. The question remained whether NeurIPS is the appropriate venue. Given the paper tackles an important question, providing non-trivial insights and equivalence they established are deemed to be correct to the expert reviewers, the AC believes the paper’s claim is significant enough and would benefit NeurIPS participants. 

As a side: One of the reviewers in the end had concern/question on the generalization bound plots (provided in anonymous link). It appears that on the right panel the train/test curves follow the same line which is questionable. Also the experiment should be compared to much further in the training time (near convergence). 

Decision: Accept (Poster)