Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors consider the evolution of Neural Tangent Kernels(NTK), characterization of neural networks in the infinite-width limit, under standard and adversarial training. The authors proposes the idea of adversarial robustness of NTK during training. Experiments on CIFAR-10 and CIFAR-100 show that NTK converges to a kernel and feature map. They have increased robustness, compared with standard training, which prevents catastrophic accuracy drop under the adversarial attacks. That property saves even after non-adversarial training.
strengths_and_weaknesses: Strengths:
•	Unusual use of NTK concept for researching adversarial robustness
•	Proposing and exploring three training dynamics, comparing their performance, training dynamics, and adversarial transferability with a list of custom metrics, considering properties of NTK. Those parts of the paper give a lot of insights about considering approaches
•	Great performance under adversarial attacks on the considered datasets

Weaknesses:
•	Authors consider only attacks, which were designed to fool standard dynamic networks
•	There is no theoretical substantiation of the obtained results
•	Numerical experiments only on CIFAR-10 and CIFAR-100

questions: •	Did you use only one attack, PGD, during experiments?
•	Is it possible to use NTK adversarial approach in other domains, for example, text? 

limitations: The limitations are clearly described in the conclusion part of the paper. Authors points related works which partially cover uncovered study features
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper conducts an emprical study on the evolution of NTK under benign and adversarial training, which considers the training process into two phases, kernel learning and lazy training. They showed that adversarial training in kernel learning obtains a different kernel from standard training, which is robust even when non-robust training is used on top of it. And adv. training on top of a fixed learned kernel could have a high robust accuracy. 
strengths_and_weaknesses: Strengths 
- The paper is well written and easy to follow.
- It reveals some new observations through considering two phases in NTK. Particularly, benigh training could produce use kernels for the later robust linear training on top of it. 

Weakness
- My major concern is that despite of the presented the empirical observations, it lacks of any tenative analysis over them. 
- Is ther any practical algorithmic insights based on the obersvations found in this paper? I think as an empirical paper, good insights for algorithms is are important. 
- Does these observations generalize beyond NTK?  
questions: See the Strength and Weakness. 
limitations: See the Weakness.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper summarizes a novel empirical study on the adversarial robustness of neural networks using a neural tangent kernel perspective comparing the dynamics of linearized neural networks and their non-linear counterparts. The main new insights of this work are:

1. The qualitative dynamics of the NTK during adversarial training are very similar as the ones of standard training, albeit the NTKs converge to very different kernels.
2. Remarkably, even during standard training, the NTK learns some robust features that can be exploited during linearized fine-tuning.
3. Linearized and centered fine-tuning of a neural network, both adversarially and standardly pretrained, can give an edge in robustness over non-linear finetuning. This does not happen for uncentered fine-tuning.
4. Adversarial attacks transfer between linearized networks and non-linear networks mostly for the uncentered networks, and after a few epochs of training.
5. Certain visualizations of the NTK eigenfunctions of an adversarially pretrained network seem semantically more meaningful than the NTK eigenfunctions of a standardly pretrained one.

strengths_and_weaknesses: ## Strenghts

1. **Timely and interesting perspective**: The NTK perspective has been the object of intense scrutiny by the generalization community in the past few years. In this regard, this work nicely complements these prior studies by filling a gap in the literature analyzing through the NTK lense different phenomena related to adversarial robustness. This will not only be useful for the robustness community, but also for the deep learning theory community at large, since studying adversarial training through this lense can give complementary information of the general dynamics of deep learning, as this is a very different algorithm. 

2. **Remarkable findings**: Personally, I find quite remarkable that benign linearized finetuning of a standardly pretrained NTK can yield some robustness. Also, the fact that you can gain a performance edge by centered and linearized fine-tuning of an adversarially pretrained network is fascinating. This is very significant as it provides very clear new evidence that sometimes the so-called "rich regime" can have a detrimental effect in performance as shown in (Ortiz-Jimenez et al. 2021).

3. **Balanced and self-contained analysis**: In general, I found this paper very easy to read and quite balanced. The authors touch upon different topics with the right amount of breadth and depth that gives the reader a good amount of new information.

4. **Opens new research avenues**: As a new empirical study on relevant deep learning phenomena, I significantly appreciate that this work gives a clear path forward for future theoretical and practical studies. In particular, I believe that the analysis of the after-kernel of adversarially trained networks could yield significant insights for robustness, while the study of the benefitial edge from linearized fine-tuning will be of broad interest to the deep learning theory community.

## Weaknesses

1. **Limited evaluation of different $\epsilon$ regimes**: I honestly believe that this work could easily become much stronger if it provided some additional analysis of the effect of the adversarial budget $\epsilon$ on its findings. It is widely known in the adversarial robustness community that the properties and dynamics of adversarially trained networks for different $\epsilon$ can be qualitatively very different. In this regard, I believe that showing the same results for other $\epsilon$, and in particular for the standard $\epsilon=8$ regime would be very valuable. For example, if the claimed edge of linearized fine-tuning is true in this regime as well (given the right evaluation), it would suggest a radically new way to improve the robustness of neural networks.
2. **Suspicious benign accuracy of centered NTK in Table 2.**: It is a bit concerning that the benign accuracy of the centered NTK in Table 2 is lower than the robust accuracy. If this is true, this would point out to some weird geometry of the loss landsape wrt input, akin to the one happening in catastrophic overfitting, which would be very surprising in my humble opinion. Without a further investigation on that front, I would say it is more likely a sign of a technical flaw in the evaluation (which could be very damaging for this work) or a typo. 
3. **Missing citations**: I have spotted a few missing citations of relevant papers that also describe the evolution of the NTK and compare it to the linearize dynamics. They should also be cited.

- G. Ortiz-Jimenez, S. Moosavi-Dezfooli, P. Frossard. "What can linearized neural networks actually say about generalization?" NeurIPS 2021.
- J. Paccolat, L. Petrini, M. Geiger, K. Tyloo, and M. Wyart, “Geometric compression of invariant manifolds in neural nets,” Journal of Statistical Mechanics: Theory and Experiment, no. 4, 2021
- A. Baratin, T. George, C. Laurent, R. D. Hjelm, G. Lajoie, P. Vincent, and S. Lacoste-Julien,“Implicit regularization via neural feature alignment,” AISTATS 2021.
- D. Kopitkov and V. Indelman, “Neural spectrum alignment: Empirical study,” in International Conference on Artificial Neural Networks (ICANN), 2020.



questions: I would encourage the authors to comment on the two outlined weaknesses before by providing results on other $\epsilon$ regimes (mostly on $\epsilon=8$) and clarifying the reasons for the lower benign accuracy of the centered kernel in Table 2. If the authors did this, I would be quite inclined to increase my score.
limitations: I see no direct negative societal impact stemming from this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper conducts empirical studies towards neural tangent kernel (NTK) in adversarial training. They figure out that NTK converges to a different kernel in adversarial training compared to clean training, and the new kernel provides adversarial robustness even when further fine-tuning on non-robust training.
strengths_and_weaknesses: Strengths:

[1] This paper provide many experiments studying the properties of adversarially trained NTK.

Weaknesses:

[1] Some paragraphs need clarifications. For example, in abstract, while most parts describing the robustness of the adversarially trained TNK, the last sentence "we find that adversarial training on top of a fixed kernel.." suddenly switch another context. Since there are a lot of things contained in this paper, please make necessary connections among different contexts.

[2] An important missing literature:

Allen-Zhu, Zeyuan, and Yuanzhi Li. "Feature purification: How adversarial training performs robust deep learning." 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS). IEEE, 2022.

Allen-Zhu et. al. (2022) provides theoretical justification on why adversarial training help purify the features. This is quite related to the 4th contribution mentioned in the submission. Please cite this paper and make comparisons or connections.

[3] It would be great if the authors could provide more insights on how people could utilize the properties of adversarially trained NTK in real practice. This paper observes interesting phenomenons for potential theoretical studies, but it could be even better if people can directly utilize the results.


questions: Besides the weaknesses part, I'm also wondering, is there any analytical formula to describe where the adversarially trained NTK converges (even for simple models)?
limitations: NA
review_title: null

Meta-review: metareview: The paper studies the evolution of NTK under adversarial training. The empirical studies show that the NTK of adversarial training converges to a different kernel compared to standard training.  All the reviewers agree that the empirical study is interesting and should be accepted. 


Missing citations: The actual notion of the NTK the authors consider is called finite-width NTK, which is proposed in the paper "Convergence analysis of deep learning via over-parameterization".
confidence: Certain
Decision: Accept