Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work uses the Neural Tangent Kernel (NTK) as a tool to study several aspects of adversarial robustness.
First, it uses the NTK to generate adversarial examples and find they transfer to the trained counterparts.
Second, it uses the eigendecomposition of the NTK to define features, which correspond to each eigenvector of the kernel matrix on the training data. It analyzes these features by visualizing them and evaluating their usefulness and robustness. 
Lastly, it analyzes the evolution empirical NTK for standard and adversarial training. In each of these experiments, it contrasts its findings with the existing literature on adversarial robustness.
strengths_and_weaknesses: Strengths
- Using the NTK to study adversarial robustness is a new contribution (I was surprised there were no prior works, except for some overlap).
- Each of the experiments are quite interesting on their own, and some of them appear to be novel. I particularly liked the eigendecomposition of features.

Weaknesses
- Overall, the main issue is that none of the experiments quite go deep enough, and overall I'm not sure what I learned from the paper. Authors argue that their results unify all the known existing phenomena. Surely, while all of their results are *consistent* with what is known in adversarial robustness and NTK, I'm not sure there was something new learned.
More details:
  - The adversarial transfer experiment is interesting, but pretty much expected given earlier works that empirically evaluated to the similarity of NTK predictions and their finite-width counterparts (e.g., https://arxiv.org/abs/1912.02803). There also appears to be prior (unpublished) work on this exact experiment: https://openreview.net/forum?id=M5hiCgL7qt. What would have been interesting to study is how transferability varies as your training regime deviates from the NTK regime; without this study, I feel that the claims about the "black box" nature of this attack is a bit overclaimed.
  - The third investigation (evolution of empirical NTK) seems to make a couple of leap of faiths (maybe they are better justified but in that case you should make them clearer). In particular, it is not obvious at all why the top eigenspectrum should correspond to more robust features for the empirical NTK and particularly for adv training (as this was only shown for the exact NTK and std training). I wonder why the authors did not try computing this explicitly (see Questions).
  - I liked the E.1 Linearized Adversarial Training experiment, but wish the authors had developed this further.
- Second issue I have with the paper is the style of writing. It is polished, but sentences are overly long/flamboyant and not to the point.

Despite these drawbacks, I appreciate the authors' breath of empirical investigations (including those in the app).
Overall, the paper would benefit from having a more focused thesis/hypothesis and tailoring the experiments to that thesis.
questions: - Have you tried visualizing the features by looking at the most/least activating images? Gradients (as visualized in the paper) are useful, but it's common to look at a few other visualizations
- Can you provide more details on how exactly you computed usefulness and robustness for the NTK features? I could not find them in the Appendix.
- Have you tried computing the robustness of the eigen-features arising from the empirical NTK? 
- For the polar angle plots, can you add a legend for the color (epoch?)?
limitations: None other than those mentioned above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper summarizes a novel empirical study on the adversarial robustness of neural networks using a neural tangent kernel perspective (both for finite-width networks and their infinite-width limit). The main new insights of this work are:
1. The adversarial perturbations of a trained infinite-width NTK match quite significantly those of the same wide-but-finite architecture trained on the same dataset.
2. For the NTK at initialization, the first eigenfunctions of the kernel tend to correspond to robust and useful features, while latter eigenfunctions mainly correspond to non-robust, but also useful features. The best performance is achieved by leveraging both types of features, and robustness can not be obtained by training only on the subspace of robust features at initialization.
3. The evolution of the finite-width NTK is much faster for adversarial training than standard training. The authors argue that this evolution is fundamental to yield robustness through feature learning.
strengths_and_weaknesses: ## Strengths

1. **Timely and interesting perspective**: The NTK perspective has been the object of intense scrutiny by the generalization community in the past few years. In this regard, this work nicely complements these prior studies by filling a gap in the literature analyzing through the NTK lense different phenomena related to adversarial robustness. This will not only be useful for the robustness community, but also for the deep learning theory community at large, since studying adversarial training through this lense can give complementary information of the general dynamics of deep learning, as this is a very different algorithm. Personally, I find the fact that there are qualitative similarities in the evolution of the NTK between adversarial training and standard training, but also significant quantitative differences in terms of speed and magnitude of the change, quite remarkable.

I also find the study of the robustness of different eigenmodes of the NTK at initialization, and the transferability of adversarial attacks between NTKs and NNs to be very interesting. However, as I point out in the weaknesses, I think there are some fuzzy technical details in these experiments that I would like the authors to clarify.

2. **Balanced and self-contained analysis**: In general, I found this paper very easy to read and quite balanced. The authors touch upon different topics with the right amount of breadth and depth that gives the reader a good amount of new information.

3. **Good contextualization with prior work**: The authors properly discuss a great deal of relevant work, both in the space of deep learning theory and adversarial robustness. I believe this will makes this paper easier to follow by both communities.

4. **Opens new research avenues**: As a new empirical study on relevant deep learning phenomena, I significantly appreciate that this work gives a clear path forward for future theoretical and practical studies. In particular, I believe that the analysis of the after-kernel of adversarially trained networks could yield significant insights for robustness, while the study of the accelerating effect on the kernel evolution of adversarial training could also help understand the rich regime of deep learning as a whole.

## Weaknesses

1. **Fuzzy technical details**: I find some technical details of this work a bit hard to follow and somewhat suspicious of possible flaws in the evaluation. Specifically, I am referring to:
- **Number of epochs of training in Fig. 4 and corresponding robustness**: The x-axis in Fig. 4 is in logarithmic scale and shows the evolution of the robustness of a neural network 100,000 epochs! This is (i) an oddly high number, and (ii) surprising for the fact that the robust accuracy is very high at the "beginning of training" (100 epochs). This does not match the previously reported evolution of the robustness of neural networks in standard training, and hence makes me seriously doubt of the validity of this experiment.  
- **Unclear usefulness metric**: I find unclear which precise metrics are used in Fig. 2 (right). In sec. 2.2. the authors introduce two metrics for usefulness and robustness in Eq. (3) and Eq. (4) but then later mention that they use clean accuracy and robust accuracy of the different eigenmodes as metrics. Knowing which of these two alternatives the authors use to plot Fig. 2 (right) is very important to contextualize the reported numbers.

2. **Weak results on transferability/black-box attacks**: Although I understand the reasons behind selling their transferability results as a black-box attack, I honestly believe that the provided evaluation (even despite my previous comments) is not enough to claim this technique as a useful adversarial black-box attack. The context of this experiment is rather limited, and has only studied the transferability between very wide networks and the NTK. These two models have been shown before to share very similar dynamics, and in this regard, it is not surprising that their adversarial attacks transfer between each other. This section would have been much stronger if it had also studied the transferability of adversarial attacks between linearized neural networks (i.e, finite-width NTK) and standard neural networks. If the transferability is high in that setting as well, then claiming this as a promising black-box attack technique would not be overclaiming.

3. **(*Minor*) Figure design**: Some graphic design aspects of the figures could be improved to help readability of the paper:
- The colormap in Fig. 3 (left) is far from ideal. The current figure is using a qualitative colormap to show a numerically diverging quantity. I would advice the authors to change it some [perceptually uniform and sequential colormap](https://matplotlib.org/stable/tutorials/colors/colormaps.html).
- The font size is very small. It is in general very hard to read the labels and titles of all the figures.


questions: I would appreciate if the authors could answer my doubts with regards to the technical details that I outlined in Weaknesses 1. I am afraid that these could be hints of potential technical flaws in this work, and hence they are are the main reasons why I am slightly leaning towards rejection. However, If the authors can alleviate my concerns, I will be very happy to significantly increase my score, as I believe this paper could be a nice contribution.


**(*Minor*) Fix citations**: I have spotted that several references point out to arXiv papers that have already been accepted to multiplee archival venues. I would encourage the authors to review their citations to make sure they correctly cite each article.
limitations: I see no direct negative societal impact stemming from this work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work generates adversarial examples by using a training-free fashion through NTK. They study the feature through the eigendecomposition of the associated kernel, and confirm that the robust features are corresponding to the top part of the eigenspectrum. They also study the dynamics of empirical NTK during adversarial training.
strengths_and_weaknesses: Strength:

1. I find the idea of connecting robust features with the eigenspectrum of NTK and studying the kernel dynamic of standard training vs adversarial training is novel. Some of the experiments are interesting.

Weakness:
1. The experiment cannot convince me that the black-box attack through NTK has a similar effect as while box attack. The strong alignment of the gradient is likely because the trained neural network is within the lazy regime, yet in reality, the neural network cannot guarantee to be within the lazy regime, i.e. adversarial training, thus making the proposed black-box attack not useful.
2. The author does not describe the technical detail clearly in the main paper. There are lots of NTK literature under different model architectures, different weight initialization, different data distribution assumptions, etc. Yet I did not find the author considering all the above situations thoroughly. Later, I notice that the author describes the initialization in appendix C, where they consider two-layer ReLU network with $W\sim N(0,0.01^2I_{m\times d})$. To me, such initialization is neither practical nor falls into the standard NTK literature. 
3. I have trouble understanding the reason for ``robust features alone is not enough''. The author defers the discussion in the appendix, where CIFAR10 binary FC_4 FC_5 suggests that adding more features may hurt the robust performance. The author however compares it with multi-classification MNIST and concludes that robust features are not sufficient for robust classification. First MNIST is a rather simple unique dataset and it's sparse. It's likely that there're so few robust features for MNIST. Second, it's unfair to compare a binary classification result vs a multi-classification result on different datasets. At least present the same plot for binary MNIST and multi-class CIFAR10.
questions: 1. It’s unclear what’s the architecture of the neural network the author describes in section 2.3, which should be given math formulas. Does the author focus on two-layer ReLU network since it dominates NTK literature? It should be clear whether the weight w a vector or a matrix; when training the neural network, whether any part of the layer is fixed; how to initialize the neural network The above questions are different throughout different NTK literature.
2. The author presents the eq(7) for learning with L2 loss, which is described by a regression problem. Then try to transfer to the classification problem by simply applying the softmax function on the top of the kernel output and applying cross-entropy loss. I’m wondering whether the author considers the training dynamics of learning with cross-entropy loss for infinite neural networks.
3. Given eq(8), does the author also consider any multiple-step adversarial example for the infinitely wide neural network?
4. The author should describe more on ``kernel generated attack’’, i.e. how to generate adversarial examples from kernel machines. Since the NTK is solely based on model initialization, does the attack only based on one random initialization or an average of multiple initializations?
5. For the top figure of figure 4, as you increase the epoch, the cosine similarity between NTK and neural network is decreasing. Moreover, it should be helpful if also plots how far the network weight is away from the initialization during training. If the learning rate is very small, then even after 10^5 epoch, there’s possible that the trained neural network is still not far from the initialization, thus lying within the lazy regime. So we should not be surprised by the result. And the author should do a thorough test of how these attacks perform compared with white-box attacks.
6. Mathematically it's not correct to use $L(f-f_t,y)$ in equation (C.30). In figure C.1 (a), the bottom figure is about epoch vs. robust accuracy, and why the highest accuracy can reach 200%?
7. Some of the figures need to enlarge the text size, i.e. figure 3, 6. 
limitations: Yes.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper applies the theory of Neural Tangent Kernel to explain insightful properties of adversarial examples.
strengths_and_weaknesses: # Strengths
- Very well written, crystal clear, both text and maths.
- Very timely since NTK gains more and more momentum
- Opens the doors to many other discoveries (both about attack and defenses)

# Weaknesses
- Too few details about the computation of the NTK (I was expecting these in the appendix) and its cost. 
- Very few statements deserve some mitigation (see below) 
questions: I would like to clarify the following minor comments
- line 68: "only require ... architecture" and training data as well, isn't it?
- line 239: "NTK ... suggest that these two scenarios are no so distinct" The scenarios are very distinct, the adversarial perturbations as well, the required average distortion for hacking images is bigger in BB than in WB. Everything is different!
- line 243: "NTK as the most faithful BB attack." I disagree. The NTK is another scenario where the attacker has only the architecture and the training data, different from WB scenario where it knows the weights and architecture, different from BB scenario where it knows nothing but has a free access.
- Fig. 2 Right. The colored boxes are not explained in the text. I suppose it shows areas of (non)useful/(non)robust features. How did you set the boundaries of these boxes?
   
limitations: Very good submission I enjoy to read.
I dislike paragraph L241-245 that I would remove, freeing some space for more explanation on the computations of the "eigen-images" for instance.
review_title: null

Meta-review: metareview: This paper is a borderline case. The review scores are quite widely spread with 4, 4, 6, 8. More specifically:

- The score 8 review is unfortunately rather short and could be more informative.
- The score 6 review argues that the paper is a nice initial investigation in this direction and would prefer accepting the paper. But they also agree that it's a borderline case and wouldn't argue strongly against rejecting the paper.
- The score 4 reviewers both responded to the rebuttals and decided to maintain their scores. Both of them found the experiments too superficial and would have preferred a more in-depth investigation.

As a result, the paper presents a difficult decision because there is nothing technically wrong about the paper, but the investigation also appears preliminary and lacking depth. My senior area chair suggested to accept the paper, so I'm going with this recommendation. I would not argue against rejecting the paper if the program chairs would prefer this outcome.
confidence: Less certain
Decision: Accept