What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?
Authors: Nikolaos Tsilivis
Julia Kempe
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Tangent Kernel
Adversarial Examples
Non Robust Features
Linearised Networks

Abstract: The adversarial vulnerability of neural nets, and subsequent techniques to create robust models have attracted significant attention; yet we still lack a full understanding of this phenomenon. Here, we study adversarial examples of trained neural networks through analytical tools afforded by recent theory advances connecting neural networks and kernel methods, namely the Neural Tangent Kernel (NTK), following a growing body of work that leverages the NTK approximation to successfully analyze important deep learning phenomena and design algorithms for new applications. We show how NTKs allow to generate adversarial examples in a ``training-free'' fashion, and demonstrate that they transfer to fool their finite-width neural net counterparts in the ``lazy'' regime. We leverage this connection to provide an alternative view on robust and non-robust features, which have been suggested to underlie the adversarial brittleness of neural nets. Specifically, we define and study features induced by the eigendecomposition of the kernel to better understand the role of robust and non-robust features, the reliance on both for standard classification and the robustness-accuracy trade-off. We find that such features are surprisingly consistent across architectures, and that robust features tend to correspond to the largest eigenvalues of the model, and thus are learned early during training. Our framework allows us to identify and visualize non-robust yet useful features. Finally, we shed light on the robustness mechanism underlying adversarial training of neural nets used in practice: quantifying the evolution of the associated empirical NTK, we demonstrate that its dynamics falls much earlier into the ``lazy'' regime and manifests a much stronger form of the well known bias to prioritize learning features within the top eigenspaces of the kernel, compared to standard training.

What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?
Nikolaos Tsilivis Center for Data Science
Julia Kempe
Center for Data Science and
New York University nt2231 @nyu. edu
Courant Institute of Mathematical Sciences New York University kempe@nyu. edu edu
Abstract
The adversarial vulnerability of neural nets, and subsequent techniques to create robust models have attracted significant attention; yet we still lack a full understand- ing of this phenomenon. Here, study adversarial examples of trained neural networks through analytical tools afforded by recent theory advances connecting neural networks and kernel methods, namely the Neural Tangent Kernel (NTK), following a growing body of work that leverages the NTK approximation to suc- cessfully analyze important deep learning phenomena and design algorithms for new applications. We show how NTKs allow to generate adversarial examples in a "training-free" fashion, and demonstrate that they transfer fool their finite- width neural net counterparts in the "lazy" regime. We leverage this connection provide alternative view robust and non-robust features, which have been suggested to underlie the adversarial brittleness of neural nets. Specifically, we define and study features induced by the eigendecomposition of the kernel better understand the role of robust and non-robust features, the reliance on both for standard classification and the robustness-accuracy trade-off. We find that such features are surprisingly consistent across architectures, and that robust features tend to correspond to the largest eigenvalues of the model, and thus are learned early during training. Our framework allows us to identify and visualize non-robust yet useful features. Finally, we shed light on the robustness mechanism underlying adversarial training of neural nets used in practice: quantifying the evolution of the associated empirical NTK, we demonstrate that its dynamics falls much earlier into the "lazy" regime and manifests much stronger form of the well known bias to prioritize learning features within the top eigenspaces of the kernel, compared to standard training.
Introduction
Despite the tremendous success of deep neural networks in many computer vision and language modeling tasks, as well in scientific discoveries, their properties and the reasons for their success are still poorly understood. Focusing on computer vision, particularly surprising phenomenon evidencing that those machines drift away from how humans perform image recognition is the presence of adversarial examples, images that are almost identical to the original ones, yet are misclassified by otherwise accurate models.
Since their discovery (Szegedy et al., 2014), a vast amount of work has been devoted to understanding the sources of adversarial examples and explanations include, but are not limited to, the close linear operating mode of neural nets (Goodfellow et al., 2015), the curse of dimensionality carried by the input space (Goodfellow al., 2015, Simon-Gabriel et al., 2019), insufficient model capacity (Tsipras et al., 2019, Nakkiran, 2019) or spurious correlations found in common datasets (Ilyas et al., 2019).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

4-/255 sign!
Prediction: Car
v.C(()))))
Prediction: Airplane
00
dg
tu Ogb
turta
Opeso
V. (e)))
V (.)()))))) (1))s)
(a)())))))))))
Figure 1: Top. Standard setup of adversarial attack, where barely perceivable perturbation is added to an image to confuse an accurate classifier. Bottom. The correspondence between neural networks and kernel machines allows to visualize decomposition of this perturbation, each part attributed to a different feature of the model. The first few features tend to be robust.
In particular, one widespread viewpoint is that adversarial vulnerability is the result of model's s sensitivity to imperceptible yet well well-generalizing features the data, so called useful non-robust features, giving rise to a trade-off between accuracy and robustness (Tsipras et al., 2019, Zhang al., 2019). This gradual understanding has enabled the design of training algorithms, that provide convincing, yet partial, remedies to the problem; the most prominent of them being adversarial training and its many variants (Goodfellow et al., 2015, Madry al., 2018, Croce et al., 2021). Yet we are far from a mature, unified theory of robustness that is powerful enough to universally guide engineering choices or defense mechanisms.
In this work, we aim to get a deeper understanding of adversarial robustness (or lack thereof) by focusing on the recently established connection of neural networks with kernel machines. Infinitely wide neural networks, trained via gradient descent with infinitesimal learning rate, provably become kernel machines with inteeiinddeenent,, but architecture dependent kernel its Neural Tangent Kernel (NTK) that remains constant during training (Jacot et al., 2018, Lee et al., 2019, Arora et al., 2019b, Liu et al., 2020). The analytical tools afforded by the rich theory kernels have resulted in progress in understanding the optimization landscape and generalization capabilities of neural networks (Du al., 2019, Arora et al., 2019a), together with the discovery interesting deep learning phenomena (Fort al., 2020, Ortiz-Jiménn diverse areas applications such as the design of better classifiers (Shankar et al., 2020), efficient neural architecture search (Chen al., 2021), low-dimensional tasks in graphics (Tancik al., 2020) and dataset distillation (Nguyen et al., 2021). While the NTK approximation increasingly utlilized, for finite width neural nets, little is known about the adversarial robustness properties of these infinitely wide models.
al., 2021), while also inspiring practical advances in
Our work inscribes itself into the quest to leverage analytical tools afforded by kernel methods, in particular spectral analysis, to track properties of interest the associated neural nets, in this case as they pertain to robustness. To this end, we first demonstrate that adversarial perturbations generated analytically with the NTK can successfully lead the associated trained wide neural networks (in the kernel-regime) to misclassify, thus allowing kernels to faithfully predict the lack of robustness those trained neural networks. In other words, adversarial (non-) robustness transfers from kernels to networks; and adversarial perturbations generated via kernels resemble those generated by the corresponding trained networks. One implication this transferability is that can analytically devise adversarial examples that do not require access to the trained model and in particular its weights; instead these "blind spots" may be calculated a-priori, before training starts.
Our contribution:
A perhaps even more crucial implication of the NTK approach to robustness relates to the understand- ing of adversarial examples. Indeed, we show how the spectrum of the NTK provides an alternative way to define features of the model, to classify them according their robustness and usefulness for

---

UTI Peat yes (CHN430, PCB)
IhC hattana coPARao Canil
Figure 2: Left: Top features for 7 different kernel architectures for car image extracted from the CIFARIO dataset when trained on car and plane images. Right: Features according to their robustness (X-axis) and usefulness (y-axis). Larger/darker bullets correspond larger eigenvalues. Useful features have >> 0.5-usefulness; shaded boxes are meant to help visualize useful-robus regions.
correct predictions and visually inspect them via their contribution to the adversarial perturbation (see Fig. 1). This in turn allows us to verify previously conjectured properties standard classifiers; dependence both robust and non-robust features in the data (Tsipras et al., 2019), and tradeoff of accuracy and robustness during training. In particular we observe that features tend to be rather invariable across architectures, and that robust features tend to correspond to the top of the eigenspec- trum (see Fig. 2), and as such are learned first by the corresponding wide nets (Arora et al., 2019a, 9a, Jacot et al., 2018). Moreover, we able visualize useful non-robust features of standard models (Fig. 4). While this conceptual feature distinction has been highly influential in recent works that study the robustness of deep neural networks (see for example (Allen-Zhr and Li, 2022, Kim et al., 2021, Springer et al., 2021)), to the best of our knowledge, none of them has explicitly demonstrated the dependence of networks Rather, these works either reveal such features in some indirect fashion, or accept their existence as an assumption. Here, we show that Neural Tangent Kernel theory endows us with natural definition of features through its eigen- degemndccomooittion and provides a way to visualise and inspect robust and non-robust features directly on the function space of trained neural networks.
such feature functions (except for simple linear models (Goh, 2019)).
Interestingly, this connection also enables us to empirically demonstrate that robust features of standard models alone are not enough for robust classification. Aiming understand, then, what makes robust models robust, we track the evolution of the data-dependent empirical NTK during adversarial training of neural networks used in practice. Prior experimental work has found that networks with non-trivial width to depth ratio which are trained with large learning rates, depart from the NTK regime and fall in the so-called "rich feature" regime, where the NTK changes substantially during training (Geiger et al., 2020, Fort et al., 2020, Baratin et al., 2021, Ortiz-Jiménez et al., 2021). In our work, which to the best of our knowledge the first provide insights on how the kernel behaves during adversarial training, we find that the NTK evolves much faster compared to standard training, simultancously both changing its features and assigning more importance the more robust ones, giving direct insight into the mechanism at play during adversarial training (see Fig. 6). In summary, the contributions of our work are the following:
We discuss how to generate adversarial examples for infinitely-wide neural networks via the NTK, and show that they transfer to fool their associated (finite width) nets the appropriate regime, yielding "training-free" attack without need to access model weights (Sec. 3).
Using the spectrum of the NTK, we give an alternative definition of features, providing a natural decomposition or perturbations into robust and non-robust parts (Tsipras et al., 2019, Ilyas et al., 2019) (Fig. 1). We confirm that robust features overwhelmingly correspond to the top part of the eigenspectrum; hence they are learned early on in training. We bolster previously conjectured hypotheses that prediction relies on both robust and non-robust features and that robustness is traded for accuracy during standard training. Further,
3

---

show that only utilizing the robust features of standard models is not sufficient for robust classification (Sec. 4).
We turn to finite-width neural nets with standard parameters study the dynamics of their empirical NTK during adversarial training. We show that the kernel rotates in a way that enables both new (robust) feature learning and that drastically increases of the importance (relative weight) of the robust features over the non-robust ones. We further highlight the structural differences of the kernel change during adversarial training versus standard training and observe that the kernel seems to enter the "lazy" regime much faster (Sec. 5).
Collectively, our findings may help explain many phenomena present in the adversarial ML. literature and further elucidate both the vulnerability of standard models and the robustness of adversarially trained ones. We provide code visualize features induced by kernels, giving unique and principled way to inspect features induced by standardly trained nets (available at (iili42/advvntkt)
//tp///gttttt com/
Related work: To the best of our knowledge the only prior work that leverages NTK theory derive perturbations in some adversarial setting is due to Yuan and Wu (2021), yet with entirely different focus. deals with what is coined generalization attacks: the process of altering the training data distribution to prevent models to generalise on clean data. Bai et al. (2021) I) study aspects of robust models through their linearized sub-networks, but do not leverage NTKs.
2 Preliminaries
We introduce background material and definitions important to our analysis. Here, we restrict ourselves to binary classification, to keep notation light. We defer the multiclass case, complete definitions and a more detailed discussion of prior work to the Appendix.
2.1 Adversarial Examples
Let be classifier, x be input (e.g. natural image) and y its label (e.g. the image class). Then, given that f is an accurate classifier on x, X is an adversarial example (Szegedy et al., 2014) for f
i) the distance d(x, x) is small. Common choices in computer vision are the Ep norms, especially the &o norm on which we focus henceforth, and
ii) f(x) + y. That is, the perturbed input is being misclassified.
Given loss function L, such as cross-entropy, one can construct an adversarial example x + 1 by finding the perturbation that produces the maximal increase of the loss, solving
n arg max L(f(x + n),
(1)
for some 0 that quantifies the dissimilarity between the two examples. In general, this is non-convex problem and one can resort to first order methods (Goodfellow et al., 2015)
x = x sign (Vx((f(x),))
or iterative versions for solving it (Kurakin et al., 2017, Madry et al., 2018). The former method is usually called Fast Gradient Sign Method (FGSM) and the latter Projected Gradient Descent (PGD). These methods are able to produce examples that are being misclassified by common neural networks with probability that approaches 1 (Carlini and Wagner, 2017). Even more surprisingly, has been observed that adversarial examples crafted to "fool" one machine learning model are consistently capable of "fooling" others (Papernot al., 2016, 2017). phenomenon that is known as the transferability of adversarial examples. Finally, adversarial training refers to the alteration of the training procedure include adversarial samples for teaching the model to be robust (Goodfellow et al., 2015, Madry et al., 2018) and empirically holds as the strongest defense against adversarial examples (Madry et al., 2018, Zhang et al., 2019).
2.2 Robust and Non-Robust Features
Despite a vast amount of research, the reasons behind the existence of adversarial examples are not perfectly clear. A line of work has argued that central reason is the presence of robust and

---

non-robust features in the data that standard models learn to rely upon (Tsipras et al., 2019, Ilyas al.. 2019). In particular training is responsible for the brittleness of neural nets. Here, we slightly adapt the feature definitions of (Iyys et al., 2019)', and extend them to multi-class problems (see Appendix A).
conjectured that reliance on useful but non-robust features during
Let D be the data generating distribution with æ E X' and y E {=1}. We define feature as function : + IR and distinguish how they perform as classifiers. Fix p. 2
1. p Useful feature: A feature & is called p-useful if
Be (==)n(((=)]==)) =
2. 7--obust feature: A feature & is called y-robust if it remains useful under any perturbation inside a bounded "ball" B, that is
Ex inf (1)p)(((=++)===) JEB
=
In general, a feature adds predictive value if it gives an advantage above guessing the most likely label, i.c. p > m Ex
[1 (. and we will speak of "useful" features in this case, omitting the p. We will call such a feature useful, non-robust if it useful, but y-robust only for 0 or very close to O, depending on context.
The vast majority of works imagines features as being induced by the activations of neurons in the net, most commonly those of the penultimate layer (representation-lugy formal definitions are in no way restricted activations, and we will show how to exploit them using the eigenspectrum of the NTK. In particular, in Sec. 4, we demonstrate that the above framework agrees perfectly with features induced by the eigenspectrum of the NTK of a network, providing a natural way to decompose the predictions of the NTK into such feature functions. In particular we can identify robust, useful, and, indeed, useful non-robust features.
features), but the previous
2.3 Neural Tangent Kernel
Let ʃ : Rd + IR be a (scalar) neural network with a linear final layer parameterized by a set of weights R" and {X, (} be a dataset of size n, with X' Raadd and y E {=1)". Linearized training methods study the first order approximation
f(x: W(+1) = f(x; w.) + fxf T (Wi+1 - Wt).
(5)
The network gradient w w:) induces a kernel function O, : Rd Rd + E. usually referred as the Neural Tangent Kernel (NTK) of the model
Ox(x,x') = f w.) T (x):
(6)
This kernel describes the dynamics with infinitesimal learning rate (gradient flow). In general, the tangent space spanned by the f w.) twists substantially during training, and learning with the Gram matrix of Eq. (6) (empirical NTK) corresponds to training along an intermediate tangent plane. Remarkably, however, in the infinite width limit with appropriate initialization and low learning rate, it has been shown that f becomes a linear function of the parameters (Jacot et al., 2018, Liu al., 2020), and the NTK remains constant (O, Oo =: -). Then, for learning with &2 loss the training dynamics of infinitely wide networks admits a closed form solution corresponding to kernel regression (Jacot et al., 2018, Lee et al., 2019, Arora et al., 2019b)
(x) = (xx T (xX, - - ))(X
(7)
where x E IRd is any input (training or testing), t denotes the time evolution of gradient descent, is the (small) learning rate and, slightly abusing notation, O X) Ruxn denotes the matrix containing the pairwise training values of the NTK, ((XxX)) (x), and similarly for ((x, X) E R". To be precise, Eq. (7) gives the mean output of the network using a weight- independent kernel with variance depending on the initialization".
'We distinguish useful and robust features based on their accuracy as classifiers, not in terms of correlation with the labels as in Ilyas et al. (2019), allowing natural extension to the multi class setting. For robustness, we consider any accuracy bounded away from zero robust, quantifying that an adversary cannot drive accuracy zero entirely.
For that reason, in the experiments, we often compare this with the centered prediction of the actual neural network. f - fo. as is commonly done in similar studies (Chizat et al., 2019).
5

---

3 Transfer Results in the Kernel Regime
In this section, show how to generate adversarial examples from NTKs and discuss their similarity to the ones generated by the actual networks. Note that for network results, we restrict ourselves to wide networks initialized in the "lazy" regime with small learning rates (the "kernel regime").
3.1 Generation of Adversarial Examples for Infinitely Wide Neural Networks
Adversarial examples arise in the context of classification, while the NTK learning process is described by a regression as in Eq. (7). The arguably simplest way to align with the framework presented in Eq. (1) is to treat the outputs of the kernel similar to logits of a neural net, mapping them to probability distribution via the sigmoid/softmax function and apply cross-entropy loss.
A simple calculation (see Appendix B, together with the generalization to the multi-class case) gives:
The optimal one step adversarial example of a scalar, infinitely wide, neural network is given by
x x ye sign (xf((x)),
for | x|1x I E, where Vxf((x)= = 0x X)
One can conceive other ways to generate adversarial perturbations for the kernel, either by changing the loss function (as previously done neural networks (e.g. (Carlini and Wagner, 2017)) or through Taylor expansion around the test input, and we present such alternative derivations in Appendix B. However, in practice we observe little difference between that approach and the one presented here.
3.2 Transfer Results and Kernel Attacks
Predictions from NTK theory for infinitely wide neural networks have been used successfully for their large finite width counterparts, so seems reasonable conjecture that adversarial perturbations generated via the kernel as in Eq. (8) resemble those directly computed for the corresponding neural net as per Eq. (2). In particular, this would imply that adversarial perturbations derived from the NTK should not only fool the kernel machine itself, but also lead wide neural nets to misclassify. While similar transfer results in different contexts have been ob- served indirectly, via the effects of the perturbation on metrics like accuracy (Yuan and Wu, 2021, Nguyen et al., 2021), we aim to look deeper to compare perturbations directly. High similarity would imply that any gradient based white-box attack on the neural net can be successfully mimicked by a "black-box" kernel derived attack.
Drike sed of
Setting. To this end, we train multiple two-layer neural networks on image classifications tasks extracted from MNIST and CIFAR-10 and compare adversarial examples generated by Eqs. (2) (attacking the neural network) and (8) (attacking the kernel). The networks are trained with small learning rate and are sufficiently large, so lie close to the NTK regime.
0.80
Foble
We track cosine similarity between the gradients of the loss from the NTK predictions and the gradients from the actual neural net as training evolves. Then, we generate adversarial perturbations from both the neural net and the kernel machine, and test whether those produced by the latter can fool the former. Full experimental details can be found in Appendix C.
Figure 3: Top. Cosine simi- larity between the loss gradi- ent of the neural net and of the NTK prediction for the same time point. Bottom. Ro- bust accuracy of neural net against its adversarial ex- amples (solid) and correspond- ing NTK examples (dashed). CIFAR 10, car vs plane.
Our experiments confirm a very strong alignment of loss gradients from the neural nets and the NTK across the whole duration of training, as can be seen in Fig. 3 (top). Then, as expected, kernel- generated attacks produce a similar drop in accuracy throughout training the networks "own" boxt attacks, eventually driving robust accuracy to 0%, as seen in Fig. 3 (bottom). We reproduce these plots for MNIST in Appendix C, leading similar conclusions.
Results.
When concerned with security aspects of neural nets, adversarial attacks are mainly characterised as either white-box box black-box

---

attacks (Papernot et al., 2017). White box attacks assume full access to the neural network and in particular its weights; prominent examples include FGSM/PGD attacks. Black box attacks, on the other hand, can only query the model to try to infer the loss gradient, either through training separate surrogate models (Papernot et al., 2016) or through carefully crafted input-output pairs fed to the target model (Chen et al., 2017, Ilyas et al., 2018, Andriushchenko et al., 2020). NTK theory and the experiments of this section suggest a threat model in which the attacker does not require access to the model its weights, nor training of substitute model. For fixed architecture and training data, all the information required for the computation of Eq. (8) is available at initialization, making the INTK-attack akin to "training free" substitution attack, and, least in the kernel-regime for wide nets considered here, as effective as white-box attacks.
4 NTK Eigenvectors Induce Robust and Non-Robust Features
This close connection between adversarial perturbations from the kernel and the corresponding neural net gives us the opportunity to bring to bear kernel tools on the study of adversarial robustness and its relation features in a more direct fashion. Several recent works leverage properties of the NTK, and specifically its spectrum, to study aspects of approximation and generalization in neural networks (Arora et al., 2019a, Basri al., 2019, Bietti and Mairal, 2019, Basri et al., 2020). Here we show how the spectrum relates to robustness and helps to clarify the notion of robust/non-robus features.
We define definefeaturee induced by the eigendecomposition of the Gram matrix ( x) -E= A.V.V. We will be most interested in the end of training, when the model has access to all the features extract from the training data X. As t oc, Eq. (7) becomes (x) = 6 x²) (),,(X
and can be decomposed as foo (x) O X)) Σ A7 v.v.y Ex=1 f() (x), where
(x),
f Rd R f (x) = X, (xx v.v.y.
(9)
Each (() can be seen as a unique feature captured from the (training) data. Note that these functions map the input to the output space, thus matching the definitions of Sec. 2.2. Also observe that all f(8's jointly recover the original prediction of the model, while each one, intuitively, should contribute something different it.
Importantly, these features induce a decomposition of the gradient of the loss into parts, each representing gradients of a unique feature as already advertised in Fig. 1. The binary case is particularly elegant as it gives rise to a linear decomposition of the gradient as
vxf(f
Σ a; (x
(10)
for some a; depending on x and y (see Appendix D). But if f(''ss 's are features, how do they look like?
Feature properties of common architectures: With these definitions in place, we of features for commonly used architectures, leveraging their associated NTK. To be consistent with the previous section, consider classification problems from MNIST (10 classes) and CIFAR-10 10 (car airplane). We compose the Gram matrices from the whole training dataset (50000 and 10000, respec- tively), and compute the different feature functions f()) using the cigendecomposition of the matrix. We estimate the useful- ness of feature (6) by measuring its accuracy on hold-out validation set, and its robustness by perturbing each input of this set, using an FGSM attack on feature f(6), We consider several different Fully Connected and Convolutional Kernels, whose expressions are available through the Neural Tangents library (Novak et al., 2020), built on top of JAX (Bradbury et al., 2018). We summarize our findings on how these features behave:
analyze the characteristics
Car features
Plane features
indwa: 1018, class act: 87.9 Indeo:: 1081, class are: M.
index: 8018, clsss acc: 67.9 Index: 8OB5, claes aCc: 72.3
Figure Non-robust, useful fea- tures earlier and later in the spec- trum, for CIFARIO car and plane.
7

---

Functions f(0 represent visually distinct features. We visualise each feature f() by plotting its gradient with respect to x. Fig. shows the gradient of the first 5 features for various architectures for specific image from the CIFAR-10 10 dataset. We observe that features are fairly consistent across models, and they are interpretable: for example the 4th feature seems to represent the dominant color of an image, while the 5th one seems to be capturing horizontal edges.
Networks use both robust and non-robust features for prediction. It has been speculated that neural networks trained in a standard (non adversarial) fashion rely on both robust and non-robust features. Our feature definition in Eq. (9) shows that this is indeed the case. The NTK of common neural networks consists of both robust features that match human expectations, such as the ones depicted in Fig. but also on features that are predictive of the true label, while not being robust to adversarial perturbations of the input (Fig. 4). Fig. 2 depicts the first 100 features of fully connected and a convolutional tangent kernel in Seefulness-Robusttee space. The upper left region of the plots shows a large amount of useful, yet non-robust features. These features seem random to human observers.
Robustness lies at the top, We observe in Fig. 2 that features corresponding to the top eigenvectors lend to be robust. This is consistent among different models and between the two datasets (see Appendix D). Since these eigenvectors are the ones fitted first during training (Arora et al., 2019a, Jacot et al., 2018), it is no wonder that the loss gradient evolves from coherence to noise, as observed in Fig. lb (in the Appendix). This also explains the apparent trade-off between robustness and accuracy of neural networks as training progresses: useful, robust features are fitted first, followed by useful, but non-robust ones. This ties in well with both empirical findings (Rahaman et al., 2019) and theoretical case studies (Basri et al., 2019, Bietti and Mairal, 2019, Basri et al., 2020) that demonstrate that low frequency functions are fitted first during training and provide favorable generalization properties and we would associate robust features with these low-frequency parts (in function space).
Robust features alone are not enough. In light of these findings, it might be reasonable to conjecture that we could obtain robust models by retaining the robust features of the prediction, while discarding the non-robust ones. The spectral approach gives principled way to disentangle features and create kernel machines keeping only the robust ones. Our results show that in general it is not possible to obtain non- non-trivial performance without compromising robustness in this fashion, strengthening the case for the necessity of data augmentation in the form of ad adversarial training (see Appendix D.3).
5 Kernel Dynamics during Adversarial Training
Given the apparent necessity for adversarial training to produce robust models, how does it achieve this goal? To shed some light this fundamental question, we depart from the "lazy" NTK regime and study the evolution of the NTK of adversarially trained models. For a neural network trained with gradient descent, as the learning rate 1) + O, the continuous time dynamics can be written as
Oc
Of Ōt
Oc Df
Ot -n - ff 0f
and
-
w
(1I)
In the NTK regime, this kernel θ; remains fixed at its initial value. However, outside this regime, it has been demonstrated, both empirically (Geiger et al., 2020, Fort et al., 2020, Baratin al., 2021, Ortiz-Jiménez et al., 2021) and theoretically (Atanasov et al., 2022), that θ is not constant during training, and is changing as the weights move. In adversarial training, moreover, there is the additional effect that at each weight update, the data changes as well. For that reason, understanding the dynamics of adversarial training requires tracking the evolution of kernel O, X2), where X, denotes the current (mini) batch of training data. Notice that the tangent vector V. (x4,) is still
describing the instantancous change of on the current batch of data, thus Or X,) is informative of the local geometry of the function space, justifying its value as quantity to be measured during adversarial training.
We train a deep convolutional architecture on CIFAR-10 10 (multiclass) with standard (SGD) and adversarial training using PGD with lo constraint. Full inplementations details and accuracy curves can be found in Appendix E, together with the reproduction of the same experiment on MNIST, where the observations are similar. We track the following quantities during training:
Kernel distance. We compare two kernels using a scale invariant distance, which quantifies the relative rotation between them, used in other works studying NTK dynamics (e.g. Fort al.
8

---

802
Figure 5: Left: Kernel Matrices for a mini batch of size 256. Left to Right: Kernel at initialization, Kernel after standard training, Kemnll after adversarial training (20 PGD steps). The standard kernll grows significantly more than the adversarial one. Right: (a) Kernel Frobenius norm evolution, and (b) concentration the top 20 eigenvalues during standard and adversarial training. Setting: CIFARIO, R0, fo = 8/255
Kand
CRAFTE
MAANT
SHANT, IOSD
(BAkin 19
CRAATA POOOTTRR
140
039
Figure 6: Left: Rotation (Eq. (12)) of the empirical NTK during standard, and adversarial training. Left to right: MNIST, standard, MNIST adversarial, CIFAR standard, CIFAR adversarial. Right: Kernel trajectories in polar space (Eq. (13)) for MNIST (left) and CIFARIO (right). Darker colors indicate earlier epochs.
(2020)):
Tr(O,0)) 0,), T
(12)
Polar dynamies. Zooming in the change that the initial kernel undergoes, define polar space on which we measure the movement of the kernel:
9 Oall
0, arccos (1 d(O,, Oo)),
(13)
| 0
where Oo, Oy are the initial and final kernel, respectively. Fig. presents a heatmap of kernel distances different time steps for both standard and adversarial training, well both training trajectories in polar space.
Concentration on subspaces. To quantify weight concentration on the top region of the spectrum, we track the (normalized) Frobenius norm of subspaces E X? Σn X² for various cut-offs p. where we have indexed the eigenvalues from largest to smallest. Fig. depicts concentration on the top 20 eigenvalues during training.
Our findings show that similar to what has been reported in prior work (Fort et al., 2020), the kernel rotates significantly in the beginning of training and then slows down for both standard and adversarial training. However, the latter case, this second phase begins a lot earlier. As Fig. 6 illuminates, the kernel moves a greater distance than when performing standard training, but after few epochs it stops both rotating and expanding; note that this is not the case for standard training where the kernel increases its magnitude substantially later training, and fact grows to have norm orders of magnitude larger than during adversarial training (see Fig. 5). In hindsight, this behavior is perhaps surprising, as each element of the kernel measures similarity between data points, and robust machine should be more conservative when estimating similarity. The observation that during adversarial training linear dynamics govern the later phase of adversarial training. It has been observed in previous works (Geiger al., 2020, Fort al., 2020, Ortiz-Jiménez, et al., 2021) that linearization after few initial epochs of rapid rotation often closely matches performance of full network training. Our
kernel becomes relatively static relatively fast might indicate that

---

results indicate that similar phenomenon occurs even under the data shift of adversarial training (see Appendix E.1 for study of linearized adversarial training). opening avenues to design robust machines more efficiently.
Moreover, endowed with the knowledge that at least for kernels trained with static data robust features lie at the top, we study polar dynamics of the top space only (see Fig. 8 in the Appendix) to observe that there is substantial rotation in this space, suggesting that robust features are learned early on not only during standard, but in particular during adversarial training. Even more interestingly, Fig. demonstrates that not only the robust features change, but their relative weight as measured by the concentration on the top-20 space is increasing simultaneously relative to standard training as well, and remains large; in fact, significantly larger than during standard training. As each eigenvalue weights the importance of the corresponding feature on the final prediction, this implies that the kernel "learns" to depend more on the most robust features.
Put together, these findings reveal different kernel dynamics during standard and adversarial training: the kernel rotates much faster, expands much less and becomes "lazy" much earlier than during standard training. Fully understanding the properties of converged adversarial kernels remains an important avenue for future work, that might allow to design faster algorithms for robust classification.
Final Remarks
We have studied adversarial robustness through the lens of the NTK across multiple architectures and data sets both in the idealized NTK regime and the "rich feature" regime. When connecting the spectrum of the kernel with fundamental properties characterizing robustness our phenomenological study reveals a universal picture of the emergence of robust and non-robust features and their role during training. There are certain limitations and unexplored themes in our work; Sec. 3 argues that transferable attacks from the NTK may be as effective white-box attacks, but this warrants in-depth study across architectures, kernels and data sets (which has not been the main focus of this work). Sec. 4 visualises features for fairly simple models, since the computation of kernel derivatives is a costly procedure. It would be interesting to use our framework to visualise features from more complicated architectures. Finally, our work in Sec. invites more research on the kernel at the end of adversarial training, similar to what has been done for standard models (Long, 2021).
We hope that our viewpoint motivate further theoretical understanding of adversarial phenomena (such as transferability) and the design of better and/or faster adversarial learning algorithms, by further analyzing the kernels from robust deep neural networks.
Acknowledgements
The authors would like thank Jingtong Su, Alberto Bietti, Yunzhen Feng, and Artem Vysogorets for fruitful discussions and feedback in various stages of this work. NT thanks Dimitris Tsipras for helpful discussion in the beginning of this project. The authors would like acknowledge support through the National Science Foundation under NSF Award 1922658. This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.
References
Z. Allen-Zhu and Y. Li. Feature purification: How adversarial training performs robust deep learning. 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 977-988, 2022.
M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein. Square attack: A query-efficient black- box adversarial attack via random search. In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, editors, Computer Vision - ECCV 2020- - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII, volume 12368 of Lecture Notes in Computer Science, pages 401-501. Springer, 2020.
S. Arora, S. Du, W. Hu, 7. Li, and R. Wang. Fine-Grained Analysis of Optimization and Generaliza- tion for Overparameterized Two-Layer Neural Networks. In K. Chaudhuri and R. Salakhutdinov,
10

---

editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 322-332. PMLR, 09-15 Jun 2019a.
S. Arora, S. S. Du, W. Hu, Z. Li, R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Ahh-BBuu, E. B. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8139-81 -148, 2019b.
A. Atanasov, B. Bordelon, and C. Pehlevan. Neural networks as kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2022.
Y. Bai, X. Yan, Y. Jiang, S. Xia, and Y. Wang. Clustering effect of adversarial robust models. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, Liang, and W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 6-14, 2021, virtual, pages 19590-29600 2021.
A. Baratin, T. George, C. Laurent, R. D. Hjelm, G. Lajoie, P. Vincent, and S. Lacoste- Julien. Implicit regularization via neural feature alignment. In A. Banerjee and K. Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pages 2269-2277. PMLR, 2021.
R. Basri, D. W. Jacobs, Y. Kasten, and S. Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 4763 4763-4772, 2019.
R. Basri, M. Galun, A. Geifman, D. W. Jacobs, Y. Kasten, and S. Kritchman. Frequency bias in neural networks for input of non-uniform density. In Proceedings the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 685-694, PMLR, 2020.
A. Bietti and Mairal. On the inductive bias of neural tangent kernels. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché--Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 12873-12884, 2019.
J. Bradbury, R. Frostig. P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //githbb//ttubb..../ com/google/jar
N. Carlini and D. A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy, SP 20/7, San Jose, CA, USA, May 22-26, 2017, pages 39-57. IEEE Computer Society, 2017.
P. Chen, H. Zhang, Y. Sharma, J. Yi, and C. Hsieh. ZOO: zeroth order optimization based black- box attacks deep neural networks without training substitute models. In B. M. Thuraisingham, B. Biggio, D. M. Freeman, B. Miller, and A. Sinha, editors, Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec@ CCS 2017, Dallas, TX, USA, November 3, 2017. pages 15-26. ACM, 2017.
W. Chen. X. Gong, and Z. Wang. Neural architecture search on imagenet in four GPU hours: A theoretically inspired perspective. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
L. Chizat, E. Oyallon, and F. R. Bach. On lazy training in differentiable programming. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pages 2233,,943, 2019.
11

---

F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion. M. Chiang, P. Mittal, and M. Hein. Robustbench: standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
S. S. Du, D. Lee, Li, Wang. and X. Zhai. Gradient descent finds global minima of deep neural networks. In K. Chaudhuri and R. Salakhutdinov. editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019. Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1675-1685, PMLR, 2019.
S. Fort, G. K. Dziugaite, M. Paul, S. Kharaghani, D. M. R. 0001, and S. Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H.-T. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
M. Geiger, S. Spigler, A. Jacot, and M. Wyart. Disentangling feature and lazy training in deep neural networks. Journal Of Statistical Mechanicx-Theor And Experiment, 2020(1 1): 13301, 2020.
G. Goh. A discussion of "adversarial examples are not bugs, they are features': : Two examples of useful, non-robust features. Distill, 2019. 3.
I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and hamnessing adversarial examples. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
A. Ilyas, L. Engstrom, A. Athalye, and J. Lin. Black-box adversarial attacks with limited queries and information. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässsan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2142-21511 PMLR, 2018.
A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Adversarial examples are not bugs, they are features. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 135-136.-0 2019.
A. Jacot, C. Hongler, and F. Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 8580-8589, 2018.
J. Kim, B.-K. Lee, and Y. M. Ro. Distilling robust and non-robust features in adversarial examples by information bottleneck. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
A. Kurakin, I. J. Goodfellow, and S. Bengio. Adversarial examples in the physical world. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Workshop Track Proceedings, 2017
J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington. Wide Neural Networks of Any Depth Evolve Linear Models Under Gradient Descent. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
C. Liu, L. Zhu, and M. Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 15954-15964. Curran Associates, Inc., 2020.
P. M. Long, Properties of the after kernel. CoRR, abs/2105.10585. 2021.
12

---

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards Deep Learning Models Resistant Adversarial Attacks. In International Conference on Learning Representations, 2018.
P. Nakkiran. Adversarial robustness may be at odds with simplicity. CoRR, abs/1901.005322 2019. 00532.
URL http:://xxxvvv.org/aa//
T. Nguyen, R. Novak, I.. Xiao, and J. Lee. Dataset distillation with infinitely wide convolutional networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
R. Novak, L.. Xiao, J. Hron, J. Lee, A. A. Alemi, J. Sohl-Dickstein, and S. S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
G. Ortiz-iiménez, S. Moosavi-Dezfooli, and P. Frossard. What can linearized neural networks actually say about generalization? CoRR, abs/2106.06770, 2021.
N. Papernot, D. McDaniel, and I. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. CoRR, feds/6605.0777, 2016.
N. Papernot, P. D. McDaniel, I. J. Goodfellow, S. Jha, Z. B. Celik, and A. Swami. Practical black-box attacks against machine learning. In R. Karri, O. Sinanoglu, A. Sadeghi, and X. Yi, editors, Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, AsiaCCS 2017, Abu Dhabi, United Arab Emirates, April 2-6, 2017, pages 506-519, ACM, 2017.
N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. A. Hamprecht, Y. Bengio, and A. C. Courville. On the spectral bias of neural networks. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5301-5310. PMLR, 2019.
V. Shankar, A. Fang. W. Guo, Fridovich-Kenn J. Ragan-Kelley, Schmidt, and B. Recht. Neural kernels without tangents. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8614-8622 PMLR, 13-18 Jul 2020.
C.
Y. Ollivier, L. Bottou, B. Schölkopf, and D. Lopez-Paz, First-order adversarial vulnerability of neural networks and input dimension. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5809-5817. PMLR, 2019.
M. Springer, M. Mitchell, and G. T. Kenyon. A little robustness goes long way: Leveraging robust features for targeted transfer attacks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoor- thi, J. T. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
13

---

C.-H. Yuan and S.-H. Wu. Neural tangent generalization attacks. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12230-12240. PMLR, 18-24 Jul 2021.
H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California. USA, volume 97 of Proceedings of Machine Learning Research, pages 7472-7482. PMLR, 2019.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Sec. 4 and Appendix.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Our work sheds light properties of adversarial examples to make mahcine learning models more reliable in the long run.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
14

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "uIK8UK0BmVn", "reviewer": "Reviewer_vnuA", "summary": "This work generates adversarial examples by using a training-free fashion through NTK. They study the feature through the eigendecomposition of the associated kernel, and confirm that the robust features are corresponding to the top part of the eigenspectrum. They also study the dynamics of empirical NTK during adversarial training.", "questions": "1. It’s unclear what’s the architecture of the neural network the author describes in section 2.3, which should be given math formulas. Does the author focus on two-layer ReLU network since it dominates NTK literature? It should be clear whether the weight w a vector or a matrix; when training the neural network, whether any part of the layer is fixed; how to initialize the neural network The above questions are different throughout different NTK literature.\n2. The author presents the eq(7) for learning with L2 loss, which is described by a regression problem. Then try to transfer to the classification problem by simply applying the softmax function on the top of the kernel output and applying cross-entropy loss. I’m wondering whether the author considers the training dynamics of learning with cross-entropy loss for infinite neural networks.\n3. Given eq(8), does the author also consider any multiple-step adversarial example for the infinitely wide neural network?\n4. The author should describe more on ``kernel generated attack’’, i.e. how to generate adversarial examples from kernel machines. Since the NTK is solely based on model initialization, does the attack only based on one random initialization or an average of multiple initializations?\n5. For the top figure of figure 4, as you increase the epoch, the cosine similarity between NTK and neural network is decreasing. Moreover, it should be helpful if also plots how far the network weight is away from the initialization during training. If the learning rate is very small, then even after 10^5 epoch, there’s possible that the trained neural network is still not far from the initialization, thus lying within the lazy regime. So we should not be surprised by the result. And the author should do a thorough test of how these attacks perform compared with white-box attacks.\n6. Mathematically it's not correct to use $L(f-f_t,y)$ in equation (C.30). In figure C.1 (a), the bottom figure is about epoch vs. robust accuracy, and why the highest accuracy can reach 200%?\n7. Some of the figures need to enlarge the text size, i.e. figure 3, 6. ", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. I find the idea of connecting robust features with the eigenspectrum of NTK and studying the kernel dynamic of standard training vs adversarial training is novel. Some of the experiments are interesting.\n\nWeakness:\n1. The experiment cannot convince me that the black-box attack through NTK has a similar effect as while box attack. The strong alignment of the gradient is likely because the trained neural network is within the lazy regime, yet in reality, the neural network cannot guarantee to be within the lazy regime, i.e. adversarial training, thus making the proposed black-box attack not useful.\n2. The author does not describe the technical detail clearly in the main paper. There are lots of NTK literature under different model architectures, different weight initialization, different data distribution assumptions, etc. Yet I did not find the author considering all the above situations thoroughly. Later, I notice that the author describes the initialization in appendix C, where they consider two-layer ReLU network with $W\\sim N(0,0.01^2I_{m\\times d})$. To me, such initialization is neither practical nor falls into the standard NTK literature. \n3. I have trouble understanding the reason for ``robust features alone is not enough''. The author defers the discussion in the appendix, where CIFAR10 binary FC_4 FC_5 suggests that adding more features may hurt the robust performance. The author however compares it with multi-classification MNIST and concludes that robust features are not sufficient for robust classification. First MNIST is a rather simple unique dataset and it's sparse. It's likely that there're so few robust features for MNIST. Second, it's unfair to compare a binary classification result vs a multi-classification result on different datasets. At least present the same plot for binary MNIST and multi-class CIFAR10.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mp5KI3BhiHr", "reviewer": "Reviewer_Ly8J", "summary": "This work uses the Neural Tangent Kernel (NTK) as a tool to study several aspects of adversarial robustness.\nFirst, it uses the NTK to generate adversarial examples and find they transfer to the trained counterparts.\nSecond, it uses the eigendecomposition of the NTK to define features, which correspond to each eigenvector of the kernel matrix on the training data. It analyzes these features by visualizing them and evaluating their usefulness and robustness. \nLastly, it analyzes the evolution empirical NTK for standard and adversarial training. In each of these experiments, it contrasts its findings with the existing literature on adversarial robustness.", "questions": "- Have you tried visualizing the features by looking at the most/least activating images? Gradients (as visualized in the paper) are useful, but it's common to look at a few other visualizations\n- Can you provide more details on how exactly you computed usefulness and robustness for the NTK features? I could not find them in the Appendix.\n- Have you tried computing the robustness of the eigen-features arising from the empirical NTK? \n- For the polar angle plots, can you add a legend for the color (epoch?)?", "limitations": "None other than those mentioned above.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Using the NTK to study adversarial robustness is a new contribution (I was surprised there were no prior works, except for some overlap).\n- Each of the experiments are quite interesting on their own, and some of them appear to be novel. I particularly liked the eigendecomposition of features.\n\nWeaknesses\n- Overall, the main issue is that none of the experiments quite go deep enough, and overall I'm not sure what I learned from the paper. Authors argue that their results unify all the known existing phenomena. Surely, while all of their results are *consistent* with what is known in adversarial robustness and NTK, I'm not sure there was something new learned.\nMore details:\n  - The adversarial transfer experiment is interesting, but pretty much expected given earlier works that empirically evaluated to the similarity of NTK predictions and their finite-width counterparts (e.g., https://arxiv.org/abs/1912.02803). There also appears to be prior (unpublished) work on this exact experiment: https://openreview.net/forum?id=M5hiCgL7qt. What would have been interesting to study is how transferability varies as your training regime deviates from the NTK regime; without this study, I feel that the claims about the \"black box\" nature of this attack is a bit overclaimed.\n  - The third investigation (evolution of empirical NTK) seems to make a couple of leap of faiths (maybe they are better justified but in that case you should make them clearer). In particular, it is not obvious at all why the top eigenspectrum should correspond to more robust features for the empirical NTK and particularly for adv training (as this was only shown for the exact NTK and std training). I wonder why the authors did not try computing this explicitly (see Questions).\n  - I liked the E.1 Linearized Adversarial Training experiment, but wish the authors had developed this further.\n- Second issue I have with the paper is the style of writing. It is polished, but sentences are overly long/flamboyant and not to the point.\n\nDespite these drawbacks, I appreciate the authors' breath of empirical investigations (including those in the app).\nOverall, the paper would benefit from having a more focused thesis/hypothesis and tailoring the experiments to that thesis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Z6nI4_H-H2x", "reviewer": "Reviewer_R7n6", "summary": "The paper applies the theory of Neural Tangent Kernel to explain insightful properties of adversarial examples.", "questions": "I would like to clarify the following minor comments\n- line 68: \"only require ... architecture\" and training data as well, isn't it?\n- line 239: \"NTK ... suggest that these two scenarios are no so distinct\" The scenarios are very distinct, the adversarial perturbations as well, the required average distortion for hacking images is bigger in BB than in WB. Everything is different!\n- line 243: \"NTK as the most faithful BB attack.\" I disagree. The NTK is another scenario where the attacker has only the architecture and the training data, different from WB scenario where it knows the weights and architecture, different from BB scenario where it knows nothing but has a free access.\n- Fig. 2 Right. The colored boxes are not explained in the text. I suppose it shows areas of (non)useful/(non)robust features. How did you set the boundaries of these boxes?\n   ", "limitations": "Very good submission I enjoy to read.\nI dislike paragraph L241-245 that I would remove, freeing some space for more explanation on the computations of the \"eigen-images\" for instance.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- Very well written, crystal clear, both text and maths.\n- Very timely since NTK gains more and more momentum\n- Opens the doors to many other discoveries (both about attack and defenses)\n\n# Weaknesses\n- Too few details about the computation of the NTK (I was expecting these in the appendix) and its cost. \n- Very few statements deserve some mitigation (see below) ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Y9HIydMWYqF", "reviewer": "Reviewer_bexg", "summary": "This paper summarizes a novel empirical study on the adversarial robustness of neural networks using a neural tangent kernel perspective (both for finite-width networks and their infinite-width limit). The main new insights of this work are:\n1. The adversarial perturbations of a trained infinite-width NTK match quite significantly those of the same wide-but-finite architecture trained on the same dataset.\n2. For the NTK at initialization, the first eigenfunctions of the kernel tend to correspond to robust and useful features, while latter eigenfunctions mainly correspond to non-robust, but also useful features. The best performance is achieved by leveraging both types of features, and robustness can not be obtained by training only on the subspace of robust features at initialization.\n3. The evolution of the finite-width NTK is much faster for adversarial training than standard training. The authors argue that this evolution is fundamental to yield robustness through feature learning.", "questions": "I would appreciate if the authors could answer my doubts with regards to the technical details that I outlined in Weaknesses 1. I am afraid that these could be hints of potential technical flaws in this work, and hence they are are the main reasons why I am slightly leaning towards rejection. However, If the authors can alleviate my concerns, I will be very happy to significantly increase my score, as I believe this paper could be a nice contribution.\n\n\n**(*Minor*) Fix citations**: I have spotted that several references point out to arXiv papers that have already been accepted to multiplee archival venues. I would encourage the authors to review their citations to make sure they correctly cite each article.", "limitations": "I see no direct negative societal impact stemming from this work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. **Timely and interesting perspective**: The NTK perspective has been the object of intense scrutiny by the generalization community in the past few years. In this regard, this work nicely complements these prior studies by filling a gap in the literature analyzing through the NTK lense different phenomena related to adversarial robustness. This will not only be useful for the robustness community, but also for the deep learning theory community at large, since studying adversarial training through this lense can give complementary information of the general dynamics of deep learning, as this is a very different algorithm. Personally, I find the fact that there are qualitative similarities in the evolution of the NTK between adversarial training and standard training, but also significant quantitative differences in terms of speed and magnitude of the change, quite remarkable.\n\nI also find the study of the robustness of different eigenmodes of the NTK at initialization, and the transferability of adversarial attacks between NTKs and NNs to be very interesting. However, as I point out in the weaknesses, I think there are some fuzzy technical details in these experiments that I would like the authors to clarify.\n\n2. **Balanced and self-contained analysis**: In general, I found this paper very easy to read and quite balanced. The authors touch upon different topics with the right amount of breadth and depth that gives the reader a good amount of new information.\n\n3. **Good contextualization with prior work**: The authors properly discuss a great deal of relevant work, both in the space of deep learning theory and adversarial robustness. I believe this will makes this paper easier to follow by both communities.\n\n4. **Opens new research avenues**: As a new empirical study on relevant deep learning phenomena, I significantly appreciate that this work gives a clear path forward for future theoretical and practical studies. In particular, I believe that the analysis of the after-kernel of adversarially trained networks could yield significant insights for robustness, while the study of the accelerating effect on the kernel evolution of adversarial training could also help understand the rich regime of deep learning as a whole.\n\n## Weaknesses\n\n1. **Fuzzy technical details**: I find some technical details of this work a bit hard to follow and somewhat suspicious of possible flaws in the evaluation. Specifically, I am referring to:\n- **Number of epochs of training in Fig. 4 and corresponding robustness**: The x-axis in Fig. 4 is in logarithmic scale and shows the evolution of the robustness of a neural network 100,000 epochs! This is (i) an oddly high number, and (ii) surprising for the fact that the robust accuracy is very high at the \"beginning of training\" (100 epochs). This does not match the previously reported evolution of the robustness of neural networks in standard training, and hence makes me seriously doubt of the validity of this experiment.  \n- **Unclear usefulness metric**: I find unclear which precise metrics are used in Fig. 2 (right). In sec. 2.2. the authors introduce two metrics for usefulness and robustness in Eq. (3) and Eq. (4) but then later mention that they use clean accuracy and robust accuracy of the different eigenmodes as metrics. Knowing which of these two alternatives the authors use to plot Fig. 2 (right) is very important to contextualize the reported numbers.\n\n2. **Weak results on transferability/black-box attacks**: Although I understand the reasons behind selling their transferability results as a black-box attack, I honestly believe that the provided evaluation (even despite my previous comments) is not enough to claim this technique as a useful adversarial black-box attack. The context of this experiment is rather limited, and has only studied the transferability between very wide networks and the NTK. These two models have been shown before to share very similar dynamics, and in this regard, it is not surprising that their adversarial attacks transfer between each other. This section would have been much stronger if it had also studied the transferability of adversarial attacks between linearized neural networks (i.e, finite-width NTK) and standard neural networks. If the transferability is high in that setting as well, then claiming this as a promising black-box attack technique would not be overclaiming.\n\n3. **(*Minor*) Figure design**: Some graphic design aspects of the figures could be improved to help readability of the paper:\n- The colormap in Fig. 3 (left) is far from ideal. The current figure is using a qualitative colormap to show a numerically diverging quantity. I would advice the authors to change it some [perceptually uniform and sequential colormap](https://matplotlib.org/stable/tutorials/colors/colormaps.html).\n- The font size is very small. It is in general very hard to read the labels and titles of all the figures.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept