Evolution of Neural Tangent Kernels under Benign and Adversarial Training
Authors: Noel Loo
Ramin Hasani
Alexander Amini
Daniela Rus
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Tangent Kernel
Adversarial Training
generalization
robustness

Abstract: Two key challenges facing modern deep learning is mitigating deep networks vulnerability to adversarial attacks, and understanding deep learning's generalization capabilities. Towards the first issue, many defense strategies have been developed, with the most common being Adversarial Training (AT). Towards the second challenge, one of the dominant theories that has emerged is the Neural Tangent Kernel (NTK) -- a characterization of neural network behavior in the infinite-width limit. In this limit, the kernel is frozen and the underlying feature map is fixed. In finite-widths however, there is evidence that feature learning happens at the earlier stages of the training (kernel learning) before a second phase where the kernel remains fixed (lazy training). While prior work has aimed at studying adversarial vulnerability through the lens of the frozen infinite-width NTK, there is no work which studies adversarial robustness of NTK during training.  In this work, we perform an empirical study of the evolution of the NTK under standard and adversarial training, aiming to disambiguate the effect of adversarial training on kernel learning and lazy training. We find under adversarial training, the NTK rapidly converges to a different kernel (and feature map) than standard training. This new kernel provides adversarial robustness, even when non-robust training is performed on top of it. Furthermore, we find that adversarial training on top of a fixed kernel can yield a classifier with $76.1\%$ robust accuracy under PGD attacks with $\varepsilon = 4/255$ on CIFAR-10.

Evolution of Neural Tangent Kernels under Benign and Adversarial Training
Noel Loo, Ramin Hasani, Alexander Amini, Daniela Rus Computer Science and Artificial Intelligence Lab (CSAIL) Massachusetts Institute of Technology (MIT) (1oo, rhasani, amini, rus} @mit.edu
Abstract
Two key challenges facing modern deep learning are mitigating deep networks' vulnerability to adversarial attacks and understanding deep learning's generalization capabilities. Towards the first issue, many defense strategies have been developed, with the most common being Adversarial Training (AT). Towards the second challenge, one of the dominant theories that has emerged is the Neural Tangent Kernel (NTK) - characterization of neural network behavior in the infinite-width limit. In this limit, the kernel is frozen, and the underlying feature map is fixed. In finite widths, however, there is evidence that feature learning happens at the earlier stages of the training (kernel learning) before second phase where the kernel remains fixed (lazy training). While prior work has aimed at studying adversarial vulnerability through the lens of the frozen infinite-width NTK, there is no work that studies the adversarial robustness of the empirical/finite NTK during training. In this work, we perform an empirical study of the evolution of the empirical NTK under standard and adversarial training, aiming to disambiguate the effect of adversarial training on kernel learning and lazy training. We find under adversarial training, the empirical NTK rapidly converges to a different kernel (and feature map) than standard training. This new kernel provides adversarial robustness, even when non-robust training is performed on top of it. Furthermore, adversarial training on top of a fixed kernel can yield a classifier with 76. 1% robust accuracy under PGD attacks with 4/255 on CIFAR-10."
find that
1 Introduction
Modern deep learning, while effective in tackling clean and curated datasets, is often very brittle to domain and distribution shifts [76]. Perhaps the most notorious failure mode of deep learning to domain shifts is adversarial examples [73]: images with small, bounded perturbations which consistently fool state-of-the-en classifiers. While work has been dedicated to mitigating [25, 18, 51], explaining [24, 39, 37], and harnessing [66, 54, 67] this peculiar behavior, the problem still remains largely open, with the state-of-the robust classifiers still falling far behind in benign accuracy compared to standard non-robust networks [18]. Indeed, tackling this idiosynerasy of deep learning seems almost impossible given that the behavior of deep models under benign data and training is still largely unexplained [13, 62, 63], let alone adversarial training.
The community has tried developed numerous theories on better interpreting [77, 27, 43] and understanding deep learning [74, 5, 38, 64, 28, 14, 31, 30]. One emerging theory is the notion of deep learning as kernel learners [48, 21, 1]. That is, deep networks trained with gradient descent learn kernel whose feature map is embedded in the tangent space of the network outputs with
Code is available at bettps//githhin httpp://ggthhu..cmmool/l//////// htt.://...cccyylll////ttrrraaaaa atk_evolution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

respect to its parameters. After a brief phase of kernel learning, neural networks behave similarly to lazy learners, i..., linear and slow in this neural tangent feature map. This two-stage theory of deep learning has been observed in vision networks such as residual networks [32] used in practice and has been the subject of many recent empirical [71, 48, 21, 7] and theoretical works [29, 35, 2]. While these reports have verified this theory for benign training. no work has looked at this property at the intersection of adversarial training. To this end, here, we perform the first empirical study of how the empirical/finite neural tangent kernel evolves over the course of adversarial training. 2 In section 2, we present our experimental setup designed to isolate the effect of adversarial training on the two stage of deep learning under the kernel learner theory: 1) kernel learning and 2) linear fitting. By using this framework, we study and show the following:
1. Similar to standard training, adversarial training results in a distinct kernel which quickly converges in the first few epochs of training (section 3)
2. Adversarial robustness can be inherited from the kernel made from adversarial train- ing, even when the second stage classifier has no access to adversarial examples during training (section 4).
3. Adversarial training is effective on top of the learned kernels given by standard training (in the frozen feature regime), providing a testbed where many of the fixed-feature assumptions present in theoretical works on adversarial training are met, while still providing high practical performance (section 7).
networks with adversarial training contained visually 4. Eigenvectors in the learned NTK of interpretable features, while the initial NTK and benign training NTK do not (section 8).
1.1 Background and Related Works
Neural Tangent Kernel. Explaining the empirical success of deep learning models from a theoretical standpoint is an exciting and active line of research that is still in its infancy. One of the most popular tools for understanding neural network behavior is the neural tangent kernel (NTK) [38, 6, 4]. Under this theory training deep networks with gradient descent in the infinite-width limit corresponds to training training kernel-based classifier, with the corresponding kernel being the NTK defined as KNTK (z, x') = Ee)-(()) (f) ( oof((')) for parameter distribution p(0) and network architecture fo- Analogously, training an infinite width Bayesian neural network corresponds to Gaussian process regression using the NNGP kernel [55, 23, 45, 52, 56, 50]. In both these infinite width limits, the underlying feature space is fixed, determined entirely by the architecture and parameter distribution.
The frozen nature of the kernel stands in contradiction to the ability of deep models to learn useful features and representations. Indeed, it has been shown theoretically, under different limiting assumptions, or with finite-widths, that feature learning does occur, with time-evolving and data- dependent kernel, (x(x,x²) = Ve. fe. (z)" fe. (z'), with dime-dependent network parameters O, [78, 29, 35, 2, 58, 59]. This learned NTK aligns itself with dataset labels to accelerate training and improve generalization. Empirical evidence supports this, with works finding that this meta-kernel evolves quickly within the early stages of network training before stabilizing [48, 21, 7, 71, 40, 8). In this new setting, there exists two stages in deep network training: 1) kernel learning and 2) linear fitting. In the first stage, the kernel rapidly evolves to align with the dataset's s features and labels, while in the second phase, the kernel changes only minimally and the network behaves linearly in the
NTK feature map (xx) Ver fx)(
regime sometimes referred to as lazy training [17, 46).
0'=0;
Adversarial Examples and adversarial training. Adversarial examples present of the most infamous failure modes of deep learning [73]. These are images within bounded pertur- bations of naturally occurring images that consistently fool deep networks over a wide arra of architectures [53]. There is much literature dedicated to designing techniques for securing networks against these attacks/25, 51, 18, 26, 44, 80, 61, 16, 42]. In this work, we focus on adversarial training with iterated projected gradient descent (PGD), which seeks to minimize
Lub E px.ppp)) (aXx'eB.(2) faannsad (x'.y) .))
That is, minimizing the worst-case loss of samples
Throughout this study, we refer to the finite width NTK as the NTK unless otherwise specified
2

---

in small neighborhood around training examples, where the inner maximization is computed during training using iterated PGD 51].
From a theoretical standpoint, much work has been devoted to explaining the causes of adversarial vulnerability and the limitations of adversarial robustness. Popular theories include the notion of there existing "robust and non-robust features" present in data [37, 76], adversarial examples being an outcome of high dimensional geometry [24, 39, 69], an outcome of stochastic gradient descent (SGD) [70, 3]. and many more [14]. Based on the NTK theory, there have been works studying the presence of adversarial examples under the NTK and NNGP kerness [22, 9, 15, 60, 10]. Recently, NTK theory has been used to generate attack methods for deep networks [75, 79]. Adversarial examples have been shown to arise in simplified linear classification settings [68. 37J, which readily transfer to the NTK settings as the NTK classifier is linear in its underlying feature representation. However, these reports on the adversarial vulnerability of NTK and NNGP kernels have been focused solely on the infinite-wice width limit of these kernels, with no literature on the adversarial robustness of the learned and data-dependent NTK that present in practical networks.
2 Experimental setup
Definitions and problem setup. We first define the necessary objects and terminologies for our experimental setup. Our scheme follows closely that of [21]; however, we consider the additional dimension of adversarial robustness in addition to benign accuracy. First, we define the parameter- dependent empirical NTK: KENTK,O (xr, x²) (o)o(z)) ( Vofo(x). In the case where I = O(t), where (t) refers the parameters of the network after training for t epochs, we use the shorthand ke KENTK,00K Next, we define three training dynamics.
1. Standard dynamics as Isunniudd = ft),), that is, network behavior with no modifications.
2. Linearized (also referred to as lazy) dynamics around epoch t: fmm,,. (a) = fo, (x) (0
Vo (x) 0'= L.e., linearized training dynamics corresponds to performing a first-
order Taylor expansion of the network parameters around point 0,. We refer to fa, as the parent network and t as the spawn epoch. Note that in lincarized dynamics, the underlying feature map (zz) = for (x) 0'=0,"
and the corresponding kernel is fixed. Fort et al. [21]
studied this regime and showed that in practice, deep networks training with SGD undergo two stages in training, in which the first phase was chaotic with a rapid changing kemnel until some relatively early epoch f, and the second stage behaves like linear training about 0,.
3. Centered linear training (or centered for short): where we subtract the parent network's output [34, 47, 49]: fcomene), = (0 - 0.) V Vor (1))
l-=00, This corresponds to linear training, with the zeroth order term removed. Now the output is strietly dependent on the difference between 0 and 6g, and the contribution of the first epochs is only through how modifies the feature map. Studying this setting lets isolate the properties of the Iearned kernel kG, without worrying about its contribution from earlier. Efficient implementation of both these linearized versions is done using forward mode differentiation in the JAX library [11].
Experimental design We follow closely that of [21], where we train networks in two distance stages with an added dimension of adversarial training. Additional details are available in appendix B. Stage 1: Standard dynamics. We train Resnet18s on CIFAR- 10 or CIFAR- 100 [41] for & epochs either using benign data (i.e. no data modification), or adversarial training, for 0 < S 100. Stage 2: Linearized dynamics. Following stage 1, we take the parent network from stage and train for an additional 100 epochs with either linearized or centered linearized training dynamics. Note that for centered training, after stage 1, the networks will output entirely zeros, as the zeroth order term has no effect, so the classifier does not have a "warm start." " In this stage, we train using benign data, with no adversarial training. Additionally, we freeze the batchnorm running mean and standard deviation parameters after stage 1, as allowing these to change would implicitly change the feature
map (xx) Vor for (x) - For all experiments in the main text, we use the standard 4/255
adversarial radius under Tioe norm, but we verify that the results hold for = 8//55 in section with additional results for g = 8/255 in the appendix.
3

---

3.30
0.20
CIFAR-100
232
0.30
0.15
132
0.75
100
0.05
90
0.00
0.00
40
80
102
127
200
Benign
CIFAR-10
1.20
Adversarial
2.3
50/50
0.05
1.2
303
1.0
4) Velocity
distance
rark
specielization
Figure 1: Evolution of the Neural Tangent Kernel under benign and adversarial training on Resnet- 18s on CIFAR-I00 (Top row) and CIFAR-I0 (Bottom Row). Networks are either trained for 100 epochs with benign or adversarial training or 50 of benign followed by 50 epochs of adversarial training (50/50). From left to right, we plot the kernel velocity, kernel distance to the final kernel, effective rank, and mean kernel specialization of the resulting kernels. (n=3)
3 Evolution of the NTK under standard and adversarial training
First, look the evolution of the kemell under different training conditions. To do this, calculate the empirical NTK on random subset of 500 glass-balanced training points on CIFAR- 10 and CIFAR- 100 for resnets trained for 100 epochs SGD under standard dynamics with either benign training or adversarial training. We also consider a third scenario where we perform benign training for 50 epochs then change to adversarial training for the remaining 50 epochs an additional control point of the effect of adversarial training. For networks with multiple outputs, the kernel matrix is rank-four tensor in RCCCxNX with C' being the class count and N being the dataset size. The calculation for these entries is given by he.e/ (x, and the resulting subelass kernel matrix Ke,e' E N Unless otherwise stated. for the statistics we present, we calculate the trace kernel, the average of the diagonal elements of the class-specific matrix: K - - Ke,--
have
Vofa (Voo" (a').
Firstly, we look the kernel distance between neighboring epochs. This kernel distance is given by: Tr(KT Ks) [Kill|| IF which equals to 0 iff K1 K2. The kernel velocity is given by finite difference between neighboring epochs (See fig. la). We also consider the kernel distance from the current epoch to the kernel the end of training (fig. 1b). From these metrics we see that both for benign and adversarial training, the kernel converges within 30 epochs to close to the final kernel, accordance with the results Fort et al. [21]. This suggests that after these few epochs, the underlying feature set is fixed, and the remainder of the training is spent performing linear classification on this feature set. Surprisingly, adversarial training also quickly converges, albeit to different kernel whose underlying feature set is more robust (as will see in later sections). the third setting, we observe small spike in kernel velocity at epoch when we swap from standard to adversarial training. The change is small compared to the initial rapid kernel evolution, suggesting that the standard training kernel is more similar to the adversarial kernel than it is to the initial NTK. Likewise, when plot the kernel distance to the final kernel, both models nearly converge after epoch 40, while in the third setting the model stabilizes at the standard training kernel before changing to the adversarial kernel after 50 epochs.
K2), ds which we approximate de
The third metric we compute the effective rank of the kernel matrix [65]. This measures the dispersion of the matrix over its eigenvectors, and is given by: erank(K) = exp(- Σm Pt log ps). A; With p. = F A, with X being the cigenvalues of the kernel matrix K. This value is bounded between when the matrix has one dominant direction and N when eigenvalues are equal. Previous work has shown that deeper networks are biased towards low effective rank (of the conjugate kernel) initialization [36]. Alternatively, the effective rank could be interpreted as the complexity of the dataset under the kernel, and existing work has shown that robust classifiers require more

---

a) Benign Traniing
b) Adversar al Training
gl Linear Component Relative Magnitude
Accurscy
Eunign Thening
Admorumal Thannng
Accuracy
Canooed
Spown apoch ID)
Spownn opoth (t)
Spawin cooch [t]
Figure 2: Performance of standard, linearized, and centered training based on kernels made from benign or standard training on CIFAR-100 (top row) and CIFAR- 10 (bottom row). Solid lines indicate benign accuracy, while dashed lines indicate adversarial accuracy. Under standard or linearized dynamics with benign training (left), networks have little to no robust accuracy, but the networks learn kernels with robust features over time, as centered training gains robustness as the kernel evolves. Centered training also sees robustness gain over adversarial training (center) at the cost of some benign accuracy. For lincarized dynamics, the relative magnitude the first-order component sharply peaks in early epochs before decaying to the spawn network full trains (right).
model capacity [14, 68], although it is unclear the precise relationship between the effective rank notion of complexity and those given in prior work relating to adversarial robustness.
As fourth metrie, consider the mean kernel specialization. As discussed earlier, for multi multi-classed outputs, the full NTK is a 4D tensor in RCxCXNXN with C class-speciff kernels corresponding to the diagonal entries of the first two dimensions. Over training, it has been observed that these
individual class-specific kernels specialize aligning themselves with their classes [71]. The kernel
specialization, defined as: KSM(c,
c- A(K4." u.5 where A(K*5, "iK",,yyy²" yyy
A(K A(K"""...1")))
S(Kee,, vv) ). i.e. the cosine similarity of the kernel matrix and the one-hot class labels. Intuitively. this compares how aligned class e specific kernel is to the labels of class c' compared other class kernels. This quantity is bounded between 0 and C' and higher diagonal entries (when = d), indicate higher specialization. We define the mean kernel specialization as C-IxC c= KSM(e, c). i.e. the average of diagonal entries of the KSM matrix. We calculate this for only CIFAR-10, as computing all 100 class-specific kernels for CIFAR- 100 is too costly. From fig. 1d, where we see that adversarial training is associated with lower mean kernel specialization, adversarial training promotes features that are more broadly shared between classes, although this needs further investigations which will be the focus of our continued effort.
take away that
Performance of Linearized and Centered Training
Next, we look at the performance of linearized and centered training, where we vary the spawn epoch at which we begin stage 2 training, i.e. the epoch / described in section 2. We then plot the benign and adversarial performance of the classifier after stage 2 training, in comparison with the performance at the end of stage training in fig. 2 for CIFAR-10 and CIFAR-100. We show the results when either stage is performed with benign training or adversarial training. Additionally, for the case of linearized training, we plot the relative magnitude of the zeroth order and first-order terms in the linearization dynamics equation. Specifically, let fo = (x) and Aʃ = (0 0,) TVo for (x) e'=
We plot the relative magnitude of the two components, given by AST
f8 averaged over the test set.
The most surprising observation is that centered training gains significant robustness over standard and linearized training dynamics, both in adversarial and benign training. Because centered training does include the zeroth order term in its predictions, all the robustness given by centered training is inherited entirely through the learned NTK, and not through adversarial training (as we perform benign training in stage 2). This lends credence to the "robust feature" hypothesis given in Ilyas et al. [37], however, now what matters is not necessarily what features are present in the data, but
5

---

(CHNAB-30)
Leealied
Cartared
spach
(5)
Figure 3:
Adversarial transferability of standard, linearized and centered networks to fully trained networks with either benign or adversarial training on CIFAR-10 and CIFAR-100. (n=3)
what features the networks learns in its kernel. On the final adversarial kernel, centered training brings about 3.7% adversarial accuracy gain over standard adversarial training on CIFAR-IO, going from 51.77 + 0.80% to 55.46 + 1.79%. Similar improvements are also seen on CIFAR- 100 with an improvement from 24.42 + 0.14% to 27.35 + 0.41. At around epoch 10, the centered network reaches 62.09 = 1.76% robust accuracy on CIFAR- 10, an over 10% performance gain.
Equally surprssing is that even under benign training, the network learns robust kernel over time, with centered training reaching 41 .144 + 9.75% after epoch 100 for CIFAR-10 10 and 14.16 + 1.03 for CIFAR-100. Despite this, the neural network under standard dynamics still sees adversarial accuracy, suggesting that the contribution from the robust learned kernel is minor compared to the contribution of the non-robust kernels, which are present early in training. Roughly speaking, the final network outputs could be written as the integral over time of instantaneous sernel-dependent functions [19]. We can verify that the contribution of the kernel later epochs is small by the relative magnitude plots in fig. 2c, which shows that the first-order additional term of linearized dynamics becomes very minor by epoch 30, meaning that the network has essentially fully fit the dataset, at which point the learned kernel reaches it maximum robustness (as seen by the centered plots). We observe that the learned kernel only gains robustness over time, but its robustness stagnates as kernel velocity slows down. Linearized training sees peak in the relative magnitude of first to zeroth order components around epoch 10-15, coinciding with the small bump in robustness for the benign training kernel under linearized training, and for the adversarial training kernel, peak in accuracy in benign and adversarial settings for linearized and centered training. We conjecture that this bump in relative magnitude correlates with the stage transition between kernel learning and linear fitting, where the kernel is well aligned with the data, but the network has not yet fully fit the training data, causing the first-order component to contain the majority of the data fit.
5 Adversarial Transferability of Linearized Dynamics
We next study the differences between standard and linearized dynamics from the perspective of ad- versarial transferability, specifically, how well do adversarial examples generated with one dynamics transfer to models with other dynamics? From the experiments described in section 2. we have a set of classifiers with standard, linearized, and centered dynamics: {/stan+): (iin, [Jcen.r) for 0 < S 100. For each of these classifiers, we generated adversarial examples on the test set with Loo PGD (details provided in appendix B), {t ). [X ). ( (Xcen.t). To see how these adversarial exam- ples compare to those generated by a network with standard dynamics, we then evaluated the accuracy of /stan, 100 on these adversarial examples: Acc (),
stan,t=100 Bxyy (,, [T( Istan,t= (z0
We then define the adversarial transferability as: T(). = Acc A.c c where
Ace'stan.- A stan.t=1o0
Stan,
1c is the accuracy of the standard dynamics classifier with benign test set. This metric measures the relative effectiveness of adversarial examples generated by another network to adversar- ial examples generated by the network itself. Values of T = 1 mean that the adversarial examples perform just as well as adversarial examples generated by the classifier itself, while 0 would imply that adversarial examples do not transfer whatsoever.
We plot these for CIFAR- 10 and CIFAR- 100 trained Resnet-18s with either benign or adversarial training in fig. Comparing the plots of benign training and adversarial training, we see that in both cases, the adversarial examples of the standard and linearized dynamics transfer similarly well to the full trained network. In contrast, the adversarial examples made by centered training are far less effective, suggesting that the feature set used by the kernel does not necessarily correspond to the

---

Table 1: Performance of stage 2 network training with parent network trained with adversarial training for 10 epochs on CIFAR-10. rj = learning rates. (n=3)
Frozzo Batchnorm
Standard Batchnomn
Purent Network
Centering
- 0.0001
-- 0.01
Centering
? - 0,0001
SGD
1 = 0.01
SGD
Benign Accuraty
78:33 1: 1.07 48.73 1.52
81 73
-.76
82.11 +: 0.68
82.21 0.06
30 1: 9.76
52.46 :: 18:47
43.22 2: 17.72
Adversarial Accuney
62.00 1.76
44.50-1 + 0.75
44.30 1.23
18.45 6.21
20.91 8.60
14.28 + 5.92
Kernel Distance
0=00
0.0140
0.0009
0.0290 0.002
0.0012 0.0001
0.0228 0.0051
0.0024
0.0151
same feature set used by a fully trained network. For standard training, the adversarial transferability plots rise much more quickly than for adversarial training, suggesting that features from early kernels used more in the final model than for adversarial training, as seen by the slower initial rise of the standard and linearized training in fig. 3b and fig. 3d. As seen earlier, centered training on earlier NTK kernels provides little robust accuracy, suggesting that some of the lack of robustness of fully trained networks could be due to the contribution of these early kernels. For standard training, the adversarial transferability of fully trained networks rises sharply in the first 15 epochs, then slowly decays over time. This suggests that the adversarial behavior of the fully trained network is closer to some of the earlier kernels than the final kernel. However, in both cases, adversarial examples generated by the initial NTK do not transfer at all to fully trained networks, suggesting that the initialization NTK analysis of adversarial examples in some theoretical work [22, 9] may not be accurately describe the behavior of adversarial examples in practical networks.
6 SGD vs Linearized Dynamics at low learning rates
In section 4, we showed that adversarial robustness could arise from linearized and centered training, in which the underlying feature set is fixed. In this section, we investigate whether freezing the kernel is necessary for robustness gains. To investigate this, we trained networks with adversarial training with standard dynamics until epoch 10. We then performed centered training on top of the resulting kernel. We chose ten epochs as it corresponds to the large jump in robust accuracy in fig. 2. The centered network serves as the control when the feature set is frozen. Then, we varied stage 2 training in two ways. First, we trained network using "centered standard" dynamics, where Scen, stan,o(*) = fo(x) - fo, (x)), that is, we subtract the network from its initialization's 's output, and train with standard gradient descent. We use learning rates of either 1 0.0001 or 1) = 0.01, corresponding to low and high learning rates. Previously, we had argued that freezing the batchnorm
running mean and standard deviation is necessary so that the feature map Vor for (I)) 0'=0,
does not
change. Here, we question this assumption and either keep the batchnorm parameters frozen or let them change.
Table I shows the resulting benign and adversarial accuracies on CIFAR-10, as well as the kernel distance between the final network's 's kernel and the kernel after stage 1. Additional results for different spawn epochs and for CIFAR- 100 reported in appendix C. We see that in the frozen batchnorm setting, both SGD variants lose robust accuracy. SGD does not freeze the kernel, we also that the kernel changes too (as seen from the kernel distance), with SGD with a larger learning rate seeing a larger change. Letting the batchnorm parameters change results in low adversarial and benign accuracy for all training models, with correspondingly higher kernel distances. This interpretation of freezing batchnorm parameters to preserve the NTTK provides a different perspective as to why more sophisticated batchnorm schemes improve performance in few-shot learning [12].
Adversarial Training on a Frozen NTK
In fig. 1, we observed that adversarial robustness could be inherited from kernel in stage of training,
even if stage 2 was performed on benign data. We now investigate whether we can gain robustness by performing adversarial training in stage 2 on top of frozen kernel. Much theoretical analysis of adversarial robustness has been in the frozen kernel regime or linear classification [22, 9, 60, 37, 68], as analysis of neural network training simplifies when the underlying feature set does not change.
For this experiment, we ran adversarial training using centered dynamics for three kernels. First, we used the initialization kernel, that is, Kı=o- Then we used the post-training kernels for networks
7

---

Table 2:
Centered network performance with benign or adversarial training in stage 2. (n=3). The left column refers the base kernel used to perform centered dynamics training with. The top row refers to whether benign or adversarial training was done using this kernel. The second row refers to whether adversarial attacks were used during testing.
Kernel
Benign Accuracy
Benign
Adversarial Accuracy
Benign Accuracy
Adversorial
Adversarial Accuracy
Standard Adversarial Training
(SGD, No Kernel)
81.58 + 0.63
51.77 0.80
Initialization Kernel
Ki_.
34.11:: 1,35
0.00 +: 0.00
10:46 4: 0.66
6.27 1: 4.45
Benign Training
86.06 0.75
41.41 9.75
84.85 0.65
76.15 1.91
beninn Adversarial Training
___ adv
79:65 1: 01.30
55. 1.79
61.25 1 0.57
57. .0 1.56
Initialization NTK
Benign Training NTK
Adversarial Training NTK
max cos(Vr, ((x()
min cos(v,, ((x))
Figure 4: Visualizations of the top eigenvectors of the initial NTK, benign training NTK, and adversarial training NTK. We either maximize the cosine similarity (top row) or minimize it (bottom).
trained either using benign training or adversarial. As baseline, compare the accuracies achieved by adversarial training under standard dynamics. These results are shown in table 2, with additional results on CIFAR-100 in appendix D)
The initialization kernel fails to learn the dataset under adversarial training, achieving near-random benign accuracy and no meaningful adversarial accuracy. Adversarial training a very limited gain in robust accuracy, improving only 2%, and most surprisingly. the benign training kernel sees a dramatic robustness gain, achieving 76.15%, surpassing the robustness of all other methods described in this paper. Similar results are reported in appendix D for CIFAR- 100, with the adversarial kernel seeing little gain and the benign kernel surpassing the robustness of all other methods. While do not have a clear explanation as to why we see such a dramatic improvement, we do have a few speculative hypotheses. Firstly, it has been argued that fitting global structure before fitting local structure can improve robust accuracy [16, 61]. Under this view, we can consider stage training as learning global structure by choosing task predictive features and stage 2 as fitting local structure by performing adversarial training. Secondly, separating the two tasks can reduce some of the instability caused by adversarial training [26. 72], as only the linear weights changed in stage 2.
From a theoretical perspective, adversarial training in linear classification far easier to analyze, and thus this regime has been the interest of several papers [68, 22, 9]. However, in practice, these analyses have not been very practically useful as neural network behavior needs to deviate far from the linear ideal in order to achieve high standard accuracy [47]. As we have shown here, linearized training can achieve high benign and robust accuracy, provided that we use the learned NTK as opposed to the initial one. This linearized regime can thus serve as middle ground where many of the analysis tools of adversarial training are applicable while still retaining high practical performance.
8 Visualizing NTK eigenvectors
In the lazy dynamics regime, the resulting classifier is linear in its features given by x(x) Furthermore, the effective classifier weights (0 0,) lie in the span of the space
Ver for (2) 1='=0,
of the features of the training set: J = [0(z1), (())))),( ((=2)...((wN)I E NxP, with P the number of parameters. The top eigenvectors of the NTK = JJT correspond to the dominant features of the dataset in the NTK feature space. While much work has looked at the properties of the NTK eigenvectors/value at initialization to understand neural network generalization or how the eigenspace evolves over training, here we visualize these eigenvectors directly. It has been shown that not only are adversarial examples generated by robustly trained networks more visually interpretable but also that they make reasonable generative models by directly maximizing the class logits log p(y (z))
8

---

CIFR-110 +=8255
0.3
Berign
Adversarial
150
100
3.0
soecializziion
<) Serign Training
Training
Lnner Component Magntude
Accuracy (N)
Dau Triinn
Acversataa Tha
Figure 5: Evolution of the Neural Tangent Kernel under benign and adversarial training on Resnet- 18s on CIFAR-10 with a larger attack radius of = 8/255 (Top row), as well as the performance of standard, linearized and centered dynamics.
w.r. to the inputs [20]. By visualizing the cigenvectors of the network directly, we disambiguate whether this property is caused by the linear on top of the eigenvectors or directly from the feature space caused by adversarial training.
To visualize these eigenvectors, we describe the following algorithm. First, we calculate the feature space representation of the top eigenvectors. This corresponds to the first columns of V in the singular value decomposition of the training set features: J UEVI, Typically P 10 and N ~ 10 so calculating this SVD directly is prohibitive. Instead, we note that EV" = U J, so the ith eigenvector V; x J u; (because Σ is a rectangular diagonal matrix), where u, the ith eigenvector of K. We can calculate K U UAA since N << P. Calculating J directly is too expensive, instead, we note that J sum of the network on training samples, with respect to the parameters.
a as (u. fo(X)), i.e., we take derivatives of weighted
Now with the feature space representation of eigenvectors vú, we visualize them by optimizing to optimizing cos, d(x;)), giving us 2.V images, with the factor arising from whether maximize or minimizing the cosine similarity. As this algorithm describes how to visualize eigenvectors for only scalar functions f. we chose the dog class of the neural network as the chosen function to visualize, however, in practice, we found that eigenvectors for other classes looked very similar. As before, computing K is too expensive for the full dataset, so we pick subset of 500 images from the training set to calculate the NTK.
The visualizations of the top 4 eigenvectors of the kernels generated for the initial NTK, the kernel after 100 epochs benign training, and the kernel after 100 epochs of adversarial training are visualized in fig. The initial NTK has visually meaningless eigenvectors which look indistinguishable from pure noise. Meanwhile, the eigenvectors after benign training seem to distinguish the edges of CIFAR- 10 classes but still largely uninterpretable. Finally, the adversarial kernel results the eigenvectors that clearly belong to different classes. This observed difference between NTK eigenvectors of the adversarial and benign NTKs provides support the "feature purification" mechanism described in Allen -hu and Li [3], in which they show that ad udversarially trained networks provably learn robust features.
Higher regime
In the previous sections, showed various properties of benign and adversarial finite NTKs under the = = 4/255 adversarial budget. However, it known that the behavior adversarial training can vary greatly depending on In this section, we verify that the results observed for 4/255 hold for = 8/ /555. We repeat all experiments with ε = 8/ 8/255. fig. 5 shows the resulting kernel evolution plots for CIFAR-10, and the resulting performance of standard, linearized, and centered training dynamics.
Consistent with our previous observations, we notice that centered dynamics an increase of robust accuracy over standard dynamics (49.15 + 3.49% vs. 35.80 + 0.76%) and that 32.29 8.36 robust accuracy be achieved from centered training the benign kernel. We observe the robust

---

overfitting phenomenon, where the best adversarial accuracy occurs earlier in training. Comparing fig. 1 to fig. 5, we observe that there slightly larger kernel distance between the benign kernel and the adversarial one with larger e values and also that the adversarial kernel takes longer to converge with larger E. This motivates future work to look at how different & values affect the stability and convergence of the finite NTK. Additionally, in appendix C and appendix D, we show that the results in section and section 7 also hold for e = 8/255.
10 Discussions, limitations and Conclusions
The experiments in this paper showed that adversarial training results in a distinct and measurable change in the NTK compared to standard training. While under both training regimes, the NTK converges quickly in the first few epochs of training, the adversarial NTK appears to have a high effective rank and lower mean kernel specialization compared to the standard training NTK. Visually, the top eigenvectors in the adversarial NTK correspond to much more visually interpretable images compared to the standard NTK and the initialization NTK.
Furthermore, showed that classifiers built top of the adversarial Kernel retain significant amounts of adversarial accuracy, despite being trained without ever seeing adversarial examples. The initial NTK results in classifiers with no adversarial robustness, and surprisingly, the standard training NTK sees a dramatic increase in adversarial robustness over time. We additionally demonstrated that frozen feature set is necessary for retaining the full robustness aspect of these kernels, as either allowing the batchnorm parameters to vary or using SGD as opposed to linearized training in the second stage results in a drop in robust accuracy.
Our results strongly motivate further research utilizing the adversarial properties of linearized training observed as defense mechanism for adversarial attacks. Furthermore, note that lincarized training costs approximately double the cost of benign training, however, adversarial training is significantly more expensive. As shown in fig. one perform "kernel early-stopping," " where we switch to linearized training on clean data so that the remainder of training can be done without adversarial training. This observation certainly motivates more efficient way gain adversarial robustness in practice, which future work can investigate further.
Limitations. We performed our analysis on robustness to an Lox PGD adversary, which is an uncertified attack but was designed to fool networks under standard dynamics and is unproven against networks under centered or linearized dynamics. More advanced state-of-the-a art adversarial defense mechanisms were not studied in this paper, nor was the optimal tuning for hyperparameters for adversarial lincarized training. Furthermore, while presented many interesting empirical insights about adversarial training on kernels, there is a great opportunity for developing the theory of the observed interplay of adversarial training and NTK evolution.
Previous research had to impose limiting assumptions, e.g., fixed kernels on NTKs due to the nonlinear nature of neural nets [68, 22, 9]. We hope that our study of adversarial training in the frozen kernel regime with learned kernels could bridge the gap between these two extremes by having simple linear dynamics but still matching exceeding the performance of standard networks. As shown in section 7, the performance of adversarial training in the frozen feature regime can significantly exceed that of standard dynamics. Moreover, based on observations, believe that centered and lincarized dynamics can be used as a defense against adversarial examples.
Acknowledgments
This research has been funded in part by the Office of Naval Research Grant Number Grant N00014- 18-1-2830, DSTA Singapore, and the J. P. Morgan AI Research program.
References
[1] L. Aitchison, A. X. Yang, and S. W. Ober. Deep kernel processes, 2021. URL https: // //openreview www//eeruurnn "errumidd-Zuvs) __Kxq..
[2] K. Aitken and G. Gur-Ari. On the asymptoties of wide networks with polynomial activations. ArXiv, abs/2006. Abs/2006.06687,, 2020.
10

---

[3] Z. Allel---nDi Zhu and Y. Li. Feature purification: How adversarial training performs robust deep learning. 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS). pages 977-988, 2022.
[4] Z. Allen-Zhu. Y. Li. and Z. Song. A convergence theory for deep learning via over- parameterization. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learn- ing Research, pages 242-252. PMLR, 09-15 Jun 2019. URL https: httpss://pooee/ngpe/// mlr. press/ wwwlennzhullla html.
[5] S. Arora, R. Ge. B. Neyshabur, and Y. Zhang, Stronger generalization bounds for deep nets via a compression approach. In ICML, 2018.
[6] S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, pages 8141-8150. Curran Associates, Inc., 2019.
[7] A. Atanasov, B. Bordelon, and Pehlevan. Neural networks as kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2022. URL
ftttp:://opereviie wwt/neeetoooeeee wtt.rooommmnnididiiiiid 1dd1NiflqAooom
[8] A. Baratin, T. George, C. Laurent, R. D. Hjelm, G. Lajoie, P. Vincent, and S. Lacoste-Julien. Implicit regularization via neural feature alignment. AISTATS, 2021.
[9] P. Bartlett, S. Bubeck, and Y. Cherapanamjeri. Adversarial examples in multi-layer random reLU networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL id=BKpNqR19JgD.
htl.pperrrviee.eeeefrrnnr net/forum?
[10] A. Blaas, A. Patané, L. Laurenti, L. Cardelli, M. Z. Kwiatkowska, and S. J. Roberts. Adversarial robustness guarantees for classification with gaussian processes. In AISTATS, 2020.
[11] J. Braddbury, R. Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke,
J. VanderPlas, S. Wanderman-Milng and Q. Zhang. JAX: composable transformations of Python+NummPy programs, 2018. URL http: //gt::////hhbbbb com/ google/jax.
[12] J. Bronskill, J. Gordon, J. Requeima, S. Nowozin, and R. Turner. Tasknorm: Rethinking batch normalization for meta-learning. In -, 03 2020.
[13] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[14] S. Bubeck and M. Sellke. A universal law of robustness via isoperimetry. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages pctpc/ap/e//2/////eeee sociates, Inc., 2021. URL https: h/wpcceedigg......e//oeee www.:nnww...pppreeee/aaaapcaccee
Curran As-
119702990050023044004446666 Pawpeeea66ddd----p.e.eea
[15] S. Bubeck, Y. Cherapanamjeri, G. Gidel, and R. T. des Combes. A single gradient step finds adversarial examples on random two-layers neural networks. In A. Beygelzimer, Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https: /ttp://///peeeeeeeeeee httt://orrmm/dd-/E111eee-eeee
[16] Y. Carmon, A. Raghunathan, L. Schmidt, J. C. Duchi, and P. S. Liang. Unlabeled data improves adversarial robustness. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: //tpc//eepoo/es/e//// www.oeedinng..euuis.s..oo cc/ppp///////pp// Paper pdf.
[17] I.. Chizat, F. Oyallon, and F. Bach. On lazy training in differentiable programming. In NewrIPS, 2019.
11

---

[18] J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smooth- ing. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1310-1320. PMIR, 09-15 Jun 2019. URL httpss:///pooceeddngggee///////t mlr. press/v99 cohen19c. hhtml19..htm..
[19] P. Domingos. Every model learned by gradient descent is approximately a kernel machine, 2020. URL https://aara wwwpeiiiorr/.ooooooe.o abs/2012. 00152.
[20] L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, B. Tran, and A. Madry. Adversarial robustness prior for learned representations, 2019. URL https:://aaattti http://aawxvv..rgaabaa//aaa wwwp////9//bb// 00945.
[21] S. Fort, G. K. Dziugaite, M. Paul, S. Kharaghani, D. M. Roy, and S. Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In NeurIPS, 2020. URL https: .tmml. paper/2020
neuriee iss. cc/
[22] R. Gao, Т. Cai, H. Li, L. Wang, C. Hsieh, and J. D. Lee, Convergence of adversarial training in overparametrized networks. CoRR, abs/1906.079) 16, 2019. URL http://arxiv..rr aba/ 07916.
[23] A. Garriga-Alonso, I. Aitchison, and C. E. Rasmussen. Deep convolutional networks as shallow gaussian processes. In International Conference on Learning Representations, 2019.
[24] Gilmer, L. Metz, F, Faghri, S. Schoenholz, M. Raghu, M. Wattenberg, and I. Goodfellow. Adversarial spheres. CoRR, 0../18801.02774, 2018. URL http:://rxiii org/ aa//a/bss//10 02774.
[25] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples, 2014. URL buttps://arxivg org/ /ab///b////// 1142..677..
[26] S. Gowal, K. Dvijotham, R. Stanforth, R. Bunel, C. Qin, J. Uesato, R. Arandjelovic, T. A. Mann, and P. Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. CoRR, abs/1810. 12715,, 2018. URL http: // //wxxxxivv.o org/ h/gg//b///// aas///aab/11./////////////////1/ 12715.
[27] S. A. Gruenbacher, M. Lechner, Hasani, D. Rus, T. A. Henzinger, S. A. Smolka, and R. Grosu. Gotube: Scalable statistical verification of comtinuous-deply models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6755-6764, 2022.
[28] S. Grunbacher, R. Hasani, M. Lechner, Cyranka, S. A. Smolka, and Grosu. On the verification of neural odes with stochastic guarantees. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11525-11553 2021.
[29] B. Hanin and M. Nica. Finite depth and width corrections to the neural tangent kernel. In International Conference on Learning Representations, 2020. URL
net/forumiide wwtmmmdddddSddddddedee
[30] R. Hasani, M. Lechner, A. Amini, L. Liebenwein, M. Tschaikowski. G. Teschl, and D. Rus. Closed-fome form continuous-depth models. arXiv preprint arXiv:2106. 2021.
[31] R. Hasani, M. Lechner, A. Amini, D. Rus, and R. Grosu. Liquid time-constant networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7657-7666, 2021.
[32] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[33] Hennigan, T. Cai, T. Norman, and I. Babuschkin. Haiku: Sonnet for JAX, 2020. URL http:// //githutt com/ wwtp/eeeeenn//e//////ed//m//de httddddhhhiiae
[34] W. Hu, Z. Li, and D. Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. International Conference on Learning Represen- tations, 2020. URL htttps://openrevier.eeeee wtt.://ee//oooomiiiiiiieiiiir/e
12

---

[35] J. Huang and H.-T. Yau. Dynamics of deep neural networks and neural tangent hierarchy. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 199 of Proceedings of Machine Learning Research, pages 4542-4551. PMLR, 13-18 Jul 2020. URL. wwt.h:ttps//// pttttps:/pproceeinng.
wall.press//ee /tt1//11////// ww11///haagg211111 .html.
[36] M. Huh, H. Mobahi, R. Zhang, B. Cheung, P. Agrawal, and P. Isola. The low-rank simplicity bias in deep networks. In -, 03 2021.
[37] A. llyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Adversarial examples are not bugs, they are features, 05 2019.
[38] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. casa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: / /ppoceedigns newrips....... cc/ p2ppe pdf.
[39] M. Khoury and D. Hadfield-Menell. On the geometry of adversarial examples. CoRR, abs/1811.00525, 2018. URL http:://aree /ttp:///axxvvv/// org/ /ttbs//81..0000 00525..
[40] D. Kopitkov and V. Indelman. Neural spectrum alignment: Empirical study. In L. Farkaš, P. Masulli, and S. Wermter, editors, Artificial Neural Networks and Machine Learning ICANN 2020, pages 168-179, Cham, 2020. Springer International Publishing.
[41] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, CIFAR, 2009.
[42] M. Lechner, R. Hasani, R. Grosu, D. Rus, and T. A. Henzinger. Adversarial training is not ready for robot learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4140-4147. IEEE, 2021.
[43] M. Lechner, R. Hasani, Z. Babaiee, R. Grosu, D. Rus, T. A. Henzinger, and S. Hochreiter. Entangled residual mappings. arXiv preprint 2rXiv:2206.01261 2022.
[44] M. Lecuyer, V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana. Certified robustness to adversarial examples with differential privacy, 2018. URL https: /ttp/v//avvvv/aa////////a//a/aaa www.:org/gaaaa abs/ /10// 03471.
[45] J. Lee, Y. Bahri, R. Novak, S. Schoenholz, J. Pennington, and J. Sohl-dickstein. Deep neural networks as gaussian processes. In International Conference on Learning Representations, 2018.
[46] J. Lee, L. Xiao, S. S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Sohl-Dickstein. Wide neural networks of any depth evolve as linear models under gradient descent. In NeurIPS, 2019.
[47] J. Lee. S. Schoenholz, J. Pennington, B. Adlam, L. Xiao, R. Novak, and J. Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. In H. Larochelle, M. Ranzato. R. Had- sell. M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15152 Curran Associates, Inc., 2020. URL https: httllhttss:///peee //ppoceedinss.. Paper. .df.
neurips.c courips.cc/ppe httpsccccpppeee/e/ httpppaapre/////ppppppppp http:///c/pppp.../....a///////a.
[48] A. Lewkowyez, Y. Bahri, E. Dyer, J. Sohl-Dickstein, and G. Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism, 03 2020.
[49] M. Long. Properties of the after kernel, 2021. URL 10585.
[50] N. Loo, R. Hasani, A. Amini, and D. Rus. Efficient dataset distillation using random feature approximation. Advances in Neural Information Processing Systems, 2022.
[51] A. Madry, A. Makelov, I. Schmidt, D. Tsipras, and Vladu. Towards deep learning models resistant to adversarial attacks, 2017. URL https: : // //wxivvvvvvv/////////aa////iia org/ http//166.06888
13

---

[52] A. Matthews, J. Hron, M. Rowland, R. E. Turner, and Z. Ghahramani. Gaussian process be- haviour in wide deep neural networks. In International Conference on Learning Representations. 2018.
[53] S. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial perturbations. CORR, abs/1610.08400, 2016.
org/ abs/ /010o///1...... 08401.
[54] L. Muñoz González. B. Biggio. A. Demontis, A. Paudice, V. Wongrassamee, E. C. Lupu, and Roli. Towards Poisoning of Deep Learning Algorithms with Back-Gradics Optimization. page 27-38. Association for Computing Machinery, New York, NY, USA, 2017. ISBN 9781 478352225 URL hwwph:tt///date org/ 21t....1111/11122..111 3140451.
[55] R. M. Neal. Priors for infinite networks (tech. rep. no. crg-tr-94-11. University of Toronto, 1994.
[56] R. Novak, L. Xiao, J. Lee, Y. Bahri, G. Yang, J. Hron, D. A. Abolafia, J. Pennington, and J. Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In International Conference on Learning Representations, 2019.
[57] R. Novak, L. Xiao, J. Hron, Lee, A. A. Alemi, Sohl-Dickstein, and S. S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference Learning Representations, 2020. URL htttt:///// https://giiii //github com/ google/neuran -aagggnnss...
[58] G. Ortiz-Jimenez, S.-M. Moosavi-Dezfooli, and P. Frossard. What linearized neural networks actually say about generalization? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https: / /ttp:///opeeeeeeeeeeeeeeee www.ew.eeeeeeeee.eeee..eeeeeeeee htt.//oommmmdddssse __sf.
[59] J. Paccolat, L. Petrini, M. Geiger, K. Tyloo, and M. Wyart. Geometric compression of invariant manifolds in neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2021 (4):044001, apr 2021. doi: 10.1088/1742 5460/////2.///1..... URL https: /doii 1t.g//1..088/ 1742-5468/abf111
[60] G. D. Palma, B. T. Kiani, and S. Lloyd. Adversarial robustness guarantees for random deep neural networks. In ICML, 2021.
[61] A. Raghunathan, S. M. Xie, F. Yang. J. C. Duchi, and P. Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
[62] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612. 2022.
[63] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175. 2022.
[64] D. A. Roberts, S. Yaida, and B. Hanin. The principles of deep learning theory, 2021. URL https://aast org/ abs//10...2.2/.. 10165.
[65] O. Roy and M. Vetterli. The effective rank: A measure of effective dimensionality. 01 2007.
[66] H. Salman, A. Ilyas, L. Engstrom, A. Kapoor, and A. Madry. Do adversarially robust imagenet models transfer better? In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3533-3545. Curran Associates, Inc., 2020. URL https: htt.::::////reeeede..e.e..r...e www.ggs.euuris..coom cttpiippsccpppapr/222000 /ttp:///5520052222211888820-0000
Paper. pdf.
[67] H. Salman, A. Ilyas, L. Engstrom, S. Vemprala, A. Madry, and A. Kapoor. Unadversarial examples: Designing objects for robust vision. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URI.
https://open /ttp::////peeeiieeeeeee
14

---

[68] L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry. Adversarially robust general- ization requires more data. Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 5019-5031, Red Hook, NY, USA, 2018. Curran Associates Inc.
[69] A. Shafahi, W. R. Huang, C. Studer, S. Feizi, and T. Goldstein. Are adversarial examples inevitable? In International Conference Learning Representations, 2019. URL https: / /openreview net/ferrute weetforunmid=rr1in foroWAAi==11WW
[70] H. Shah, K. Tamuly, A. Raghunathan, P. Jain, and P. Netrapalli. The pittalls of simplicity bias in neural networks. In NeurlPS, 2020. URL https: htt.httts//ppooeediiii //woceeedno/geoeeoeo ctpppppprccpppp////cppppppppppep 2067 -0tttt0-aa..a.a.aaaa..... html.
[71] H. Shan and B. Bordelon. Rapid feature evolution accelerates learning in neural networks. ArXiv, abs/2105. 14301,, 2021.
[72] C. Sitawarin, Chakraborty, and D. Wagner. Improving adversarial robustness through progressive hardening. In 03 2020.
[73] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna. D. Erhan, Goodfellow, and R. Fergus. Intriguing properties of neural networks. -. 12 2013.
[74] N. Tishby and N. Zaslavsky. Deep learning and the information bottleneck principle. CoRR, abs/1503.02206, 2015. URL httt::/ttax/ii../////////
/103//553... 02406.
[75] N. Tsilivis and J. Kempe. The NTK adversary: An approach to adversarial attacks without any model access, 2022. URL
net/forum?id M5hiclgL77
[76] D. Tsipras, S. Santurkar, L.. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. -, 04 2019.
[77] C. Vorbach, R. Hasani, A. Amini, M. Lechner, and D. Rus. Causal navigation by continuous- time neural networks. Advances in Neural Information Processing Systems, 24:12425- 2021.
[78] G. Yang and E. Hu. Feature learning in infinite-width neural networks. In -n-, 11 2020.
[79] C.-H. Yuan and S.-H. Wu. Neural tangent generalization attacks. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12200-12240, PMLR, 18-24 Jul 2021. html.
URL https: ptttts::/proceeeeee /ttp///proeeen...////////// mlr
[80] H. Zhang, Y. Yu, J. Jiao, Xing, L. E. Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 7472-7482. PMLR, 09-15 15 Jun 2019. URL https: //proceedings.
mlr. wwwppprrssssseeeeeeeee html.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] Yes we supported all claims made in the abstract.
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We do include theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen-
tary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "zPBcPKH4S5_", "reviewer": "Reviewer_7GsC", "summary": "This paper summarizes a novel empirical study on the adversarial robustness of neural networks using a neural tangent kernel perspective comparing the dynamics of linearized neural networks and their non-linear counterparts. The main new insights of this work are:\n\n1. The qualitative dynamics of the NTK during adversarial training are very similar as the ones of standard training, albeit the NTKs converge to very different kernels.\n2. Remarkably, even during standard training, the NTK learns some robust features that can be exploited during linearized fine-tuning.\n3. Linearized and centered fine-tuning of a neural network, both adversarially and standardly pretrained, can give an edge in robustness over non-linear finetuning. This does not happen for uncentered fine-tuning.\n4. Adversarial attacks transfer between linearized networks and non-linear networks mostly for the uncentered networks, and after a few epochs of training.\n5. Certain visualizations of the NTK eigenfunctions of an adversarially pretrained network seem semantically more meaningful than the NTK eigenfunctions of a standardly pretrained one.\n", "questions": "I would encourage the authors to comment on the two outlined weaknesses before by providing results on other $\\epsilon$ regimes (mostly on $\\epsilon=8$) and clarifying the reasons for the lower benign accuracy of the centered kernel in Table 2. If the authors did this, I would be quite inclined to increase my score.", "limitations": "I see no direct negative societal impact stemming from this work.\n", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strenghts\n\n1. **Timely and interesting perspective**: The NTK perspective has been the object of intense scrutiny by the generalization community in the past few years. In this regard, this work nicely complements these prior studies by filling a gap in the literature analyzing through the NTK lense different phenomena related to adversarial robustness. This will not only be useful for the robustness community, but also for the deep learning theory community at large, since studying adversarial training through this lense can give complementary information of the general dynamics of deep learning, as this is a very different algorithm. \n\n2. **Remarkable findings**: Personally, I find quite remarkable that benign linearized finetuning of a standardly pretrained NTK can yield some robustness. Also, the fact that you can gain a performance edge by centered and linearized fine-tuning of an adversarially pretrained network is fascinating. This is very significant as it provides very clear new evidence that sometimes the so-called \"rich regime\" can have a detrimental effect in performance as shown in (Ortiz-Jimenez et al. 2021).\n\n3. **Balanced and self-contained analysis**: In general, I found this paper very easy to read and quite balanced. The authors touch upon different topics with the right amount of breadth and depth that gives the reader a good amount of new information.\n\n4. **Opens new research avenues**: As a new empirical study on relevant deep learning phenomena, I significantly appreciate that this work gives a clear path forward for future theoretical and practical studies. In particular, I believe that the analysis of the after-kernel of adversarially trained networks could yield significant insights for robustness, while the study of the benefitial edge from linearized fine-tuning will be of broad interest to the deep learning theory community.\n\n## Weaknesses\n\n1. **Limited evaluation of different $\\epsilon$ regimes**: I honestly believe that this work could easily become much stronger if it provided some additional analysis of the effect of the adversarial budget $\\epsilon$ on its findings. It is widely known in the adversarial robustness community that the properties and dynamics of adversarially trained networks for different $\\epsilon$ can be qualitatively very different. In this regard, I believe that showing the same results for other $\\epsilon$, and in particular for the standard $\\epsilon=8$ regime would be very valuable. For example, if the claimed edge of linearized fine-tuning is true in this regime as well (given the right evaluation), it would suggest a radically new way to improve the robustness of neural networks.\n2. **Suspicious benign accuracy of centered NTK in Table 2.**: It is a bit concerning that the benign accuracy of the centered NTK in Table 2 is lower than the robust accuracy. If this is true, this would point out to some weird geometry of the loss landsape wrt input, akin to the one happening in catastrophic overfitting, which would be very surprising in my humble opinion. Without a further investigation on that front, I would say it is more likely a sign of a technical flaw in the evaluation (which could be very damaging for this work) or a typo. \n3. **Missing citations**: I have spotted a few missing citations of relevant papers that also describe the evolution of the NTK and compare it to the linearize dynamics. They should also be cited.\n\n- G. Ortiz-Jimenez, S. Moosavi-Dezfooli, P. Frossard. \"What can linearized neural networks actually say about generalization?\" NeurIPS 2021.\n- J. Paccolat, L. Petrini, M. Geiger, K. Tyloo, and M. Wyart, “Geometric compression of invariant manifolds in neural nets,” Journal of Statistical Mechanics: Theory and Experiment, no. 4, 2021\n- A. Baratin, T. George, C. Laurent, R. D. Hjelm, G. Lajoie, P. Vincent, and S. Lacoste-Julien,“Implicit regularization via neural feature alignment,” AISTATS 2021.\n- D. Kopitkov and V. Indelman, “Neural spectrum alignment: Empirical study,” in International Conference on Artificial Neural Networks (ICANN), 2020.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "R4uhmkRyNHE", "reviewer": "Reviewer_47zX", "summary": "This paper conducts empirical studies towards neural tangent kernel (NTK) in adversarial training. They figure out that NTK converges to a different kernel in adversarial training compared to clean training, and the new kernel provides adversarial robustness even when further fine-tuning on non-robust training.", "questions": "Besides the weaknesses part, I'm also wondering, is there any analytical formula to describe where the adversarially trained NTK converges (even for simple models)?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n[1] This paper provide many experiments studying the properties of adversarially trained NTK.\n\nWeaknesses:\n\n[1] Some paragraphs need clarifications. For example, in abstract, while most parts describing the robustness of the adversarially trained TNK, the last sentence \"we find that adversarial training on top of a fixed kernel..\" suddenly switch another context. Since there are a lot of things contained in this paper, please make necessary connections among different contexts.\n\n[2] An important missing literature:\n\nAllen-Zhu, Zeyuan, and Yuanzhi Li. \"Feature purification: How adversarial training performs robust deep learning.\" 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS). IEEE, 2022.\n\nAllen-Zhu et. al. (2022) provides theoretical justification on why adversarial training help purify the features. This is quite related to the 4th contribution mentioned in the submission. Please cite this paper and make comparisons or connections.\n\n[3] It would be great if the authors could provide more insights on how people could utilize the properties of adversarially trained NTK in real practice. This paper observes interesting phenomenons for potential theoretical studies, but it could be even better if people can directly utilize the results.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B-KAWFZcR18", "reviewer": "Reviewer_ecap", "summary": "The authors consider the evolution of Neural Tangent Kernels(NTK), characterization of neural networks in the infinite-width limit, under standard and adversarial training. The authors proposes the idea of adversarial robustness of NTK during training. Experiments on CIFAR-10 and CIFAR-100 show that NTK converges to a kernel and feature map. They have increased robustness, compared with standard training, which prevents catastrophic accuracy drop under the adversarial attacks. That property saves even after non-adversarial training.", "questions": "•\tDid you use only one attack, PGD, during experiments?\n•\tIs it possible to use NTK adversarial approach in other domains, for example, text? \n", "limitations": "The limitations are clearly described in the conclusion part of the paper. Authors points related works which partially cover uncovered study features", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n•\tUnusual use of NTK concept for researching adversarial robustness\n•\tProposing and exploring three training dynamics, comparing their performance, training dynamics, and adversarial transferability with a list of custom metrics, considering properties of NTK. Those parts of the paper give a lot of insights about considering approaches\n•\tGreat performance under adversarial attacks on the considered datasets\n\nWeaknesses:\n•\tAuthors consider only attacks, which were designed to fool standard dynamic networks\n•\tThere is no theoretical substantiation of the obtained results\n•\tNumerical experiments only on CIFAR-10 and CIFAR-100\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "3IFuOimxigZ", "reviewer": "Reviewer_36HK", "summary": "This paper conducts an emprical study on the evolution of NTK under benign and adversarial training, which considers the training process into two phases, kernel learning and lazy training. They showed that adversarial training in kernel learning obtains a different kernel from standard training, which is robust even when non-robust training is used on top of it. And adv. training on top of a fixed learned kernel could have a high robust accuracy. ", "questions": "See the Strength and Weakness. ", "limitations": "See the Weakness. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n- The paper is well written and easy to follow.\n- It reveals some new observations through considering two phases in NTK. Particularly, benigh training could produce use kernels for the later robust linear training on top of it. \n\nWeakness\n- My major concern is that despite of the presented the empirical observations, it lacks of any tenative analysis over them. \n- Is ther any practical algorithmic insights based on the obersvations found in this paper? I think as an empirical paper, good insights for algorithms is are important. \n- Does these observations generalize beyond NTK?  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept