The NTK Adversary: An Approach to Adversarial Attacks without any Model Access

1 INTRODUCTION . Adversarial examples , a term coined by Szegedy et al . ( 2014 ) , are inputs of a machine learning model that are carefully crafted so that the model is mistaken on them . Their existence , together with the empirically verified susceptibility of many classifiers to them ( Szegedy et al. , 2014 ; Carlini & Wagner , 2017 ) , imply an important security headache for modern machine learning . The ultimate pursuit for robust and immune models calls for an extensive understanding of the different versions of adversarial examples that a model may encounter . In most cases , an adversarial perturbation ( an instance of small non random noise ) is added to training or test data to form an adversarial example . This procedure is called an adversarial attack . There has been a large body of work on designing such attacks . The most successful attacks exploit model information through its gradient with respect to the input . By perturbing the data in directions of the largest change in loss function , they are able to drop the performance of naive classifiers to almost zero ( Goodfellow et al. , 2015 ; Moosavi-Dezfooli et al. , 2016 ; Carlini & Wagner , 2017 ; Kurakin et al. , 2017 ; Madry et al. , 2018 ) . While successful , these gradient-derived attacks require access to the parameters of the trained model - which might be unrealistic for practical applications . To relax this requirement , so called black-box attacks only query the model . Observation of the model outputs allows to estimate the aforementioned gradients either by training substitute models ( Papernot et al. , 2016 ; 2017 ; Liu et al. , 2017 ) or by finite-difference methods ( Chen et al. , 2017 ; Ilyas et al. , 2018 ) . Nonetheless , to some extent even these strategies require access to the model . Our contribution : In this work , we leverage recent advances in the theoretical understanding of trained neural networks to devise adversarial data perturbations that , surprisingly , require neither access to the model nor training of a substitute model , and do not use any model information whatsoever , apart from the model architecture . We thus reveal an interesting phenomenon : the susceptibility of trained neural networks to adversarial examples does not depend on their specific weights ; instead these blind spots may be calculated a-priori , prior to initialization , even before the training starts . Although this is already known to some extent from prior work that trains substitute models , we demonstrate that this notion holds from first principles that only require the a-priori description of the model . Our powerful , highly agnostic attack is derived from Neural Tangent Ker- nel theory ( Jacot et al. , 2018 ; Lee et al. , 2019 ; Arora et al. , 2019b ) , which allows to describe the dynamics of the gradient-descent based training procedure of a neural net as a kernel regression , with an architecture- and data-dependent ( but parameter-agnostic ) kernel ( NTK ) . At a high-level of abstraction , our approach exploits the NTK description of the function the model has learned at the end of training , and proceeds to maximally perturb this function . To illustrate more precisely , let ( x , y ) ∈ Rd × { −1 , +1 } be an input sample , η ∈ Rd a small perturbation , and f : Rd → R denote the model after training has converged . Then , as we show , in the NTK regime : f ( x+ η ) ≈ f ( x ) + ηT z , ( 1 ) for some z ∈ Rd that depends on the training data and the NTK kernel only . Although the above is derived for regression models , we show that this simple expression naturally leads to a family of adversarial attacks on classifiers , depending on the choice of constraints1 imposed on η . We also cover the general case of multiclass classifiers , which is similar in spirit to Eq . ( 1 ) . As our experiments on MNIST and CIFAR suggest , we are able to confuse classifiers with a rate that in some cases approaches current benchmarks that require model access . In summary , in this work we initiate the study of “ NTK-derived ” adversarial perturbations and make the following contributions : 1 . We derive expressions for the output of a converged model on adversarial inputs , based on the theory of the Neural Tangent Kernel ( Sec . 3 ) . 2 . We propose a family of simple attacks that exploit the aforementioned description . These attacks either depend on analytical expressions afforded by the NTK or on finite approximations of the true quantities ( “ empirical ” NTK ) , obtained by initialized replicas of the attacked models ( Sec . 3 ) . 3 . We experimentally verify the efficacy of our approaches on a simple wide single hidden layer model that operates in the NTK regime ( Arora et al. , 2019a ) ( Sec . 4.1 ) and show that it favorably compares to the strongest benchmark ( PGD attack ) , without any access to model weights . 4 . Further , we demonstrate that our attack works in practical settings : fully connected and convolutional , undefended , networks , trained in standard ways ( cross entropy minimization , stochastic gradient descent , default initialization , etc ) are quite susceptible to NTK noise . We show this both in the case where the noise is generated from the NTK of the model itself , even though the assumptions of NTK theory do not necessarily hold ( Sec . 4.2 ) , as well as for transfer attacks with adversarial perturbations generated by models within the NTK regime ( Sec . 4.3 ) . 2 BACKGROUND AND RELATED WORK . Our work bridges two as yet disparate areas : generation of adversarial examples for trained neural networks at inference time and the description of the dynamics of neural network training in the infinite width limit , with the Neural Tangent Kernel . Adversarial Examples . Let f be a classifier , x be an input ( e.g . a natural image ) and y its true label ( e.g . the image class ) . Then , given that f is an accurate classifier on x , we define x̃ to be an adversarial example ( Szegedy et al. , 2014 ) for f if the following two hold : 1. d ( x , x̃ ) is small for some distance function d. Common choices in computer vision applications are the ℓp norms . 2. f ( x̃ ) ̸= y . That is , the perturbed input is being misclassified . There are many approaches on how to generate such examples and they are often classified into two large areas : white-box and black-box attacks2 . White-box attacks assume perfect knowledge of the 1While we focus on ℓ∞-attacks in this work , we remark that the introduced framework is much more general and the adoption of other similarity measures is equivalent to optimizing Eq . ( 1 ) ( a linear objective ) with the desired constraint on η . 2While we use the word “ attack ” multiple times , it must be emphasized that the existence of adversarial examples and the search for robust models are not security-only concerns . They align very well with concepts , such as interpretability and human-like perception ( Tsipras et al. , 2019 ; Ilyas et al. , 2019 ) . model under-attack and they mainly rely on optimization procedures to maximize the “ confusion ” of the model ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Moosavi-Dezfooli et al. , 2016 ; Kurakin et al. , 2017 ; Carlini & Wagner , 2017 ; Madry et al. , 2018 ) . The most widely adopted method is called Projected Gradient Descent ( Kurakin et al. , 2017 ; Madry et al. , 2018 ) and performs multiple gradient steps on the input space to discover adversarial examples . Formally , if L ( f ( x ) , y ) denotes a loss function , then , in the context of ℓ∞ adversaries , setting x0 = x the update formula for PGD is : xt+1 = Πx+S ( xt + α · sgn ( ∇xtL ( f ( xt ) , y ) ) , t = 0 , . . . , N − 1 , ( 2 ) where Πx+S ( · ) projects the input inside the allowed ℓ∞ ball , and α is a gradient step - hyperparameter . xN is the final adversarial example . Such attacks are the most powerful to date . Black-box attacks , on the other hand , are not allowed to estimate the aforementioned gradient directly , but rather are restricted to querying the model without accessing the parameters . A lot of these attacks proceed by collecting pairs of inputs and predictions from the model , then train another classifier f̂ on them , attack f̂ with white-box attacks and finally use those generated adversarial samples to attack f ( Papernot et al. , 2016 ; 2017 ; Liu et al. , 2017 ) . The empirical success of such strategies relies on the phenomenon of transferability of adversarial examples ( Szegedy et al. , 2014 ; Papernot et al. , 2016 ; Tramèr et al. , 2017 ; Liu et al. , 2017 ) : it has been observed that adversarial inputs designed for successfully fooling one classifier , manage to also fool others . A related line of research abandons the need for creating substitute models , at the cost of allowing multiple curated queries to the model to infer the loss function ( Chen et al. , 2017 ; Ilyas et al. , 2018 ) . Finally , hard-label black box attacks have been considered in the literature , in which an attacker observes only the prediction of f ( Brendel et al. , 2018 ; Cheng et al. , 2019 ) . The adversarial attack we propose does not fall into any of these categories , since it does not query the target model at all . More related to the scope of this work is the so-called “ no-box ” setting ( Chen et al. , 2017 ; Bose et al. , 2020 ; Li et al. , 2020 ) , where an adversary is prohibited from querying the model under-attack . However , to the best of our knowledge , practical attacks under this threat model do train substitute neural networks , making them incomparable to our work . Neural Tangent Kernel ( NTK ) . NTK theory plays an increasing role in the quest to theoretically understand deep learning and the elusive concepts of optimization and generalization in overparametrized networks . It connects deep learning to Gaussian Processes and kernels . The seminal work of Neal ( 1996 ) considered the infinite width limit of neural networks with one hidden layer and showed that at initialization the network is a function drawn from a Gaussian Process with an associated kernel that depends on the activation function . Recently , this limit ( and non asymptotic versions of it ) has been revisited either for deep random networks where only the last layer is being trained ( Lee et al . ( 2018 ) ; de G. Matthews et al . ( 2018 ) ; Novak et al . ( 2019 ) ) or fully trained networks with gradient descent ( Jacot et al. , 2018 ; Lee et al. , 2019 ; Arora et al. , 2019b ) . In the latter case , Jacot et al . ( 2018 ) identified a particular quantity that governs the dynamics of the training procedure of a neural network with randomly initialized parameters θ computing an output f ( x ; θ ) : Rd → R , which they called Neural Tangent Kernel . The NTK is defined as the expectation of the inner product between the gradients of the function with respect to the random initialization of its parameters θ evaluated on all inputs xi , xj , i.e. , h ( xi , xj ) = Eθ∇θf ( xi ; θ ) T∇θf ( xj ; θ ) . ( 3 ) While first studied in fully connected networks , it is possible to define neural tangent kernels for any sane architecture ( Yang ( 2020 ) ) . In the infinite width limit3 ( and under appropriate initialization ) , the training dynamics of the network with infinitesimal learning rate can be described by a differential equation with a constant NTK . Moreover , crucially , in the case of squared loss this gives rise to a closed-form description of the training dynamics with the NTK . For ease of presentation we restrict to the case where the expected network output at initialization is zero ; then the expected output of the network at convergence of training becomes ( Lee et al. , 2018 ) : f∞ ( X̃ ) = H ( X̃ , X ) H ( X , X ) −1Y , ( 4 ) where X ∈ Rn×d is a collection of n d-dimensional training data , X̃ ∈ Rn′×d is a test set and H : Rn1×d × Rn2×d → Rn1×n2 is the NTK ( for fixed input datasets , H is a matrix ) . For some 3for non asymptotic statements see Arora et al . ( 2019b ) . models , this kernel can be computed in closed form analytically by computing the expectation over the initializations . It can also be approximated by an empirical kernel via averaging over a few initialized instances ( Jacot et al. , 2018 ; Lee et al. , 2019 ) . Finally , there has also been work on the intersection of the two areas , though with very distinct emphasis to ours . Recent work ( Bubeck et al . ( 2021 ) ; Bartlett et al . ( 2021 ) ) studies the existence of adversarial examples in overparametrized random neural networks , exploiting some local linearity of the functions computed by such networks . The only prior work that leverages NTK theory to derive perturbations ( published almost concurrently with the preparation of our work ) is due to ( Yuan & Wu , 2021 ) , yet with different focus and without our modification to the standard approach to the optimization problem , that generates the perturbations ( see at the end of Sec . 3 ) . That work is concerned with what is coined as generalization attacks : the process of altering the training data distribution to prevent models to generalise on clean data . In contrast , our work analyses trained models to analytically derive their weakness to adversarial perturbations at inference time . Besides that , our work introduces analytical expressions for the perturbed data and the output of the network on them , something that does not follow from the methods of Yuan & Wu ( 2021 ) ( even if extended to the problem we consider ) .

Summary: This paper proposes the use of Neural Tangent Kernel (NTK) to generate transferrable adversarial examples without access to the target model.  The main contributions are:  1. The derivation of adversarial perturbation under NTK setting.  2. Illustration of different attack scenarios using NTK-attack (e.g. no training needed but several initializations, training-data-agnostic, etc).  3. Good transferability performance on MNIST and CIFAR-10 on the tested model compared to PGD attack (Table 4).  ====== Post-rebuttal feedback  I thank the reviewer for providing the responses. However, I don't think they address my major concerns.  Technically, as the reviewer admitted, there is a similar paper published on the same topic (Yuan & Wu, 2021). Although (Yuan & Wu, 2021) focused on poisoning attacks, the fact that poisoning attack is essentially a bilevel optimization means it actual includes the studied inference-attack on NTK as a special case, by treating the target model as fixed and setting the time to infinity in NTK (as the authors also pointed out). In addition, I don't think doing a first-order approximation then solving the approximated problem is a "difference" as the authors' tried to argue; to me, this will lead to a suboptimal solution, not even an innovation.  Contribution-wise; applying adversarial attack on NTK at inference time seems a trivial extension of PGD attack, as other reviewers also mentioned. The considered transfer attack does not have a solid demonstration of why the result is significant and does not compare to SOTA transfer attacks in the same setting, like no-box attack.