The NTK Adversary: An Approach to Adversarial Attacks without any Model Access

1 INTRODUCTION . Adversarial examples , a term coined by Szegedy et al . ( 2014 ) , are inputs of a machine learning model that are carefully crafted so that the model is mistaken on them . Their existence , together with the empirically verified susceptibility of many classifiers to them ( Szegedy et al. , 2014 ; Carlini & Wagner , 2017 ) , imply an important security headache for modern machine learning . The ultimate pursuit for robust and immune models calls for an extensive understanding of the different versions of adversarial examples that a model may encounter . In most cases , an adversarial perturbation ( an instance of small non random noise ) is added to training or test data to form an adversarial example . This procedure is called an adversarial attack . There has been a large body of work on designing such attacks . The most successful attacks exploit model information through its gradient with respect to the input . By perturbing the data in directions of the largest change in loss function , they are able to drop the performance of naive classifiers to almost zero ( Goodfellow et al. , 2015 ; Moosavi-Dezfooli et al. , 2016 ; Carlini & Wagner , 2017 ; Kurakin et al. , 2017 ; Madry et al. , 2018 ) . While successful , these gradient-derived attacks require access to the parameters of the trained model - which might be unrealistic for practical applications . To relax this requirement , so called black-box attacks only query the model . Observation of the model outputs allows to estimate the aforementioned gradients either by training substitute models ( Papernot et al. , 2016 ; 2017 ; Liu et al. , 2017 ) or by finite-difference methods ( Chen et al. , 2017 ; Ilyas et al. , 2018 ) . Nonetheless , to some extent even these strategies require access to the model . Our contribution : In this work , we leverage recent advances in the theoretical understanding of trained neural networks to devise adversarial data perturbations that , surprisingly , require neither access to the model nor training of a substitute model , and do not use any model information whatsoever , apart from the model architecture . We thus reveal an interesting phenomenon : the susceptibility of trained neural networks to adversarial examples does not depend on their specific weights ; instead these blind spots may be calculated a-priori , prior to initialization , even before the training starts . Although this is already known to some extent from prior work that trains substitute models , we demonstrate that this notion holds from first principles that only require the a-priori description of the model . Our powerful , highly agnostic attack is derived from Neural Tangent Ker- nel theory ( Jacot et al. , 2018 ; Lee et al. , 2019 ; Arora et al. , 2019b ) , which allows to describe the dynamics of the gradient-descent based training procedure of a neural net as a kernel regression , with an architecture- and data-dependent ( but parameter-agnostic ) kernel ( NTK ) . At a high-level of abstraction , our approach exploits the NTK description of the function the model has learned at the end of training , and proceeds to maximally perturb this function . To illustrate more precisely , let ( x , y ) ∈ Rd × { −1 , +1 } be an input sample , η ∈ Rd a small perturbation , and f : Rd → R denote the model after training has converged . Then , as we show , in the NTK regime : f ( x+ η ) ≈ f ( x ) + ηT z , ( 1 ) for some z ∈ Rd that depends on the training data and the NTK kernel only . Although the above is derived for regression models , we show that this simple expression naturally leads to a family of adversarial attacks on classifiers , depending on the choice of constraints1 imposed on η . We also cover the general case of multiclass classifiers , which is similar in spirit to Eq . ( 1 ) . As our experiments on MNIST and CIFAR suggest , we are able to confuse classifiers with a rate that in some cases approaches current benchmarks that require model access . In summary , in this work we initiate the study of “ NTK-derived ” adversarial perturbations and make the following contributions : 1 . We derive expressions for the output of a converged model on adversarial inputs , based on the theory of the Neural Tangent Kernel ( Sec . 3 ) . 2 . We propose a family of simple attacks that exploit the aforementioned description . These attacks either depend on analytical expressions afforded by the NTK or on finite approximations of the true quantities ( “ empirical ” NTK ) , obtained by initialized replicas of the attacked models ( Sec . 3 ) . 3 . We experimentally verify the efficacy of our approaches on a simple wide single hidden layer model that operates in the NTK regime ( Arora et al. , 2019a ) ( Sec . 4.1 ) and show that it favorably compares to the strongest benchmark ( PGD attack ) , without any access to model weights . 4 . Further , we demonstrate that our attack works in practical settings : fully connected and convolutional , undefended , networks , trained in standard ways ( cross entropy minimization , stochastic gradient descent , default initialization , etc ) are quite susceptible to NTK noise . We show this both in the case where the noise is generated from the NTK of the model itself , even though the assumptions of NTK theory do not necessarily hold ( Sec . 4.2 ) , as well as for transfer attacks with adversarial perturbations generated by models within the NTK regime ( Sec . 4.3 ) . 2 BACKGROUND AND RELATED WORK . Our work bridges two as yet disparate areas : generation of adversarial examples for trained neural networks at inference time and the description of the dynamics of neural network training in the infinite width limit , with the Neural Tangent Kernel . Adversarial Examples . Let f be a classifier , x be an input ( e.g . a natural image ) and y its true label ( e.g . the image class ) . Then , given that f is an accurate classifier on x , we define x̃ to be an adversarial example ( Szegedy et al. , 2014 ) for f if the following two hold : 1. d ( x , x̃ ) is small for some distance function d. Common choices in computer vision applications are the ℓp norms . 2. f ( x̃ ) ̸= y . That is , the perturbed input is being misclassified . There are many approaches on how to generate such examples and they are often classified into two large areas : white-box and black-box attacks2 . White-box attacks assume perfect knowledge of the 1While we focus on ℓ∞-attacks in this work , we remark that the introduced framework is much more general and the adoption of other similarity measures is equivalent to optimizing Eq . ( 1 ) ( a linear objective ) with the desired constraint on η . 2While we use the word “ attack ” multiple times , it must be emphasized that the existence of adversarial examples and the search for robust models are not security-only concerns . They align very well with concepts , such as interpretability and human-like perception ( Tsipras et al. , 2019 ; Ilyas et al. , 2019 ) . model under-attack and they mainly rely on optimization procedures to maximize the “ confusion ” of the model ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Moosavi-Dezfooli et al. , 2016 ; Kurakin et al. , 2017 ; Carlini & Wagner , 2017 ; Madry et al. , 2018 ) . The most widely adopted method is called Projected Gradient Descent ( Kurakin et al. , 2017 ; Madry et al. , 2018 ) and performs multiple gradient steps on the input space to discover adversarial examples . Formally , if L ( f ( x ) , y ) denotes a loss function , then , in the context of ℓ∞ adversaries , setting x0 = x the update formula for PGD is : xt+1 = Πx+S ( xt + α · sgn ( ∇xtL ( f ( xt ) , y ) ) , t = 0 , . . . , N − 1 , ( 2 ) where Πx+S ( · ) projects the input inside the allowed ℓ∞ ball , and α is a gradient step - hyperparameter . xN is the final adversarial example . Such attacks are the most powerful to date . Black-box attacks , on the other hand , are not allowed to estimate the aforementioned gradient directly , but rather are restricted to querying the model without accessing the parameters . A lot of these attacks proceed by collecting pairs of inputs and predictions from the model , then train another classifier f̂ on them , attack f̂ with white-box attacks and finally use those generated adversarial samples to attack f ( Papernot et al. , 2016 ; 2017 ; Liu et al. , 2017 ) . The empirical success of such strategies relies on the phenomenon of transferability of adversarial examples ( Szegedy et al. , 2014 ; Papernot et al. , 2016 ; Tramèr et al. , 2017 ; Liu et al. , 2017 ) : it has been observed that adversarial inputs designed for successfully fooling one classifier , manage to also fool others . A related line of research abandons the need for creating substitute models , at the cost of allowing multiple curated queries to the model to infer the loss function ( Chen et al. , 2017 ; Ilyas et al. , 2018 ) . Finally , hard-label black box attacks have been considered in the literature , in which an attacker observes only the prediction of f ( Brendel et al. , 2018 ; Cheng et al. , 2019 ) . The adversarial attack we propose does not fall into any of these categories , since it does not query the target model at all . More related to the scope of this work is the so-called “ no-box ” setting ( Chen et al. , 2017 ; Bose et al. , 2020 ; Li et al. , 2020 ) , where an adversary is prohibited from querying the model under-attack . However , to the best of our knowledge , practical attacks under this threat model do train substitute neural networks , making them incomparable to our work . Neural Tangent Kernel ( NTK ) . NTK theory plays an increasing role in the quest to theoretically understand deep learning and the elusive concepts of optimization and generalization in overparametrized networks . It connects deep learning to Gaussian Processes and kernels . The seminal work of Neal ( 1996 ) considered the infinite width limit of neural networks with one hidden layer and showed that at initialization the network is a function drawn from a Gaussian Process with an associated kernel that depends on the activation function . Recently , this limit ( and non asymptotic versions of it ) has been revisited either for deep random networks where only the last layer is being trained ( Lee et al . ( 2018 ) ; de G. Matthews et al . ( 2018 ) ; Novak et al . ( 2019 ) ) or fully trained networks with gradient descent ( Jacot et al. , 2018 ; Lee et al. , 2019 ; Arora et al. , 2019b ) . In the latter case , Jacot et al . ( 2018 ) identified a particular quantity that governs the dynamics of the training procedure of a neural network with randomly initialized parameters θ computing an output f ( x ; θ ) : Rd → R , which they called Neural Tangent Kernel . The NTK is defined as the expectation of the inner product between the gradients of the function with respect to the random initialization of its parameters θ evaluated on all inputs xi , xj , i.e. , h ( xi , xj ) = Eθ∇θf ( xi ; θ ) T∇θf ( xj ; θ ) . ( 3 ) While first studied in fully connected networks , it is possible to define neural tangent kernels for any sane architecture ( Yang ( 2020 ) ) . In the infinite width limit3 ( and under appropriate initialization ) , the training dynamics of the network with infinitesimal learning rate can be described by a differential equation with a constant NTK . Moreover , crucially , in the case of squared loss this gives rise to a closed-form description of the training dynamics with the NTK . For ease of presentation we restrict to the case where the expected network output at initialization is zero ; then the expected output of the network at convergence of training becomes ( Lee et al. , 2018 ) : f∞ ( X̃ ) = H ( X̃ , X ) H ( X , X ) −1Y , ( 4 ) where X ∈ Rn×d is a collection of n d-dimensional training data , X̃ ∈ Rn′×d is a test set and H : Rn1×d × Rn2×d → Rn1×n2 is the NTK ( for fixed input datasets , H is a matrix ) . For some 3for non asymptotic statements see Arora et al . ( 2019b ) . models , this kernel can be computed in closed form analytically by computing the expectation over the initializations . It can also be approximated by an empirical kernel via averaging over a few initialized instances ( Jacot et al. , 2018 ; Lee et al. , 2019 ) . Finally , there has also been work on the intersection of the two areas , though with very distinct emphasis to ours . Recent work ( Bubeck et al . ( 2021 ) ; Bartlett et al . ( 2021 ) ) studies the existence of adversarial examples in overparametrized random neural networks , exploiting some local linearity of the functions computed by such networks . The only prior work that leverages NTK theory to derive perturbations ( published almost concurrently with the preparation of our work ) is due to ( Yuan & Wu , 2021 ) , yet with different focus and without our modification to the standard approach to the optimization problem , that generates the perturbations ( see at the end of Sec . 3 ) . That work is concerned with what is coined as generalization attacks : the process of altering the training data distribution to prevent models to generalise on clean data . In contrast , our work analyses trained models to analytically derive their weakness to adversarial perturbations at inference time . Besides that , our work introduces analytical expressions for the perturbed data and the output of the network on them , something that does not follow from the methods of Yuan & Wu ( 2021 ) ( even if extended to the problem we consider ) .

Summary: In this paper the authors introduce and study a new approach to generate Adversarial Attacks for image classification models. The core of their approach lies in leveraging Neural Tangent Kernel (NTK) formulations of Deep Neural Networks. By using the analytic and empirical NTK versions of common network architectures they are able to generate adversarial samples without requiring white- or black-box access to the model under attack. The work establishes a new link between formerly disjoint research fields of Adversarial Attacks and NTKs. They show empirically that attacks generated using a simple model with closed-form, analytic NTK transfer reasonably well to other models that are far from the kernel regime. In addition they also study the adversarial samples generated from an empirical, sampled kernel via auto-differentiation and show their success in attacking other models. The authors provide ablations on MNIST and CIFAR-10 for standard models such as the analytically solvable Random Combination of Features and deep models such as Fully Connected Networks and Convolutional Networks in the LeNet-family.