Strength in Numbers: Trading-off Robustness and Computation via Adversarially-Trained Ensembles

1 INTRODUCTION . Deep neural networks have demonstrated state-of-the-art performance in a wide range of application domains Krizhevsky et al . ( 2012 ) . However , researchers have discovered that deep networks are in some sense ‘ brittle ’ , in that small changes to their inputs can result in wildly different outputs ( Huang et al. , 2017 ; Jia & Liang , 2017 ; Szegedy et al. , 2013 ) . For instance , practically imperceptible ( to human ) modifications to images can result in misclassification of the image with high confidence . Not only are networks susceptible to these ‘ attacks ’ , but these attacks are also relatively easy to compute using standard optimization techniques ( Carlini & Wagner , 2017b ; Goodfellow et al. , 2014 ) . These changes are often referred to as adversarial perturbations , in the sense that an adversary could craft a very small change to the input in order to create an undesirable outcome . This phenomenon is not unique to image classification , nor to particular network architectures , nor to particular training algorithms ( Papernot et al. , 2016 ; 2017 ) . Adversarial attacks can be broken into different categories depending on how much knowledge of the underlying model the adversary has access to . In ‘ white-box ’ attacks the adversary has full access to the model , and can perform both forward and backwards passes ( though not change the weights or logic of the network ) ( Carlini & Wagner , 2017a ; Goodfellow et al. , 2014 ) . In the ‘ black-box ’ setting the adversary has no access to the model , but perhaps knows the dataset that the model was trained on ( Papernot et al. , 2016 ; 2017 ) . Despite several recent papers demonstrating new defences against adversarial attacks ( Akhtar & Mian , 2018 ; Guo et al. , 2017 ; Liao et al. , 2017 ; Song et al. , 2017 ; Tramèr et al. , 2018 ; Warde-Farley & Goodfellow , 2016 ; Xie et al. , 2017 ; Yuan et al. , 2017 ) , recent papers have demonstrated that most of these new defences are still susceptible to attacks and largely just obfuscate the gradients that the attacker can follow , and that non-gradient based attacks are still effective Uesato et al . ( 2018 ) ; Athalye et al . ( 2018 ) . Exploring Tradeoff of Computation and Robustness In many safety-critical application domains ( e.g . self-driving cars ) , robustness is extremely important even if it comes at the cost of increased computation . This motivated the central question considered by this paper : Is it possible to increase adversarial robustness of a classifier at the cost of increased computation ? There are a number of possibilities to employ extra computation available at runtime . We can use a much larger model that requires more time to run , execute the original model multiple times and aggregate the predictions , or instead of using a single model , make predictions from a portfolio or ensemble of models . While researchers have proposed the use of portfolios and ensembles as a mechanism to improve adversarial robustness Abbasi & Gagné ( 2017 ) ; Thilo Strauss ( 2017 ) , our experimental results indicate that stronger adversaries are able to attack the ensembles successfully . Contributions In this paper , we study and analyze the trade-off of adversarial robustness and computation ( memory and runtime ) . We propose the use of adversarial training of ensemble of models and through an exhaustive ablative analysis make the following empirical findings : • increased computation and/or model size can be used to increase robustness , • ensembles on their own are not very robust , but can be made robust through adversarial training where the ensemble is treated as a single model , • adversarially trained ensembles are more robust than adversarially trained individual models requiring the same amount of parameters/computation Related Work Recently , Tramèr et al . ( 2018 ) investigated the use of ensembles for adversarial robustness . However , their goal and approach was quite different from the technique we are investigating . In Tramèr et al . ( 2018 ) , the authors generated adversarial perturbations using an ensemble of pre-trained models in order to transfer the example to another model during training . This procedure decouples adversarial example generation from the current model , and consequently the model being trained can not simply ‘ overfit ’ to the procedure for generating adversarial examples , which they generally took to be single-step attack methods . The authors demonstrated strong robustness of the resulting trained model to black-box attacks . By contrast , in this paper we investigate using an ensemble of models as our predictive model , and we train the models using multi-step adversarial training . We show increased robustness to both black-box and white-box adversarial attacks using this strategy . 2 PRELIMINARIES . Here we lay out the basics of attacking a neural network by the generation of adversarial examples . Denote an input to the network as x ∈ Rd with correct label ŷ ∈ Y ⊂ N , and let mθ : Rd → R|Y| be the mapping performed by the neural network which is parameterized by θ ∈ Rp . Let L : Y×R|Y| → R denote the loss we are trying to minimize ( e.g. , the cross-entropy ) . When training a neural network we seek to solve minimize E ( x , y ) ∼D L ( ŷ , mθ ( x ) ) ( 1 ) over variable θ , where D is the data distribution . Given any fixed θ we can generate ( untargeted ) adversarial inputs by perturbing the input x so as to maximize the loss . We restrict ourselves to small perturbations around a nominal input , and we denote by B this set of allowable inputs . For example , if we restrict ourselves to small perturbations in ` ∞ norm around a nominal input xnom then we could set B = { x | ‖x− xnom‖∞ ≤ } where > 0 is the tolerance . A common approach for generating adversarial examples is projected gradient descent Carlini & Wagner ( 2016 ) , i.e. , to iteratively update the input x by x̃k+1 = ΠB ( x̃ k + η∇xL ( y , mθ ( x̃k ) ) ) , ( 2 ) where typically x0 = x + for some noise , η > 0 is a step-size parameter and ΠB denotes the Euclidean projection on B . We add noise to the initial point so that the network can ’ t memorize the training dataset and mask or obfuscate the gradients at that point Uesato et al . ( 2018 ) ; Athalye et al . ( 2018 ) , in other words the added noise encourages generalization of adversarial robustness to the test dataset . If instead of using the gradient we just use the sign of the gradient then this is the fast-gradient-sign method Goodfellow et al . ( 2014 ) . Empirically speaking , for most networks just a few steps of either of these procedures is sufficient to generate an x̃ that is close to xnom but has a different label with high confidence . In this paper we are primarily concerned with the performance of ensembles of models when trained with adversarial training Madry et al . ( 2017 ) . In adversarial training we train a network to minimize a weighted sum of two losses ( where the relative weighting is a hyper-parameter ) . The first loss is the standard loss of the problem we are trying to solve on the normal training data , e.g. , the cross-entropy for a classification task . The second loss is the same function as the first loss , except evaluated on adversarially generated data , where typically the adversarial data is generated by attacking the network at that time-step . In other words we replace the problem in eq . ( 1 ) with minimize E ( x , y ) ∼D ( L ( ŷ , mθ ( x ) ) + ρL ( ŷ , mθ ( x̃ ) ) ) ( 3 ) where ρ ≥ 0 is the weighting parameter and x̃ is an adversarial example generated from x at model parameters θ using , for example , the update in eq . ( 2 ) . This problem is usually approximated by sampling and minimizing the empirical expectation . 3 ADVERSARIALLY-TRAINED ENSEMBLES . In this section we lay out the basic strategy of using ensembles of models to increase robustness to adversarial attacks . The notion of ensemble used here simply involves taking k separatelyparameterized models and averaging their predictions . If the output of network i as a function of input x and with network parameters θi is given by p ( ·|x , θi ) = mθi ( x ) , then the output of the ensemble is p ( y|x ) = 1 k k∑ i=1 p ( y|x , θi ) . Alternatively , we could consider using a ‘ gating network ’ to generate data-dependent weights for each model rather than a simple average , though we found the performance to be similar . Using ensembles to improve the performance of statistical models is a very old idea ; see , e.g . Opitz & Maclin ( 1999 ) for a survey . The basic intuition is that several weak models can be combined in such a way that the ensemble performs better than any individual , and is sometimes explained as being caused by the errors of the models ‘ cancelling ’ with one another . In order to ensure that the models are actually producing different outputs the diversity of the models must be maintained . This can be done in several ways , such as bootstrapping the data , whereby each model gets a slightly different copy of the data , or using totally different model types or architectures . In the case that the model training procedure is convex , and if all models architectures are the same and are getting the same data , then the models in the ensemble would be expected to converge on the same parameters . In the case of neural networks however , the model training procedure is not convex and so our strategy for maintaining diversity is very simple—initialize each model differently . Due to the nature of training neural networks it is likely that differently initialized networks will converge ( assuming they do , in fact , converge ) to different points of the parameter space . The insight that only different initialization is required is not new , previous papers have observed that different initialization is sufficient for uncertainty estimation Lakshminarayanan et al . ( 2016 ) ; Osband et al . ( 2016 ) . Different initialization for networks has an appealing interpretation . If we take a Bayesian approach to the classification problem , then we have a prior over possible model parameters , p ( θ ) , a likelihood of the data , p ( D|θ ) , and a probability of a label y given an input and a model , p ( y|x , θ ) . The ‘ Bayes-optimal ’ classification of a new data point x is given by y ? = argmaxy ∫ θ p ( y|x , θ ) p ( D|θ ) p ( θ ) . This classifier is optimal in the sense that no other classifier can outperform it on average , given the same model class and knowledge of the prior and likelihood ; however , the formulation is intractable for all but small problems . We can consider approximating it by the following approach , sample initial parameters from the prior p ( θ ) and run an iterative procedure to ( approximately ) maximize the likelihood p ( D|θ ) . Very loosely speaking , we can consider this procedure as approximately sampling from the posterior over models p ( θ|D ) ∝ p ( D|θ ) p ( θ ) . Consequently , we output the classification y ? = argmaxy k∑ i=1 p ( y|x , θi ) , Algorithm 1 Adversarial ensemble training using PGD under ` ∞ norm constraint input : k neural networks mθi , i = 1 , . . . , k ; attack steps N ; step sizes η , η̂ ; initial variance σ ; adversarial loss weighting ρ ; perturbation width δ initialize : neural network parameters θ0i randomly , i = 1 , . . . , k for time-step t = 0 , 1 , . . . , do sample input minibatch ( x , ŷ ) ∼ D initialize x̃0 = x+ where ∼ N ( 0 , σ2I ) define B = { x′ | ‖x− x′‖∞ ≤ δ } for k = 0 , . . . , N − 1 do x̃k+1 = ΠB ( x̃ k + η̂∇xL ( ŷ , 1 k k∑ i=1 mθi ( x̃ k ) ) ) end for update parameters for each i = 1 , . . . , k : θt+1i = θ t i − η∇θi L ( ŷ , 1 k k∑ j=1 mθtj ( x ) ) + ρL ( ŷ , 1 k k∑ j=1 mθtj ( x̃ N ) ) end for i.e. , the best guess of the ensemble . The role of initialization therefore is that of sampling from our prior over possible model parameters . Adversarial training of Ensembles Up to this point we have discussed the use of ensembles for improving classification performance and approximating the Bayes optimal classifier . Typically speaking neural networks appear to not benefit much from ensembling in terms of nominal performance . Here , however , we make the claim that adversarially trained ensembles of networks provide a level of robustness to adversarial attacks . When using ensembles the loss function for adversarial training in ( 3 ) is replaced by the mean of the loss over the k models , i.e. , now we want to solve minimize E ( x , y ) ∼D ( L ( ŷ , 1k ∑k i=1mθi ( x ) ) + ρL ( ŷ , 1 k ∑k i=1mθi ( x̃ ) ) ) ( 4 ) over variables θi , i = 1 , . . . , k , and where x̃ is an adversarial example generating by attacking the entire ensemble . The exact procedure is outlined in Algorithm 1 . We demonstrate empirically in the numerical results section that this procedure increases robustness to adversarial inputs . Following these results , we offer an analysis and hypothesis why ensembles outperform single models , even when controlling for number of parameters .

Summary: This paper presents a new adversarial training defense whereby an ensemble of models is trained against both benign and adversarial examples. The authors demonstrate on the CIFAR-10 dataset that the ensemble has improved robustness against a wide variety of white-box and transfer-based black-box attacks compared to other adversarial training techniques. The results appear significant but would greatly benefit from more thorough experiments.