MODEL-AGNOSTIC META-ATTACK: TOWARDS RELI-
Year: 2021

Abstract:
The vulnerability of deep neural networks to adversarial examples has motivated an increasing number of defense strategies for promoting model robustness. However, the progress is usually hampered by insufficient robustness evaluations. As the de facto standard to evaluate adversarial robustness, adversarial attacks typically solve an optimization problem of crafting adversarial examples with an iterative process. In this work, we propose a Model-Agnostic Meta-Attack (MAMA) approach to discover stronger attack algorithms automatically. Our method learns the optimizer in adversarial attacks parameterized by a recurrent neural network, which is trained over a class of data samples and defenses to produce effective update directions during adversarial example generation. Furthermore, we develop a model-agnostic training algorithm to improve the generalization ability of the learned optimizer when attacking unseen defenses. Our approach can be flexibly incorporated with various attacks and consistently improves the performance with little extra computational cost. Extensive experiments demonstrate the effectiveness of the learned attacks by MAMA compared to the state-of-the-art attacks on different defenses, leading to a more reliable evaluation of adversarial robustness.

heading: 1 INTRODUCTION
text: Deep neural networks are vulnerable to maliciously crafted adversarial examples (Biggio et al., 2013; Szegedy et al., 2014; Goodfellow et al., 2015), which aim to induce erroneous model predictions by adding small perturbations to the normal inputs. Due to the threats, a multitude of defense strategies have been proposed to improve adversarial robustness (Kurakin et al., 2017b; Madry et al., 2018; Liao et al., 2018; Wong & Kolter, 2018; Zhang et al., 2019b; Dong et al., 2020a; Pang et al., 2020). However, many defenses have later been shown to be ineffective due to incomplete or incorrect robustness evaluations (Athalye et al., 2018a; Uesato et al., 2018; Carlini et al., 2019; Croce & Hein, 2020b; Dong et al., 2020b; Tramer et al., 2020), making it particularly challenging to understand their effects and identify the actual progress of the field. Therefore, developing methods that can evaluate adversarial robustness accurately and reliably is of profound importance.
Adversarial attacks are broadly adopted as an indispensable solution to evaluate adversarial robustness for different models. One of the most prominent attacks is the projected gradient descent (PGD) method (Madry et al., 2018), which generates an adversarial example by performing iterative gradient updates to maximize a classification loss (e.g., cross-entropy loss) w.r.t. the input. Although recent methods improve upon PGD by introducing different loss functions (Carlini & Wagner, 2017; Gowal et al., 2019; Croce & Hein, 2020b; Sriramanan et al., 2020) and adjusting the step size (Croce & Hein, 2020b), most of them adopt hand-designed optimization algorithms, such as vanilla gradient descent, momentum (Polyak, 1964), and Adam (Kingma & Ba, 2015). However, it has been shown that these attacks can be sub-optimal (Croce & Hein, 2020b; Tramer et al., 2020), arousing an overestimation of adversarial robustness for some defenses. It is thus imperative to develop more effective optimization algorithms for improving adversarial attacks. Nevertheless, designing a generic optimization algorithm in a hand-crafted manner is non-trivial, considering the different defense models with varying network architectures, defense strategies, and training datasets.
To overcome this issue and develop stronger adversarial attacks, in this paper, we propose a ModelAgnostic Meta-Attack (MAMA) approach to learn optimization algorithms in adversarial attacks automatically. In particular, we parameterize the optimizer with a recurrent neural network (RNN)
to mimic the behavior of iterative attacks, which outputs an update direction at each time step during adversarial example generation. By learning to solve a class of optimization problems with different data samples and defense models, the RNN optimizer could benefit from long-term dependencies of iterative attacks and exploit common structures among the optimization problems. To improve and stabilize training, we then propose a prior-guided refinement strategy, which imposes a prior given by the PGD update rule on the outputs of the RNN optimizer, whose parameters are trained via the maximum a posteriori (MAP) estimation framework. Consequently, the RNN optimizer can learn to yield more effective update directions than hand-designed ones.
Despite the effectiveness, the learned optimizer may not generalize well to attacking unseen defenses due to potential overfitting to the defenses used for training. Although training a different optimizer for each defense is possible, the training process can be time-consuming and inconvenient in practice when it is utilized to benchmark adversarial robustness. To endow the learned optimizer with a better generalization ability, we develop a model-agnostic training algorithm with a gradient-based metatrain and meta-test process to simulate the shift from seen to unseen defenses. Therefore, the learned optimizer can directly be deployed to attack unseen defenses without retraining or fine-tuning.
Extensive experiments validate the effectiveness and generalization ability of the learned optimizers for attacking different defense models. We also demonstrate the flexibility of our method integrating with various attacks, including those with different loss functions (Carlini & Wagner, 2017; Gowal et al., 2019; Croce & Hein, 2020b) and initialization (Tashiro et al., 2020). Our method consistently improves the attacking performance over baselines, while introducing little extra computational cost. By incorporating our method with an orthogonal technique — output diversified initialization (ODI) (Tashiro et al., 2020), we achieve lower robust test accuracy on all 12 defense models that we study than other state-of-the-art attacks, including the MultiTargeted (MT) attack (Gowal et al., 2019) and AutoAttack (Croce & Hein, 2020b), leading to a more reliable evaluation of adversarial robustness. Moreover, MAMA provides a plug-and-play module in adversarial attacks, which may benefit future adversarial attacks if new improvements have been made in different aspects.

heading: 2 PRELIMINARIES AND RELATED WORK

heading: 2.1 ADVERSARIAL ATTACKS
text: Let f(x) : x ∈ RD → z ∈ RK denote a classifier, which outputs the logits z := [z1, ..., zK ] over K classes for an input image x. The prediction of f is Cf (x) = arg maxi=1,...,K zi. Given the true label y of x, adversarial attacks aim to generate an adversarial example x̂ that is misclassified by f (i.e., Cf (x̂) 6= y), while the distance between the adversarial example x̂ and the natural one x measured by the `∞ norm is smaller than a threshold as ‖x̂ − x‖∞ ≤ . Although we introduce our approach based on the `∞ norm only, the extension to other `p norms is straightforward. The adversarial example x̂ can be generated by solving a constrained optimization problem as
x̂ = arg max x′
L(f(x′), y), s.t. ‖x′ − x‖∞ ≤ , (1)
where L is a loss function on top of the classifier f(x). For example, L could be the cross-entropy loss as Lce(f(x), y) = − log py where py = ezy/∑Ki=1 ezi denotes the predicted probability of class y, or the margin-based CW loss (Carlini & Wagner, 2017) as Lcw(f(x), y) = maxi6=y zi − zy . A lot of gradient-based methods (Goodfellow et al., 2015; Kurakin et al., 2017a; Carlini & Wagner, 2017; Madry et al., 2018; Dong et al., 2018) have been proposed to solve this optimization problem. The projected gradient descent (PGD) method (Madry et al., 2018) performs iterative updates as
x̂t+1 = ΠB (x) ( x̂t + αt · sign(∇xL(f(x̂t), y)) ) , (2)
where B (x) = {x′ : ‖x′ − x‖∞ ≤ } denotes the `∞ ball centered at x with radius , Π(·) is the projection operation, and αt is the step size at the t-th attack iteration. PGD initializes x̂0 randomly by uniform sampling within B (x) and adopts a fixed step size α in every iteration. To develop stronger attacks for reliable robustness evaluation, recent improvements upon PGD include adopting different loss functions (Gowal et al., 2019; Croce & Hein, 2020b; Sriramanan et al., 2020), adjusting the step size (Croce & Hein, 2020b), and designing sampling strategies of initialization (Tashiro et al., 2020). For the update rules, besides the vanilla gradient descent adopted in PGD, recent attacks (Dong et al., 2018; Gowal et al., 2019) suggest to use the momentum (Polyak, 1964)
and Adam (Kingma & Ba, 2015) optimizers. In contrast, we aim to enhance adversarial attacks by learning optimization algorithms in an automatic way.

heading: 2.2 ADVERSARIAL DEFENSES
text: To build robust models against adversarial attacks, numerous defense strategies have been proposed, but most defenses can be evaded by new adaptive attacks (Athalye et al., 2018a; Tramer et al., 2020). Among the existing defenses, adversarial training (AT) is arguably the most effective defense technique, in which the network is trained on the adversarial examples generated by attacks. Based on the primary AT frameworks like PGD-AT (Madry et al., 2018) and TRADES (Zhang et al., 2019b), improvements have been made via ensemble learning (Tramèr et al., 2018; Pang et al., 2019), metric learning (Mao et al., 2019; Pang et al., 2020), semi-supervised learning (Alayrac et al., 2019; Carmon et al., 2019; Zhai et al., 2019), and self-supervised learning (Chen et al., 2020a; Hendrycks et al., 2019; Kim et al., 2020). Due to the high computational cost of AT, other efforts are devoted to accelerating the training procedure (Shafahi et al., 2019; Wong et al., 2020; Zhang et al., 2019a). Recent works emphasize the training tricks (e.g., weight decay, batch size, etc.) in AT (Gowal et al., 2020; Pang et al., 2021). However, it has been shown that the robust test accuracy of numerous adversarially trained models can be degraded significantly by using stronger attacks (Croce & Hein, 2020b; Tramer et al., 2020), indicating that the reliable evaluation of adversarial robustness remains an imperative yet challenging task. In this paper, we focus on evaluating adversarial robustness of AT models due to their superior robustness over other defense techniques.

heading: 2.3 META-LEARNING
text: Meta-learning (learning to learn) studies how to learn learning algorithms automatically from a meta perspective, and has shown promise in few-shot learning (Finn et al., 2017) and learning optimizers (Andrychowicz et al., 2016). The latter is called learning to optimize (L2O). The first L2O method leverages a coordinate-wise long short term memory (LSTM) network as the optimizer model, which is trained by gradient descent on a class of optimization problems (Andrychowicz et al., 2016). The optimizer can also be trained by reinforcement learning (Li & Malik, 2017). Bello et al. (2017) propose to adopt an RNN controller to generate the update rules of optimization algorithms, which can be described as a domain specific language. L2O is then extended to efficiently optimize black-box functions (Chen et al., 2017). Recent methods focus on improving the training stability and generalization of the learned optimizers by designing better optimizer models and training techniques (Lv et al., 2017; Wichrowska et al., 2017; Chen et al., 2020b).
The general idea of meta-learning has been investigated in adversarial attacks. Most related methods learn to generate adversarial examples by using convolutional neural networks (CNNs), which usually take clean images as inputs and return adversarial examples/perturbations (Baluja & Fischer, 2017; Poursaeed et al., 2018). Differently, our method adopts an RNN model as the attack optimizer to mimic the behavior of iterative attacks. Empirical results validate the superiority of our approach compared to the CNN-based generators. Meta-learning has also been used in black-box attacks (Du et al., 2020) and AT defenses (Xiong & Hsieh, 2020; Jiang et al., 2021). It is noteworthy that Xiong & Hsieh (2020) propose a similar approach to ours that uses an RNN optimizer to learn attacks for AT. Although the RNN optimizer is also adopted in our approach, we improve the attacking performance and generalization ability of the RNN optimizer with new techniques, as presented in Sec. 3. More detailed comparisons between our work and Xiong & Hsieh (2020) are shown in Appendix A.

heading: 3 METHODOLOGY
text: In this paper, we propose a Model-Agnostic Meta-Attack (MAMA) approach to improve adversarial attacks by learning optimization algorithms automatically. MAMA permits learning to exploit common structures of the optimization problems (1) over a class of input samples and defense models. We first present the instantiation of the learned optimizer with a recurrent neural network (RNN) and the corresponding training procedure in Sec. 3.1. The learned optimizer is expected to not only perform well on the data samples and defenses for which it is trained but also possess an excellent generalization ability to other unseen data samples, defenses, and more optimization steps (i.e., attack iterations) than training. Therefore, we propose a model-agnostic training algorithm simulating train/test shift to ensure generalization of the learned optimizer, as detailed in Sec. 3.2. An overview of our proposed MAMA approach is provided in Fig. 1.

heading: 3.1 META-ATTACK WITH RECURRENT NEURAL NETWORKS
text: Different from the existing attack algorithms, we propose a meta-attack that learns an optimizer in adversarial attacks parameterized by an RNN model mφ with parameters φ to mimic the behavior of iterative attacks. The learned optimizer is capable of optimizing the loss function L in Eq. (1) for different data samples (x, y) and defense models f . At each step, mφ takes gradient information of the loss function (1) at the current adversarial example as input and outputs an update direction for generating the adversarial example. This procedure can be expressed as
x̂t+1 = ΠB (x) (x̂t + αt · gt) , [gt,ht+1] = mφ (∇t,ht) , (3)
where αt is the step size,∇t = ∇xL(f(x̂t), y) denotes the gradient of the loss function L at x̂t, and ht denotes the hidden state representation of the optimizer, which contains rich temporal gradient information to capture long-term dependencies. The output gt of the RNN model is used to update the adversarial example x̂t. We apply a tanh transformation to the last layer of the RNN model to normalize and smoothen the update direction gt since the adversarial perturbations are required to be smaller than the threshold . In particular, we adopt the coordinate-wise long short term memory (LSTM) network (Andrychowicz et al., 2016) as the RNN modelmφ, which can significantly reduce trainable parameters, making training much easier and faster.
Given a distribution F over defenses f and a data distribution D over (x, y) pairs, the parameters φ can be trained by maximizing the expected loss over the entire trajectory of optimization as
J (φ;F ,D) = Ef∼F,(x,y)∼D [ T∑ t=1 wtL(f(x̂t), y) ] , (4)
where x̂t is obtained by the optimizer model mφ through Eq. (3), wt ∈ R≥0 is the weight for the loss at the t-th step to emphasize different importance on the loss at different steps, and T is the total number of optimization steps. For example, we could let wt = 1[t = T ] to focus on the loss at the final adversarial example since it would be used to attack the classifier, but using a different setting forwt can make training easier. For training the optimizer networkmφ, we can maximize the objective (4) using gradient ascent on φ. The expectation in Eq. (4) is approximated by mini-batches of randomly sampled defense models f and data points (x, y). The gradient w.r.t. φ is calculated by truncated backpropagation. Besides the general framework of learning to optimize (Andrychowicz et al., 2016; Chen et al., 2020b; Xiong & Hsieh, 2020), there exist particular challenges in training the optimizer due to the optimization problem of adversarial attacks as illustrated below.
Vanishing gradient problem. Note that the optimization problem (1) is a constrained one, which is different from unconstrained problems commonly studied in previous works (Andrychowicz et al., 2016; Chen et al., 2020b; Lv et al., 2017; Wichrowska et al., 2017). As in Eq. (3), we need to project the adversarial example onto the `∞ ball B (x) after every optimization step, which is essentially the clip function. However, optimizing Eq. (4) requires backpropagation through the clip function, which can result in a number of zero gradient values when the update exceeds the `∞ ball. This can consequently lead to the vanishing gradient problem for a long optimization trajectory. To address this problem, we apply the straight-through estimator (Bengio et al., 2013) to get an approximate gradient of the clip function as ∇x′ΠB (x)(x′) = I , where I denotes the identity matrix. In this way, the optimizer can effectively be trained to optimize the constrained problem (1).
Algorithm 1 Model-Agnostic Meta-Attack (MAMA) algorithm
Require: A data distribution D over (x, y) pairs, a set of defense models F = {fj}Nj=1, the number of defenses n in the meta-test set, the objective function J in Eq. (5), attack iterations T , batch size B, learning rates β, γ, balancing hyperparameter µ. Ensure: The parameters φ of the learned optimizer. 1: for iter = 1 to MaxIterations do 2: Randomly draw N − n defenses from F to get F̂ , and let F̄ = F\F̂ consist of the other n defenses; 3: for fr in F̂ do . Meta-train 4: Randomly draw a mini-batch Br = {(xri , yri)}Bi=1 from D; 5: For (xri , yri) ∈ Br , compute x̂rit from φ for defense fr by Eq. (3) for t = 1 to T ; 6: end for 7: Compute J (φ; F̂ ,D) using the defenses fr in F̂ and the corresponding mini-batches Br; 8: φ′ = φ+ β∇φJ (φ; F̂ ,D); 9: for fe in F̄ do . Meta-test 10: Randomly draw a mini-batch Be = {(xei , yei)}Bi=1 from D; 11: For (xei , yei) ∈ Be, compute x̂eit from φ′ for defense fe by Eq. (3) for t = 1 to T ; 12: end for 13: Compute J (φ′; F̄ ,D) using the defenses fe in F̄ and the corresponding mini-batches Be; 14: φ← φ+ γ∇φ(J (φ; F̂ ,D) + µJ (φ′; F̄ ,D)); 15: end for
Training instability problem. Training the learned optimizer may suffer from an instability problem (Chen et al., 2020b), leading to unsatisfactory performance. To further stabilize training of the learned optimizer mφ and improve its performance, we propose a prior-guided refinement strategy which leverages an informative prior to refine the outputs of the optimizer. In particular, the prior is given by the PGD update rule (Madry et al., 2018) which performs generally well on many defenses. We specify the prior distribution of gt to be an isotropic Gaussian distribution asN (sign(∇t), 12λt ), whose mean is the PGD update direction sign(∇t). The variance of the distribution is 1/2λt with λt being a hyperparameter. Under the maximum a posteriori (MAP) estimation framework, the objective function (4) can be viewed as the likelihood while the prior is equivalent to minimizing the squared `2 distance as λt‖gt − sign(∇t)‖22. We provide a formal derivation of MAP in Appendix B. Therefore, the overall objective function of training the optimizer becomes
J (φ;F ,D) = Ef∼F,(x,y)∼D [ T∑ t=1 ( wtL(f(x̂t), y)− λt‖gt − sign(∇t)‖22 )] , (5)
where {λt}Tt=1 can be seen as another set of weights corresponding to the regularization terms at each step. In this way, the optimizer can learn from the informative prior given by the PGD update rule and explore potential better directions simultaneously. To maximize the objective function (5), a basic technique is using gradient ascent on φ. We call it Basic Meta-Attack (BMA).

heading: 3.2 MODEL-AGNOSTIC META-ATTACK ALGORITHM
text: To make robustness evaluation more convenient after training, the learned optimizer should generalize well to new data samples, defense models, and more optimization steps that are not seen during training. In practice, we observe that the data-driven process of optimizing Eq. (5) (i.e., BMA) enables to obtain a generalizable optimizer on new data samples with more optimization steps, but the learned optimizer may not necessarily generalize well to new defenses, as shown in Sec. 4.3. This is because the defenses usually adopt different network architectures and defense strategies, making it hard for an attack optimizer to generalize well across a variety of defenses (Tramer et al., 2020).
To improve the generalization ability of the learned optimizer when attacking unseen defenses, we propose a model-agnostic training algorithm for meta-attack (i.e., MAMA). This method simulates the shift between defenses during training with a gradient-based meta-train and meta-test procedure, inspired by model-agnostic meta-learning (Finn et al., 2017; Li et al., 2018). After MAMA training, the learned optimizer can directly be used to attack unseen defenses, which will be more convenient and user-friendly for carrying out robustness evaluation once novel defense methods are presented.
Specifically, assume that we have a set of N defenses (denoted as F) in training, where we reuse the notation F without ambiguity. To simulate the shift between defense models in training and testing,
at each training iteration we split F into a meta-train set F̂ of N −n defenses and a meta-test set F̄ of the other n defenses. The model-agnostic training objective is to maximize the loss J (φ; F̂ ,D) on the meta-train defenses, while ensuring that the update direction of φ also leads to maximizing J (φ; F̄ ,D) on the meta-test defenses, such that the learned optimizer will perform well on true test defenses. During meta-train, the optimizer parameters φ are updated by the one-step gradient ascent
φ′ = φ+ β∇φJ (φ; F̂ ,D), where β is the meta-train learning rate. During meta-test, we expect the updated parameters φ′ to exhibit a good generalization performance on the meta-test defenses, which indicates that J (φ′; F̄ ,D) is also maximized. The meta-train and meta-test can be optimized simultaneously as
max φ J (φ; F̂ ,D) + µJ (φ+ β∇φJ (φ; F̂ ,D); F̄ ,D), (6)
where µ is a hyperparameter to balance the meta-train loss and the meta-test loss. The objective (6) is optimized by gradient ascent. After accumulating the gradients from meta-train and meta-gradients from meta-test, the optimizer is trained to perform well on attacking both meta-train and meta-test defenses. The gradient and meta-gradient flows in the computation graph are illustrated in Fig. 1(b). We summarize the overall MAMA algorithm in Alg. 1.

heading: 4 EXPERIMENTS
text: In this section, we present the experimental results to validate the effectiveness of BMA and MAMA for adversarial attacks. We follow a simple-to-complex procedure to conduct experiments. Sec. 4.2 first shows the advantages of the learned optimizer in BMA when considering one defense at a time. Sec. 4.3 then proves that the learned optimizer possesses good cross-data, cross-iteration, and cross-model generalization abilities, respectively. Sec. 4.4 demonstrates that BMA and MAMA can achieve state-of-the-art performance by comparing with various top-performing attacks on 12 defense models. More details about the baseline attacks and defenses are provided in Appendix C and D, respectively. We mainly present the results based on the `∞ norm in this section. The results based on the `2 norm and standard deviations of multiple runs are provided in Appendix E.

heading: 4.1 EXPERIMENTAL SETUP
text: The hyperparameters of baseline attacks are set according to the common practice or their original implementations. Specifically, in Sec. 4.2 and Sec. 4.3, we set the attack iterations as T = 20 and the fixed step size as α = 2/255 according to (Madry et al., 2018). The learned optimizer in our method also adopts the same setting for fair comparison. In Sec. 4.4, we adopt the official hyperparameter configurations of the baseline attacks since we assume that their hyperparameters have been tuned appropriately. Our BMA and MAMA attacks follow a similar setup with 100 iterations, 20 random restarts, and ODI initialization. In all experiments, we use a two-layer LSTM with 20 hidden units in each layer as the optimizer model. We set wt = 1 and λt = 0.1 in Eq. (5) and study different λt through an ablation study. We set the learning rates as β = 0.0001 and γ = 0.001 in MAMA with µ = 1.0. The mini-batch size is B = 32.

heading: 4.2 EFFECTIVENESS AND ADAPTIVENESS OF BMA
text: We first train a different attack optimizer for each defense by BMA. We consider four typical AT defenses on CIFAR-10 (Krizhevsky & Hinton, 2009) — HYDRA (Sehwag et al., 2020), Pre-Training (Hendrycks et al., 2019), Overfitting (Rice et al., 2020), and FastAT (Wong et al., 2020).
Comparison with hand-designed optimizers. Since most existing adversarial attacks adopt handdesigned optimizers, we first compare the performance of our learned optimizer with these handdesigned ones. Specifically, we adopt the CW loss (Carlini & Wagner, 2017) as L in Eq. (1) and set
Table 2: Classification accuracy (%) of four defenses on CIFAR-10 against PGDCE, PGDDLR, MT, ODI-R10, APGDDLR and their extensions by integrating with our proposed BMA method. The default attack iterations are set as T = 20 as detailed in Sec. 4.1, and we also involve another 100-step APGDDLR as APGDDLR-100.
Defense Fixed Step Size Automatic Step Size
PGDCE PGDDLR MT ODI-R10 APGDDLR APGDDLR-100 Baseline +BMA Baseline +BMA Baseline +BMA Baseline +BMA Baseline +BMA Baseline +BMA
HYDRA 60.38 60.27 59.16 59.02 57.63 57.54 57.50±0.01 57.41±0.01 58.89 58.74 58.47 58.37 Pre-training 57.89 57.69 57.86 57.66 55.16 55.09 55.11±0.01 55.04±0.01 57.83 57.44 57.44 57.15 Overfitting 56.08 55.87 55.29 55.13 55.71 55.64 52.68±0.02 52.57±0.01 55.19 54.84 54.60 54.45 FastAT 46.82 46.70 48.19 48.01 43.85 43.72 43.76±0.01 43.66±0.01 48.02 47.53 47.18 46.85
0 5 10 15 20 25 30 Iteration
58.0
58.5
59.0
59.5
60.0
60.5
61.0
A cc
ur ac
y (%
)
= 0.0 = 0.01 = 0.1 = 0.5 = 1.0
Figure 2: The accuracy curves of HYDRA w.r.t. training iterations of BMA with different λ. HYDRA Pre-Training Overfitting FastAT
40
44
48
52
56
60
A cc
ur ac
y (%
)
1,000 2,000 5,000 10,000
Figure 3: Cross-data generalization evaluation with different numbers of training data points on CIFAR-10.
= 8/255. We compare our learned optimizer (in BMA) with vanilla gradient descent (used in PGD (Madry et al., 2018)), momentum (Polyak, 1964) (used in MI-FGSM (Dong et al., 2018)), nesterov accelerated gradient (Nesterov, 1983) (used in NI-FGSM (Lin et al., 2019)), and Adam (Kingma & Ba, 2015) (used in (Uesato et al., 2018; Gowal et al., 2019)). Besides, we also adopt a CNN-based adversarial example generation method (Adv-CNN) (Baluja & Fischer, 2017) for comparison. We report the classification accuracy of the four defenses against these attacks in Table 1. We can see that BMA with the learned optimizer leads to lower accuracy of the defense models than the existing attacks, demonstrating that BMA can discover stronger adversarial attacks automatically.
Ablation study of the prior-guided refinement strategy. We conduct an ablation study to investigate the effects of the prior-guided refinement strategy introduced in Sec. 3.1. Fig. 2 shows the accuracy of HYDRA along the training iterations of BMA with λt := λ = 0.0, 0.01, 0.1, 0.5, and 1.0, respectively. After integrating the PGD update rule as a prior with λ > 0, the learned optimizer achieves faster convergence and better performance than that with λ = 0. The results indicate that the proposed prior-guided refinement strategy can benefit from the informative prior given by the PGD update rule and explore potential better update directions simultaneously. Moreover, as λ increases, to a certain extent the convergence tends to be faster, but the performance tends to be similar. Therefore, we select λ = 0.1 for all experiments in the training phase of BMA.
Adaptiveness. The learned optimizer is generally compatible with previous methods on improving adversarial attacks in different aspects. To verify this, we incorporate our method with PGD variants using the cross-entropy loss (PGDCE) (Madry et al., 2018), the difference of logits ratio loss (PGDDLR) (Croce & Hein, 2020b) and PGDDLR with varying step size (APGDDLR) (Croce & Hein, 2020b), the MultiTarged (MT) attack (Gowal et al., 2019) that performs targeted attack for each class, and the output diversified initialization method (Tashiro et al., 2020) with 10 random restarts (ODI-R10). Note that these methods only change the loss function L or the starting point x̂0, which do not affect the learning process of BMA. Table 2 shows the results of the baseline methods and their extensions by integrating with the learned optimizer in BMA. The results verify that integrating BMA can consistently enhance the performance of each method. The mechanism of combining BMA is concise and easy to implement, yet steadily improving the different attack methods.

heading: 4.3 GENERALIZATION EVALUATION
text: Cross-data generalization. We first evaluate the cross-data generalization ability of the learned optimizer in BMA on unseen data points. Specifically, we randomly select 1, 000, 2, 000, 5, 000, and all 10, 000 data points from the CIFAR-10 test set as the training set for BMA, respectively, and adopt the entire test set of CIFAR-10 for evaluation. The performance on the entire test set is shown in Fig. 3. We can see that the learned optimizers using different numbers of data points achieve
similar attack performance on the four defense models, indicating that the data-driven procedure enables the learned optimizer to obtain a good generalization ability to new unseen data samples.
Cross-iteration generalization. We then evaluate the generalization ability of the learned optimizer in BMA to more attack iterations than training. Fig. 4 shows the curves of model accuracy w.r.t. the number of attack iterations ranging from 1 to 100. We also mark the number of iterations used for training BMA at 20. The results on four defense models verify that BMA can consistently outperform the baseline attacks with varying attack iterations. It demonstrates that the learned optimizers also have a good generalization ability across attack iterations.
Cross-model generalization. The training process may face inconvenience in practice when training a different optimizer for each defense. A better property of the learned optimizer is more generalizable for adversarial attacks on any unseen defenses. We thus consider the challenging and practical scenario — training the optimizer on several available defense models and evaluating its generalization ability to other unseen defense models. This setting requires the optimizer to learn more general knowledge for adversarial attacks. For choosing the defense models in training, we mainly consider using a set of diverse defenses. Thus we randomly sample four models to train the optimizer from TRADES (T) (Zhang et al., 2019b) , AWP (A) (Wu et al., 2020), Unlabeled (U) (Carmon et al., 2019), Hydra (H) (Sehwag et al., 2020), and SAT (S) (Huang et al., 2020). MAMA randomly selects three defenses as meta-train and the other one as meta-test in each iteration. We compare MAMA with two BMA variants. The first one trains an optimizer for each training defense and chooses the best optimizer for evaluation (BMAbs), and the second one optimizes Eq. (5) using the training defenses as an ensemble (BMAen). Table 3 shows that MAMA is more effective than the two methods, indicating that the generalization ability of the optimizer can be improved by adopting the model-agnostic algorithm. The effect of meta-test components is listed in Appendix E.3.

heading: 4.4 COMPARISON WITH STATE-OF-THE-ART ATTACKS
text: In this section, we perform a large-scale experiment to compare the performance of our methods with various state-of-the-art attacks for robustness evaluation of 12 defense models on CIFAR-10. We consider both BMA that learns a different attack optimizer for each defense model and MAMA that only learns one generalizable optimizer adopting the model-agnostic training algorithm to test all defenses. We compare our methods with several typical and state-of-the-art attacks, including 1) PGDCE (Madry et al., 2018), 2) PGDCW, 4) APGD (Croce & Hein, 2020b) with the DLR loss, 3) MT, 5) AutoAttack (AA) (Croce & Hein, 2020b), and 6) ODI (Tashiro et al., 2020). PGDCE, PGDCW, and ODI use 100 iterations with step size 2/255 and 20 restarts. APGD also uses the same iterations and restarts. MT performs targeted attacks for 9 classes except the true one with 2 restarts and 100 attack iterations. We adopt the official implementation of AA. We integrate our methods with ODI to establish stronger attacks. Our methods also use 100 attack iterations with 20 ODI restarts. MAMA is trained on defense models {T,A,U,H} with the model-agnostic algorithm. Table 4 shows the classification accuracy of these defense models against different adversarial attacks. We also show the clean accuracy and the reported robust accuracy of these models in the
corresponding papers. Our methods achieve lower robust test accuracy on all 12 defense models than the baseline attacks, including MT and AA, demonstrating that ours can discover stronger adversarial attacks automatically than the existing attacks with hand-designed optimizers. Meanwhile, our methods are not only stronger but also less complex than AA. Besides, the results also demonstrate that MAMA can achieve a comparable performance of BMA, showing that the learned optimizer has a good generalization ability to unseen defense models due to the involved model-agnostic training algorithm, and could be flexibly used to evaluate the robustness of future defenses.
Robustness evaluation on other datasets. We directly apply the learned optimizer in MAMA on CIFAR-10 to evaluate adversarial robustness on CIFAR-100 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009), without updating the optimizer by new datasets or defenses. We adopt = 8/255 on CIFAR-100 and = 4/255 on ImageNet. During testing, there is a significant shift between training and testing defense models for MAMA. Despite these difficulties, MAMA can consistently improve the performance on most defenses in Table 5. The results verify that MAMA obtains a good generalization ability for adversarial attacks on unseen datasets, benefiting from the data-driven learning procedure automatically.
Running time. Fig. 5 shows the running time of AA, ODI, and BMA (MAMA) based on a single RTX 2080-Ti GPU. BMA is slightly slower than the baseline ODI, but significantly faster than AA.

heading: 5 CONCLUSION
text: In this paper, we propose a Model-Agnostic Meta-Attack (MAMA) approach to automatically discover stronger attack algorithms. Specifically, we learn an optimizer in adversarial attacks parameterized by an RNN model, which is trained over a class of data samples and defense models to produce effective update directions for crafting adversarial examples. We further develop a modelagnostic learning approach with a gradient-based meta-train and meta-test procedure to improve the generalization ability of the learned optimizer to unseen defense models. Extensive experimental results demonstrate the effectiveness, adaptiveness, and generalization ability of MAMA on different defense models, leading to a more reliable evaluation of adversarial robustness.
ETHICS STATEMENT
Our work aims at evaluating the adversarial robustness of defense models more accurately and reliably. It could identify the security problems of deep learning models before they are deployed to avoid potential harm caused by adversarial examples. We are unaware of any direct negative impact of this work. One limitation of our work is that the proposed methods (i.e., BMA and MAMA) cannot directly be used to attack defenses with obfuscated gradients. We still need to design some adaptive attack strategies (e.g., Expectation over Transformation (Athalye et al., 2018b) for randomized defenses) to apply our methods.
REPRODUCIBILITY STATEMENT
We provide the code for reproducing the results in the supplementary material.

heading: A MORE DISCUSSIONS ON RELATED WORK
text: Xiong & Hsieh (2020) propose a similar approach to ours that uses an RNN optimizer to learn attacks for adversarial training. The differences between our method and Xiong & Hsieh (2020) are three-fold. First, the motivations are different: we propose to discover stronger adversarial attack algorithms while they aim to learn robust models by adversarial training. Second, based on the different goals, the techniques are different: we develop a prior-guided refinement strategy and a model-agnostic training algorithm to improve the attacking performance and generalization ability of the RNN optimizer while they do not adopt these techniques and use a relatively smaller number of attack iterations (i.e., T = 10). Third, the empirical evaluations are different: we mainly conduct experiments to show the attack performance on various defense models while they mainly consider the defense performance of adversarial training. Overall, our method differs from Xiong & Hsieh (2020) in the aspects of motivation, technique, and experiment. Despite using a similar Learningto-Learn framework, our method has its own contributions in developing an effective attack method for evaluating adversarial robustness.

heading: B EXPLANATION OF MAXIMUM A POSTERIORI
text: Assume that the output of the RNN optimizer is a random variable gt. The objective of the optimizer is to maximize the loss L at x̂t after applying gt, as in Eq. (3). By using the cross-entropy loss as L, we actually want to minimize py (i.e., the probability of the true class) given x̂t. We let p(y|gt) = 1py such that we want to maximize p(y|gt). Thus p(y|gt) can be viewed as the conditional likelihood. By assuming the prior distribution of gt to be a Gaussian one as p(gt) ∼ N (sign(∇t), 12λt ), we can estimate the posterior distribution of gt using Bayes’ theorem
p(gt|y) ∼ p(y|gt)p(gt).
Under the maximum a posteriori (MAP) framework, we would like to solve
arg max log p(gt|y) = arg max{− log py − λt‖gt − sign(∇t)‖22} = arg max{L(f(x̂t), y)− λt‖gt − sign(∇t)‖22}.
Be accumulating the loss at each step, we can obtain Eq. (5).

heading: C TECHNICAL DETAILS OF ADVERSARIAL ATTACKS
text: In this section, we introduce more related backgrounds and technical details for reference.
PGD. One of the most commonly studied adversarial attacks is the projected gradient descent (PGD) method (Madry et al., 2018). PGD iteratively crafts the adversarial example as
x̂t+1 = ΠB (x) ( x̂t + αt · sign(∇xL(f(x̂t), y)) ) ,
where B (x) = {x′ : ‖x′ − x‖∞ ≤ } denotes the `∞ ball centered at x with radius , Π(·) is the projection operation, and αt is the step size at the t-th attack iteration. L could be the cross-entropy loss as Lce(f(x), y) = − log py where py = ezy/∑Ki=1 ezi denotes the predicted probability of class y, or the margin-based CW loss (Carlini & Wagner, 2017) as Lcw(f(x), y) = maxi6=y zi − zy . MI-FGSM. Dong et al. (2018) propose to adopt the momentum optimizer (Polyak, 1964) in adversarial attacks, which can be formulated as
gt+1 = µ · gt + ∇xL(f(x̂t), y) ‖∇xL(f(x̂t), y)‖1 ; x̂t+1 = ΠB (x)(x̂t + α · sign(gt+1)),
where α is a fixed step size and µ is a hyperparameter.
NI-FGSM. Lin et al. (2019) propose to adopt the Nesterov accelerated gradient (Nesterov, 1983) in adversarial attacks, as
xnest = x̂t+α ·µ ·gt; gt+1 = µ ·gt+ ∇xL(f(xnest ), y) ‖∇xL(f(xnest ), y)‖1 ; x̂t+1 = ΠB (x)(x̂t+α · sign(gt+1)).
Adam. Some attacks (Uesato et al., 2018; Gowal et al., 2019) propose to adopt the Adam optimizer (Kingma & Ba, 2015) in adversarial attacks. By replacing the sign gradient in PGD by the Adam optimization rule, we can get the attack based on Adam easily.
Adv-CNN. Baluja & Fischer (2017) first propose to generate adversarial examples against a target classifier by training feed-forward CNN networks. To efficiently craft adversarial examples under white-box attack, we directly feed the gradients of adversarial images into the CNN, and output the update direction as
g′t = G (L(f(x̂t), y)) ; x̂t+1 = ΠB (x) (x̂t + αt · g ′ t) ,
where G indicates the feed-forward CNN.
AutoAttack (AA). Croce & Hein (2020b) first propose the Auto-PGD (APGD) algorithm, which can automatically tune the adversarial step sizes along with the optimization trend. As for the adversarial objective, they develop a new difference of logits ratio (DLR) loss as
LDLR(f(x), y) = − zy −maxi 6=y zi zπ1 − zπ3 ,
where z is the logits and π is the ordering which sorts the components of z. Finally, the authors propose to group APGDCE and APGDDLR with FAB (Croce & Hein, 2020a) and square attack (Andriushchenko et al., 2020) to form the AutoAttack (AA).
ODI attack. The Output Diversified Sampling (ODS) (Tashiro et al., 2020) belongs to a sampling strategy that aims to maximize diversity in the models’ outputs when crafting adversarial examples. The formulation of ODI attack derived from ODS can be described as
x̂k+1 = ΠB (x) ( x̂k + αt · sign(vODS(x̂k, f,wd)) ) , vODS(x̂k, f,wd) =
∇x(wTd f(x̂k)) ‖∇x(wTd f(x̂k))‖2 ,
where vODS(·) generates output-diversified starting points, and wd is sampled from the uniform distribution over [−1, 1]C . MultiTarged (MT) attack. MT (Gowal et al., 2019) performs targeted attack for all other classes except the true one, which iteratively crafts the adversarial example as
x̂t+1 = ΠB (x) ( x̂t − αt · sign(∇xL(f(x̂t), yc)) ) ,
where yc belongs to the specific targeted attack class.

heading: D REFERENCE CODE OF ADVERSARIAL DEFENSES
text: We provide the architectures and code links for the defenses used in our paper in Table 6. Please note that the checkpoints of evaluating adversarial robustness are released in their code implementations.

heading: E MORE EXPERIMENTS
text: E.1 EFFECTIVENESS UNDER THE `2 NORM
Apart from the results based on the `∞ norm, we study the effectiveness based on the `2 norm. The performance of our learned optimizer with these hand-designed ones is presented in this sec-
tion. Specifically, we adopt the CW loss (Carlini & Wagner, 2017) and set = 0.1. We compare our learned optimizer (in BMA) with vanilla gradient descent (used in PGD (Madry et al., 2018)), momentum (Polyak, 1964) (used in MI-FGSM (Dong et al., 2018)), nesterov accelerated gradient (Nesterov, 1983) (used in NI-FGSM (Lin et al., 2019)), and Adam (Kingma & Ba, 2015) (used in (Gowal et al., 2019; Uesato et al., 2018)). For fair comparison, we set the attack iterations as T = 20 and the fixed step size as α = 0.025 in all methods. The learned optimizer in BMA is trained with 20 attack iterations on the entire test set of CIFAR-10. We report the classification accuracy of the two defenses including Overfitting (Rice et al., 2020) and Robustness (Engstrom et al., 2019) against these attacks in Table 7. BMA with the learned optimizer leads to lower accuracy of the defense models than the existing attacks with hand-designed optimizers, demonstrating the effectiveness of BMA based on the `2 norm.
E.2 RANDOMNESS EXPERIMENTS
To avoid the randomness caused by a single experiment, we report the standard deviations of multiple runs on CIFAR-10 in Table 8. We can see that the learned optimizers using different runs achieve similar attack performance on the four defense models including HYDRA (Sehwag et al., 2020), Pre-training (Hendrycks et al., 2019), Overfitting (Rice et al., 2020) and FastAT (Wong et al., 2020). The results also demonstrate the stability of BMA to existing attacks.
E.3 ESTIMATE THE EFFECT OF META-TESTING COMPONENT
Effect of different µ. The meta-train and meta-test can be optimized simultaneously as Eq. (6), and µ is the hyperparameter to balance the meta-train and meta-test losses. To estimate the effect of different µ, we sample four models to train the optimizer from TRADES (T), AWP (A), Unlabeled (U), Hydra (H), and SAT (S), and test the performance on Pre-Training for different µ. We can see in Table 9, a proper value 1.0 gives the best performance, indicating that the meta-train and meta-test should be equally learned.
Effect of different numbers of meta-train and meta-test. Apart from the results based on different hyparameters, we study the effectiveness for different numbers of meta-train and meta-test. In the experiments, we attempt to samplem domains as meta-train, and remaining n domains as meta-test, thus named Sm-Tn. We report the classification accuracy of the defense model on different numbers of domains in Table 10. We can see that the best performance can be obtained by setting m as 3 and n as 1, which is also adopted in all experiments of MAMA.
E.4 AUTOMATIC STEP SIZE
We also attempt to apply the automatic step size scheduler (Croce & Hein, 2020b) to MAMA in Table 11, which adopts 100 iterations and 20 restarts based on ODI on the different defenses including TRADES (Zhang et al., 2019b), Pre-training (Hendrycks et al., 2019), HYDRA (Sehwag et al., 2020). The results show that integrating the automatic step size does not further improve the overall performance for robustness evaluation. We think that combining MAMA with ODI can already converge to sufficiently good results for robustness evaluation and may not be improved by integrating other techniques such as automatic step size.
E.5 OTHER TYPES OF DEFENSES
We further conduct experiments by considering other types of defenses rather than adversarial training. Specifically, we include a certified defense — CROWN-IBP (Zhang et al., 2020) and a randomized defense — JEM-1 (Grathwohl et al., 2020). We compare our proposed methods BMA and MAMA with AutoAttack (AA), which is the most effective baseline. We re-train the attack optimizer for each model in BMA and directly use the learned optimizer (i.e., the same as Table 4) in MAMA. To counter the randomness of JEM-1, we average the gradients over 3 runs (known as Expectation over Transformation (EoT) (Athalye et al., 2018b)). The results are shown in Table 12. Our methods consistently achieve better performance than AutoAttack.

Review summaries:
This paper proposes an automatic approach for attacking classifiers, by approximating a possible adaptive attack that can take place on a newly-published defense. The methodology (MAMA) applies meta machine learning techniques, by training it on different defenses to let it grasp what might be a good signal to follow when attacking an unknown classifier. The authors then test this strategy and a naive version of it (that is, the one that is only trained on the attacks, and not fine tuned on a test set) against other attacks proposed in literature. Results suggest a modest improvement against defenses hardened with adversarial training.

This paper proposes a model-agnostic meta-attack and achieves promising adversarial attack performance compared with state-of-the-art adversarial attack algorithms. The proposed algorithm overcomes many issues, such as the vanishing gradient problem, training instability problem, generalization to other defense models. The detailed experiments support the proposed method.