Meta Adversarial Training
Authors: Jan Hendrik Metzen
Nicole Finnie
Robin Hutmacher
Venue:  (2021)
Keywords: robustness
adversarial examples
adversarial training
physical-world adversarial attacks
adversarial patch
universal perturbation

Abstract: Recently demonstrated physical-world adversarial attacks have exposed vulnerabilities in perception systems that pose severe risks for safety-critical applications such as autonomous driving. These attacks place adversarial artifacts in the physical world that indirectly cause the addition of universal perturbations to inputs of a model that can fool it in a variety of contexts. Adversarial training is the most effective defense against image-dependent adversarial attacks. However, tailoring adversarial training to universal perturbations is computationally expensive since the optimal universal perturbations depend on the model weights which change during training. We propose meta adversarial training (MAT), a novel combination of adversarial training with meta-learning, which overcomes this challenge by meta-learning universal perturbations along with model training. MAT requires little extra computation while continuously adapting a large set of perturbations to the current model. We present results for universal patch and universal perturbation attacks on image classification and traffic-light detection. MAT considerably increases robustness against universal patch attacks compared to prior work.  

Under review as a conference paper at ICLR 2021
META ADVERSARIAL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Recently demonstrated physical world adversarial attacks have exposed vulnerabili- ties in perception systems that pose severe risks for safety-critical applications such as autonomous driving. These attacks place adversarial artifacts in the physical world that indirectly cause the addition of universal perturbations inputs of a model that can fool it in a variety of contexts. Adversarial training is the most effective defense against image-dependent adversarial attacks. However, tailoring adversarial training to universal perturbations is computationally expensive since the optimal universal perturbations depend on the model weights which change during training. We propose meta adversarial training (MAT), novel combination of adversarial training with meta-learning. which overcomes this challenge by meta meta-lcarning universal perturbations along with model training. MAT requires little extra computation while continuously adapting a large set of perturbations to the current model. We present results for universal patch and universal perturba- tion attacks on image classification and traffic-light detection. MAT considerably increases robustness against universal patch attacks compared to prior work.
INTRODUCTION
Deep learning is currently the most promising method for open-world perception tasks such as in automated driving and robotics. However, the use in safety-critical domains is questionable, since
lack of robustness of deep learning- -ased perception has been demonstrated (Szegedy et al. 2014
Goodfellow al. 2015 Metzen et al. 2017 Hendrycks & Dietterich 2019).
Physical-world adversarial attacks (Kurakin al. [017 Athalye al.) 2018; Braunegg et al. 2020) are one
Undefended
Meta Adversarial Training (MAT)
335
0.65
of most problematic failures in robust- ness of deep learning. Examples of such attacks are fooling models for traffic sign recognition (Chen et al.
0.76
2018 Eykholt et al., 20188 b] Huang 2019), face recognition (Sharif 2016 2017). optical flow es- timation (Ranjan et al., 2019). per- son detection Thys et al. 2019 Wu
et al. cct al.
eet al.) 2020b Xu et a 2020), and
Figure 1: Illustration of a digital universal patch attack
LiDAR perception (Cao et al. 2019). In this work, we focus on sub- sets of these physical-worlde attacks: local ones which place a printed pat- tern in a scene that does not over- lap with the target object (Lee & Kolter, 2019) Huang et 2019) and global ones which attach mainly- translucent sticker on the lens of camera (Li et al., 2019). Note that these physical-world attacks have responding digital digital-domain attacks. in which the attacker directly modifies the signal after was received by the sensor and before it is processed by the model. The corresponding digital-domain
against an undefended model (left) and model defended with meta adversarial training (MAT, right) on Bosch Small Traffic Lights (Behrendt & Novak 2017). A patch can lead the undefended model to detect non-existent traffic lights and miss real ones that would be detected without the patch (bot- tom left). In contrast, the same patch is ineffective against a MAT model (bottom right). Moreover, a patch optimized for the MAT model (top right), which bears resemblance to traffic lights, does not cause the model to remove correct detections.

---

Under review as a conference paper at ICLR 2021
attack for the adversarial camera sticker is type of universal adversarial perturbation (Moosavi Dezfooli et al.) 2017), while the digital adversarial patch attack (Brown et al. 2017 Anonymous
2020) corresponds to physical patch attacks (Lee & Kolter 2019 Huang et al. 2019).
We focus on increasing robustness against higital-domain attacks. Digital-domain attacks are strictly stronger than the corresponding physical-world attacks since they allow the attacker to have complete control over the change of the signal. In contrast, physical-worf attacks need to be invariant under non-controllable effects such as scale, rotation, object position, and light conditions, which cannot be controlled by the attacker. Therefore, a system robust against digital-domain attacks is also robust against the corresponding physical-world attacks.
Currently, the most promising method for increasing robustness against adversarial attacks is ad- versarial training (Goodfellow al., 2015 Madry al. 2018). Adversarial training simulates an adversarial attack for every mini-batch and trains the model to become robust against such an attack. Adversarial training against digital universal perturbations or patches is complicated by the fact that these attacks are computationally much more expensive than image-dependent adversarial attacks and existing approaches for speeding up adversarial training (Shafahi et al. 2019 Zhang et al. 2019; Zheng et al.) 2020) are not directly applicable. Existing approaches for tailoring adversarial training to universal perturbations or patches either refrain from simulating attacks in every mini- batch (Moosavi- (Moosavi-Dezfooll Dezfooli et al. 2017 Hayes & Danezis 2018 Perolat al. 2018). which bears the risk that the model easily overfits these fixed or rarely updated universal perturbations. Alternative approaches use proxy attacks that are computationally cheaper such as "universal adversarial train- ing" (UAT) (Shafahi et al., 2018) and "shared adversarial training" (SAT) (Mummadi et al. 2019). These approaches face the challenge of balancing the implicit trade-off between simulating universal perturbation attacks accurately and keeping computation cost of the proxy attacks small.
We propose meta adversarial training (MAT which falls into the category of proxy attacks. MAT combines adversarial training with meta-learning. We summarize the key novel contributions of MAT and refer to BectionB]oor details:
MAT amortizes the cost of computing universal perturbations by sharing information about optimal perturbations over consecutive steps of model training, which reduces the cost of generating strong approximations of universal perturbations considerably. In contrast to UAT (Shafahi et al. 2018), MAT uses meta-learning for sharing of information rather than joint training, which empirically generates stronger perturbations and more robust model.
MAT meta-learns a large set of perturbations concurrently. While a model easily overfits a single perturbation, even if it changes in UAT, overfitting is much less likely for larger set of perturbations such as those generated with MAT.
MAT encourages diversity of the generated perturbations by assigning random but fixed target classes and step-sizes to each perturbation during meta-learning. This avoids that many perturbations focus on exploiting the same vulnerability of a model.
We perform an extensive empirical evaluation and ablation study of MAT on image classification and traffic-light detection tasks against variety of attacks to show the robustness of MAT against universal patches and perturbations (see Section[4). We refer to Figure I for an illustration of MAT for universal patch attacks against traffic light detection.
2 RELATED WORK
We review work generating universal perturbations, defending against them, and meta-learning.
Generating Universal Perturbations Adversarial perturbations are changes to the input that are crafted with the intention of fooling a model's 's prediction on the input. Universal perturbations are special case in which one perturbation needs to be effective on the majority of samples from the input distribution. Most work focuses on small additive perturbations that are bounded by some norm constraint. For example, Moossaaiieaiiiieiotooolll et al. ( (2017) proposed the first approach by extending the DeepFool algorithm -Moosavi-Dezfool iterative fast gradient sign method (Kurakin et al. (017)) for generating universal perturbations on
al. (0016). Similarly, Metzen al. (2017) extended the
semantic image segmentation. Mopuri et al. (2017 2018 presented iata-indeeendent attacks and
Code will he publicly released upon acceptance and he found in the supplementary material.
2

---

Under review as a conference paper at ICLR 2021
Hayes & Danezis 2018 proposed using a generative model for learning a diverse distribution of universal perturbations. Li et al. (2019) presented a physical-world attack in which a translucent sticker is placed on the lens of a camera, which adds a universal perturbation to the image taken by the camera, and showed that this can fool an image classification system.
Other types of universal perturbations are so-called adversarial patches (Brown et al. 2017). In these universal patch attacks, the adversary can arbitrarily modify a small part of the image, typically a connected rectangular area, while leaving the remaining part of the image unchanged. Following Athalye et al. (2018), randomizing conditions such as location, rotation, scale, and lighting during the attack can make the universal patch sufficiently effective fool the model when it printed out and placed in the physical world. Later work has generalized these physical-worf attacks to object detection (Lee & Kolter 2019: Huang et al. 2019) and optical flow estimation (Ranjan et al. 2019).
Defending Universal Perturbations First works for defending against universal perturbations are based on training a model against a fixed or slowly updated set/distribution of universal perturbations: Doosavi-Dezfooli et al. (2017) precompute a set of universal perturbations that are used during training, Hayes & Danezis (2018) learn generative model of universal perturbations, and Perollat ett al. (2018) build a slowly increasing set of universal perturbations concurrent to model training. A shortcoming of these approaches is that the model might overfit the fixed or slowly changing distribution of universal perturbations. However, re-computing universal perturbations in every mini-batch from scratch is prohibitively expensive. To address this issue, SAT Mummadi et al. (2019) trains a model against so-called shared perturbations. These shared perturbations do not have to be universal but only need to fool the model fixed subset of the batch. However, since the shared perturbations are recomputed in every strong perturbations from random initialization. In contrast, our method meta-learns strong initial perturbations. In UAT (Shafahi et al. 2018), training the neural network's ' weights and updating single universal perturbation happen concurrently, which scales to a large dataset. However, our experiments in Section 4 indicate that single incrementally and slowly updated perturbation is not sufficiently strong and diverse for making a model robust against all possible universal perturbations. Instead, our method meta-learns large and diverse collection of perturbations during training.
it assumes few gradient steps are sufficient to find
For defending against adversarial patches, Chiang et al. (2020) proposed an approach of extending iiterval-boune propagation (Gowal et al., (019)) to the patch threat model. While this allows certification of robustness, only scales to tiny patches and reduces clean accuracy considerably. Wu et al.] (2020a) proposed the "defense against occlusion attack", which applies adversarial training to inputs perturbed with input-dependent adversarial patches placed at specific positions determined, for example, by the input gradient magnitude. Since they generate patches from scratch, they require an expensive optimization of the patch for every training batch. Moreover, robustness against stronger attacks such as those proposed in Section 4.1 remains unclear. [Saha et al.] (2019) hypothesize that vulnerability of object detectors against adversarial patches stems from contextual reasoning. Accordingly, they propose Grad-defense which penalizes strong dependence of object detections on their context in a data-driven manner, where dependence is determined by Grad-CAM (Selvaraju al., 2019 Lastly, some non-ad versarilll data augmentation techniques resemble the universal adversarial patch scenario: they add a Gaussian noise patch (Lopes et al. 2019) or a patch from a different image (CutMix) (Yun et al. 2019) to each input. CutMix conceptually very similar to the out-of-conten defense (Saha et al. 2019). However, as demonstrated in our experiments in Section 4 even though these approaches increase robustness against occlusions, they are unlikely to increase robustness against universal patch attacks.
Meta-Learning Gradient-based meta-learning methods such as MAML (Finn et al. 2017) or REPTILE (Nichol et al. 2018) allow learning initial parameters for class of optimization tasks, so that one can find close-to-optimal parameters on a novel task from the distribution with a small number of gradient steps. Moreover, meta-learning can also be used to learn the task optimizer itself such as by Xiong & Hsieh (020)) in the context of adversarial training. While it is common to meta-learn iitiill weights for neural networks, we propose that these algorithms can also be used to meta- meta-leam initial values for universal perturbations. In this work, combine REPTILE with adversarial training because of the low computational overhead of REPTILE; however, in principle other gradient- pradeent-based meta-learning methods could also be used as part of our method.
3

---

Under review as a conference paper at ICLR 2021
3 META ADVERSARIAL TRAINING
In this section, we propose a novel combination of adversarial training with meta-learning that trains models to be robust against universal perturbations.
3.1 PRELIMINARIES
Let DD be a distribution over d-dimensional datapoints E [0, 1] and corresponding labels y, 0 model parameters to be optimized, and L a loss function. Moreover, let S be the set of allowed perturbations and F be function that applies perturbation & E S to datapoint, potentially dependent and some randomness r ~ R. For universal perturbations (Moosavi-Dezfooli et al. 2017), one may choose S { || I <) for some small and (.r, pl | g]. A Alternatively for universal patch attacks (Brown et al. 2017). we may define a mask m E {0, 133 set S = [0, 1j and let ()) = (1 - (m, r]) x + TTm, rT) with 7' = T, ...... To being sequence of stochastic transformations T; and some randomness r ~ R governing the stochasticity of the transformations. That is, each patch and mask are consistently transformed with T. e.g., translated, scaled, and rotated, and the transformed patch is applied to the input where the transformed mask is 1 and the input remains unchanged otherwise.
the label
Following Mummadi et al. (2019), we define the universal adversarial risk as
Puni
max EES D,r
E
[(0, F(,, (.r.
where we drop the explicit dependence of Puni on S, D, and R. Generally, we are interested in finding model parameters that minimize the universal adversarial risk, denoted as 0* arg ming Punn((), This corresponds to the standard min-max saddle point formulation of adversarial training introduced Madrr et al. (2018). where we incrementally update the model parameters A by computing 0 based on Ve, Pune (Bt) (or more precisely an approximation of (f))( However, in contrast to standard adversarial training, the inner maximization problem is optimized over expected value with respect to the data distribution and potential randomness R. making it more expensive to solve (even approximately). As the optimal & of the inner maximization step i of the outer minimization depends on the parameter value O,, this maximization of &, needs to be repeated in every step of the outer minimization, making the direct minimization of Puna(() (0) intractable.
Existing work has addressed this in different ways. One approach (Moosavi-Dezfooli et al. 2017 Hayes & Danezis 2018 Perolat et al., 2018 relaxes the explicit dependence & on 6, and computes a set or distribution over & for some parameter checkpoints of f, and then applies these perturbations to the model while updating its parameters A. One shortcoming that the outer minimization of can converge to a value for which the precomputed set distribution over & loses its effectiveness even though there are still some other choices of & which are effective for 6. Another approach is proposed by Muummaii et al. (011): they instead replace the distribution D in the inner maximization with the current batch of the outer minimization. The effectiveness this procedure hinges on the ability to efficiently approximate this inner maximization with few gradient steps. In summary, the main challenge of using adversarial training to increase model robustness against universal perturbations is efficiently approximating & for the current 0, in every step of the outer minimization.
3.2 META-LEARNING UNIVERSAL PERTURBATIONS
In contrast with the aforementioned approaches and similar to UAT (Shafahi et al. 2018). we exploit the property that one step of the outer minimization only applies small change to 6; thus, for consecutive steps and /. 1+1 of the outer minimization, the resulting inner maximization problems for finding & and E++1 are closely related (Zheng et al. 2020). UAT exploits this property by initializing the inner maximization at + 1 with the (approximate) solution for & and performs single gradient step on a single batch in the inner maximization at t+ 1. A potential shortcoming of this method is that it uses only single gradient-step and thus implements joint training of parameters and perturbation, which does not allow capturing higher-order derivatives of the loss function (Nichol et al. 2018 and may therefore learn suboptimal initial parameters.
In order to address this shortcoming, we propose an approach to meta-learn initial values for universal perturbations- by approaching the optimization problems (&., ).M. with gradient-based meta-laarning:
4

---

Under review as a conference paper at ICLR 2021
in parallel to updating 0g in the outer minimization, meta-leam an initialization E,, which we refer to as the "meta- penettapeturbatiion at time step t of the outer minimization. That allows for approximating the inner-optimization problem of ξ with few gradient steps. More precisely, use the REPTILE (Nichol et al., 2018) meta-learning algorithm with the iterative fast gradient sign method (I-FGSM) (Kurakin et al. 2017) task learner. In the inner maximization,
employ K iterations of I-FGSM r), v))
with the following update: < E, and < ( ITs
{t + a sgn (V F(r,
where & indexes the inner maximization iterations, IIs denotes projection on the set S, and a the step size of I-FGSM. The key difference compared to standard I-FGSM and PGD (Madry et al. 2018 is
that the initialization ( is neither constant nor randomly sampled but meta-learned.
The resulting perturbation <(K) is used two-fold: first, it is used with the REPTILE meta-learner for updating E with the following update: E++1 = (1 - -), + ag(k) where a the learning rate of REPTILE. Second, <(K) is used in the next step of the outer minimization as an approximation of the optimal & for the sample (r,, y) with randomness r. Learning the universal perturbation in UAT can be seen as a special case of our procedure for K 1 and = 1.
We estimate the expected loss D(zx)) ((-) in the I-FGSM task learner based on a single sample (zt, v) D,r R. Moreover, we use the same sample in all K steps of I-FGSM time well as in the outer minimization step of updating 6, This provides us benefits of reduced variance and more efficient computation, however, at the cost of a biased estimate of Pani I-FGSM will
converge to an { that is overfit to (x. y) and r. Compared to perturbation optimized over the entire distributions DD and R. E will incur a higher loss on the sample. Nevertheless, since we typically choose the number of I-FGSM steps K < 10, we expect only weak overfitting and the gains from reduced variance more than compensates for the increased bias.
3.3 META-LEARNING DIVERSE COLLECTIONS OF PERTURBATIONS
While the procedure proposed in Section 3.2 allows for meta-learning of a single pettappettubattonn E, one such meta-perturine perturbation can easily get trapped in local optimum, from which gradient-based meta-learning cannot easily escape. For instance, in classification task with C classes, there will likely be at least C' local optimal perturbations. More precisely, for each class, there is at least one optimum which corresponds to the perturbation that maximizes the model's prediction of this class. Hence, we propose a meta-learning approach that learns not just a single meta-perturbation but rather an entire set P of P meta-pene pettaaperturbaionn, where the chosen P should be proportional to C. For each sample, we select one of these meta-perturbations that will be used for initializing I-FGSM and later get updated by REPTILE.
However, meta-learning a set of mnta-peeturbattons in this way with the same optimizer and ob- jective will not automatically result in diverse set of petarpeeturbattons.. For instance, many of the perturbations might focus on exploiting similar weaknesses of a model such as triggering a misclassification the same class. Moreover, utilizing the same task learner with the same step size might result in perturbations with similar properties. To alleviate this optimization problem, we encourage diversity of the generated set of meta- petttappeeubrbiions:. for every meta- meta-perturee secea-perturbation. we randomly assign target and perform targeted I-FGSM attack. This avoids many perturbations converging to similar patterns that fool the model into predicting the same target. Moreover, we also assign randomly chosen fixed step size for I-FGSM to every meta- vetta-petturbatoo.. Larger step sizes correspond to meta-perturbation that explore the space of allowed perturbations more globally while smaller step sizes result in more fine- fnniine attacks. We empirically evaluate the effectiveness of these heuristics in Section
3.4 META ADVERSARIAL TRAINING
We summarize the proposed meta adversarial training (MAT) in Algorithm | The function INIT" (see Algorithm 2 in the appendix) is responsible for initializing P consisting of P meta-perturbations E, along with corresponding targets y/ torget and step-sizes a; For classification tasks, we select the target as of the classes in round-robin fashion. Moreover, select the step size -niffrrlyy from [0.0001, 0.1]. We initialize the meta meta- perturbations by either sampling uniform randomly from [0, 1]d by (sub-sampling) an actual data-point, which corresponds to on-manifold initialization
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Meta Adversarial Training
Input: data D. initial parameters application fet. F, loss-fct. L, REPTILE learn-rate o
2: P INITT (D, "data") # Initialize P tuples of petturpeturb,tionn, target, and step size
3: fmar steps of outer minimization, we drop subscript t below for readability
4: for t in f maa - 1} do
5: (ar, y) ~ D # Sample datapoiint
6: # Select meta-pert. =\t and corresponding target yfarget, step-size a, and randomness from P
7:
E. ytargett a, r <- SELECT (P.I. 0,, C, R)
8:
Ç & I-FGSM K (=, Iz, ytrrget a.r)) Inner maximization initialized with meta-pert. =
9:
Pert + Frr, S. Apply perturbation & to
10:
0 OPT 0, pert,, y) # Outer minimization step with optimizer OPT, for instance SGD
11:
E (1- o)-o)=+oo of Meta-learning update of meta-perturbation E with REPTILE
Pt UPDATE(P, # Replace updated meta-perturbation in P
13: end for
akin to CutMix (Yun et al. 2019). This data-initialization was concurrently proposed by Yang et al. (2020b), and Yang et al. (2020a) found that such texture patches can be adversarial even without further optimization. The function SELECT (see Algorithm 3 in the appendix) uniform randomly samples F trials of (E, y gtagget a) from P with randomness maximizes the loss ((,,,)))
R and returns the trial which
Line 8-11 present the core of MAT consisting of (i) inner maximization of perturbation & that was initialized from meta-perturbation with I-FGSM K (Line 8), (ii) a step of outer minimization of 9g with optimizer like SGD a pair of perturbed input xeert and corresponding label y (Line 10), and (iii) the meta-learning update of the respective meta-perturbation E with REPTILE (Line 11). While Algorithm shows the procedure for batch size equal to one, we can easily run it also for larger batch sizes. The only required change is that REPTILE E-based meta-learning can deal with the situation where the same pertuueetunnttionn selected and optimized for several elements in batch. In this case, the meta-learning update becomes = (1 - (1-o)= + o Ei=1 & for N perturbations & that were initialized with the same meta-perturbation E.
We briefly summarize the main advantages of MAT compared to prior work: as opposed to UAT, MAT meta-lcarns diverse set of meta perturbations with I-FGSM concurrently model training rather than jointly training model parameters and a single perturbation with FGSM. Compared to SAT (Mummadi al. 2019), MAT does not treat every inner maximization problem indepen- dently but meta-learns strong initializers, allowing MAT to find stronger perturbations with no more computational cost than standard adversarial training (see Section A). In contrast the work of Moosavi-Dezfooli et al. (2017); Hayes & Danezis (2018); Perolat et al. (018)) MAT computes novel perturbations in every iteration of model training (outer minimization). We would also like note that MAT meta-learns perturbations but not model weights and thus results in standard trained model that does not require test-time adaptation.
4 EXPERIMENTS
We evaluate the performance of MAT, ablated versions of MAT, and baselines. We present results for universal patch attacks on image classification on Tiny ImageNet (Tin) (results for universal perturbations attacks are provided in Section C.1.2). In addition, we present results for a universal patch attack for an object detection task on the Bosch Small Traffic Lights Dataset (Behrendt & Novak 2017). For detailed description of the implementation of the experiments, we refer to Section B in the appendix. In order to compare different methods for increasing robustness against universal patches, a reliable way of evaluating their robustness required, which we outline next.
4.1 RELIABLE ROBUSTNESS EVALUATION
We outline strong attacks for reliably evaluating the robustness of trained models against universal perturbations and patches. Importantly, we do not use the meta-learned perturbations E, as this might result in biased robustness evaluation. Instead, extend PGD (Madry et al. 2018) in similar

---

Under review as a conference paper at ICLR 2021
way as Mummadi et al. (2019) by rewriting Puns from Equation to (un)(
max p(B,g) with SE5
((0,5)=
E [(0 ((r,)))))
(rr
and then use the estimate
p(0,E) N E F(n,
based on samples (I); 16) (6) ~ D and r; ~ R. Finally, we define stochastic projected gradient descent (S-PGD) as g(0) S and g(k) = IIs [g(k-1) + a sgu( (e g(k=1)))]. Note that S-PGD uses different 24; Ts Ty in every step when estimating A.
In general, S-PGD will converge to local optima; namely, {(K) obtained after K steps of S-PGD will not necessarily be the global maximizer of ((0, To account for this, we propose three extensions of S-PGD: Firstly, since the initialization of Eo will generally affect the quality of {(K), we propose an alternative initialization akin CutMix (Yun al., 2019) where we initialize g(0) based on datapoint x ~ D. For universal patch attacks, we downsample or crop x to the patch size, whereas for universal perturbation attacks, we scale its intensity range such that x E S. This initialization becomes even more effective if we sample many x ~ DD and select the one for initializing g(0) which would maximize ~(0, ( We denote this initialization as data initialization.
Secondly, we take inspiration from recently proposed low-frequency attacks (Guo et al. 2019 Sharma ett al. 2019) : we modify the process of adding perturbation to an input Flz,, LP(C, u), r.). where LP C, u) denotes a low-pass filter with cutoff- freuufffreuencyy u. To achieve this, we follow Jo & Bengio 2017) and create centered radial mask with radius u. The patch is transformed into frequency space and multiplied by the radial mask. The result is transformed back to image space and thus yields the patch to be applied the image. While this makes the attack weaker in principle since only low-frequency perturbations are possible, we observe that in practice, it can lead to more well-behaved optimization problem and result in S-PGD converging to stronger perturbations.
Thirdly, we perform a transfer attack, in which we run an attack after every epoch of model training. We initialize c(0) with one of the g(k) found in previous epochs, namely the one that would maximize p(0, 5 { After every 5 epochs, we run an additional S-PGD attack from randomly initialized ę (()), This transfer attack helps identify cases where universal perturbations found in early epochs remain effective against the model but in later epochs are no longer found when running S-PGD attacks from random data initialization.
4.2 IMAGE CLASSIFICATION ON TINY IMAGENET
We evaluate robustness against universal patches of size 24x24 pixel that cover approximately 14% of the image. Patches are randomly translated from the center of the image by at most 26 pixels.
We train every model for 75 epochs with SGD, initial learning rate of 0.033, cosine decay learning rate scheduler, momentum 0.9, and a batch size of 128. We use a ResNet (He et al. 2016). train it from scratch, and follow Xie & Yuille (2220) by replacing batch normalization with group normalization (Wu & Hc. 2019) and weight standardization (Qiao et al. 2019). We use к = 5 iterations of I-FGSM in AT (Madry et al. 2018). SAT (Mummadi et al., 2019). and MAT. For UAT (Shafahi et al.) 2018), we use K = 1 following their recommendation. For SAT, we use sharedness 128. We note that all adversarial training baselines were trained against patch attacks.
For every setting, we perform 5 independent runs. We evaluate the robustness against 2500- step-S-PGD with a batch size of 64 and random initialization, data initialization (data sam- ples resized to patch size), and low-frequency filter, and the transfer attack (see Section 4.1). For the S-PGD settings, we perform a grid search (see Table 4) over step sizes a e (0.0001, 0.00033, 0.001, 0.0033, 0.01, 0.033, 0.1} and momentum for every trained model and report the minimal accuracy. Finally, we report the minimal accuracy across all attacks.
{0, 0.9, 0.99) independently
Results are summarized in Table D (more details can be found in Section We observe that model trained with standard empirical risk minimization offers no robustness against any of the
7

---

Under review as a conference paper at ICLR 2021
Table 1: Accuracy (mean over 5 runs) of different methods on Tiny ImageNet clean data (CI.) and against universal patch attacks with random init (RI), init with cropped image patch (DI), low-frequency filter (LF), transfer attacks (Tr), and worst across all four attacks (Min).
SETTING
CL
DI
LF
TR
MIN
STANDARD
0.55
0.03
0.03
0.07
0.03
0.02
CUTMIX YUN AL.
2019)
0.57
0.03
0.03
0.07
0.03
0.02
PATCHUNIFORM LOPES ET AL.
2019
0.57
0.04
0.06
0.10
0.04
0.03
AT
MADRY ET AL. 2018)
0.57
0.06
0.07
0.12
0.17
0.06
SAT
MUMMADIET AL.
2019
0.58
0.07
0.07
0.12
0.33
0.06
UAT
SHAFAHI ET AL.
2018)
0.49
0.41
0.09
0.15
0.12
0.09
MAT
(FULL)
0.59
0.56
0.55
0.53
0.55
0.53
(RANDOM INIT)
0.58
0.56
0.32
0.26
0.52
0.23
(UNTARGETED)
0.58
0.56
0.56
0.56
0.50
0.50
(F = 1)
0.58
0.53
0.53
0.50
0.52
0.48
(K = 1)
0.58
0.55
0.53
0.53
0.46
0.45
(a = 1.0)
0.58
0.57
0.55
0.55
0.50
0.50
evaluated attacks. Similarly, PatchUniform (akin to Lopes et al. (2019) but with uniform rather than GaussianNoise) CutMix (Yun et al.] 2019) augmentation do not offer robustness against strong patch attacks. When comparing the baseline adversarial defenses (Madry et al.] 2018 Mummadi et al. 2019; Shafahi al. 2018), none of them exhibit more than trivial robustness. UAT provides relatively high robustness against S-PGD with random initialization, even though we perform an extensive grid search over hyperparameters of S S-PGD. However, this robustness does not carry over to other attack variants. Closer inspection of AT and SAT (see Figure 4 in the appendix) shows that also for these methods only very specific combinations of learning rate and momentum allow for discovery of effective patches from random initialization. In summary, since S-PGD with random initialization, single step-size, and momentum value is often used as the default evaluation, we suspect that some prior work might have overestimated the robustness of existing methods.
In contrast, MAT (full) with standard parameters (INIT with data initialization and P 1000 meta- perturbations, targeted attacks, F = 5 in SELECT, K 5 iterations in I-FGSM, REPTILE learning rate a 0.25) shows high robustness against all attack variants. When ablating MAT, choosing a random initialization in INIT is most problematic it results in similar but less severe overfitting to randomly initialized attacks as UAT. Also, ablating towards joint training (o = 1.0 and K 1) deteriorates performance relative to meta-learning in MAT (full). In addition, enforcing diversity in meta-perturbaatios via targeted attacks in MAT (full) is responsible for a small increase in robustness compared to untargeted learning of notaappeturbations. Finally, taking the worst over F samples in SELECT outperforms random sampling (F = 1).
We observed that baseline adversarial training procedures like AT, SAT, and UAT are very sensitive to the step size a used in the inner maximization. Although we have tuned their step sizes to some extent, there might be some effective with which their performance would improve. However, MAT does not require us to tune at all since it uses different learning rates for every meta peeta--ertubbtioo. and thus it covers a broad range of step sizes automatically. Most importantly, MAT offers increased robustness without affecting clean performance. In contrast, MAT acts as an effective regularizer and reduces overfitting compared to standard training and achieves the strongest clean performance among all methods, surpassing standard training by percentage points. Even against the strongest patches, MAT only loses 2 percentage points accuracy relative to standard training on clean data, despite the relatively large patch size. We provide illustrations of patches in Section D and show in Section A that MAT has similar computational cost as standard adversarial training.
4.3 OBJECT DETECTION ON BOSCH SMALL TRAFFIC LIGHTS DATASET
We evaluate robustness of a traffic light detector based on YoloV3 Redmon & Farhadi 2018). This attack scenario is good proxy for physical-wordh attacks automated driving systems since traffic light detection crucially relies on camera- based perception. We add 64x64 patches to 1280x704 images, covering 0.45% of the image, and add random translations from the center by up to (512, 282)
8

---

Under review as a conference paper at ICLR 2021
Table 2: Mean recall (left) and mean average precision (right) of different methods clean data (CI.) and against universal patch attacks with random init (RI), init with a cropped image patch (DI) and low-frequency filter (LF) on Bosch Small Traffic Lights (Behrendt & Novak (017).
Recall
C1.
RI
DI
LF
MIN
mAP
Ct.
RI
DI
LF
MIN
STANDARD
0.47
0.37
0.38
0.43
0.37
0.41
0.09
0.10
0.16
0.09
UAT
0.47
0.45
0.45
0.45
0.45
0.41
0.40
0.29
0.26
0.26
MAT (DEFAULT)
0.45
0.43
0.43
0.43
0.43
0.41
0.39
0.35
0.35
0.35
(+DATA INIT)
0.45
0.43
0.43
0.42
0.42
0.41
0.38
0.39
0.39
0.38
(K 1)
0.46
0.44
0.43
0.44
0.43
0.41
0.40
0.31
0.27
0.27
(K 1K=1P=1 0.46
0.44
0.43
0.43
0.43
0.41
0.40
0.21
0.17
0.17
pixels. We train the models with ADAM for 15 epochs with batch size and learning rate 0.0001. We replace batch normalization by group normalization with weight standardization.
We evaluate the effectiveness of universal patch attacks using two metrics: mean Average Precision (mAP) and mean recall over classes for a fixed confidence threshold. While mAP captures both non non-existent detections caused by the patch (false positives) and correct detections missed by the model (false negatives), mean recall focuses only on the latter. In other words, recall captures "blindness" attacks (Saha et al. 2019) that could be more dangerous in real life scenarios. We set the confidence threshold to 0.3, the non-maximum suppression threshold to 0.1, and the IOU threshold for evaluating true positives to 0.1, respectively.
For attacks with S-PGD, we run 4000 steps with batch size of 4, fix momentum to 7 0.9, and perform a grid search (see T over step sizes a E (0.1, 0.01, 0.001, 0.0001 } and cutoff' frequency u E (25, 50, 100) of the optional low-pass filter. Moreover, we also conduct a grid search over three options for the loss maximized by the attacker: the standard loss also used for model training, the standard loss subtracting the objectness loss as proposed by [Saha al. (2019) and the standard loss ignoring all false positives. The last loss variant is also well suited for "blindness" attacks since accentuates false negatives. For MAT (default), we use random initialization in INIT, P = 10 retta-erturbattoons F = 1 in SELECT, K 3 iterations in I-FGSM, and learning rate a 0.25 REPTILE as the default configuration.
In general, there are no systematic differences between training methods on clean data. Standard training faces a drop in mean recall when universal patch is added. Thus, standard training is likely to be susceptible to physical-world blindness attacks, which could for example cause the model to ignore real traffic lights. In contrast, UAT and all variants of MAT very robust against the tested blindness attacks even in the digital domain. In terms of the mAP, UAT faces a considerable drop for patch attacks initialized with data crops and with low-frequency filter. A similar but weaker effect can also be observed for MAT with the default configuration. Both methods therefore detect non-existent traffic lights on the patch or in its vicinity. Interestingly. false positive detections often resemble traffic lights (see Figure [ and Section Eb. Despite this resemblance, human would not be fooled by these patches. The same holds true for MAT with data initialization in INIT: its high mAp indicates that is very robust in terms of prevention of additional false positives. When ablating MAT, we observe that its mAP deteriorates as the configuration approaches UAT (K and P 1). We conclude that all aspects of MAT are essential for achieving maximal robustness.
summarizes the results (more details can be found in Section
5 CONCLUSION
We propose meta adversarial training (MAT), a novel combination of adversarial training with meta- learning that allows the increase of model robustness against universal perturbations and patches with little computational overhead. Moreover, can be fooled by stronger attacks. In contrast, MAT remains robust against all evaluated attacks. Our results show that further research into attacks or alternatively scaling up certification procedures (Chiang et al. 2020) is required for reliably evaluating robustness against universal perturbations. On the other hand, our results also indicate that physical-word attacks will become considerably more difficult against models trained with MAT.
show that prior work, which was assumed to be robust,

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tiny ImageNet Visual Recognition Challenge.
com/ Accessed: 2020-02-05.
Authors Anonymous. On Adversarial Patch Attacks Against Semantic Segmentation. In International
Joint Conferences on Artificial Intelligence Organization (IJCAI), 2020. submitted.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial
examples. In International Conference on Machine Learning (ICML), 2018. URL http://
arxiv.org http///orggabb//177..o
Karsten Behrendt and Libor Novak. A deep learning approach to traffic lights: Detection, tracking, and classification. In JEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017.
A. Braunegg, Amartya Chakraborty, Michael Krumdick, Nicole Lape, Sara Leary, Keith Manville, Elizabeth Merkhofer, Laura Strickhart, and Matthew Walmer. APRICOT: Dataset of Physical Adversarial Attacks on Object Detection. In 16th European Conference on Computer Vision (ECCV), 2020.
Tom Brown, Dandelion Mane, Aurko Roy, Martin Abadi, and Justin Gilmer. Adversarial patch. In Conference on Neural Information Processing System (NIPS), 2017. URL htttp:://axxiiii orq/pdf/1712. 09665. pdf arXiv: 1712.09665.
Yulong Cao, Chaowei Xiao, Dawei Yang, Jing Fang, Ruigang Yang, Mingyan Liu, and Bo Li. Adversarial objects against based autonomous driving systems. CoRR, htss/907.00418, 2019.
URL http:://axttait org abs/1907. 05418
Shang- Tse Chen, Cory Comelius, Jason Martin, and Duen Horng Chau. Robust physical adversarial attack on Faster R-CNN object detector. CoRR, ths/1804.05810 2018. URL httt:://arxttt
org/abs 1804. 05810
Ping-yeh Chiang, Renkun Ni, Ahmed Abdelkader, Chen Zhu, Chris Studor, and Tom Goldstein. Cer- tified defenses for adversarial patches. In International Conference on Learning Representations, 2020.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Florian Tramer, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Physical Adversarial Examples for Object Detectors. In 12th USENIX Workshop on Offensive Technologies, July 2018a. URL
abs/ 1ess/1860 /1077 07769
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust Physical- World Attacks on Deep Learning Visual Classification. In Computer Vision and Pattern Recognition (CVPR), 2018b.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML), 2017. URL
http:://rxixg org/ abs/1703 03400
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICLR), 2015.
Sven Gowal, Krishnamurthy (Dj) Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovie, Timothy Mann, and Pushmeet Kohli. Scalable verified training for provably robust image classification. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Chuan Guo, Jared S. Frank, and Kilian Q. Weinberger. Low frequency adversarial perturbation. Association for Uncertainty in Artificial Intelligence (UAI), 2019.
org/
abs 0bss/1099.om 08758
Hayes and G. Danezis. Learning universal adversarial perturbations with generative models. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 43-49, May 2018.
10

---

Under review as a conference paper at ICLR 2021
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), 2016. URL https://
arxiv. hwlgg/abs/iin 1002.1112.000 03385
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations (ICLR), 903. 12261
2019. URL
Yi Huang, Adams Wai Kin Kong, and Kwok-Yan Lam. Adversarial signboard against object detector. In British Machine Vision Conference (BMVC), 2019.
Jason Jo and Yoshua Bengio. Measuring the tendency of learn surface statistical regularities. wwwpaawwwwwwv..aa//aaaaaaa/ 11111
CoRR, abs/1711.. 1561, 2017. URL
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. International Conference on Learning Representations (Workshop), April 2017.
Mark Lee and J. Zico Kolter. On physical adversarial patches for object detection. International Conference on Machine Learning (Workshop), 2019. URL http::tttt:tt..ottttttt/a http:aawwww..ggga///////aoa /106
11897
Juncheng Li, Frank R. Schmidt, and J. Zico Kolter. Adversarial camera stickers: A physical camera- based attack on deep learning systems. International Conference on Machine Learning (ICML), 2019. BRLhttp:: //t::///aaiiv... wttp//////...///0000/2
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D. Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. CoRR, 2019. URL
http::/arxiv org/ htrgg/bbse abs/ 02611
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018. URL https: //axivv. org abs////////a aawaaassia 000066666360033...
Jan Hendrik Metzen, Mummadi Chaithanya Kumar, Thomas Brox, and Volker Fischer. Universal Adversarial Perturbations Against Semantic Image Segmentation. International Conference on Computer Vision (ICCV), October 2017.
Seyed-Mohsen Moosavi-Dezfoole Alhussein Fawzi, and Pascal Frossard. DeepFool: simple and accurate method to fool deep neural networks. In Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA, 2016.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Hawaii, USA, 2017.
Konda Reddy Mopuri, Utsav Garg, and R Venkatesh Babu. Fast feature fool: A data independent approach to universal adversarial perturbations. In Proceedings of the British Muchine Vision Conference (BMVC), 2017.
Konda Reddy Mopuri, Aditya Ganeshan, and R Venkatesh Babu. Generalizable data-free objective for crafting universal adversarial perturbations. arXiv preprint arXiv: 1801.08092, 2018.
Chaithanya Kumar Mummadi, Thomas Brox, and Jan Hendrik Metzen. Defending against universal perturbations with shared adversarial training. In The IEEE International Conference Computer Vision (ICCV), October 2019.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order metanninnn algorithms. CoRR, abs/1803.02999, 2018 URL http:://axxlv.rrrt/
http://axvvvv.//// wwtp://vw.../////....33 02999
Julien Perolat, Mateusz Malinowski, Bilal Piot, and Olivier Pietquin. Playing the Game of Universal Adversarial Perturbations. 17Xiv:8009 00890078022000 [cs, stat], September 2018. URL http:// /arxiv
org/abs 07802
arXiv: 1809.07802.

---

Under review as a conference paper at ICLR 2021
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization, arXiv preprint arXiv:1903. 2019.
Anurag Ranjan, Joel Janai, Andreas Geiger, and Michael J. Black. Attacking optical flow. In International Conference on Computer Vision (ICCV), 2019.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. CoRR, abs/ 10d/188040277, 2018. UtLppttt://axxxvvcoom
Aniruddha Saha, Akshayvarun Subramanya, Koninika Patil, and Hamed Pirsiavash. Adver- sarial Patches Exploiting Contextual Reasoning in Object Detection. arXiv e-prints, art. arXiv:1910.00068, Sep 2019.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Basel Localization. International Journal of Computer Vision, October 2019. ISSN 1573-1405.
Ali Shafahi, Mahyar Najibi, Zheng Xu, John P. Dickerson, Larry S. Davis, and Tom Goldstein. Universal adversarial training. AAAI Conference on Artificial Intelligence (AAAI), 2018. URL
http://arxiv/ org/ abs / 11111111111 11304
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neu- ral Information Processing Systems 32, pp. 3358-3369, Curran Associates, Inc., 2019.
7/papers nips. wttpccpppppppppeeeeeceeeeeeeeeep 8537 adversaria teweell-traiinnnn-eeeeee for wwrffoorre...ereeeeeeeeeeeeeeeee pdf
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Accessorize a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS '16, pp. 1528-1540. New York, NY, USA, 2016. ACM.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Adversarial generative nets: Neural network attacks on state-of-the-ar face recognition. December 2017. URL https/
www.iii.org/ab///1/008 00349
Yash Sharma, Gavin Weiguang Ding, and Mareus A. Brubaker. On the effectiveness of low frequency perturbations. International Joint Conference on Artificial Intelligence (IJCAI), 2019. URL
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Simen Thys, Wiebe Van Ranst, and Toon Goedem. Fooling automated surveillance cameras: adversar- ial patches to attack person detection. In Conference on Computer Vision and Pattern Recognition (Workshop), April 2019. htt.://pp//aaawwooo/////////////
org wwtp:///////aaeeeeee http:///////9..000.0000 08653)
Tong Wu, Liang Tong, and Yevgeniy Vorobeychik. Defending against physically realizable attacks on image classification. In International Conference on Learning Representations (ICLR), 2020a.
URL htttps://aaxiiii http::///xxvv.or//..o/. wttp://.wwr//b//1...... http//ass.1..0... 09552
Yuxin Wu and Kaiming He. Group Normalization. International Journal of Computer Vision, 2019.
Zuxuan Wu, Ser-Nam Lim, Larry Davis, and Tom Goldstein. Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectors. 16th European Conference on Computer Vision (ECCV), 2020b. BtLhttts::// /ttp::///aavvvv.a org/ abs 1910.191011 14667
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. In International Conference on Learning Representations (ICLR), 2020. URL https: httppp/:///xxvvoo////.// abs/ 1906. 03787
Yuanhao Xiong and Cho-Jui Hsieh. Improved Adversarial Training via Learned Optimizer. 16th European Conference on Computer Vision (ECCV), 2020.
12

---

Under review as a conference paper at ICLR 2021
Kaidi Xu, Gaoyuan Zhang, Sijia Liu, Quanfu Fan, Mengshu Sun, Hongge Chen, Pin- Yu Chen, Yanzhi Wang, and Xue Lin. Adversarial T-shirt! Evading Person Detectors in A Physical World. 16th European Conference on Computer Vision (ECCV), 2020. URL
1910. 11099
Chenglin Yang, Adam Kortylewski, Cihang Xie, Yinzhi Cao, and Alan Yuille. PatchAttack: A Black-box Texture- based Attack with Reinforcement Learning. 16th European Conference on Computer Vision (ECCV), 2020a.
Xiao Yang, Fangyun Wei, Hongyang Zhang, and Jun Zhu. Design and Interpretation of Universal Adversarial Patches Face Detection. In 16th European Conference on Computer Vision (ECCV), 2020b.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In The IEEE International Conference Computer Vision (ICCV), October 2019.
Dinghuai Zhang. Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong- You only propagate once: Accelerating adversarial training via maximal principle. In Advances in Neural Information Processing Systems 32, pp. 227-238. Curran Associates, Inc., 2019.
Haizhong Zheng, Ziqi Zhang, Juncheng Gu, Honglak Lee, and Atul Prakash. Efficient adversarial training with transferable adversarial examples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
13

---

Under review as a conference paper at ICLR 2021
META ADVERSARIAL TRAINING SUPPLEMENTARY MATERIAL
A COMPUTATIONAL COST OF META ADVERSARIAL TRAINING
Computational cost of MAT dominated by the number of forward passes dip and backward passes Tlpp through the network for a single iteration of model training (the outer loop). Adversarial training (AT) with K-step PGD incurs (K + 1) (n + Tipp) cost for one iteration: K (n + bbp) for generating the perturbation and (n.p nbp)) for one step of model training. Similarly, the cost of MAT (with REPTILE as our experiments) for inner maximization and one step outer minimization is (K + (nfp + Tbbp). Additionally, selecting meta- metaa-erturbatton from F samples in Algorithm meta-perturbation is sampled randomly in this case and its loss need be computed. The cost of REPTILE itself is negligible, because it is simple convex combination. Therefore, the cost of MAT (F 1) and that of AT are comparable and MAT (F 1) clearly outperforms AT in Table Also for small F such as F = 5 for MAT (full) in Table MAT+REPTILE does not incur considerably higher cost than AT for the same K. The key point is that due to better initialization from the set of petaaperturbbtiinns small K can be chosen for MAT, whereas K would need to be very large for PGD in order to create equally strong attacks.
3 incurs a cost of F fp for F > 1. The additional cost is for
because the
B IMPLEMENTATION DETAILS
B.1
SUBPROCEDURE INIT
We present details on subprocedure INIT in Algorithm 2 Two relevant parameters of INIT are described below.
meta adversarial training (MAT) meta-learns set P of P meta-perturbations E,, where indexes P. Similar the attack initialization described in Section 4.1
Initialization of Pettrbbrtussssioos As described in Section 3.2
these meta- pectappetuurbations can be initialized in INIT in two ways as follows:
Random initialization: sampling randomly from uniform distribution over the space of allowed perturbations S.
Data initialization: this initialization sub-samples actual data points from the training dataset and corresponds to an on- manifold initialization that follows the data distribution. To generate universal patches, downsample crop the data points. To create universal perturbations, we scale the intensity of the data points to the range S.
Number of Meta-Perturbations The number of meta-perturbations P is chosen roughly propor- tional to the number of classes of the dataset regardless classification or object detection tasks. We choose P 1000 for Tiny ImageNet, which has 200 classes. For Bosch Small Traffic Lights Dataset, we choose P = 10 because the dataset only has 4 classes
Algorithm 2 INIT
1: Input: number of meta-perturb. P, data D, initialization type "init"
2: P +0
3: for in (l, do
4: target mod C' # round-robin target class, C' being the number of classes
5: "randoomm then
6: E~ UNIFORM( (pdethc)
7:
else if init "data" then UNIFORM (Plymytareet # Select datapoint labeled with target class uniformly E RESIZE vr dpatch) # Resize datapoint to appropriate dimensionality
8:
9:
end if
11: a LOGUNNIFRMM 0.1)
P+ ((E farget, ,o))
13: end for
14: Return P
14

---

Under review as a conference paper at ICLR 2021
Algorithm SELECT
1: Input: number of trials F' set of meta-perturb. P, input æ, label y, parameters 0, application fct. F, loss-fct. f,, random generator R.
2: Sopt> Vopt
pargett 'opp UNIFORM(P) # Sample uniform randomly from P
3: Topt ~R
4: F then
target 5: Return (Sopt: Vopt
Gopt l'opt)
6: end if
7: lopt f- L(0, (F(x, Eot, Eopt, Sopt::oot)) )o
v)) loss perturbed data, according label for parameters 0
8: for i in {l,...,
......... F} do
9: # Find worst (in terms of loss) out of F trials
10: E, y.target, a UNIFORM
11:
r R
12:
+ L(O, (F(x,,,r),
13:
ifl > Topt then
(Eope: Hopt 15: lopt f-
target
Cropt Topt) ) (E. Marget a.
end if
17:
end for
18: Return (Zopt: Hopt
target
Coopts Topt)
B.2 SUBPROCEDURE SELECT
We present details on the sub-procedure SELECT in Algorithm 3 We note that the special choice F = corresponds to a uniform random sampling of a meta-perturbation (and corresponding target class, step-size, and randomness). For F 1, SELECT requires F additional evaluations of the loss functions (and thus forward passes through the model) since the sample with the maximal loss is selected.
B.3 IMAGE CLASSIFICATION ON TINY IMAGENET
To evaluate the robustness of a model trained with MAT against universal patch attacks and universal perturbation attacks, we compare its performance with other training approaches such as Standard, CutMix (Yun et al., 2019). PatchUniform. adversarial training (AT) (Madry et al. 2018). shared adversarial training (SAT) (Mummadi et al., 2019), and universal adversarial training (UAT) (Shafahi et al. (018). Please note that this evaluation is an ablation study of MAT, namely, we configure MAT in way that is similar to each training approach. Detailed configurations are shown for universal patches in Table 3 and for universal perturbations in Table
Model Architecture For each setting, we train a ResNet VI model (He et 2016) from scratch on the Tiny ImageNet dataset (Tin). This ResNet model contains 4 residual stacks, where each stack consists of 2 residual blocks. The stacks have 64, 128, 256, and 512 channels and spatial resolution of 64x64, 32x32, 16x16, 8x8, respectively. We employ ReLU its activation function. Each convolutional layer has a stride of 1, kernel size of 3, group normalization with weight standardization, SAME padding, "he_normal" kernel initialization, and weight decay of 1100 on the kernel weights.
Model Training Each model is trained with 24x24 pixel patches applied to the 64x64 pixel input images, namely, a patch covers approximately 14% of the image. Patches are randomly translated from the center of the image by up to 26 pixels during training. We train each model with SGD for 75 epochs, an initial learning rate of 0.033, a cosine decay learning rate scheduler, momentum of 0.9, and batch size of 128. For each setting, perform 5 independent runs with 5 different seeds. Details regarding adversarial training procedures are shown the Table The most crucial parameter for AT, SAT, and UAT is the step size of I-FGSM. For AT, follow MAT and sample the step size per datapoint randomly from a log-uniform distribution over [0.0001, 0.1). Since UAT and SAT only update a single patch per batch, this random sampling strategy not feasible on ber-batch level. Instead, we use a fixed a: more specifically, we use 0.2 for SAT such that I-FGSM can reach any value in [0, 1| in K=5 iterations. Since UAT updates the perturbation iteratively over
15

---

Under review as a conference paper at ICLR 2021
SETTING
CUTMIX
PATCHUNIFORM
AT
SAT
UAT
MAT (FULL)
PATCH INITIALIZATION
RANDOM
RANDOM
RANDOM
RANDOM
WORST OVER F SAMPLES
1
REPTILE LEARNING RATE d
0.25
NUMBER PATA-PATCHES P
1000
I-FGSM STEP-SIZE
00.0001, 0.1]
0.2
0.01
0.0001, 0.1]
I-FGSM LTERATIONS
128
Table 3: Configuration of training procedures against universal adversarial patch attacks for image classification. We denote irrelevant entries as
Parameter
Values
Patch Initialization
random, data
Step Size a
0.0001, 0.00033, 0.001, 0.0033, 0.01, 0.033, 0.1 0, 0.9. 0.99
Momentum
Cutoff Frequency
off, 12
Number of iteration (S-PGD)
2500
Total Step Size Decay
0.01
Batch Size
64
Table 4: Configuration grid of attacks against the classification tasks.
the batches, smaller value for a is feasible here and we employ a 0.01. We did not tune these choices for extensively but note that MAT does not require any tuning of the step size
propose strong attacks for reliably evaluating robustness of trained models against universal patch attacks and universal perturbation attacks by optimizing the perturbations using S-PGD. We propose two initialization methods for S-PGD. Additionally, we utilize the low-frequency attack described above. The S-PGD step size a is exponentially decayed with total decay of 0.01. For evaluation of each model's robustness, perform S-PGD attacks over the parameter grid given in Table The attack results can be found in Subsection C.1
Model Evaluation As described in Section 4.1
B.4
OBJECT DETECTION ON BOSCH SMALL TRAFFIC LIGHTS DATASET
We describe the experimental details for training robust traffic light detectors against universal patch attacks.
Model Training For each training procedure, we train a Yolo V3 model (Redmon & Farhadi 2018 from scratch on Bosch Small Traffic Lights Dataset (Behrendt & Novak (017). The model has three network outputs each scale implemented in the original paper. For each DarkNet conv layer, replace batch normalization with group normalization and use weight standardization. To interpret the network outputs of Yolo V3, set the confidence threshold to 0.3. This means only the predictions with an objectness score > 0.3 count as valid predictions. The non-mannee mon-maximum suppression threshold set to 0, that means we prune the predictions when their bounding boxes overlap with IoU 0.1.
Each model is trained with 64x64 pixel patches applied to the input images resized to 1280x704 both width and height of the resized images are multiple of because grid cell's size is 32x32; thus, a patch covers 0.45% of the image. Patches are randomly translated from the center of the image by up to (512, 282) pixels during training. We ensure that translated patches do not overlap with any ground-truth traffic-light annotation. We train the model with ADAM for 15 epochs, learning rate of 0.0001, cosine decay learning rate scheduler, and batch size 1. We compare the accuracy against universal patch attacks of UAT and MAT variants. The configuration details are shown in 5
initial
Model Evaluation As described in Section 4.3 in order to evaluate the effectiveness the universal patches well the robustness of the model, apply metries the evaluation procedure
16

---

Under review as a conference paper at ICLR 2021
SETTING
UAT
MAT (DEFAULT)
MAT(-DATA)
MAT (K=l)
MAT(K=I,PEI)
PATCH INTIAALZATTON
RAND
RAND
DATA
RAND
RAND
WORST OVER F
1
REPTILE LEARNING RATE
1.0
0.25
0.25
0.25
0.25
NUMBER PATCHES P
10
10
10
1
I-FGSM STEP-SIZE
0.01
00.0001, 0.1]
00.0001,0.1 0.1]
[0.0001, 0.1]
[0.0001.0.1]
I-FGSM ITERATIONS K
Table Configuration of MAT with different approaches against universal adversarial patch attacks for object detection.
Parameter
Values
Patch Initialization
random (RI), data crop (DI)
Number of Steps (S-PGD)
4000
Batch Size
Step Size a
0.1. 0.01, 0.001, 0.0001
Total Step Size Decay
0.01
Momentum
0.9
Cutoff Frequency u
25, 50, 100, 250
Loss
standard, no objectness loss, ignoring false positives
Table 6: Configuration grid of attacks against the detection tasks.
mean Average Precision (mAP) and mean recall over classes with the IoU threshold of 0.1, which determines true positives between predicted bounding boxes and the ground truth. For generating universal patches, we use S-PGD with 4000 steps with batch size of 4. To find the strongest patches, we perform grid search over step sizes a E {0.1, 0.01, 0.001, 1.0001, a fixed momentum of 0.9, and a cutoff frequency u. {25, 50, 100, 250} of an optional low-pass filter. In addition, we also conduct grid search over three different options for the loss that is maximized by the attacker 1) the standard loss that is also used during training. 2) the standard loss subtracting the objectness loss, and 3) the standard loss ignoring all false positives. Similar to the previous initialization approaches in Section B.3 the perturbations for these attacks are initialized two different ways randomly or from cropped image of Bosch Small Traffic Lights Dataset. Each configuration is unique combination of an initialization, a step size, a cutoff frequency, and a loss found through the grid search. The parameter grid is summarized in Table 6 More result details can be found in Section
C RESULT DETAILS
C.I IMAGE CLASSIFICATION ON TINY IMAGENET
C.1.1 ROBUSTNESS AGAINST UNIVERSAL PATCH ATTACKS
In Figure we compare learning curves of MAT models against the transfer attack (see Section between different settings during training. The left plot shows that initializing the meta-perturbations via data initialization leads higher universal adversarial accuracy compared random initialization.. The middle plot shows that the model trained with targeted meta petta-peturbbtions is more robust than model trained with untargeted meta-perturbations, because targeted meta- peta-perturbations allow for greater diversity. The right plot shows results of randomly choosing a patch (F 1), selecting the worst patch from F = 5 samples, and selecting the worst patch from F = 10 samples. Training the model with more than sample (F > 1) improves the model's robustness but robustness saturates for F = while larger F' increases computational cost.
Figure 3 shows learning curves of ablated versions of MAT against the transfer attack. In accordance with Table training with larger number of meta- peteapperturbaions P, more iterations in I-FGSM, and with REP TILE learning rate smaller than 1.0 consistently improves robustness.
While Table shows the worst accuracy of a setting against all attacks of the grid search, Figure 4 summarizes the accuracy of all attacks of the grid search in box plot. Each value is averaged over
17

---

Under review as a conference paper at ICLR 2021
0.2
patca Inttolination
data
Fale
tageted artads
0.0
Epoch 60
Fpoch
Epoch
Figure 2: Comparison of MAT models' learning curves against the transfer attack for generating universal patches. (Left) pertureeturnation initialization in MAT between data initialization and random initialization. (Middle) MAT using targeted attacks and untargeted attacks for updating meta-perturbations. (Right) MAT uses the worst meta-perturbation chosen from different numbers of samples F.
independent runs with 5 different seeds for each training procedure. Each model is evaluated against three patch attack procedures: data, random, and low frequency. Configurations with large variance indicate that the model might appear to robust if hyperparameters of the attack chosen badly, This effect is particularly pronounced for AT and SAT against the random initialized S-PGD attack, where only very few attack configurations are able strongly degrade performance.
Moreover, the results exhibit that MAT is the only model robust against the data initialization attacks. None of the attacks reduce MAT's accuracy below 0.5 regardless of initialization methods. As discussed before, attacks through data initialization are more effective than through random initialization and attacks employing low frequency filter are most effective on MAT with random initialization (MATr). Nevertheless, MATr still shows stronger robustness than all other approaches except MAT.
C.1.2 ROBUSTNESS AGAINST UNIVERSAL PERTURBATION ATTACKS
We present an analogous evaluation as Section 4.2 for a universal perturbation attack. We use the same dataset, neural architecture, and training pipeline but train the models specifically for universal perturbation attacks. We allow universal perturbations with ||]|| S 20/255 0.078. In comparison with training models against universal patch attacks, the key difference is that the models are trained specifically for universal perturbation attacks instead of universal patch attacks. Following the training configuration shown in Table 7 we train ResNet VI models with 4 training approaches Standard, AT, UAT, and MAT. We do not present results for SAT (Mummadi et al. (219)) since we have found a stable configuration of hyperparameters for this setting; however, the results of Mummadi et al. (2019) indicate that SAT should perform slightly better than AT when configured appropriately. We evaluate the robustness the models against the same attacks as for patch attacks.
SETTING
AT
UAT
MAT
PATCH INITIALIZATION
RANDOM
RANDOM
RANDOM
WORST OVER F SAMPLES
REPTILE LEARNING RATE o
0.25
NUMBER OF META-PERTUREATION P
1000
I-FGSM STEP-SIZE
[0.0001, 0.02]
0.01
[0.0001, 0.02]
I-FGSM ITERATIONS K
Table Configuration of MAT with different approaches against universal perturbation attacks for image classification.
The evaluation results are summarized in Table g In comparison with the results of universal patch attacks in Table I we notice few interesting differences: firstly, clean accuracy is degraded for all variants of adversarial training compared to standard training. This indicates trade-off between clean performance and robustness in this AT and UAT made non-trivial gains in robustness, whereas their robustness did not improve against universal patches addressed in Section 4.2 Thirdly, the accuracy of UAT in Table g shows that UAT
Secondly, in contrast standard training,
18

---

Under review as a conference paper at ICLR 2021
0.6
Universal Patch Accuracy
0.4
patches
0.3
(UAT)
50
0.2
100
0.1
200
1000
0.0
0.6
Universal Patch Accuracy
0.5
0.4
iterations K
0.3
(UAT)
0.2
3
0.1
10
0.6
Universal Patch Accuracy
0.4
0.3
patch
0.25
0.5
0.1
(UAT)
0.0
10
20
30
40
50
60
70
Epoch
Figure 3: Ablation study of MAT in the aspects where differs from UAT Shaaahi et al. 2018) Shown is accuracy on Tiny ImageNet against a patch generated with the transfer attack: UAT learns single perturbation/patch while MAT learns an ensemble (upper plot). UAT performs one iteration of I-FGSM while MAT performs multiple (middle). UAT uses joint training (a = 1), while MAT uses full REPTILE with < 1 (bottom plot).
SETTING
CL
RI
LF
TR
MIN
STANDARD
0.55
0.03
0.04
0.03
0.03
0.03
AT (ADDRY ET AL.
2018)
0.48
0.33
0.26
0.33
0.27
0.21
UAT SHAFAHI ET AL.
2018
0.46
0.23
0.23
0.23
0.17
0.17
MAT
(FULL)
0.48
0.42
0.42
0.42
0.39
0.39
Table 8: Accuracy (mean 5 runs) of different methods against universal perturbation attacks Tiny ImageNet.
19

---

Under review as a conference paper at ICLR 2021
against universal patch attacks
0.6
Rand Intt
0.4
0.2
0.0
Data Irt
0.6
Low Frea
0.4
0.2
0.0 Standard
CurMix
Patchtinifomm
AT
SAT
UAT
MAT
MATY
Figure 4: Robustness against universal patch attacks: results correspond to Table but show distribu- tion of accuracy over elements of the grid search rather than only the worst accuracy. Each value is averaged across runs (with 5 different seeds) each configuration per training approach. MATr is the MAT model trained with randomly initialized meta-perturbatiioss, whereas MAT represents the model trained with meta meta-perturbations through data initialization. The rows correspond to three different universal patch attacks.
Universal Perturbation Accuracy
0.6
training approach
0.5
AT
MAT
0.4
Standard
UAT
0.3
0.2
0.1
0.0
10
20
30
40
50
60
70
Epoch
Figure 5: Comparison of learning curves of Standard training, AT, UAT, and MAT against the transfer attack for generating universal perturbations.
overfits less strongly to the randomly initialized S-PGD attack compared to the universal patch attacks Table Despite these differences, MAT considerably outperforms other methods terms of robustness also in this setting.
Figure 5 shows the learning curves of those training approaches against the transfer attack for generating universal perturbations. Notably, while MAT is less robust in the early phase of training, reaches significantly higher level of robustness in the end.
Figure shows the box plot corresponding to Table 8 The accuracy of MAT models is above 0.4 for all three attacks and shows little variance. In contrast, UAT and AT are robust against certain attack configurations but against optimally configured attack, accuracy degrades to 0.25 or less. This shows that evaluating robustness reliably requires strong set of attacks and their well-tuned hyperparameters.
20

---

Under review as a conference paper at ICLR 2021
Accuracy model against universal perturbation attacks
Rand
0.0
0.6
0.4
0.6
0.4
0.2
0.0
Standard
UAT
MAT
Figure 6: Robustness against universal perturbation attacks: results correspond to Table 8 but show distribution of accuracy over elements of the grid search rather than only the worst accuracy. Each value is an average across runs (with 5 different seeds) for each configuration per training approach. The rows correspond three different universal patch attacks.
C.2 OBJECT DRTECTION ON BOSCH SMALL TRAFFIC LIGHTS DATASET
We show the box plots corresponding Table in Figure 7 for recall and mAP, respectively. Notably, only the recall of the standard model can be reduced considerably (meaning true positives can be hidden) and this requires an appropriately configured attaek. Interestingly, a low-frequency attack not effective for reducing the recall of any model. In contrast, low-frequency attacks are the most effective ones for reducing mAP, that is: for causing false positive detections. While randomly initialized S-PGD is not successful at reducing the mAP of any model besides the standard model, many low-frequency attacks of varied attack configurations reduce mAP of most models (except MAT data) considerably. In contrast, S-PGD from data initialization be effective but fails in most cases to reduce mAP for all but the standard model.
by against patch
model agpinzt antvorsal
0.5
0.
0.4
04
0.2
02
0.
0.6
0.6
0A
02
0.
06
Standent
HAT
IAT (dnfeult) C+ datal
(K
PK11
Standerd
CAT
MAT (oefuutt Co dats)
[Km13
[K=1,P
Figure 7: Recall (left) and mean Average Precision (right) by model against universal patch attacks on Bosch Small Traffic Lights Dataset. MAT (default): meta perturbation is initialized -andolyyrandomly. MAT (+data): meta perturbation is initialized from cropped image. Both MAT (default) and MAT (+data) have I-FGSM iteration K=3 and number of patches
10. MAT(K=1)): I-FGSM iteration K=1 and number of patches P 10. MAT(K 1, P 1): I-FGSM iteration K=1 and number of patches P 1.
21

---

Under review as a conference paper at ICLR 2021
D ILLUSTRATION OF PATCH ATTACKS ON TINY IMAGENET
We illustrate universal patch attacks on models trained on Tiny ImageNet in Figure Note that these the strongest patches found against these models during the grid search. Oftentimes, the generated patch resembles the target class: examples for this are the low-frequency attack on a standard model (fooling it to mistake chimpanzee for a police van), the random initialization attack against the SAT model (fooling it to mistake the chimpanzee for ladybug), the data initialization attack against the SAT model (fooling
mistake the chimpanzee for an orange), or the cow-freuuency attack against the UAT model (fooling it into classifying the input as a fire salamander based on the characteristic texture of the patch). While these misclassifications can explained, a human would very likely still classify the inputs chimpanzees. Attacks on MAT (full) fail generate interpretable patches; however, transferring patches generated for other models (such as the shown ones) to MAT does not cause misclassification either.
E ILLUUSTRATION OF PATCH ATTACKS ON BOSCH SMALL TRAFFIC LIGHTS DATASET
We illustrate universal patch attacks on models trained on Bosch Small Traffic Lights Dataset in Figure p Note that these are the strongest patches found against these models during the grid search in terms of the mAP. These patches often invoke high confidence false detections. However, MAT with data initialization does not show any false positives.
For the patches found for MAT (Data Init), we show the progress of the patches during the attack in Figure TO Similarly, Figure 11 shows the patches' evolution during attack the standard model. Note that patches converge fairly quickly, namely, running attacks longer would not make them stronger. Moreover, all three patches for MAT converge to a red-cyan pattern and the patches for data and random initialization exhibit very similar patterns. This indicates that this pattern is actually a minimizer of the loss with large basin of attraction. However, as Figure Segure/ssowe shows, it does not really fool the model. Finally, Figure 12 shows the training of the patch shown in Figure [
F ILLUSTRATION OF PERTURBATION ATTACKS ON TINY IMAGENET
We illustrate universal perturbation attacks on models trained on Tiny ImageNet in Figure 13 Note that these are the strongest perturbations found against these models during the grid search.
22

---

Under review as a conference paper at ICLR 2021
Rand Init
Data Init
Low Freq
pop bottle
lifeboat
police van
Standard
pop bottle
bucket
police van
CutMix
PatchUniform
pop bottle
acorn
beer bottle
beer bottle
chimpanzee
chimpanzee
AT
ladybug
orange
chimpanzee
SAT
chimpanzee
European fire salamander
UAT
chimpanzee
chimpanzee
chimpanzee
MAT (full)
Figure 8: Best random initialization, data initialization, and low-frequency attacks against the respective models from Table I chimpanzee.
The model prediction is given above each plot. The correct label
23

---

Under review as a conference paper at ICLR 2021
Random
Data
Low Frequency
Standdadd
S8
766
O 0918
OLD 0.58
OS
2093
0.86
0.94
URT
0.86
35
040
MAT (Defocclt
KAT Datta Ialt)
68
O
MAT (K-1))
98
8n
0,98
0997
MANT (K=1, P=1)
0
0.9995
69
$999
Figure 9: Best (in terms of mAP) random initialization, data-crop initialization and low-frequency The patch location is fixed for better comparison.
attacks against the models from Table 2
24

---

Under review as a conference paper at ICLR 2021
Data, Learning Rate: 0.01
Random, Learning Rate: 0.1
Low-Frequency Cutoff: 100, Learning Rate: 0.01
Figure 10: Training of patches against MAT (Data Init) model. Attack from data-crop initialization
with step size of 0.01 (top). Attack from random initialization with step size of 0.1 (center). Low- frequency attack with cutoff frequency 100 from data-crop initialization with step size of 0.001.
Data, Learning Rate: 0.01
Random, Learning Rate: 0.01
Low-Frequency Cutoff: 100, Learning Rate: 0.001
Figure II: Training patches against standard model: Attack from data-crop initialization with step size of 0.01 (top). Attack from random initialization with step size of 0.01 (center). Low-frequency attack with cutoff frequency 100 from data-crop initialization with step size of 0.001
Low-Frequency Cutoff: 100, Learning Rate: 0.001
Figure 12: Training of patch from Figure I This patch is generated with a low- low- oow-frequency attack with cut-off frequency 100, learning rate of 0.001 and starting from data-crop initialization.
25

---

Under review as a conference paper at ICLR 2021
Rand Init
Data Init
Low Freq
chimpanzee
chimpanzee
chimpanzee
AT
chimpanzee
chimpanzee
chimpanzee
MAT
turnstlle
chimpanzee
UAT
picket fence
confectionery
mantis
Standard
Figure 13: Best random initialization, data initialization, and low-frequency perturbation attacks
against the respective models from Table g The model prediction is given above each plot. The correct label is chimpanzee.
26


Reviews:
[{"review_id": "vNXSdIzOLiy", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel meta adversarial training method. In particular, to improve the robustness against digital domain attacks, the proposed meta adversarial training (MAT) combines adversarial training and meta-learning, which reduces the computing cost compared with adversarial training by generating a set of stronger perturbations. The proposed approach sounds with extensive experimental results.\n\n* Pros:\n\n1.\tThe authors study a crucial issue of adversarial attacks, namely, a robust defense to perturbations. This problem is practical and of great importance to many areas, such as autonomous driving, robotics, etc.\n\n2.\tThe proposed MAT is designed for robust defense by combining adversarial training and meta-learning. \n\n3.\tIn MAT, meta-learning is used to meta-learn a set of initial perturbations and followed by a few gradient steps for adaptations. This method can avoid overfitting and the generated perturbations will try to focus on diverse weaknesses of the model, and thus the model is more robust. \n\n4.\tThe authors conducted comprehensive experiments, including image classification and object detection, with both qualitative analysis and quantitative results provided to show the effectiveness of the proposed model.\n\n* Cons:\n\n1.\tIn meta-training process, REPTILE is used to update the model $\\cal M$. However, it seems that in the testing stage, $\\cal M$ is directly used rather than adapted using training data from the testing task. As a result, the learned model $\\cal M$ is more like a pretrained model, rather than meta learning. It would nice if the authors can explain the details of testing process, and the design choice.\n\n2.\tIn Algorithm 1, it would be nice to explain in more details of the choices of INIT and SELECT functions, and their impacts to the algorithm performances. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "orhfBCxdYQ", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#########################################################################\n\nSummary: \nThis paper uses meta learning to learn adversarial perturbations to improve the model robustness against universal patch attacks. \n\n#########################################################################\n\nPros:\n1. The proposed meta adversarial training (MAT) claims to learn a large & diverse collection of universal perturbations that aids the robust training. \nSpecifically, this paper leverages the “reptile” meta learning method to learn the better initial values of universal perturbations, which may lead to better universal perturbations for updating the model. \nThe meta learner can be trained in parallel to the robust training. \n\n2. The writing is sound. The logic flow is clear. \n\n#########################################################################\n\nCons:\n1 This paper mentioned existing defenses overfit universal perturbation. Would the proposed method MAT relieve this overfitting issue? \n\n2 How does the adversarial training (e.g., against l_p norm bounded attack) resist the universal perturbation, compared with specially designed defenses such as MAT? \nIn table 1, why AT cannot defense transfer attack? Could it be possible that the baseline methods are improperly configured?  \n\n3 How & why does the initialization of the universal perturbations matters most? (Since the author claims the meta learning can better help the initialization.)\nTo be specific,  are there any justifications & explanations on the better initialization for aiding robustness? \n\n4 Another drawback of this paper is the unclear/incomplete technical descriptions (including Section A.)\n#########################################################################", "extra_scores": {}, "extra_text": {}}, {"review_id": "BaI-fhWumPy", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the use of meta learning and adversarial training to defend against universal perturbations. This approach tries to learn a set of perturbations through meta-learning and train the model to defend against such attacks. Experimental results on Tiny ImageNet and the Bosch small traffic lights dataset show the method is effective in defending against universal patch-based attacks. \n\nAlthough the main idea of the paper is sound and empirical evaluations promising, there are some clarity issues with the writing of the paper: \n1. In Algorithm 1, important operations such as INIT, SELECT, and UPDATE are described informally in the text instead of defined properly. \n2. What is the attack method used in Table 1? The columns only describe the initialization. \n3. Is the AT model in Table 1 trained using patch-based attacks or usual whole image attack? \n4. The notations used in Section 3.2 is for \\xi is very confusing and too overloaded. What do the notations \\xi(\\theta_t_1), \\xi(\\theta_t_i)_{i=1}^N, \\xi_t^m, \\xi_{T=k}, \\xi_{t=K} mean? The authors are using notations liberally without definitions. \n5. What does S-PGD stand for? \n\nAlso, in learning meta-perturbations the authors use different step sizes \\alpha in I-FGSM and targeted attacks from different classes for a diverse collection of attacks. The same idea of diversifying the attacks can also be incorporated in AT, SAT or UAT training. Have the authors tried this when performing comparisons? \n\nOverall I think this paper is a solid piece of work, but the clarity of presentation has to be improved.  \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors and reviewers for the discussions. Reviewers agreed the work is interesting but there are some aspects of the paper that need improvements. In particular, authors need to better address concerns raised by R5. Given all, I think the work still needs a bit more work before being accepted.  
Decision: Reject