A Curious Case of Remarkable Resilience to Gradient Attacks via Fully Convolutional and Differentiable Front End with a Skip Connection
Authors: Leonid Boytsov
Ameya Joshi
Filipe Condessa
Venue: Accepted by TMLR ()

Abstract: We experimented with front-end enhanced neural models where a differentiable and fully convolutional model with a skip connection added before a frozen backbone classifier. By training such composite models using a small learning rate for about one epoch, we obtained models that retained the accuracy of the backbone classifier while being unusually resistant to gradient attacks—including APGD and FAB-T attacks from the AutoAttack package—which we attribute to gradient masking.

Although gradient masking is not new, the degree we observe is striking for fully differentiable models without obvious gradient-shattering—e.g., JPEG compression—or gradient-diminishing components. The training recipe to produce such models is also remarkably stable and reproducible: We applied it to three datasets (CIFAR10, CIFAR100, and ImageNet) and several modern architectures (including vision Transformers) without a single failure case.

While black-box attacks such as the SQUARE attack and zero-order PGD can partially overcome gradient masking, these attacks are easily defeated by simple randomized ensembles. We estimate that these ensembles achieve near-SOTA AutoAttack accuracy on CIFAR10, CIFAR100, and ImageNet (while retaining almost all clean accuracy of the original classifiers) despite having near-zero accuracy under adaptive attacks.

Moreover, adversarially training the backbone further amplifies this front-end “robustness”. On CIFAR10, the respective randomized ensemble achieved 90.8±2.5% (99% CI) accuracy under the full AutoAttack while having only 18.2±3.6% accuracy under the adaptive attack (ε = 8/255, L∞ norm). While our primary goal is to expose weaknesses of the AutoAttack package—rather than to propose a new defense or establish SOTA in adversarial robustness—we nevertheless conclude the paper with a discussion of whether randomized ensembling can serve as a practical defense.

Code and instructions to reproduce key results are available. https://github.com/searchivarius/curious_case_of_gradient_masking

Published in Transactions on Machine Learning Research (08/2025)
A Curious Case of Remarkable Resilience to Gradient At- tacks via Fully Convolutional and Differentiable Front End with a Skip Connection
Leonid Boytsov* Bosch Center for Artificial Intelligence, Pittsburgh, USA
Ieo &boytson info
Ameya Joshi
Bosch Center for Artificial Intelligence, Pittsburgh, USA
Filipe Condessa*
Bosch Center for Artificial Intelligence, Pittsburgh, USA
Reviewed on OpenReview: https: // openreview. net/ forum? id= kt 7m
Abstract
We experimented with front-end enhanced neural models where a differentiable and fully convolutional model with a skip connection added before a frozen backbone classifier. By training such composite models using a small Icarning rate for about one epoch, we obtained models that retained the accuracy the backbone classifier while being unusually resistant to gradient attacks including APGD and FAB-T attacks from the AutoAttack package which attribute to gradient masking.
Although gradient masking is not new, the degree we observe is striking for fully differen- tiable models without obvious gradient-shattering e.g., JPEG compression or gradient- diminishing components. The training recipe produce such models is also remarkably stable and reproducible: We applied to three datasets (CIFAR10, CIFAR100, and Ima- geNet) and several modern architectures (including vision Transformers) without single failure case.
While black-box attacks such as the SQUARE attack and zero-order PGD can partially over- come gradient masking, these attacks easily defeated by simple randomized ensembles. We estimate that these ensembles achieve near-SOTA AutoAttack accuracy CIFAR10, CIFAR100, and ImageNet (while retaining almost all clean accuracy of the original classi- fiers) despite having near-zero accuracy under adaptive atlacks.
Moreover, adversarially training the backbone further amplifies this front-end "robustness". On CIFAR10, the respective randomized ensemble achieved 90.8+2.5% (99% CT) accuracy under the full AutoAttack while having only 18.2+3.6% accuracy under the adaptive at- tack (s = 8/255, - norm). While our primary goal is to expose weaknesses of the Au- toAttack package-ratee robustness we nevertheless conclude the paper with a discussion of whether randomized ensembling can serve as practical defense.
than to propose a new defense or establish SOTA in adversarial
Code and instructions to reproduce key results are available. searchivarriss/urriuus.cco wttp://ssuuuuuuuuueeeeeeeeeeeeee saaiss____graddeet_meeeeeee maaniingaasinggggg
https://githun com/
Work done outside the scope of Amazon employment, primarily while employed by Bosch. Work completed while employed by Bosch.

---

Published in Transactions on Machine Learning Research (08/2025)
1 Introduction
In the field of adversarial machine learning, the AutoAttack gained considerable popularity as a standard approach to evaluate model robustness. According to Google Scholar, mid 2025 core AutoAttack papers (Croce & Hein 2020a b Andriushchenko et al.] 2020) had been cited about 4400 times. AutoAttack is the "engine" of the RobustBench benchmark Crocee et al. (2021) which is a popular benchmark tracking progress in adversarial machine learning. However, the reliability of AutoAtaack itself, as well as that of its individual "componenis," " has received little to no discussion in the literature. Tramèr et al. (2020) noted in passing that AutoAttack overestimated robust accuracy of several models, which had nearly zero accuracy under adaptive attacks, but they did not discuss the actual failure modes of AutoAllack.
simply AutoAttack- has
To partially address this gap, we have carried out a case study where AutoAttack is applied to front- end "'nhancedd" models. We have found such composite models to be surprisingly resistant to standard projected gradient (PGD) attacks as well as to gradient attacks from AutoAttack (Croce & Hein 2020a
We attribute this behavior to gradient masking {Athalye et al. 2018: Tramer et al. 2020) and provide supporting evidence. Interestingly, the FAB attack from AutoAttack was previously shown to be resilient to gradient masking (Croce& & Hein 2020b). Yet, we have found it to be ineffective as well.
Despite the fact that RobustBench does not accept randomized models (Croce et al. 2021 users may apply AutoAttack to randomized defenses unaware that AutoAttack can fail against them (in fact, this was our experience). Moreover, AutoAttack was shown to be effective against complex randomized defenses (Croce & Heinn 2020a). Thus, it was reasonable to expect AutoAttack to be successful against a simple randomized ensemble as well. When AutoAttack detects randomized defense it only produces an casy-to-miss warning and suggests using the randomized version of the allack. However, the randomized version of AutoAtlack has only gradient attacks, which- we find- can be ineffective in the presence of gradient masking.
Even though the gradient masking phenomenon is not new, the degree of this masking is quite remarkable for fully differentiable models that do not have gradient-shattering components such as JPEG compression or components that are expected to cause diminishing gradients (we also carried out sanity checks to ensure there were no numerical anomalies, see $ AA.2). In particular, the models retain full accuracy on original (non- adversarial) data, but "basic" 50-step PGD with 5 restarts degrades performance of front-end enhanced model by only a few percent. At the same time it is possible to circumvent gradient masking with nearly 100% success rate using a variant of the backward-pass differentiable approximation Athalye et al. 2018
Although black-box attacks such as the SQUARE attack and the zero-order PGD can be effective against gradient masking, these attacks are easily defeated by combining gradient-masking models into simple ran- domized ensembles. We estimate that these ensembles schieve near-SOTA AutoAttack accuracy on CIFAR1O, CIFAR100, and ImageNet despite having virtually zero accuracy under adaptive atlacks. Quite interestingly, adversarial training of the backbone classifier can further increase the resistance of the front-end enhanced model to gradient attacks. On CIFAR10, the respective randomized ensemble achieved 20.8+2.5% accuracy under AutoAutack (99% confidence interval).
We do not aim to establish SOTA in adversarial robustness. Our primary goal is to expose weaknesses of AutoAttack. In that, our results further support the argument that adaptive attacks designed with the complete knowledge of model architecture are crucial in demonstrating model robustness (Tramrr et al., 2020). We nevertheless believe that automatic evaluation is useful bascline and we hope that our findings can help improve robustness and reliability of such evaluations. Beyond exposing weaknesses of AutoAttack, we also examine whether randomized ensembling can serve as a practical defense, and conclude the paper with a discussion of its potential. Code and instructions to reproduce key results are available
2 Related Work
Our paper focuses on a specific sub-field concerned with test-time attacks and defenses for neural-network classifiers on image data, which dates back only to 2013 (Zegedy et al., 2014 Biggio et al. 2013 For a
2

---

Published in Transactions on Machine Learning Research (08/2025)
detailed overview, we address the reader to recent surveys CChkrraborty et al. 2021: Akhtar et al. 2021 However, broader field of adversarial machine learning has a longer history with earlier applications such as spam and intrusion detection (Dalvi et al. 2004 Lowd & Meek, 2005 Biggio & Roli 2018).
Originally, most practical classifiers were linear and despite successes in attacking them there was a belief that attacking complex and non-linear models would be difficult. (Biggio & Roli, 2018). This assumption was partly rooted in the intuition that increased model complexity could make decision boundaries harder to approximate or exploit. However, neural networks while non-linear- turned out to be vulnerable in ways
similar to even more so than their linear counterparts (Biggio et al., 2013 Szegedy et al., 2014). As Biggio & Roli (2018) noted, adversarial machine learning gained quite bit of momentum after the publication by Szegedy et al. (2014). In addition to showing that deep neural networks were vulnerable to small adversarial perturbations, Szegedy et al. (0144) found that these perturbations were (to a large extent) transferable
among models.
Since then, the field has been growing exponentially, with more than twelve thousand papers uploaded to arXiv from the beginning of 2018 to mid-2025 (Carlini 2019). Despite such explosive growth, adversarial training is arguably the only reliable defense. Nevertheless, it does not provide theoretical guarantees. Some
assurances can be provided by provable defenses, but they have limited applicability (Croce & Hein 2020a;
Akhtar et al. 2021
Quite a few additional empirical (sometimes sophisticated) defenses have been proposed, but they have been routinely broken using strong adaptive attacks (Athalye et al.) 2018 Tramer al.) 2020). This, in turn, motivated development of automatic evaluation suites, which can reduce manual effort often required to execute adaptíve attacks. Among them, AutoAttack (Croce & Hein 2020b) is arguably the most popular package, which includes several lyperparaameeer-fiee gradient attacks well as query-efficient black-box SQUARE attack (Andriushcbenko et al.) 2020). About one year later, Croce et al. (2021) created the RobustBench benchmark to track progress in adversarial machine learning where adversarial robustness is measured using the accuracy under AutoAttack.
Although Croce& & Hein (2020a) advertised AutoAttack as a reliable attack, Tramer et al. (2020) noted in passing that AutoAttack overestimated robust accuracy of models in several cases, but without providing details. In that, research weaknesses of AutoAttack appears to be scarce. For example, Lorenz et al. (2021) question the general transferability of Robust.Benc findings to more realistic settings, but not reliability of the attack itself. In particular, our finding regarding extreme gradient masking appears to be unknown. Mothodologically,, the most related paper is by Tramèr et al. (2020), who emphasized the importance of adaptive attacks. The main difference is that we locus on documenting one specific use of (extreme) gradient masking and simple randomization.
Note that gradient masking is typically associated with gradient shattering via non-differentiable components and randomization as well as components encouraging vanishing gradients (Athalye et al. 2018). but not with fully differentiable components with skip connections such as the denoising front end used in our study (see Figure 1 in & 4.1b
We found only one case in which a fully differentiable convolutional network became attacks due to a special training procedure, namely, training with differential privacy
more resistant to PGD
[Beniich et al. 2021 Tursynbek et al. 2020). However, the seemingly increased robustness was observed only on MNIST (LCunn et al.! 2010) and in the low accuracy zone (sce Fig. 1a in the paper by |Boenisch] et al. 2021 In contrast, in our case gradient-masked models achieve an accuracy of about 80-90% under standard PGD attack while remaining fully breakable (see Table I.
Randomized defenses fall under the broader category of moving-targe defenses Evans et al. 2011 Al- though our paper focuses on the weaknesses of AutoAttack rather than on proposing a new defense through randomization or obfuscation, it is worth discussing several typical approaches. This discussion covers some key approaches, but it not intended to be exhaustive. Randomization and obfuscation broadly aim to prevent attackers from reproducing the exact data or model conditions used at prediction time. Existing approaches fall into two broad categories: the first involves random pscudo-random modifications to the input data, while the second randomizes the model architecture, its parameters, or combination thereof.
3

---

Published in Transactions on Machine Learning Research (08/2025)
For example, Taran et al. 2019) use ensemble of models, each operating on differently perturbed imput data. A perturbation is created by (1) first transforming input using an invertible parameterized transfor- mation Wa, (2) modifying the result using another parameterized transform Pk. and (3) then mapping it back using the inverse transformation W.'. The choice of transformations is determined by secret keys unknown to the attacker. However, there is no inference-time randomization and the defense can polen- tially be circumvented by black-box attacks such as the SQUARE attack (Andriushchenkoo et al.] 2020). In contrast, Qin et al. (2021) proposed to add Gaussian noise before passing imputs to the model method that aims to circumvent black- box attacks. Randomized smoothing also relies on randomly perturbing imput data multiple times. It uses majority voting over noisy predictions to create a robust classifier, which also provides certification (Cohen et al. 2019) 2
Model randomization inchudes a variety of defense strategies, with randomized ensembles being one of the simplest and most natural. In this approach, a subset of models (possibly just a single one) is selected at inference time to make a prediction. Even if the attacker has access all models in the ensemble, they cannot know in advance which subset will be used for a given input. While we do not know what. the first paper is that formally introduced randomized ensembles a defense, [Sengupta et al. (2019) proposed game-theoretic extension of this idea, where optimal sclection probabilities are computed based on cach model's performance on benign and adversarial imputs. A key limitation of their analysis is the assumption that the adversary generates attacks using only one model at a time. In practice, it may be possible to construct adversarial examples that transfer across all models an attack strategy not considered in their work. For example, one can generate adversarial samples by attacking a model ensemble (Liu et al. 2017 Gubri et al. 2022
That said, it is possible to reduce transferability among ensemble models. For example, Bui et al. 20221 proposed method that combines minimizing the cross-entropy loss for one model in the ensemble on an adversarial example with maximizing the loss for other models that misclassify the same input. Importantly, it is not always the same model whose loss is maximized-cad adversarial example is associated with a different weighting of loss components, depending on whether each model predicts correctly. Although the additional maximization term discourages correct predictions, it also encourages mispredicting models to produce highly uncertain outputs, with post-softmax probabilities approaching uniform distribution. As result, the influence of such mispredictions the ensemble's overall prediction remains limited (the prediction is based on summing logits of all models in the ensemble).
Sharif et al. (209)) proposed a variant of adversarial training in which adversarial examples are generated using targeted attacks. Instead of searching for arbitrary perturbations that maximize the cross-entropy loss for the correct label y, they define a random label mapping mly)-called a derangement- such that m(y) # y for all y. In other words, m is a randomly selected permutation that never maps a label to itself. Given an input with label y, they then construct adversarial examples that minimize the cross-entropy loss with respect to the target label m(y), which is different from true label y. To promote diversity within the ensemble, a different derangement is sampled for each model.
Although not suggested for reducing intra-model transferability, Sitawarin el al. 2024) proposed light ad- versarial training where a target model is trained using adversarial examples generated for a set of publicly available models. This can also be applied to a multi-model scenario. To this end, at cach training step one can sample a pair of models A and B. Then, they generale an atlack using the model A and apply it to the adversarial training of model B.
Bai et al. (2024) obscrved that attacks exhibit limited transfcrability between an adversarially trained model and a model trained on original (clean) data-a finding we also confirm in our study (see $ A.1I. They exploited this insight by linearly combining the logits of the robust and standard classifiers (without retrain- ing) and then applying a non-differentiable nonlinear transformation (with three parameters) to the result. The parameters of this transformation are optimized to maximize robustness.
"Certification formal guarantee that the classifier's prediction does not change within certain neighborhood, defined by certification radius R. In other words, ensures that the prediction is unaffected by any adversarial perturbation with radius at most R.
4

---

Published in Transactions on Machine Learning Research (08/2025)
A disadvantage of random ensembling is that it requires training and hosting of multiple models. A more efficient alternative is self-ensembling, which introduces randomness directly into a single model during training and/or inference. A basic example is test-time dropout, which as we found despite its simplicity can fool the SQUARE attack. It is also worth noting that randomization of input data with subsequent application of the same model e.g,, as in the randomized smoothing (Cohen el. al. 2019 can be seen as
a form of self-ensembling as well.
A number of more sophisticated self-ensembling techniques have also been proposed. Liu et al. (2018) add random noise layer before each convolutional layer, injecting Gaussian perturbations into the layer's input,
Dong et al. (2022) found that using different normalization approaches (e.g., batch normalization vs. instance normalization) reduces transferability of attacks, and they propose a random normalization aggregation approach, which randomly selects the normalization method for each layer that uses normalization.
Fu al. (2011) proposed to train models using randomly selected precision. They combine random-pre -prciiiinnnnnnn inference (RPI) and random-precision training (RPT): RPI selects inference precision at run-time to exploit low attack transferability across precisions, while RPT randomly switches training precisions with switchable batch normalization to further reduce attack transferability and enhance robustness.
3
Background and Notation
3.1
Preliminaries
We consider c-class image classification problem with neural network ho(x), where 0 is vector of network parameters. We will also use fa(z) to denote a special front-end network, which is optionally prepended to a standard vision classification backbone such as ResNet (He et al. 2016). Each input image is a vector x of dimensionality = w h-3 3 (width x height the number of color channels), which "resides" " in a normed vector space. In our work we use exclusively the L norm and use (x(x) to denote an - ball of the radius centered æ.
Given input x, the network estimates class-membership probabilities of input by producing vector of dimensionality e whose elements are called soft labels:
ho(x): Rd + [0, 1/
The actual prediction (a hard label) is taken to be the class with the highest probability:
arg max ho
For training purposes the accuracy of prediction is measured using a loss function The value of the loss function L for input x and true output is denoted as:
I(ho(a),v)
In this work-at it is common for classification problems- we use exclusively the cross-entropy loss.
3.2 Adversarial Example and Threat Models
Adversarial examples are intentionally (and typically inconspicuously) modified inputs designed to fool a machine learning model. In the untargeted attack scenario, the adversary does not care about specific class outcomes as long the model can be tricked to make an incorrect prediction, In the targeted attack scenario, attacker's 's objective is to make the model predict a specific class, c.g., to classify all images of pedestrians as motorcycles. The original unmodified inputs are often termed as clean data. The accuracy of the model on clean data is typically termed as clean accuracy. In contrast, the accuracy of the model on the adversarial examples is termed as robust accuracy.
In this paper, we consider adversarial input perturbations whose - norm does not exceed a dataset-specific threshold e. In other words, adversarial examples must. be inside the ball (x(x;), where æ is the original input.
5

---

Published in Transactions on Machine Learning Research (08/2025)
Different types of attacks require different levels of knowledge about the model and its test time behavior. The attacker can query the model (possibly limited number of times) by passing an input æ through the model and measuring one or more outcomes such as:
model prediction arg max; ho());
soft labels ho(x;;
the value of the loss (z). y);
the gradient with respect to input data VoL(hh(z).y),
In some scenarios, the attacker additionally know the model architecture and weights. However, in the case of randomized models, it would not be possible to know test-time values of random parameters. In our study, we consider three levels of access:
blackbbor attacks, which obtain model predictions arg max, ho()); the full soft label distri- bution he(z). In the latter case, the attacker can also compute the loss value L(he(z), v). Black-box attacks are commonly referred to as query-based attacks.;
gradient attacks, which can additionally access gradients (2L(ha(z), v)
full train-time access attacks, which can additionally "know" all the training-time parameters such as model architecture and weights, but not the test-time random parameters.
3.3 Standard and Adversarial Training
Given a classifier hx(r) the standard training procedure aims to minimize the empirical risk on the clean, i.e. original training set D:
min Σ L(ha(z),v)
(1)
To train models resistant adversarial attacks Szegedy al. (2014) and Goodfellow al. (2015) proposed to expand replace the original training set with adversarial examples generated by the model. This general idea has several concrete realizations, which are broadly termed as adversarial training. As noted by Goodfellow et al. (2015). "the adversarial training procedure can be seen minimizing the worst-case error when the data is perturbed by an adversary." Using a popular formalization of this statement due to Madry et al. 2018)) an adversarial training objective can be written as the following minimax optimization problem:
min
max L(hg(f),v) (E)
E.ED
The inner maximization problem in Equation 2 corresponds to generation of adversarial examples for a given data point and a model. This problem generally intractable: In practice, it is solved approximately using methods such as projected gradient descent (Biggio et al.] 2013 Madry et al.] 2018). In Section 4.2 we describe several approaches for finding adversarial examples (including PGD), which are used our experiments.
Note that these gradients computed with respect input image, rather than with respect model parameters. Indeed, the objective of the attack find the input maximizing the loss rather than the model parameters (that minimize the loss).

---

Published in Transactions on Machine Learning Research (08/202))
Conv2d BN RELU block (repeats depth times)
CONV (NxN channels) BN + RELU
CONV (NxN cnannels) BN RELU
CONV (3xN channels) RELU
CONV (Nx3 channels)
Figure 1: Denoising front end (DnCNN) architecture proposed by Zhang al. 2017 ): Note the skip connection.
4 Models and Attacks
4.1 Model Architectures and Training
In our experiments we use several lypes of models:
Standard (non-robust) models, i.e., models trained using standard objective (sce Equation D
Random ensembles of three coandard/non-roohe models;
A pre-trained classifier is combined with a front end, and the resulting model is further trained using a modified variant of adversarial training, with the backbone frozen. The backbone can be either
robust or standard (i.e., non-robust);
Random ensembles of three models with front ends;
Models trained using a traditional (multi-cpoch) variant of adversarial training (sce Equation 2
In that we use three types of popular backbone models:
Standard ResNet models of varying depth (He et al. 2016
Wide ResNet with depth 28 and widening factor 10 Zagoruyko & Komodakis. 2016
VIT with varying patch sizes Dosoviiskky et al. 2021
A random ensemble is simply set of three models, which are selected randomly and uniformly each time obtain predictions ha(x) compute gradients (zL(ho(z),)) It is noteworthy that randomization may unintentionally occur in Torch models that are inadvertently left in the training mode. In such case, the model behaves similarly to a randomized ensemble due to enabled test-time use of dropout (Srivastaval et al. 2014).
In the case of front-end enhanced models, an input image first passed through a pre-processing network fo, (x), also referred to as the front end. Following Norouzzadeh et al. (2021), we use a popular Gaussian Denoiser model DrCNN (Zhang al. (017)) to process the image. DnCNN is a fully-convolutionna neural network with skip connection (see Figure D.. The rationale behind this architecture is to learn residual mapping: instead of directly predicting the clean image, the network learns to estimate the noise compo- nent. This enables the model to approximate an identity transformation for clean regions while selectively correcting corrupted regions. Such a residual formulation simplifies the learning problem and accelerates convergence, especially when the corruption is relatively small.
The front end is prepended standard vision backend such as ResNet (He al. 2016) or ViT @Dsovvtskki al. 2021 which denote ha, (Ir). The full network is represented by composite function ha, (Jo, (x)).

---

Published in Transactions on Machine Learning Research (08/2025)
Unlike many other front-end defenses (Athalye et al. 2018), DnCNN does not explicitly shatter obfuscate gradients. Moreover, DrCNN is fully differentiable model with a skip connection, which is commonly used to improve gradient flow (He et al. 2016). Note that we use DrCNN without changing its architecture.
We train the front-end enhanced model using adversarial training (see Equation 2 with a "frozen" " backbone
model: This means that only parameters of the front end So, (x) are modified (using PGD to generate adversarial examples). In the case of the ResNet (and WideResNet) backends (He et al. 2016 Zagoruyko & Komodakis 2016), we disable updates of the batch-normalizztion layers (loffe & Szegedy 2015). Although
our training procedure is somewhat similar to the approach of Norouzzadeh et al. 20021 there two
crucial differences:
We use the original adversarial training objective without additional denoiscr-specifie losses;
We
use extremely small (on the order of 10-9) learning rate and train for at most epoch. Increasing the learning rale exlending training time does not yield a model that both resisls strong gradient attacks and retains the clean model's accuracy. In contrast, training a strong adversarial model on CIFAR datascts typically requires dozens of epochs.
Our training procedure also differs from that of the original DrCNN, which was trained in fully supervised manner to remove noise. Specifically, Zhang et al. (2017) added random noise to clean images and trained the denoiser to reconstruct the original clean versions.
As mentioned above, several models in our work are trained using a more conventional adversarial training approach. In this case, there is no front end and no parameter freezing. These models are trained for dozens of epochs using learning rates and schedules similar to those used standard training.
4.2 Attacks
Gradient attacks used in our work require access to the loss function [(ho(x), v), which is used in a slightly different way for largeled and untargeted attacks. In the case of an untargeted attack, the adversary needs to find a point (in the vicinity of the clean imput z) with the smallest "likelihood" of the correct class y. This entails solving the following constrained maximization problem:
f = arg max ((ha(z).)))
(3)
wEB.(2)
In the case of a targeted attack, the adversary "nudges" the model towards predicting a desired class ú instead of the correct one y. Thus, they minimize the loss of class ÿ. This requires solving the following constrained minimization problem:
a = arg min L(ho(r),))
(4)
(x)
There are situations in which the model's loss surface or computational pathway causes gradient- masientt-baeed ad- versarial attacks to appear ineffective not because the model is genuinely robust, but because the gradients are mislcading, noisy, uninformative. This phenomenon is commonly referred to gradient masking
grudient obfusculion (Athalye et al. 2018 Tramer el al. 2020
There are different approaches to solving Eqqation 3 or Equation Most commonly, they include first-order gradient- basod mothods, their zero-order approximations, and random search. In what follows we bricfly survey attacks used in our work. For simplicity of exposition, we focus on the unlargeted attack case.
Projected Gradient Descent (PGD). Projected Gradient Descent (Biggio et al., 2013 Madry et al. 2018) is an iterative optimization procedure, where in each iteration we make a small step the direction of the loss gradient and project the result onto B, (x)) to make sure remains sufficiently close to the initial clean example:
Fn proji, (nn a- sign V (x)(((())))))
(5)

---

Published in Transactions on Machine Learning Research (08/2025)
where sign(z) is an element-wise sign operator and a is step size. PGD with restarts repeats this iterative procedure multiple times starting from different random points inside (x(x) and produces several examples. Among those, it selects a point with the maximum loss.
Zero-order (iffereeifferennc) PGD. The zero-order PGD is different from the standard PGD in that it uses a finite-difference approximation for VxL(ho(z),)) 9) instead of actual gradients. To simplify the presentation, the description below denoles g(æ)) = L(hg(2), 9). The classic two-sided finite difference method for estimating gradients of the function g(x) uses the following equation:
Vxg(x); g(00, 21.............................. Ti + a Ta) - go .; a, a,.,,, ...)
(6)
However, this compulation can be quite expensive as it requires (dd) inferences, where d is the number of pixels in the image. To reduce complexity, we estimate gradients using block-wise differences.
First organize all variables into blocks. Then, instead of changing single variable at time, add the same a to all variables in a block and additionally divide the estimate in Eq. (6 by the number of variables in a block. This basic procedure works well for low-dimensional CIFAR images, but not for ImageNet. For ImageNet, we first divide an image into equal-size squares. Then, we combine squares into bigger blocks.
AutoAttack. The AutoAttack package (Croce & Hein 2020a or simply AutoAttack is a suite of attacks that has several variants of a gradient (PGD) attack as well as the query-efficient black-box SQUARE attack (Andriushchenko et al., 2020). Note that both the SQUARE attack and the zero-order PGD attacks are query-based attacks. One of the key contributions of Croce Hein (2020b) was an Automatic Parameter- free PGD attack with two different loss functions, which are abbreviated as APGD-CE and APGD-DLR.
In addition to this, APGD can be combined with Expectation Of Transformation (EOT) algorithm (Athalye et al. 2018). which is used to approximate gradients of randomized models. EOT was shown to be effective against randomized defenses (Croce & Hein 2020a). The third algorithm is targeted FAB attack, which was shown to be effective against gradient masking (Croce & Hein 2020b).
The standard version of AutoAttack includes all gradient attacks without EOT and the SQUARE attack. The rand version of AutoAltack has only APGD with EOT where the number of restarts set to one, but the number of gradient estimation iterations set to 20.
Transfer attack. A transfer attack is an adversarial black- box attack where an example created to fool one (source) model is appliod to a different (i.c., target) model. This exploits the phenomenon that adversarial examples often transfer across models, even when they differ in architecture, training data, or parameters (Segedy al. 2014).
Attacks that bypass the front end. To circumvent the gradient-maskinn front end, we use two sim- ilar "removal" approaches. They both exploit the fact that front end network is close the identity transformation, i.c., fe, (x) x.
The first approach is a variant of the Backward Pass Differentiable Approximation (BPDA) approach with straight-through gradient estimator (Athalye et al. 2018). We use a previously described PGD attack and only change the way we compute gradients of the front-end enhanced network ha (fo, (z)). In a general case of a However, due to the front end fo, (x) being approximately equal to the identity transformation, we use Vxlu, (x)) directly.
gradient estimator, one would use Vehe, (=)1== (a) instead of Vxhe, (fe, (x)).
Another approach is variant of transfer attack that simply generates the adversarial example å for the network (x) and input z. Then, the adversarial example a is passed to full composite network (Jo, (x)) [ To generale an adversarial example, we use the (gradient) APGD altack from Attokllack
Croce & Hein 2020b).
4Once again, exploit the fact that S0, (2) x.
9

---

Published in Transactions on Machine Learning Research (08/2025)
5 Experiments
5.1 Experimental Setup
We use three datasets: CIFARIO, CIFAR100 (Krizhevsky 2009), and Ima- geNet (Deng et al. 2009] all of which have RobustBench leaderboards for 1 attacks (Croce al. 2021] CIFARIO/ 100 images have small resolution of 32 by 32 pixels, while ImageNet models use the resolu- tion 224 x 224. We use attack radii that are the same as in RobustBench, namely, E 8/255 for CI- and E = 4/255 for ImageNet.
Evaluation Protocol.
FARIO/CIFALI
All our datasets have large test sets. However, rumning AutoAttack on the full test sets is quite expensive. is also unnecessary since we do not aim to establish a true AutoAttack SOTA. Thus, we use randomly selected images: 500 for CIFARIO/ CIFAI000 and 200 for ImageNet. The uncertainty in accuracy evaluation due this sampling is estimated using 99% confidence intervals. Note that in some cases attack is perfectly successful and every image is classified incorrectly. In such cases, the sample variance is zero and the confidence interval cannot be computed.
Models. For main experiments we use four types of models that can be classified along two dimensions:
Presence of the front end;
Single-model vs a three-model randomized ensemble.
Here we only note that four backbone types are used for all three datasets. The only exception a WideResNet, which is used only for CIFARIO and CIFAR100.
A detailed description of these models and their training procedures given in Section 4.1
Attacks. We used a combination of attacks described in Section 4.2
Standard 50-step PGD with restarts;
Zero-order (fiite-diiferrncc) 10-step PGD with restart. For ImageNet, pixels are first grouped into
8x8 squares. For CIFAR datascts, which have very low resolution, the square size is one pixel (no such grouping occurs). Furthermore, these blocks randomly grouped (5 squares in group).
Various combinations of attacks from AutoAttack (Croce & Hein 2020a). Unless specified otherwise:
When attacking single (i.e., a ron-oondodized model) we used APGD attacks without EOT (Atha-
( Iye et al. 2018). Together with the SQUARE attack (Andriushchenko et al. 2020 and FAB-T Croce
Hein 2020b) this combination a standard AutoAttack.
To attack randomized ensemble, we use an expanded rand variant of AutoAttack. AutoAttack rand has only APGD attacks with EOT (denoted as APGD/EOT). However, also use the SQUARE attack.
BPDA PGD with 10 steps and 5 restarts. Transfer attack using AutoAttack APGD.
5.2 Discussion of Results
Summary of results In this section we briefly review results showing that models with diffcrentiable front ends can, indeed, appear much more robust than they actually are. In particular, when such models are assembled into randomized ensemble, they appear to achieve near-SOTA robust accuracy virtually without degradation on clean test sets. We will then try to answer the following research questions:
Is this behavior due to gradient masking?
Is AutoAttack in principle effective against randomized ensembles?
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Accuracy for unprofettaa/Saandddd-raiingg models with front end (sce table notes).
Single Model
Model
No Attack
PGD
APGD+FAB-T (AutoAttack)
SQUARE (AutoAttack)
PGD
PGD
(standard)
(0-order)
(BPDA)
CIFAR10
ResNet-18
9113%
0%
110%
3.12%
ResNet-18 +front
91133%
88+4%
75+5%%
1+1%
5+.2%
0%
VIT-base-16x16
97+22%
0+1%
1+1%
10+:4%
VIT-base-166 16x16 +front
97+2%
96--2%
85++4%
1+2%
10+33%
0+11%
CIFAR100
ResNet-18
6816%
01:1%
111%
41.3%
ResNet-18 +front
68.15%
65-15%
48416%
24:1%
14.2%
0.11%
VIT-base- 16x16
84.5%%
1111
24.2%
84.3%
VIT-base-16x16 front
84.5%%
80.15%
59.6%
3122%
81.4%
1111
ImageNet
ResNet-18
69.99%
0%
2+:2%
23+8%
ResNet-18 +front
7011%%
68.99%
28.18%
2113%
23.88%
0%
VIT-baso-16xe 16x16
83.77%
0%
1818%
60119%
VIT-bbae-16xxIe +front
82.8%%
7918%
4819%
2118%
62:19%
0%
Randomized Ensemble
Model
No Attack
PGD
APGD/EOT
SQUARE (AutoAttack)
PGD
PGD
(standard)
(AutoAttack)
(0-order)
(BPDA)
CIFAR10
ResNet-18 3x
92*33%
2+2%
76+5%
6616%
ResNet-18 +-rnnt 3x
91.14%
89.14%
70.6%%
74.5%%
64.55%
3122%
VIT-base-16xe 3x
97.12%
111%
7815%
64.16%
VaT- 16x16 + front
96.12%
96.12%
7415%
81144%
65.16%
111%
CIFAR100
ResNet-18 3x
69-55%
5+.3%
51+6%
49+5%
ResNet-18 +front 3x
66.16%
6745%
4011%%
5446%%
4611%
81:3
VIT-bbae-116x 3x
85t44%
2+22%
64116%
4615%
VIT- brr-bos--16x66 +front 3x
86144%
8115%
5416%
63.5%
4516%%
3111
ImageNet
ResNet-18 3x
69.19%
0%
2.12%
23.18%
ResNet-18 +front 3x
70188%
68+99%
42+10%
48+99
33+.9%
0+:2%
VIT-base- 111-bsse-16x66 3x
83.77%
0%
18+7%
58+10%
WWT-hase-16x16 front 3x
8248%
80-77%
63-19%
69-19%
68+99
0%
Test sample has 500 images for CTEARI0/CTFARIQ and 200 images for ImageNet:
We show 99% confidence intervals (CT) when applicable: Note that CIs not be computed for attacks with 100% success rate.
APGD/EOT+SQUARE attack.
combination of APGD with EOT (as rand version of AutoAttack) followed by SQUARE
We do not run APGD or BPDA attacks when the accuracy 0% under the standard PGD allack.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Summary of results for auproteceel/standardttraaiiii models with a front end (see table notes).
Dataset
E-attack
Single model
Best Randomized Ensembles
Best RobustBench (June 2025)
(stand. train.)
(adv. train. with frozen backbones)
original dataset
extra data
Attack
AutoAttack
PGD
AutoAttack
No
AutoAttack
Attack
(APGI//EOT+SQUARE) (BPDA)
Attack
(standard)
Attack
(standard)
CIFARIO
8/255
95.13%
9612%
74.1%%
5122%
92.2%
66.6%
93.7%
73.7%
CTFAR100
8/255
84+5%
86+44%
54+7%
3+11%
69.2%
36.9%
75.2%
42.7%
ImageNet 4/255
85++4%
82+8%
63++9%
0%
78.9%
59.6%
Attack = = 8/255. Test sample has 500 images for CIFAR10/CIFARI0 and 200 images for ImageNet:
We show 99% confidence intervals (CT) when applicable: Note that CIs can not be computed for attacks with 100%
success rate.
Table 3: Accuracy for arversar models with a front end on CIFAR10 (see table notes).
Single model
e-train
No Attack
+APGD-CE
+APGD-T
+FAB-T
+SQUARE
Transfer (AutoAttack)
2/255
95.2%
92.6%
92.4%
92.1%
41.0++4.3%
15.24-3.2%
4/255
92.2%
90.8%
90.6%
90.6%
56.4:444%
32%6=+11%
8/255
84.8%
83.8%
82.6%
82.6%
60.8+4.3%
43.6+4.4%
Randomized Ensemble
e-train
No Attack
+APGD-CE
APGD-DLR/EOT
+SQUARE
Transfer (AutoAttack)
2/255
94.4%
93.8%
92.4%
90.8+:2.5%
18.2+33.69 6.6
4/255
92.2%
91.4%
90.6%
89.0.12.8%
35.614.3%
8/255
84.8%
83.2%
82.2%
80.213.5%
45.214.4%
Attack 8/255.
Test sample has 500 images: We show 99% confidence intervals (C1).
To simplify exposition we do not post Cls for intermediate results, but only for the final attacks.
In our experiments we focused primarily on combining standard models with a front end (except for C1- FAR10). This was done to reduce experimentation cost. Indeed, training adversarially robust models for ImageNet computationally intensive while high-accuracy models trained using the standard objective and original/clean data are readily available.
The key outcomes of these experiments presented in Tables and Table 1
includes results only for ResNet-18 (He et al. 2016 and ViT (Dosovitskiy et al. 2021 Expanded variants of Table I which include
additional models are provided in the Appendix A.3 (Tables 10
12
Using ResNet18 on CIFAR-10, we experimented with removing the skip connection from the DrCNN front- end. By design, the skip connection allows the convolutional layers focus on learning how to suppress corruption of small magnitude, effectively acting as ncar-identity transformation. Without the skip connec- tion, training makes virtually no progress even after dozens of epochs. Interestingly, increasing the learning rate in this case leads to behavior similar to that of standard adversarial training.
From Table 2 it can be seen that randomized ensembles, which include front-end enhanced models, rival the best models on RobustBench (Croe el al. 2021), even when RobustBench models were trained using additional data. At the same time, we do not observe degradation on clean data, which indicates that a front-end should largely preserve original images. However, all front-end enhanced models have nearly zero accuracy under the BPDA PGD attack. At the same time, neither the gradient attacks nor the SQUARE (Andriushhhenko et al. 2020) attack particularly effective.
12

---

Published in Transactions on Machine Learning Research (08/2025)
The SQUARE attack is quite effective against single model without front end. On CIFAR10 and CTFAR100 datasets (see Table 1 and Tables 10 11 in Appendix A.3 it degrades accuracy of all models to nearly zero. For ImageNet and ViT, the accuracy under attack is also quite low (about 20%). However, it is much less successful against randomized ensembles, even when individual models are without front ends. We also find that in some cases the vero-order PGD can be more successful than other attacks. However, the attack success rate is still well below the expected 100%. To reiterate, both zero-order PGD and the SQUARE attack are particularly ineffective against randomized ensembles with front-end enhanced models.
What happens when an adversarially trained model is combined with front end? We carried out a de- tailed investigation using a sample of CIFARIO im- ages and presented results in Table X The main dif- ference here is that we report accuracy after each at- tack from AutoAttack (Croce & Hein 2020a). Thus, we can how successful each type of attack is.
Table 4: Accuracy under attack for adversarially-trained models on CIFAR10 for standard vs zero-Order PGD (see table notes).
No
PGD (0-order)
PGD (standard)
Attack
From Table 3 can see that adversarial training even with small 2/255, makes the model nearly "impervious" to gradient attacks when such a model is combined with a front-end. For example, a combi- nation of all gradient attacks degrades the accuracy of a single front-end "protected" model from 95, 95.2% to only 92.4%. At the same time, the true robust accuracy of this model does not exceed 15.2% (we know that the accuracy under the transfer attacks is about 15%, but it might be possible to further re- duce it using some unknown approach). In contrast, the SQUARE allack could reduce accuracy to only 41%. In the case of a randomized ensemble, though, it is not effective and the combination of attacks can reduce accuracy only 90.8%. Curiously enough, adversarial training scems to increase resistance to both gradient and black-box attacks.
ResNet-18
65.122%
53.12%
42.12%
ResNet-50
761.2%
62.12%
53.12%
ResNet-101
78+-2%
65+22%
524-2%
All models are trained using a standard adversar- ial training procedure without a front end (training 8/255). Test sample has 2000 images: 99% confidence in- tervals are shown.
Is it gradient masking? It should now be clear that AutoAttack (and/or or its individual attacks) can greatly exaggerate performance of both single mod- els and (especially) randomized ensembles. There are several indicators that this is primarily due gradient masking. First, if plot accuracy of the models for attacks of varving radius (as recom- mended by Tramer et al. 2020), we can see that the accuracy of adveesarialll-trrinee models with- out front ends quickly decreases as the attack radius & grows. This happens for at least two reasons:
100%
Resnel-18 +front
Reenet-50 *front
70%
Resnet-152 +front
acouracy
Resnet-18
Resnet-50
152
0.00
0.25
0.50
0.75
1.00
Figure 2: CIFAR10: Accuracy under attack for differ- ent attack radii for models with a front-end and models that underwent traditional (multi-epoch) adversarial training without using a front-end (denoted with AT). Accuracy of front-end equipped models decreasing much slower than the accuracy of AT models, which is indicative of gradient masking.
Since the model is trained to resist pertur- bations of a radius at most & = 8/255, it is easily "fooled" when the attack radius in- creases;
Large perturbations can alter the underly- ing class semantics, so we should not expect the model to predict the initial class as the radius grows.
In contrast, the accuracy of the front-end equipped models declines noticeably slower and does not reach zero when 1, which is the maximum possible distortion (remember that pixel intensity ranges from 1).
13

---

Published in Transactions on Machine Learning Research (08/2025)
Second, we see that black-box attacks, namely SQUARE and zero-order PGD, can be substantially more effective than standard gradient attacks. In that, for typical adversarially trained model, the zero-order PGD is less effective (see Table 4p. This suggests that the gradients provide little useful information about the true optimization direction- a characteristic phenomenon of gradient masking.
Third, we can see that front-end bypassing attacks such as BPDA PGD-make front-end "protected" models much less accurate. The ineffectiveness of standard gradient- based attacks, contrasted with the success of stronger approaches such as transfer attacks, is well-rocognized indicator of gradient masking Tramèr et al. (2020).
Although gradient masking is a known phenomenon, it is umusual to observe in fully differentiable models without gradient-diminnshingg or gradient-shattering components. For example, Trrmmrr et al. (020)) describe a case when gradient masking was caused by an additional "log-softmaxing" which is known to reduce "gradient ftows". However, in our case, a gradient-masking culprit a front end, which is fully-convolutional network with skip connection and balch-nnormalization layers (see $ A.22 where we describe checking for numerical abnormalities). Both of these architectural components were instrumental to training deep neural
networks, which otherwise would suffer from vanishing exploding gradients (Szegedy et al. 2014 He et al. 2016).
Is AutoAttack in principle effective against randomized ensembles? We observed that Au- loAtlack can be inelfective against randomized en- sembles. In particular, on a sample of CIFARIO im- ages, we can achieve an accuracy of 90.8 (99% CI) for a model that has true robust accuracy be- low 20%. Is AutoAttack, in principle, weak against randomized ensembles or is there something special about our ensemble models?
Table 6: Accuracy under attack for diverse ran- domized ensembles of varying adversarial robust- ness on CIFAR10 (500 images).
s-train
PGD (ensemble) average)
PGD (full ensemble)
AutoAttack
2/255
301:2%
6416%%
29155%
First of all, we confirmed that even standard PGD can be extremely effective against randomized en- sembles where individual models underwent only standard training using clean data. Indeed, from Appendix A.3 we see that the accuracy of such ensembles is nearly zero under a standard PGD atlack. This was an unexpected finding because such atlacks do not use averaging techniques such EOT and each gradient computation uses mix of models with different random weights.
4/255
47+.2%
67.15%
42.16%
8/255
52+3%
6416%
4716%
Table 1 and Tables 10 11
A more careful investigation showed that the lack of EOT-bocause all models in the ensemble share the same architecture, the standard PGD finds adversarial examples that work for all models in the ensemble. To prevent this, we trained ensembles using three different types of backbone models, namely, ResNet-152 (He et al. 2016), wide ResNNet-28-10 (Zagoruyko & Komodakis 20166,, and ViT (Dosovitskiy et al. 2021). We did not use any front ends and trained three ensembles using three values of attack e: 2/255, 4/255, and 8/255. For each model in the ensemble we computed the accuracy under standard PGD attack (with 8/255) and then obtained ensemble averages, which ranged from 30% to 52% depending the training-time
As expected, the smaller E was during training, the lower was the robust accuracy of the models. However, when we attacked a randomized ensemble using a 50-stop PGD with five restarts, we achieved the accuracy close to 65% (see Table Gj. Thus, the randomized ensemble of diverse models can fool the standard PGD attack without EOT. In contrast, the accuracy under AutoAttack was slightly lower (as expected) than ensemble-average robust accuracy (sce Table thus, confirming that AutoAttack can indeed be very effective against randomized ensembles.
Can randomized ensembling offer practical defense? While our primary focus is not on designing or benchmarking a specific cellomizzedenssmmbll scheme, worth discussing whether such an approach could meaningfully boost robustness. To this end, we should first consider plausible attack scenarios. SSttaaarin
14

---

Published in Transactions on Machine Learning Research (08/20) (025)
Table 5: Calculating the lower bound for the accuracy of the randomized ensemble under transfer attack with a single source model.
We calculate a lower bound for the expected accuracy of the randomized ensemble under a transfer attack, assuming the following:
An attacker creates attack by using one model at time, i..., there is ensembling of source models;
source model one of the models used in the randomized ensemble;
Individual models in the randomized ensemble are selected with equal probabilities.
According Table (in A.1
A(i > i) Ro (sell-attack);
Ag-s 2 44.2% if both i and k are different. ResNet models (altacking a Res using different ResNet model);
Ap- 73.6% if i is a VIT model and & is ResNet model (attackiig VIT model using a ResNet model);
As+i 2 65.4% if i is ResNet model and k is a ViT model (attacking a ResNet model using a ViT model);
Ak-i 55.9% if both i and & are ViT models (attacking ViT model using different ViT model).
First, there is a 10% chance that the source model and the target model are the same. In this case, the average accuracy is near zero. For a source ResNet model, in four cases out of 10, Ak-s 44.2% (when target model is also ResNet model) and in five cases out of ten A 73.6% (when the target is ViT). Likewise for source VII model, in four cases out of ten Ag- 2 55.9% and in five cases out of ten Ag+s 65.4%. Therefore, for source ResNet model, the lower bound for the expected accuracy is:
0% 0.1 + 44.2% 0.4 + 73.6% 0.5 R 54.6%.
For source ViT model, the lower bound for the expected accuracy is:
0% 0.1 + 55.9% 0.4 65.4% 0.5 R 55%
15

---

Published in Transactions on Machine Learning Research (08/2025)
Table 7: Calculating the expected accuracy of the randomized ensemble under the transfer attack that uses the all-model ensemble.
We calculate the expected accuracy of the randomized ensemble under attack, assuming the following:
The attacker creates an attack using an ensemble model, which includes all models;
Individual models in the randomized ensemble selected with equal probabilities.
According to Table (in $
A.1
ensemble- 18.1% if is ResNet model;
ensemble-s 26.1% if is a ViT model.
Because VIT model is randomly selected 50% of the time (in the ensemble) the expected accuracy is equal to:
18.1% - 0.5 + 26.1% 0.5 = 22%
et al. [2024] Qin et al. (2021) showed (hat randomization thwarts several popular query-based atlacks (which we also confirm for the AutoAttack case). Therefore, a typical attacker has to rely on transfer attacks or on hybrid approaches that combine transfer attacks with query-based attacks.
Assume that the defender has n models and makes a prediction by randomly selecting a model according to probabilities {pr}- The attacker has models from which they select source model k to generate adversarial example. Given such setup, the expected accuracy of the randomized ensemble is:
12 Ag-ii Pis
A(k) Σ
where Ak-z is the expected accuracy of model i when making prodiction transferred from model k. The best the attacker can do in this situation is to select k: arg mink A(k) that minimizes the expected accuracy under attack. In the worst case for the defender, the attacker has detailed knowledge about model architectures and parameters and, thus, be able to compute A(k) even without querying the randomized ensemble. Nonotheless, even in the absence such knowledge, the attacker may be able to estimate A(k) by probing the ensemble for different values of e-albeit this may not always be practical due a significant reduction in efficiency (a reliable estimate requires lot of queries for each k).
adversarial example
To make things concrete, let us consider 10 models used in our transferability study for ImageNet (see A.1 and Table H. This study includes five ResNet and five ViT models trained using clean data. Because these models were not trained adversarially, even small-radius PGD attacks "drive" their accuracy to zero, i.e., A(i () 2 0. However, A(k + i) is well above zero for i 7 k.
To simplify the presentation, we moved the calculations of the expected accuracy under attack to Table 5 Tn summary, no matter what the source model is, the defender is guarantcod to have the accuracy under attack above 54%. This about. 23 percentage point decrease compared to average accuracy on the clean data, which is 77%. However, it is still markedly better than nearly 0% accuracy under attack, and importantly, the clean data accuracy is fully retained.
Instead of using individual models, the altacker can use ensembles as source models, which are known to produce stronger attacks (Liu et al.) 2017 Gubri et al., 2022). A particularly unfortunate for the defender situation when the attacker can ensemble all models to generate an attack. In this case (sce Table 7 for details), the accuracy under attack is only 22%, which is 2.5x decrease compared the case when the attacks are generated using single model.
16

---

Published in Transactions on Machine Learning Research (08/2025)
It should now be clear that randomization be a practical defense only if one combines models with low mutual attack transferability. Further exploration of the feasibility of this approach is out of the scope of this work. However, we would like to mention several approaches to achieve this objective. First, Bai cct al. (2024) found there is only limited transferability among models trained using adversarial attacks with different strengths, which also confirmed by our own experiments (see Table in S A.T. Not only transfer is limited when the source model is trained on clean-data, but, surprisingly, adversarial examples coming from more robust models do not work well for less robust models. Second as discussed in 2 there are additional training approaches that reduce mutual transferability among models (Sharif et al., 2019 Bui ct al. 2021 Sitawarin et al. 2024). Third, one can modify existing architectures in controlled ways, e-g-, by changing normalization approaches (Dong et al. 2022) or precision (Fu et al. 2021). This can be used to further improve the architecture ural diversity of the randomized ensemble.
6 Conclusion
We discovered that training a fully differentiable front end, which also has a skip connection, with a frozen classification model can result in a model that is unusually resistant to gradient attacks while retaining accuracy on original data. We provided evidence that this behavior due to gradient masking. Although the gradient masking phenomenon not new, the degree of this masking remarkable for fully differentiable models lacking gradient-shattering components like JPEG compression other elements known to diminish gradients. The objective of this paper is to document this phenomenon and expose weaknesses of AutoAttack (Croce & Hein 2020a) rather than to establish new SOTA in adversarial robustness. We conclude the paper with a discussion of whether randomized ensembling can function as a practical defense.
7 Acknowledgments
This work was sponsored by DARPA grant HR11002020006.
References
Naveed Akhtar, Ajmal Mian, Navid Kardan, and Mubarak Shah. Advances in adversarial attacks and defenses in computer vision: A survey. CoRR, abs/2108.00400, 2021.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: A query-efficient black-box adversarial attack via random search. In ECCV (23), volume 12368 of Lecture Notes in Computer Science, pp. 484-501. Springer, 2020.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses adversarial examples. In ICML, volume 80 of Proceedings of Machine Learning Research, pp. 274-283. PMLR, 2018.
Yatong Bai, Mo Zhou, Vishal M. Patel, and Somayeh Sojoudi. Mixednuts: Training-free free accuracy-robustnes balance via nonlinearly mixed classifiers. Trans. Mach. Learn. Res., 2024, 2024.
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning, Pattern Recognit., 84:317-331, 2018
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Sindic, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In FCML//PDD (3), volume 8190 of Lecture Notes in Computer Science, pp. 387 402. Springer, 2013.
Franziska Boenisch, Philip Sperl, and Konstantin Böttinger. Gradient masking and the underestimated robustness threats of differential privacy in deep learning. CoRR, abs/2105.07985, 2021.
Tuan-Anh Bui, Trung Le, He Zhao, Paul Montague, Olivier Y. de Vel, Tamas Abraham, and Dinh Phing. Improving ensemble robustness by collaboratively promoting and demoting adversarial robustness. AAAI, pp. AAAT Press, 2021.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Nicholas Carlini. A complete list of all (arxiv) adversarial example papers. htttps///iicholas.. cww.ihhhlaaac..aniii... carlini. com/writings
wawplleepppps....eeo papers.html 2019. Last accessed in September 2022:
The list has been regularly updated at the moment of access.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. A
survey on adversarial attacks and defences. CAAI Trans. Intell. Technol., 6(1):25-45, 2021.
Jeremy Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smooth- ing. In ICML, volume 97 of Proceedings of Mochine Learning Research, pp. 1310-1320. PMLR, 2019.
Francesco Croce and Matthias Hcin. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free al lacks. In JCML, volume 119 of Proceedings of Machine Learning Research, pp. 2206 2216. PMLR, 2020a.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 2196-2205, PMLR, 2020b.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustnoss benchmark. In NewrIPS Datasets and Benchmarks, 2021.
Nilesh N. Dalvi, Pedro M. Domingos, Mausam, Sumit K. Sanghai, and Deepak Verma. Adversarial classifi- cation. In KDD, pp. 99-108. ACM, 2004.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical TEFE Computer Society, 2009.
image database. In CVPR, pp.
Minjing Dong, Xinghao Chen, Yunhe Wang, and Chang Xu. Random normalization aggregation for adver- sarial defense. In NeurIPS, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
David Evans, Anh Nguyen-Tuong, and John C. Knight. Effectiveness of moving larget defenses. In Moving Target Defense, volume 54 of Advances in Information Security, pp. 29 48. Springer, 2011.
Yonggan Fu, Qixuan Yu, Meng Li, Vikas Chandra, and Yingyan Lin. Double-win quant: Aggressively winning robustness of quantized deep neural networks via random precision training and inference. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 3492 3504. PMLR, 2021.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegody. Explaining and harnessing adversarial examples. In ICLR (Poster), 2015.
Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Koushik Sen. Efficient and transferable adversarial examples from bayesian neural networks. In UAI, volume 180 of Proceedings of Machine Learning Research, pp. 738-748. PMLR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770 778. IEEE Computer Society, 2016.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448-456. .rg, 2015.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images, 2009.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: hllp://yann..eoee www.n./weeccc... com herdb /ee////d//m/sss, 2, 2010.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsich. Towards robust neural networks via random self-ensemble. In ECCV (7), volume 11211 of Lecture Notes in Computer Science, pp. 381-397. Springer, 2018.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song, Delving into transferable adversarial examples and black-box attacks. In ICLR (Poster). OeenReview.nct, 2017.
Peter Lorenz, Dominik Strassel, Margret Keuper, and Janis Keuper. Is robustbeeuch/auttotttcck a suitable benchmark for adversarial robustness? CoRR, abs/2112.01601, 2021.
Daniel Lowd and Christopher Meek. Adversarial learning. KDD, pp. 641 647. ACM, 2005.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR (Poster). OpenReview net,, 2018
Mohammad Sadegh Norouzzadeh, Wan- Yi Lin, Leonid Boytsov, Leslie Rice, Huan Zhang, Filipe Condessa, and J Zico Kolter. Empirical robustification of pro-trained classifiers. In ICML 2021 Workshop on Adver- sarial Machine Learning, 2021.
Zeyu Qin, Yanbo Fan, Hongyuan Zha, and Baoyuan Wu. Random noise defense against query-based black- box attacks. Tn NewrIPS, pp. 7650- 7660, 2021.
Sailik Sengupla, Tathagata Chakraborti, and Subbarao Kambhampati. MTDeep: Moving targel defense to boost the security of deep neural nets against adversarial attacks. Proc. GameSec, 2019.
Mahmood Sharif, Lujo Bauer, and Michael K. Reiter. n-ml: Mitigating adversarial examples via ensembles of topologically manipulated classifiers. Colli,
2019.
Chawin Sitawarin, Jaewon Chang, David Huang, Wesson Altoyan, and David A. Wagner, Pubdef: Defending against transfer attacks from public models. In ICLR. OpenReview.net, 2024.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Iyy Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929-199) 1958, 2014.
Christian Szegedy, Wojciech Zaremba, Llya Sutskever, Joan Bruna, Dumitru Erhan, lan J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR (Poster), 2014.
Olga Taran, Shideh Rezacifar, Taras Holotyak, and Slava. Voloshynovskiy. Defending against. adversarial attacks by randomized diversification. In CVPR, pp. 11226 11233. Computer Vision Foundation IEEE, 2019.
Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive allacks to adver- sarial example defenses. NewrIPS, 2020.
Nurislam Tursynbek, Aleksandr Petiushko, and Ivan V. Oseledets. Robustness threats of differential privacy. CoRR, abs/2012.0782s 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC. BMVA Press, 2016.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang, Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising. IEFE Trans. Image Process., 26(7):3142-311 2017.
19

---

Published in Transactions on Machine Learning Research (08/2025)
A Appendix
A.1 Transfer Attack Study
We have carried out a study of the effectiveness of transfer attacks using two different scenarios, in particular, to assess leasibility of using randomized ensembles as practical defenses. Attacks were generated using a 50-step PGD: E 8/255 for CIFAR datasets and = = 4/255 for ImageNet. We used the same subsets of CIFARIO, CIFAR100, and ImageNet as in the main experiments.
In the first scenario, we assessed transferability among ten different architectures: Five of the models belong to the ResNet class (from ResNet-11 to ResNet-152) (He et al. 2016) and five the ViT (Dosovitskiy al, 2021) class. ViT models differed in their sizes (tiny, small, base) and input patch sizes. In addition to measuring "clean" accuracy and accuracy under a PGD attack, we computed the accuracy under transfer attack for several source-model architectures. A source model can be either single model or an ensemble, but& a target model is always one of the above-mentioned ResNet or VIT models. The source-modelss included:
The same model as the larget model (the case of self-attack).
An individual model whose architecture is different from the target model. To simplify presentation, we only show accuracy for source models producing the strongest attack (i.e., that led to the lowest accuracy). In doing so, we show results for the best ResNet and the best VIT model separately.
An ensemble of models. We tested two types of ensembles: (1) a leave-one-out ensemble that does not include the larget model, and (2) an all-inclusive ensemble that has all models of given type (or types). All ensembles were additionally divided into three groups:
An ensemble of ResNet models, which has either all ResNet models (an inclusive ensemble) or all ResNet models except the target model (a leave-one-out ensemble):
An ensemble of ViT models, which has either all ViT models (an inclusive ensemble) or all VIT models except the target model (a leave-one-out ensemble);
An all-models ensemble, which has either all 10 models (an inclusive ensemble) or all models except the target one (a leave-one-out ensemble);
To simplify presentation, we only show summary results obtained by averaging accuracies for all target models of the same class. From Table g we can see that the accuracy under the self-attack is always under 5%. This is expected because models were not trained adversarially. However, attacks from different architectures are less successful, especially attacks from ResNet to VIT models and vice versa. Morcover, generating attacks using ensembles produces stronger altacks, which is well known from the literature {Liu et al. 2017 Gubri et al. 2022). Finally, we note that on ImageNet we observe the lowest transferability of attacks.
In our next experiment, we lested five ResNet and two ViT architectures on CIFARIO. Each architecture variant was trained using different scenario. Scenarios include (1) training on clean data and (2) adversarial training with different attack strengths: from 2/255 to 8/255. The attack radius was e = 8/255.
From Table g we can see that transferring an attack from a model trained on clean data to a model trained with the attack radius & = 2/255 reduces the accuracy by only 6.7 percentage points. This is totally unsurprising since the clean-data model not supposed produce strong attacks for robust models. However, transferring attacks in the other direction-ice from the model trained adversarially with E 2/255 to model trained on clean data data-yields 34.7% accuracy, still massively higher than the 0.4% achieved under the self-attack.
A.2 Gradient Tracing: Checking for Numerical Instabilities
To make sure our results were not affected by numerical instabilities (or by exploding and vanishing gradi- ents), "traced" gradients for ResNet-152 (He et al. 2016) backbone all three datasets. We used PGD
20

---

Published in Transactions on Machine Learning Research (08/2025)
Table 8: Transfer attack accuracy among different model architectures (see table notes). We show averages for ResNet and ViT models.
CIFARIO
CIFAR100
ImageNet
target
ResNet
ViT
ResNet
VIT
Res
VIT
source
Source model and larget models are different
No attack
92.7%
96.5%
73.0%
80.3%
74.7%
79.1%
Self-attack
0.3% (-92%)
0.6% (-96%)
2.2% (-71%)
1.4% (-79%)
0.0% (-75%)
3.1% (-76%)
Best ResNet
33.2% (-60%)
80.3% (-16%)
31.5% (-41%)
59.9% (-20%)
44.2% (-31%)
73.6% (-5%)
Best VIT
67.0% (-26%)
12.9% (-84%)
50.4% (-23%)
16.9% (-63%)
65.4% (-9%)
55.9% (-23%)
Leave-One Out (LOT) ensembles: target models are excluded
ResNet ensemble
24.3% (-68%) 63.8% (-33%)
19.6% (-53%) 46.6% (-34%)
32.9% (-42%
71.1%
(-8%)
ViT ensemble
53.9%
(-39%) 10.0% (-87%)
39.2% (-34%) 12.3% (-68%)
60.6% (-14%)
55.2%
(-24%)
All ensemble
25.6%
(-67%) 39.0% (-58%)
20.9% (-52%) 23.2% (-57%) 34.9% (-40%) 59.1% (-20%)
All-inclusive ensembles: All models of given type are included
ResNet ensemble
15.4%
(-77%) 62.8% (-34%)
11.4% (-62%) 46.9% (-33%)
13.8% (-61%) 71.0% (-8%)
ViT ensemble
52.9% (-40%) 6.4% (-90%
38.8% (-34%)
6.8% (-73%)
60.5% (-14%)
15.4% (-64%)
All ensemble
17.8%
(-75%) 35.3% (-61%)
13.9% (-59%) 17.8% (-63%)
18.1% (-57%) 26.1% (-53%)
We experiment with five ResNet models and five VIT models: Best ResNet and Best. VIT denote the most effective attack using ResNet or VIT. Model are trained using "clean" data, i.c., without adversarial attacks. In round brackets we show the loss of accuracy compared to "clean" accuracy (no attack), which we round to nearest integer.
Table 9: CIFARIO: average transfer attack accuracy among models with the same architecture, but trained with different attack strengths: round brackets we show the loss of accuracy compared "clean" accuracy (no attack). Diagonal row shows the self-attack accuracy (attack is generated using the same model). We tested five ResNet and two VIT architectures.
e-train
No attack
Accuracy under transfer attack
target
no adv. train.
2/255
4/255
8/255
source
adv. train
93.7%
0.4% (-93.4%)
83.1% (-6.7%)
83.1% (-3.8%)
75.3% (-2.6%)
2/255
89.8%
34.7% (-59.0%)
34.6% (-55.3%)
60.9% (-26.0%)
65.9% (-12.0%)
4/255
86.9%
52.2%
(-41.5%)
51.8% (-38.0%)
46.9%
(-40.0%)
60.4%
(-17.5%)
8/255
77.9%
72.5%
(-21.2%)
67.1%
(-22.7%)
61.7%
(-25.2%)
51.8% (-26.1%)
21

---

Published in Transactions on Machine Learning Research (08/2025)
with 50 steps and single restart. We did this for all datasets using sample of 500 images. We compared
gradients between two scenarios:
Attacking a model trained on clean data;
Attacking a front-end "enhanced" model that was adversarially trained.
We found no apparent anomalies. In particular:
No gradient was ever NaN or =oo;;
In both cases only a modest fraction (10-30%) of absolute gradient values were small (< 0.01);
In both cascs, mean and median gradients (per image) tended be close zero;
99% of gradients absolute values were under 2000;
In that, the distribution of gradient values for front-end "enhanced" models tended to be more skewed with more extreme values in lower or higher quantiles (by 1-2 order of magnitudes).
A.3 Additional Experimental Results
22

---

Published in Transactions on Machine Learning Research (08/2025)
Table 10: Detailed results for CTFAR1O (see table notes).
Single Model
Model
No Attack PGD
APGD+FAB-T (AutoAttack)
SQUARE (AutoAttack)
PGD
PGD
(standard)
(0-order)
(BPDA)
ResNet-18
9113%%
0%
110%
31:2%
ResNet-18 +front
9113%%
8814%%
7515%
1+1%
51.2%
0%
ResNet-30
95.33%
0%
0+1%
41.3
ResNet-50 front
95..3%%
93.33%
78:15%
24.2%
54.3%
0111%
ResNet-152
94.33%
0%
11.2%
61.3%
ResNet-152 +front
94+33%
92+33%
81+4%%
2+22%
7+:3%
0%
WideResNee-28-
96+t2%
0%
0%
84.3%
WideltesNet-2S--10 front
95.33%
9113%
65.1%
111%
1044%
111%
VIT-base- 16x16
97122%
011%
111%
1014%
VIT-base-16x16 +frrnt
97+-2%
96--2%
85+44%
14:2%
10+33%
0+11%
Randomized Ensemble
Model
No Attack PGD
APGD/EOTTSSQARR SQUARE (AutoAttack)
PGD
PGD
(standard) (AutoAttack)
(0-order)
(BPDA)
ResNet-18 3x
92.13%
2.12%
7611%%
66.16%
ResNet-18 +front 3x
91.14%
89.14%
70.6%
7415%
64.15%
3..2%
ResNet-30 3x
95+33%
3+:2%
80+44
69+5%
ResNet-30 +front 3x
96+-2%
94+33%
71+5%
81+4%
69+5%
5+22%
ResNet-152
9413%%
31:2%
84144%
7015%
ResNet-152 front 3x
95122%
9413%
7115%
7915%
7415%
61.3%
WideResNNet-28- 3x
96-12%%
34.2%
77+55%
45+6%
2122-e0 + front 3x
95--3%%
92433%
60.16%
791-1%
5145%
31.2%
VIT-base-16x16 3x
97.22%
1+1%
7845%
6416%
VIT-base--6616 +front 3x
96-22%
96122%
74+5%
8144%
65+6%
1+1%
Test sample has 500 images: 99% confidence intervals are shown. Each random ensemble has models.
APGD/EOT+SQUARRF is combination of APGD with FOT (as in rand version of AutoAttack) followed by SQUARE attack.
Best outcomes shown in bold. Best results computed separately for single models and ensembles.
We show 99% confidence intervals: They cannot be computed when the attack has 100% success rate. We do additional PGD attacks (except 0-order PGD) when the accuracy 0% with the standard PGD atlack.
23

---

Published in Transactions on Machine Learning Research (08/2025)
Table 11: Detailed results for CIFARI00 (see table notes).
Single Model
Model
No Attack PGD
APGD+FAB-T (AutoAttack)
SQUARE (AutoAttack)
PGD
PGD (BPDA)
(standard)
(0-order)
ResNet-18
6816%
0+1%
111%
41.3%
ResNet-18 +front
6845%
6515%
4816%
2+1%
41.2%
0+1%
ResNet-30
7515%
1+0%
1+2%
613%
ResNet-50 front
77..5%
70.6%
53.16%
21.2%
74.3%
111%
ResNet-152
76.15%
11.2%
2111%
91.%
ResNet-152 +front
74+55%
71+5%
51+6%
5+.2%
11+4%
2+2%
WideResNeet22--11
76+5%%
2+1%
0+1%
743%
WideltesNet-2S--10 front
75.5%%
6715%
35.5%
211%
813%
311%
VIT-base- 16x16
8415%
111%
212%
843%
VTT-base-16x16 +frrnt
84+55%
80+-5%
5916%
34.2%
844%%
141%
Randomized Ensemble
Model
No Attack
PGD
APGD/EOTTSSQARR SQUARE (AutoAttack)
PGD
PGD
(standard) (AutoAttack)
(0-order)
(BPDA)
ResNet-18 3x
69.15%
5..3%
51166%
49.15%
ResNet-18 +front 3x
66.16%
67.15%
40.66%
5416%
4616%%
8.13%
ResNet-30 3x
73+5%
6+:2%
57+66%
49+6%
ResNet-30 +front 3x
75+-5%
72+6%
46+16%
58+66%
51+6%
9+.3%
ResNet-152
7615%
1013%
5816%
4916%
ResNet-152 front 3x
7515%
7115%
4816%
5716%
5316%
1014%
WideResNNet-28- 3x
784.5%
5+.2%
5245%
3145%
2122-e0 front 3x
764.5%
7045%
3116%
52466
3316%
54.2%
VIT-base-16x16 3x
85+.4%
2+2%
64+6%
4645%
VIT-base--6616 +front 3x
86+44%
8115%
5416%
63+5%
45.6%
3+1%
Test sample has 500 images: 99% confidence intervals are shown. Each random ensemble has 3 models. APGD/EOT+SQUARRF is combination of APGD with FOT (as in rand version of AutoAttack) followed by SQUARF attack.
Best outcomes shown in bold. Best results computed separately for single models and ensembles. We show 99% confidence intervals: They cannot be computed when the attack has 100% success rate. We do additional PGD attacks (except 0-order PGD) when the accuracy 0% with the standard PGD atlack.
24

---

Published in Transactions on Machine Learning Research (08/2025)
Table 12: Detailed results for ImageNet (see table notes).
Single Model
Model
No Attack
PGD
APGI+FFABTT (AutoAttack)
SQUARE (AutoAttack)
PGD (0-order)
PGD
(standard)
(BPDA)
ResNet-18
6919%
0%
2+22%
2318%
ResNet-18 +front
70188%
6849%
2818%%
2+:3%
2318%
0%
ResNet-50
7618%%
0%
715%
4019%
ResNet-50 + front
76.17%
66.1%%
31..9%
8.15%
40110%
0%
ResNet-152
78.18%
0%
12.15%
4619%
ResNet-152 +foott
78+8%
74+9%
26+88%
16+6%
47+9%
0%
VIT-hase-16x16
83+7%
0%
18+8%
60+.9%
VIT-bbae-16xx16 ffoot
8218%
7918%
4819%
2118%
62.19%
0%
Randomized Ensemble
Model
No Attack
PGD
APGD//OOT+SQAARE
SQUARE (AutoAttack)
PGD (0-order)
PGD
(standard)
(AutoAttack)
(BPDA)
ResNet-18 3x
6911%
0%
2+2%
2348%
ResNet-18 + front 3x
70.18%
68.19%
42.110%
48.19%
33.99%
0.12%
ResNet-50 3x
76.18%
0%
8.14%
3818%
ResNet-50 + front 3x
76.17%
72.8%%
50.99%
55.19%
50.19%
0%
ResNet-152 3x
78+88%
0%
12+5%
44+9%
ResNet-152 ffront 3x
78+77%
74+8%%
48+10%
53+99%
53+.9%
0%
VIT-base-16xil 16x 3x
83.17%
0%
1817%
58410%
VIT'-baso-16x16 +front
8218%
8017%
63.1%%
69.19
68419%
0%
Test sample has 200 images: 99% confidence intervals are shown. Each random ensemble has 3 models. APGD/EOT+SQUARE combination of APGD with EOT (as rand version of AutoAttack) followed by SQUARE altack.
Best outcomes are shown bold. Best results are computed separately for single models and ensembles. We show 99% confidence intervals: They cannot be computed when the attack has 100% success rate. do not run additional PGD attacks (except 0-order PGD) when the accuracy 0% with the standard PGD attack.
25


Reviews:
[{"review_id": "ZbXFXu8Lm5", "reviewer": "Reviewer_wEQx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This submission thoroughly investigates its claims by evaluating the front-end enhanced models on different datasets and architectures. However, the clarity of the writing could be improved to allow the reader to follow the reading more easily. Specific examples are mentioned in the \"Requested Changes\" section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission explores the effect of using a differentiable, fully convolutional front-end layer with skip connections before a frozen backbone on gradient-based adversarial attacks. Furthermore, this submission questions the reliability of the increasingly popular AutoAttack package, which includes multiple gradient-based attacks. The authors suggest AutoAttack may overestimate the robustness of some models, specifically those that use randomized ensembles of models including the differentiable front end.\n\nThe authors found that the differentiable, fully convolutional model can seemingly preserve the classifier’s ability to make accurate predictions on clean data, while increasing robustness to gradient-based attacks. For non-gradient based attacks, a single front-end enhanced model does perform significantly worse, with almost 0% accuracy in some cases. However, when using randomized ensembles of the same front-end enhanced models, the accuracy for non-gradient-based attacks rivals that of the current state-of-the-art models. This discrepancy suggests that AutoAttack may not be as reliable as previously claimed.\n\nThe authors suggest that this may be due to an unexpected phenomenon seen in these front-end layers known as gradient masking, typically seen in “gradient shattering components” such as JPEG compression. The authors investigate and provide reasons as to why they believe gradient masking is responsible for this behavior.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Proposed Adjustments to strengthen your work:\n\n1. In the first sentence in the abstract, \"a frozen backbone classifier was prepended by a differentiable and fully convolutional model with a skip connection\" sounds as if the front-end layer comes before the classifier, not before the backbone. This is clarified later on, but I'd suggest rewriting to avoid a potential misinterpretation from the reader.\n\n2. In the first paragraph of the introduction, standardize the references to AutoAttack to improve clarity. Currently \"The AutoAttack\", \"AutoAttack\", and \"The AutoAttack package\" are all used which could confuse the reader.\n\n3. In the first sentence of the second paragraph in the Related Work section, the connection between the linear and non-linear classifiers is not well established. The sentence moves quickly from discussing linear classifiers and their susceptibility to attacks to previous beliefs that complex classifiers were more difficult to attack. Consider being more explicit in the wording of this sentence.\n\n4. For the caption of Figure 1, I suggest changing \"shortcut connection\" to \"skip connection\" as it term used most in the paper and is used in the title.\n\n5. For the first set of bullets in section 4.1, I would suggest being more explicit in the description of tested models. Specifically, for the second bullet, it sounds as if standard and robust are being used synonymously when the first bullet uses standard and non-robust synonymously. \n\n6. For the final set of bullets in section 4.1, explain why the model is trained for at most one epoch.\n\n7. In the first sentence of the paragraph discussing Zero-order PGD in section 4.2, if going to make a comparison between zero-order PGD and smooth PGD, smooth PGD should be briefly introduced before this section. I would suggest removing this line or adding a short paragraph discussing smooth PGD to establish necessary context.\n\n8. In the third sentence of paragraph 5 in section 5.2, explain why the accuracy of the Transfer attack is used as the \"true robust accuracy of this model\".\n\n9. In the \"Is it gradient masking\" section in section 5.2, provide more information for each of the three supporting reasons as to why the exaggerated performance is due to gradient masking. A sentence or two highlighting why these reasons could be an indicator would help solidify your rationale for a more general audience.\n\n10. Sentence 3 of the Conclusion is copied and pasted four times throughout the paper. While this is a good way to describe the gradient masking phenomenon for differentiable models, refrain from using this exact sentence multiple times throughout the writing. You could either remove some of the instances of this sentence of rephrase them to cover the same information with different wording.\n\nTo secure my recommendation, go through the submission and correct spelling and grammar issues to the best of your ability. Proposed adjustments:\n\n1. In the second sentence of the third paragraph in the related works section, the unnecessary use of colons negatively affects the readability of this section. Consider separating the sentences using commas or new sentences.\n\n2. For the first bullet in the first set of bullets in section 4.1, there are too many commas being used, and the use of \"i.e.\" seems forced. Simplify this bullet to improve readability\n\n3. In the first sentence of the third block of text in section 4.1, correct \"randomly a uniformly\".\n\n4. In the last sentence of the block of text in the AutoAttack section of section 4.2, correct \"AuttoAttack\"", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UqZiKOwHOk", "reviewer": "Reviewer_5CEV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. Targets an important question: The robustness of deep learning models is a significant problem.\n2. Provides interesting discussions based on the experimental results.\n3. Includes both CNN and ViT in the experiments.\n\n# Weakness\n1. Paper organization: The organization of the paper could be improved. Specifically, the authors mix their proposed techniques with prior works in one section, making the core contribution unclear. More detailed comments are in the \"Requested Changes\" section.\n2. Unclear contribution: It seems that the \"front-end enhanced neural models, where a frozen backbone classifier is prepended by a differentiable and fully convolutional model with a skip connection,\" were not proposed by the authors, as the model structures follow those in DnCNN (Zhang et al., 2017) and Norouzzadeh et al. (2021). Also, the attack methods are not new, as detailed in Sec. 4.2. Based on my understanding, the authors merely modified the settings of prior attack methods to challenge the \"front-end enhanced neural models.\" Consequently, the claim of \"methodological contributions\" may not be valid. Moreover, the statement \"further supports the thesis...\" does not sufficiently contribute because the authors have not clearly delineated what is new compared to prior works.\n3. Validity of experimental settings is not clarified: The authors claim that \"we use randomly selected images: 500 for CIFAR10/CIFAR100 and 200 for ImageNet\", however, no prior works are cited for this setting, and no experimental results are provided to prove its validity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work conducted experiments on front-end enhanced neural models where a frozen backbone classifier was prepended by a differentiable and fully convolutional model with a skip connection. It was observed that the resulting models, after training, were unusually resistant to gradient attacks. The authors suspected that the reason for the observed resilience is the gradient masking phenomenon.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Besides the \"Weaknesses\" listed above, I have the following detailed comments:\n\n1. Some minor consistency issues: e.g., \"Croce et al. (2021)\" vs. \"(Croce & Hein, 2020a)\".\n2. The abstract is relatively longer than average, and several sentences in the abstract merely repeat content from the introduction.\n3. The explanation of how prior attacks are processed is placed in the Methods section, which may be confusing to first-time readers, as it is unclear whether it is part of the preliminaries or the proposed method. Moving this introduction to prior works to the \"Preliminaries\" section might be a better option.", "broader_impact_concerns": "No concerns on the ethical implications of this work", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "56yK8n7dio", "reviewer": "Reviewer_nhBn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed framework is simple and easy to follow. It includes a front-end enhanced neural model that combines a frozen backbone classifier with a convolutional model, which enhances adversarial robustness against gradient attacks.\n2. The experiments are extensive. The evaluation includes multiple datasets, such as CIFAR-10/100 and ImageNet. Compared with baselines, the proposed framework achieves superior performance.\n3. The discussion on the limitations of existing evaluation approaches, particularly regarding the reliability of the AutoAttack package could benefit the community.\n\nWeaknesses:\n1. There exist many randomized defense techniques, such as [a,b,c,d], which should be discussed or compared.\n2. Although the discussion on the limitations of existing evaluation approaches can be valuable, more discussion of a potential reasonable evaluation of randomized ensemble could be more valuable.\n3. The design of the proposed randomized ensemble is highly correlated to the adversarial transferability of selected models. The adversarial transferability study could be included to demonstrate its influence.\n\n[a]. Defending against adversarial attacks by randomized diversification. CVPR 2019.\n\n[b]. Certified adversarial robustness via randomized smoothing. ICML 2019.\n\n[c]. Double-win quant: Aggressively winning robustness of quantized deep neural networks via random precision training and inference. ICML 2021.\n\n[d]. Random normalization aggregation for adversarial defense. NeurIPS 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores front-end enhanced neural models that combine a frozen backbone classifier with a differentiable and convolutional model to improve the adversarial robustness. Specifically, the authors utilize several models for random ensembles with front ends, resulting in models that maintained the accuracy of the backbone classifier while exhibiting significant resistance to gradient attacks. The study demonstrates that this resistance is primarily due to gradient masking. The proposed framework as well as the observations have been validated in multiple datasets and model types. The authors also discuss the limitations of black-box attacks and propose the use of randomized ensembles to counteract them. While the goal of the paper is not to establish SOTA performance in adversarial robustness, the findings contribute methodological insights and emphasize the importance of adaptive attacks in evaluating model robustness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned in the weakness.\n1. Please discuss or compare some relevant randomized defense.\n2. Please discuss and explore the reasonable evaluation of randomized ensemble.\n3. Please provide an adversarial transferability study.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed adding a denoising front end (DnCNN) architecture from Zhang et al. (2017) with a skip connection to pre-trained image classification backbones. The empirical results on multiple datasets and model architectures show that training this layer along can have the gradient masking effect that obfuscates the tested white-box attacks (in a non-adaptive setting), while it is not effective against tested non-gradient based and black-box attacks. All reviewers appreciate the extensive empirical analysis, but share similar concerns as listed below.

Based on review feedback, author response and my own reading, I suggest a mandatory revision and will accept the revised version only if all the requirements are met.

- Presentation and organization: please follow the reviewers' comments to improve the presentation and organization.
- Novelty to DnCNN: please elaborate on the difference to the original DnCNN paper (both in terms of architecture changes and the scope of the research). Also, do the claimed results hold only for DnCNN but not for other types of similar front-end architectures? How critical is the skip connection? Adding some ablation study on the importance of the proposed architecture for the claims will add more clarity.
- Experiments on transfer attacks: please add the transfer attack results, as discussed with the reviewers.
Decision: Accept (poster)