SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Naman D Singh*  
University of Tübingen  
Tübingen AI Center
Francesco Croce*  
University of Tübingen  
Tübingen AI Center
Matthias Hein  
University of Tübingen  
Tübingen AI Center
While adversarial training has been extensively studied for ResNet architectures and low resolution datasets like CIFAR-10, much less is known for ImageNet. Given the recent debate about whether transformers are more robust than convnets, we revisit adversarial training on ImageNet comparing ViTs and ConvNeXts. Extensive experiments show that minor changes in architecture, most notably replacing PatchStem with ConvStem, and training scheme have a significant impact on the achieved robustness. These changes not only increase robustness in the seen  $\ell_{\infty}$ -threat model, but even more so improve generalization to unseen  $\ell_1 / \ell_2$ -attacks. Our modified ConvNeXt, ConvNeXt + ConvStem, yields the most robust  $\ell_{\infty}$ -models across different ranges of model parameters and FLOPs, while our ViT + ConvStem yields the best generalization to unseen threat models.
Adversarial training [36] is the standard technique to obtain classifiers robust against adversarial perturbations. Many works have extensively analyzed several aspects of adversarial training, leading to further improvements. These include changes in the training scheme, i.e. using different loss functions [66, 43, 40], tuning hyperparameters like learning rate schedule or weight decay, increasing the strength of the attack at training time, adding label smoothing and stochastic weight averaging [19, 41]. Also, complementing the training set with pseudo-labelled [3] or synthetic [20] data, and via specific augmentations [45] yields more robust classifiers.
However, these works focus on small datasets like CIFAR-10 with low resolution images, and models using variants of ResNet [22] as architecture. When considering a more challenging dataset like ImageNet, the space of possible design choices regarding model, training and evaluation protocols becomes much richer, and little work exists exploring this domain. For example, it has been recently shown that architectures like vision transformers (ViTs) [16, 50], mixers [49, 54] and hybrids borrowing elements of different families of networks [17, 64] are competitive or outperform convolutional networks on ImageNet, while they hardly work on e.g. CIFAR-10. In turn, ConvNeXt [34] has been proposed as modern version of ResNet which closes the gap to transformer architectures.
In this work we study the influence of architecture and training schemes on the robustness of classifiers to seen and unseen attacks. We focus on two extreme ends of architectures used for ImageNet, ViT and ConvNeXt (isotropic vs non-isotropic, attention only vs convolution only, stem with large vs small patches), and study Isotropic ConvNeXt as an intermediate architecture. We focus on training ImageNet models robust with respect to the  $\ell_{\infty}$ -threat model (i.e. the perturbations have bounded  $\ell_{\infty}$ -norm), but additionally track the robustness to unseen  $\ell_{1}$ - and  $\ell_{2}$ -attacks in order to reveal architectural components which lead to better generalization of robustness. This, in fact, is a desirable
* Equal Contribution. Correspondence to: naman-deep.singh@uni-tuebingen.de
Figure 1: AutoAttack  $\ell_{\infty}$ -Robust accuracy ( $\epsilon = 4/255$ ) on ImageNet across different architectures and training schemes at resolution  $224 \times 224$ . With our training scheme and replacing PatchStem with ConvStem in ConvNeXt and ViT, we outperform over a range of parameters/FLOPs existing models in terms of  $\ell_{\infty}$ -robust accuracy, e.g. our ConvNeXt-T-CvSt with 28.6M parameters achieves  $50.2\%$  robust accuracy, an improvement of  $5.8\%$  compared to the ConvNeXt-T of [14].
property since typically adversarially trained models are still vulnerable to attacks not seen at training time [28]. In particular, we show via extensive experiments that
- using a convolutional stem instead of a patch stem in most cases leads to consistent gains in  $\ell_{\infty}$ -robustness and quite a large boost in the unseen  $\ell_{1}$ - and  $\ell_{2}$ -threat models.
- leveraging SOTA pre-trained clean models as initialization allows us to train using heavy augmentation, in contrast to what has been suggested by prior work [2, 14].
- increasing image resolution at test time yields improvements in  $\ell_{\infty}$ -robust accuracy. This is surprising as for the  $\ell_{\infty}$ -threat model with fixed radius, higher resolution yields a stronger attack.
Combining our various improvements, we obtain state-of-the-art  $\ell_{\infty}$ -robust accuracy for perturbation size  $\epsilon = 4/255$  at resolution  $224 \times 224$  on ImageNet, see Fig. 1. We obtain  $50.2\%$  robust accuracy for small and  $56.3\%$  robust accuracy for large models, improving by  $5.8\%$  and  $2.8\%$  respectively upon prior works. Furthermore, we improve SOTA when fine-tuning these models to a larger radius,  $\epsilon_{\infty} = 8/255$  and to other datasets. We make our code and robust models publicly available. $^{1}$
In [36] adversarial training (AT) of a classifier  $f_{\theta}:\mathcal{X}\to \mathcal{Y}$  is introduced as the optimization problem
$$
\min  _ {\theta} \sum_ {(x _ {i}, y _ {i}) \in \mathcal {D}} \max  _ {\delta : \| \delta \| _ {p} \leq \epsilon_ {p}} \mathcal {L} \left(f _ {\theta} \left(x _ {i} + \delta\right), y _ {i}\right) \tag {1}
$$
where  $\theta$  are the parameters of the classifier  $f_{\theta}$ . The goal is to make  $f_{\theta}$  robust against  $\ell_p$ -bounded perturbations, that is the classifier should have the same prediction on the  $\ell_p$ -ball  $B_p(x_i, \epsilon_p)$  of radius  $\epsilon_p$  centered at  $x_i$ . In practice, the inner maximization is approximately solved with projected gradient descent (PGD),  $\theta$  is optimized with SGD (or its variants) on the training set  $\mathcal{D}$ , and cross-entropy loss is used as objective function  $\mathcal{L}$ . AT has been shown to be an effective defense against adversarial perturbations, and many follow-up works have modified the original algorithm to further improve the robustness of the resulting models [66, 61, 20, 43, 40].
The most popular threat models are  $\ell_{\infty}$ - and  $\ell_2$ -bounded perturbations. Several other attacks have been explored, including  $\ell_1$ - or  $\ell_0$ -bounded, adversarial patches, or those defined by neural perceptual
<https://github.com/nmndep/revisiting-at>
metrics like LPIPS [30]. A well-known problem is that models trained to be robust with respect to one threat model need not be robust with respect to unseen ones [28]. Multiple-norm adversarial training [53] aims to overcome this but degrades the performance compared to adversarial training in each individual threat model.
Training recipe. [19, 41] analyzed in detail the influence of several fine-grained choices of training hyperparameters (batch size, weight decay, learning rate schedule, etc.) on the achieved robustness. [24, 5] suggest to use a standard classifier pre-trained either on a larger dataset or with a self-supervised task as initialization for adversarial training for improved robustness. However, all these works focus on CIFAR-10 or other datasets with low resolution images. In [38], the effect of several elements including clean initialization, data augmentation and length of training on adversarial training with ViTs is analyzed: however, their study is limited to CIFAR-10 and Imagenette [25], and we show below that some of their conclusions do not generalize to ImageNet and other architectures. For ImageNet, [2] argued that ViTs and ResNets attain similar robustness for several threat models if training scheme and architecture size for both are similar, while [14] proposed a recipe for training  $\ell_{\infty}$ -robust transformers, in particular XCiT [17]: basic augmentation, warm-up for the radius  $\epsilon_p$  and large weight decay. In contrast, [44] use the standard training scheme for a large ViT-B for 300 epochs and obtain the most  $\ell_{\infty}$ -robust model on ImageNet. In concurrent work, [32] study  $\ell_{\infty}$ -training schemes for ConvNeXt and Swin-Transformer which are outperformed by our models, in particular regarding generalization to unseen attacks.
Architectural design choices. Searching for more robust networks [60, 26] studied the effect of architecture design choices in ResNets, e.g. width and depth, on adversarial robustness for CIFAR-10. Recently, [42] did a similar exploration on ImageNet. [48] compared the robustness to seen and unseen attacks of several modern architectures on CIFAR-10 and Imagenette, and conclude that ViTs lead to better robust generalization. Finally, [63] found that using smooth activation functions like GELU [23] improves the robustness of adversarially trained models on ImageNet, later confirmed for CIFAR-10 as well [19, 41].
Modern architectures. Since [16] showed that transformer-based architectures can be successfully used for vision tasks, a large number of works have introduced modifications of vision transformers [50, 33, 55] achieving SOTA classification accuracy on ImageNet, outperforming ResNets and variants. In turn, several works proposed variations of training scheme and architectures to make convolutional networks competitive with ViTs [47, 59], in particular leading to the ConvNeXt architecture of [34]. Moreover, hybrid models combine typical elements of ViTs and CNNs, trying to merge the strengths of the two families [17, 13, 64]. While these modern architectures perform similarly well on vision tasks, it is open how they differ regarding adversarial robustness. [18, 21] compared the robustness of normally trained ViTs and ResNets to  $\ell_p$ -norm bounded and patch attacks concluding that the ranking depends on the threat model.
3 Influence of Architecture on Adversarial Robustness to Seen and Unseen Attacks
In this section we analyze existing architectures and their components regarding adversarial robustness on ImageNet. In particular, we are interested in studying which elements are beneficial for adversarial robustness  $(i)$  in the threat model seen at training time and  $(ii)$  to unseen attacks. Since it is the most popular and well-studied setup, we focus on adversarial training w.r.t.  $\ell_{\infty}$ , and use  $\ell_{2}$ - and  $\ell_{1}$ -bounded attacks to measure the generalization of robustness to unseen threat models. While we focus on the influence of architecture on the achieved robustness in this section, we analyze the influence of pre-training and augmentation in Sec. 4. All results are summarized in Table 1 which provides the full ablation for this and the next section. We select the version T for ConvNeXt and S for Isotropic ConvNeXt and ViT, since these have comparable number of parameters and FLOPs.
Training setup. If not stated otherwise, we always use adversarial training [36] w.r.t.  $\ell_{\infty}$  with perturbation bound  $\epsilon = 4 / 255$  and 2 steps of APGD [8] for the inner optimization on ImageNet-1k using resolution  $224\times 224$  (we refer to App. C.2 for a comparison of APGD-AT vs PGD-AT). Note that we use the  $\ell_{2 - }$  or  $\ell_1$  -threat model just for evaluation, we never do adversarial training for them. We use a cosine decaying learning rate preceded by linear warm-up. In this section, we use a basic
training setting, that is only random crop as basic augmentation and 50 epochs of adversarial training from random initialization. More details are available in App. A.
Evaluation setup. For all experiments in the paper, unless specified otherwise, we evaluate the robustness using AutoAttack [8, 9], which combines APGD for cross-entropy and targeted DLR loss, FAB-attack [7] and the black-box Square Attack [1], on the 5000 images of the ImageNet validation set selected by RobustBench [6] at resolution 224x224. The evaluations are done at the following radii:  $\epsilon_{\infty} = 4 / 255$ ,  $\epsilon_{2} = 2$ , and  $\epsilon_{1} = 75$ . For some marked experiments which involve a large number of evaluations we use  $\mathrm{APGD}_{\mathrm{T - DLR}}$  (40 iterations, 3 restarts) with the targeted DLR loss [8] which is roughly off by  $1 - 2\%$  compared to full AutoAttack.
3.2 Architecture Families and Convolutional Stem
Choice of architectures. While some of the novel components of ConvNeXts are inspired by the Swin Transformer [33], the ConvNeXt is a modified version of a ResNet, using progressive downsampling and convolutional layers. On the other end of the spectrum of network families we find ViTs, as a non-convolutional architecture using attention and with an isotropic design after the PatchStem, see Fig. 2. Additionally, [34] introduce Isotropic ConvNeXt, which creates tokens similarly to ViTs, which are then processed by a sequence of identical residual blocks (with no downsampling, hence the name isotropic) as in ViTs, until a final linear classifier. The blocks of a ViT rely on multi-head self-attention layers [56], while Isotropic ConvNeXt replaces them with depthwise convolutions. It can be thus considered an intermediate step between ConvNeXt and ViT. By considering ViT and ConvNeXt, we cover two extreme cases of modern architectures. While it would have been interesting to include other intermediate architectures like XCiT or Swin-Transformer, our computational budget did not allow this on top of the already extensive experiments of the present paper.
Convolutional stem. [62] propose to replace the patch embedder in ViTs with a convolutional block (ConvStem) consisting of several convolutional layers alternating with normalization and activation layers. While the PatchStem in ViTs divides the input in disjoint patches and creates tokens from them, the ConvStem progressively downsamples the input image until it has the right resolution to be fed to the transformer blocks. [62] show that this results in a more stable standard training which is robust to the choice of optimizer and hyperparameters.
As adversarial training is more difficult than standard training, we check if adding a ConvStem has a similar positive impact there as well. As both ViTs and Isotropic ConvNeXts divide the input image into  $16 \times 16$  patches, we use for both the convolutional stem of [62], which has four convolutional layers with stride 2, each followed by layer normalization and GELU activation. Since ConvNeXt uses a PatchStem with  $4 \times 4$ -patches, the ConvStem of ViTs would not work. Therefore we design a specific convolutional stem with only two convolutional layers with kernel size 3 and stride 2, see Fig. 2. Adding a ConvStem has only minimal impact on the number of parameters and FLOPs of the models (see Table 2), especially for ConvNeXt.
More details on the convolutional stem design, and ablation studies on its effect on unseen threat models can be found in App. B. The effect of ConvStem on model size is discussed in App. C.4.
Figure 2: Convolutional stem for ConvNeXt. We use two convolution layers, each followed by LayerNorm and GELU to replace the 4x4 patch of the standard ConvNeXt model (PatchStem).
3.3 Effect of Architecture and ConvStem vs Patchstem on Adversarial Robustness
In Table 1 we first check for the basic setting described above (random init., basic augment.) the effect of using a convolutional stem (+ CvSt) vs PatchStem for the three architectures: for both isotropic architectures it notably improves the robustness as well as clean accuracy for adversarially trained models, whereas the improvements for the ConvNeXt are marginal. However, all three architectures show huge improvements in the generalization of robustness to unseen  $\ell_1$ - and  $\ell_2$ -attacks when using ConvStem instead of PatchStem. Interestingly, the  $\ell_1$ - and  $\ell_2$ -robustness of the ConvStem models is quite similar despite larger differences in  $\ell_{\infty}$ -robustness. Our results in this basic training setup show
Table 1: Influence of ConvStem, strong (clean) pre-training and heavy data augmentation. We show an ablation of using strong (clean) pre-training, heavy data augmentation and longer training for all models with and without ConvStem. Across all architectures using ConvStem, stronger pre-training, heavy augmentation and longer training improve adversarial robustness. We boldface the best performance for every metric within each architecture family.
Architecture	Training Scheme	Adversarial Training w.r.t. ℓ∞
clean	ℓ∞	ℓ2	ℓ1
ConvNeXt-T	random init., basic augment.	64.0	37.1	28.9	8.8
+ strong clean pre-training	69.4 +5.4	41.6 +4.5	42.4 +13.5	18.9 +10.1
+ heavy augmentations	71.0 +1.6	46.5 +4.9	38.1 -4.3	14.9 -4.0
50 → 300 epochs	72.4 +1.4	48.6 +2.1	38.0 -0.1	14.9 +0.0
ConvNeXt-T + CvSt	random init., basic augment.	64.2	37.6	40.2	18.6
+ strong clean pre-training	69.1 +4.9	42.2 +4.6	42.4 +2.2	19.7 +1.1
+ heavy augmentations	69.1 +0.0	47.5 +5.3	47.8 +5.4	24.6 +4.9
50 → 300 epochs	72.7 +3.6	49.5 +2.0	48.4 +0.6	24.5 -0.1
Isotropic ConvNeXt-S	random init., basic augment.	60.5	31.7	20.8	4.0
+ strong clean pre-training	68.0 +7.5	38.9 +7.2	38.3 +17.5	17.0 +13.0
+ heavy augmentations	64.5 -3.5	38.5 -0.4	36.5 -1.8	17.7 +0.7
50 → 300 epochs	69.0 +4.5	44.2 +5.7	36.6 +0.1	14.9 -2.8
Iso-ConvNeXt-S + CvSt	random init., basic augment.	62.1	33.5	40.0	23.8
+ strong clean pre-training	69.5 +7.4	41.8 +8.3	46.2 +6.2	26.9 +3.1
+ heavy augmentations	69.5 +0.0	42.1 +0.3	47.2 +1.0	28.9 +2.0
50 → 300 epochs	70.2 +0.7	45.9 +3.8	49.2 +2.0	27.9 -1.0
ViT-S	random init., basic augment.	61.5	31.8	35.5	15.1
+ strong clean pre-training	66.8 +5.3	39.1 +7.3	34.5 -1.0	12.6 -2.5
+ heavy augmentations	65.2 -1.6	39.2 +0.1	37.3 +2.8	16.2 +3.6
50 → 300 epochs	69.2 +4.0	44.0 +4.8	37.5 +0.2	15.1 -1.1
ViT-S + CvSt	random init., basic augment.	62.8	34.4	39.2	20.4
+ strong clean pre-training	71.2 +8.4	44.3 +9.9	47.1 +7.9	23.2 +2.8
+ heavy augmentations	69.9 -1.3	44.0 -0.3	47.1 +0.0	25.9 +2.7
50 → 300 epochs	72.5 +2.6	48.1 +4.1	50.4 +3.3	26.7 +0.8
that the ConvStem is a simple and effective architecture modification which has significant impact on robustness with respect to seen and unseen threat models. We also note that this finding generalizes for the modifications of the training scheme proposed in the next section. It is particularly remarkable that even for the ConvNeXt architecture, this little change from PatchStem to ConvStem has such a huge effect on the robustness with respect to the unseen  $\ell_1$ -and  $\ell_2$ -threat models.
4 Effect of Strong Pre-Training and Heavy Data Augmentation on Robustness
As the next step after the architecture modification of the last section, we now identify in an ablation study in Table 1 strong pre-training and heavy augmentations as key elements of the training scheme which boost adversarial robustness on ImageNet. The effect of other fine-grained training hyperparameters like weight decay and label smoothing is discussed in App. D.3, where we show that varying them around our default values does not have any impact on robustness.
4.1 Strong Pre-Training
It is well-known that pre-training on large datasets improves the performance on downstream tasks, e.g. models pre-trained on ImageNet-21k or JFT achieve significantly higher accuracy on ImageNet-1k than those trained on ImageNet-1k from random initialization [58]. Similarly, it has been observed that pre-training on larger datasets like ImageNet-21k or ImageNet-1k helps to achieve better adversarial robustness on CIFAR-10 [24, 38]. However, our goal is not to use a larger training set than ImageNet-1k but to better utilize the given dataset ImageNet-1k. As adversarial training on ImageNet is a non-trivial optimization problem, where runs can fail completely, see [14], we propose to initialize adversarial training from a fully trained (on ImageNet-1k) standard model. Although the clean model
is clearly not adversarially robust, starting from a point with low clean loss considerably helps in the first stages of adversarial training. Short fine-tuning of clean models to become adversarially robust has been discussed in [27, 10] but without achieving robustness comparable to full adversarial training.
In the following we use as initialization the standard models available in the timms library or from the original papers, refer App. D.7 for further details. For the models with convolutional stem no such models are available. For these we initialize the ConvStem randomly and the remaining layers with the weights of the corresponding non-ConvStem models, then do standard training for 100 epochs to reach good clean accuracy (see App. B.3 for details and performance of these classifiers). Interestingly, changing from random initialization to initialization with strongly pre-trained models significantly improves clean as well as  $\ell_{\infty}$ -robust accuracy across architectures, see Table 1. The effects on the isotropic architectures, which are known to be harder to train, are even higher than for the ConvNeXt. At this stage the ViT-S + ConvStem already has  $2.5\%$  better  $\ell_{\infty}$ -robustness ( $44.3\%$  vs  $41.8\%$ ) than the XCiT-S model of [14]. Finally, using an even better model, i.e. pre-trained on ImageNet-21k, as initialization did not provide additional benefit, see App. D.1.
4.2 Heavy Data Augmentation
The second aspect we analyze is how to take advantage of heavy augmentation techniques like RandAugment [12], MixUp [67] and CutMix [65], which are crucial for the performance of standard models, in adversarial training. In particular, [2] note that for ViTs using heavy data augmentation with adversarial training leads to model collapse, and thus design a schedule to progressively increase the augmentation intensity. Similarly, [14] report better performance for robust ViTs when only weak augmentations (random crop, horizontal flip, color jitter) are used, and that ConvNeXt training even diverges if heavy augmentations are not excluded. Finally, [44] could train with heavy augmentations but using a much larger ViT-B model and a longer training of 300 epochs (which allows them to have a less steep learning rate warm-up phase). Instead we show that initializing adversarial training with a well-trained standard classifier makes it possible to use heavy augmentation techniques from the beginning, and benefit from them, even for small models and shorter training schedules. In contrast, in our experiments, training a ConvNeXt with heavy augmentations from random initialization failed.
As heavy augmentations we use RandAugment, CutMix, MixUp, Random Erasing [68] (with hyperparameters similar to [34], see App. A). We also use Exponential Moving Average (EMA) with a decay of 0.9999 and label smoothing with parameter 0.1 for consistency with [34]. However, we observe little to no effect of EMA on robustness.
In Table 1 one can see that adding heavy data augmentation on top of strong pre-training significantly boosts the performance of the ConvNeXts in clean and  $\ell_{\infty}$ -robust accuracy. Interestingly, only the ConvNeXt + ConvStem has further strong improvements in the unseen  $\ell_{2}$ - and  $\ell_{1}$ -threat models. For the isotropic architectures, heavy augmentations have marginal effect on  $\ell_{\infty}$ -robustness, although they improve robustness to unseen attacks. Isotropic architectures require longer training to benefit from heavy augmentations. We highlight that at this point our ConvNeXt-T and ConvNeXt-T + ConvStem already have higher (+2.1% and +3.1% respectively) robustness w.r.t.  $\ell_{\infty}$  than the ConvNeXt-T from [14], and similarly our ViT-S + ConvStem and Isotropic ConvNeXt + ConvStem outperform XCiT-S [14] (detailed comparison to SOTA in Table 2). A further comparison of heavy augmentation and the 3-Aug scheme of [14] can be found in in App. D.2.