SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The ConvNeXt-T trained for 50 epochs shows a small increase in  $\ell_{\infty}$ , while the one trained for 300 epochs gets slightly lower robustness. For ViT-B with 50 epochs we see a drop of  $0.4\%$  in robust accuracy although clean accuracy goes up by  $0.2\%$  (which is intuitively correct as adversarial training starts from a better point with 21k-pre-trained weights). Overall we did not see any sign of gains in robustness when using a model pre-trained on a larger dataset. This finding is line with what [38] witnessed when using ImageNet-1k and 21k as initialization while training robust models for CIFAR-10.
Table 16: Ablation for data augmentations. We show how varying levels of data augmentations influences robustness across all threat models in our training paradigm. The best performing augmentation setup for each model per perturbation is in bold. All models are trained for 50 epochs. Results for basic and heavy augmentation are taken from Table. 1.
Augmentation	ViT-S + ConvStem	ConvNeXt-T + ConvStem
clean	l∞	l2	l1	clean	l∞	l2	l1
basic	71.2	44.3	47.1	23.2	69.1	42.2	42.4	19.7
3-Aug [14]	70.1	43.6	46.1	23.4	71.2	45.3	45.6	19.6
heavy	69.9	44.0	47.1	25.9	69.1	47.5	47.8	24.6
D.2 Ablating data augmentation components.
A complete ablation study of which augmentations are useful for adversarial training on ImageNet is beyond the scope of this paper. The results of Table 1 suggest that heavy augmentations works better than the basic augmentation (random crop). In this section we test additionally the intermediate augmentation suggested in [14] called 3-aug (basic + radom horizontal flip + color-jitter), in particular we are interested in the performance of the ConvStem models and the generalization beyond the  $\ell_{\infty}$ -threat model used for training. In Table 16 we show for two models, ViT-S + ConvStem and ConvNeXt-T + ConvStem, each trained for 50 epochs the results of basic, 3-Aug [14] and heavy augmentation. Note, that the clean pre-trained model was trained with the heavy augmentations protocol. We can see that using heavy augmentation yields the best robustness across all threat models for the ConvNeXt-T+ConvStem (+2.2% in  $\ell_{\infty}$ -robustness) while having slightly worse clean accuracy (-1.1%). For the ViT-S+ConvStem the results are mixed, basic augmentation performs best, heavy augmentation is second and 3-Aug the worst of all three. However, the gap between basic and heavy augmentation is marginal and thus we stick to heavy augmentation throughout the paper. It is an interesting open question if there exist intermediate or completely different augmentation schemes which would yield improved robustness for ImageNet. A more detailed analysis for the effects of data augmentation on adversarial robustness for small resolution datasets like CIFAR-10 or SVHN can be found in [31].
D.3 Effect of weight decay and label smoothing on adversarial robustness
For our training scheme which already yields strong results (50 epochs of adversarial training with strong pre-training and strong augmentation), we test the effect of varying the weight decay and label smoothing parameters. We check on a logarithmic scale other weight decay values than our default value 0.05. In Table 17, we see that for both ViT and ConvNeXt neither smaller nor larger weight decay yields any improvement. Moreover, we observe that the label smoothing parameter has, compared to weight decay, less influence on the final robustness. For the ViT we get a small improvement of  $0.2\%$  when using label smoothing coefficient 0.2 compared to our default value of 0.1, while for the ConvNeXt the results get  $0.2\%$  worse for label smoothing coefficient 0.2. Then, we decided to keep a label smoothing coefficient of 0.1. Finally, we note that a weight decay of 0.05 and label smoothing 0.1 are the values used for clean training of the ConvNeXt in [34].
D.4 Clean and robust accuracy trade-off
Fig. 5 shows the clean vs robust accuracy trade-off for ImageNet models. In the 'small' size category, our models, i.e. ViT-S-ConvStem (black circle) and ConvNeXt-T-ConvStem (red circle), have the best clean accuracy except for XCiT-S of [14], which however has more than  $7\%$  worse robust accuracy. In 'medium' size category our models were only trained for 50 epochs (see Table 2) but outperform all SOTA models in terms of robust accuracy. On top of that, our ConvNeXt-S-ConvStem has both the best clean and robust accuracy in this size range. For 'large' sized models only the ViT-B of [44] is in a comparable range. Their model has around  $0.6\%$  higher clean accuracy than our ConvNeXt-B-ConvStem but  $2.6\%$  lower robust accuracy. In general, our models mostly show a better clean vs robust accuracy trade-off as compared to existing models in literature.
Table 17: Effect of weight decay and label smoothing coefficient. Varying the weight decay and label smoothing coefficient around our default values (marked below with a *), which are the ones suggested for clean training in [34], does not lead to a consistent improvement in  $\ell_{\infty}$  robustness.
Architecture	Weight decay	Label smoothing
0.5	0.05*	0.005	0.0	0.1*	0.2
ViT-S + ConvStem	37.2	44.0	44.0	44.0	44.0	44.2
ConvNeXt-T + ConvStem	27.5	47.5	47.2	47.4	47.5	47.3
Figure 5: AutoAttack  $\ell_{\infty}$ -Robust accuracy ( $\epsilon = 4/255$ ) vs Clean accuracy on ImageNet across different architectures. All our models are better than their SOTA counterparts in robust accuracy with only negligible difference in clean accuracy, if at all. ConvStem is written as CvSt.
D.5 Increased image resolution
We further discuss the effect of varying the test-time resolution for  $\ell_{\infty}$  robust models. All evaluations in the following are done with  $\mathrm{APGD}_{\mathrm{T - DLR}}$  (40 iteration, 3 target classes) on 1000 test points to limit computational cost, as increasing resolution significantly impacts the number of FLOPs.
Larger models. Similar to 'small' models, 'large' models trained for 50 epochs also show improvements in both clean accuracy and  $\ell_{\infty}$  robustness, see Fig. 6. This phenomenon is seen for both ViT and ConvNeXt with and without ConvStem, with peak robustness at resolution 256x256 for all classifiers.
$\ell_{2}$  radius scales with resolution. Since  $\ell_{\infty}$  bounds are applied element-wise, increasing the resolution, i.e. the input dimension, does not change how much each input component can be modified. Thus one can argue that for fixed radius, the  $\ell_{\infty}$ -threat model is for increasing resolution/dimension at least as strong as the original one. In fact, as each dimension can be manipulated independently one can argue that increasing dimension makes the  $\ell_{\infty}$ -threat model more powerful. However, with a bound  $\epsilon_{2}$  w.r.t.  $\ell_{2}$  and input dimension  $d$ , the average element-wise perturbation is  $\epsilon_{2} / \sqrt{d}$ . If one keeps the  $\epsilon_{2}$  constant for increasing dimension, it seems clear that the threat model becomes weaker.
Figure 6: (Robust) accuracy improves with increasing test-time image resolution. Evaluation of clean (left) and  $\ell_{\infty}$ -robust accuracy (right) for the 50 epoch 'large' models from Table 2 when varying the input resolution at test-time.
Figure 7:  $\ell_2$ -(robust) accuracy for varying test-time image resolution. Evaluation of clean (left) and  $l_2$ -robust accuracy (right) for the top models from Table 1 when varying the input resolution at test-time. The radius is  $\epsilon_2 = 2$  for image resolution 224 and rescaled as  $\epsilon_2 = 2\frac{d}{224}$  for varying  $d$  (this means that the perturbation budget per pixel is constant), and can be read in the figure on the right.
In order to keep the perturbation budget per dimension constant we thus increase  $\epsilon_{2}$  as a function of  $\sqrt{d}$ . Note however that when increasing  $\epsilon_{2}$  the maximal perturbation per pixel or the number of pixels which can be maximally perturbed grows as well. Thus one can argue that the threat model becomes stronger when scaling with  $\sqrt{d}$  when increasing  $d$ . In summary, we see that the comparison across image resolutions for  $\ell_{2}$  (and similarly for  $\ell_{1}$ ) is non-trivial. We decided to consider the scenario where the  $\ell_{2}$ -treat model is at least not becoming less powerful with increasing dimension and thus increase  $\epsilon_{2}$  with  $\sqrt{d}$ . The resulting  $\ell_{2}$ -robust accuracies for varying resolution can be found in Fig. 7, where we see that the ConvStem- models are very stable in robust accuracy even for much higher resolution.
D.6 Fine-tuning to other datasets
We study whether it is possible to transfer adversarial robustness from ImageNet to other datasets via fine-tuning our robust models. We consider datasets with images of resolution either similar to ImageNet, i.e. Flowers-102 [39], or smaller like CIFAR-10 and CIFAR-100 [29], and fine-tune the  $\epsilon_{\infty} = 8 / 255$  robust models from Table 4. We fine-tune with 2 steps (Flowers-102) or 10 steps (CIFAR-10, CIFAR-100) adversarial training at  $\epsilon = 8 / 255$  on all target datasets, with RandAug as augmentation, for 20 epochs, as in [14], with learning rate (LR) warm-up of 2 epochs: for Flowers-102 we set peak LR to 4e-3, a weight decay of 5e-3 and batch size 30, for CIFAR-100 peak LR is 2e-4, weight decay 0 and batch size 1024, for CIFAR-100 peak LR is 2e-4, weight decay 5e-3 and batch size 256. To adapt the networks to the new datasets, we replace the final linear classifier with one suitable for the new number of classes, and for smaller resolution inputs, we change from ConvNeXt + ConvStem the strides from 2 to 1 in the convolutional layers of the stem and in the first downsampling block in ConvNeXt. The robust accuracy is computed on the full test sets with AutoAttack in the resolution of the dataset to which we finetune, e.g.  $32 \times 32$  for CIFAR-10/100.
Table 18: Specific configurations of pre-trained models we use from timm. As all models have a lot of pre-trained weights available on timm, we explicitly list the configuration (also includes pre-trained weight path) of the specific architectures.
Architecture	Configuration
ConvNeXt-T	convnext_tiny.fb_in1k
ConvNeXt-S	convnext_small.fb_in1k
ConvNeXt-B	convnext_base.fb_in1k
ViT-S	deit_small_batch16_224
ViT-M	deit3_medium_batch16_224
ViT-B	vit_base_batch16_224.augreg_in1k
D.7 More pre-training details
In this section we give some additional details related to our models. As we heavily rely on pretraining in our work, the origin of pre-trained models on  $\text{timm}$  are listed in Table 18. Most ViTs and ConvNeXts available at  $\text{timm}$  have different versions (patch sizes, resolutions) and even pre-training (ImageNet-21k, -12k, -1k). Among those we use ViTs with patch size 16 trained at resolution of 224 on ImageNet-1k only. Similarly, for ConvNeXt we use the ones trained only with ImageNet-1k and resolution 224. For ViT-M, we use pre-trained weights of a DeiT-3 model from [51], because ViT-M was not available in our desired configuration of ImageNet-1k only pre-training. Note that ViT and DeiTs have the same architecture and differ only in the training scheme. All the clean pre-trained models use heavy augmentations protocol except ViT-M (DeiT-3-M) which uses only 3 augmentations. Finally, for Isotropic ConvNeXt-S we use the weights of convnextIso_small_1k_224_ema from the original repository.