SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[59] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476, 2021.
[60] Boxi Wu, Jinghui Chen, Deng Cai, Xiaofei He, and Quanquan Gu. Do wider neural networks really help adversarial robustness? In NeurIPS, 2021.
[61] Dongxian Wu, Shu-tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurIPS, 2020.
[62] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dólar, and Ross Girshick. Early convolutions help transformers see better. In NeurIPS, 2021.
[63] Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, and Quoc V Le. Smooth adversarial training. arXiv preprint arXiv:2006.14536, 2020.
[64] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR, 2022.
[65] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In CVPR, 2019.
[66] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
[67] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[68] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, 2020.
1. Appendix A ... Experimental details (training parameters, etc.).
2. Appendix B ... Experiments specifically related to ConvStem.
3. Appendix C ... Cost, design-choice comparisons and testing more architectures.
4. Appendix D ... Ablations on training setup and image resolution experiments.
In the following we provide the details relative to the setup used for the various experiments. First, we describe the general setup (App. A.1), then detail the variations for different settings, if they differ from the main regime.
A.1 General experimental setup
In all the experiments on ImageNet [15], unless stated otherwise, we use training parameters similar to [44]. All models are trained with AdamW [35] as optimizer with a cosine decay scheduler for the learning rate (LR), weight decay of 5e-2 and momenta terms  $\beta_{1}$  and  $\beta_{2}$  set to 0.9 and 0.95 respectively. We employ a peak LR of 1e-3 (chosen to be in the same range as the values of the original ConvNeXt [34] and DeiT [51] works), slightly higher than [44]. When training for 50 or 100 epochs the peak LR is attained at epoch 10, when using 250 or 300 epochs at epoch 20. We also use label smoothing coefficient of 0.1 and exponential moving average (EMA) with decay rate 0.999. For the inner maximization in adversarial training we use 2-step APGD [8] with an  $\ell_{\infty}$  radius of  $4 / 255$  and cross entropy as objective function.
Computational expense. All the training runs were conducted on a multi-GPU (8) setup using NVIDIA A100s with 40GB memory. Around 20,000 hours of single-GPU run-time was expended during this work.
ConvStem models clean training. All the models with ConvStem (barring the one in 'random init., basic augment.' row in Table 1) were normally (non-adversarially) trained for 100 epochs with the non-ConvStem part of the model initialized with ImageNet-1k pre-trained weights from timm. Heavy augmentations (App. A.2) were used for this pre-training. For 'medium' and 'large' category models in Table 2, we only did standard training for 50 epochs to save computational load given the network size, still achieving similar clean performance as their non-ConvStem counterparts initialized with weights from timm (see Table 10). The checkpoint with highest accuracy was then used as the starting point for adversarial training. The specific timm configuration for pre-trained weights for every model is available in Table 18.
Robustness evaluation. Unless stated otherwise, all robustness evaluations are done using complete standard AutoAttack [8, 9] (includes 4 different attacks) on the complete ImageNet validation set (5k points) from RobustBench [6]. As is the common practice, we do the best (most robust) checkpoint selection in a post-hoc fashion using 1-step APGD on a separate validation set of 4k ImageNet-1k images. Note that we use a validation set disjoint from the RobustBench ImageNet test set.
Calculation of model size. We compute the number of FLOPs and parameters using the fvcore library. In further sections, we again differentiate models as 'small' (< 30M parameters), 'medium' ( $\approx$  50M params) and 'large' ( $\geq$  80M params) depending on their parametric size.
A.2 Pre-training, augmentations and longer training
For the first three rows in 'Training Scheme' column in Table 1 we do 50 epoch adversarial training. For the specific cases of the 'Training Scheme' column, we have the following settings (the list is incremental):
- random init., basic augment. It has random initialized weights with RandomCrop as augmentation.
$^{2}$ https://github.com/facebookresearch/fvcore/tree/main/fvcore
- + strong clean pre-training. Adversarial training is initialized with ImageNet-1k pre-trained weights from timm with RandomCrop as augmentation.
- + heavy augmentations. It uses the augmentations similar to [34], including CutMix [65], MixUp [67] with strength 0.8, RandAugment [12] with two layers, magnitude 9 and a random probability of 0.5. Label smoothing is set to 0.1, Exponential Moving Average (EMA) with factor 0.999 and random erasing with probability 0.25. Unlike [34], we do not use any stochastic depth.
-  $50 \rightarrow 300$  epochs. We adversarially train for 300 epochs instead of 50 and peak LR is achieved at epoch 20.
The two Isotropic ConvNeXt models in Table 1 use the pre-trained weights from the original repository<sup>3</sup> as these are not available on timm.
A.3 Scaling-up and comparison to SOTA models
For the results in Sec. 5, all our runs were initialized with ImageNet-1k pre-trained weights and use the heavy augmentations protocol from the previous section. Adversarial training was done with 2-step APGD for either for 50, 250 or 300 epochs (as listed in the 'Epochs' column in Table 2). For all 'ours' models in Table 2 in 'small' category the effective batch size was 1392, for 'medium' it was 1024. In 'large' category, all ViT-B and ConvNeXt-B models have an effective batch size of 864 and 756 respectively. Unlike for 'small' models, in 'medium' and 'large' category the checkpoint selection is done using 10-step  $\mathrm{APGD_{CE}}$  attack (first attack in AutoAttack), since 1-step  $\mathrm{APGD_{CE}}$  returns several checkpoints in a very small numerical range of robustness values.
A.4 Increasing test-time resolution
When using image resolution other than 224x224, we keep the same pre-processing, only resolution is changed. For final resolution NS (new size) and scale-ratio (SR) 0.875 (same as during training), the images are resized with bicubic interpolation to NS/SR and then a center crop to NS is done. Note that we do not fine-tune the models with higher resolution images.
A.5 Fine-tuning to larger radius
Fine-tuning to larger radii (Sec. 6) uses our ViT-S + ConvStem, ConvNeXt-T + ConvStem and 50 epochs ConvNeXt-B + ConvStem models from Table 2 as initialization. We employ the standard setting with heavy augmentations and adversarially train for 25 epochs at the  $\ell_{\infty}$  radius of 8/255 with a peak LR of 1e-4 attained at epoch 5.
B Design Choices for ConvStem
In this section, we give a detailed account of how ConvStem can be designed for different architectures and how it affects model size (FLOPs, parameters) and robustness. All the ConvStem configurations discussed below had the non-ConvStem part of the model initialized with pre-trained ImageNet-1k weights and the ConvStem with random initialization. This setup was chosen to manage the computation load. Afterwards, for the best configurations selected, we clean trained the model with ConvStem before training them adversarially.
B.1 A closer look at the convolutional stem
We here investigate which elements of the convolutional stem are most relevant for better robustness of ConvNeXt-T to unseen threat models. Standard ConvNeXt has a 4x4 convolution in the initial stem (see Fig. 2, right column) that downloads the input resolution by 4 with non-overlapping patches. This means any possible ConvStem design is limited to a maximum of two downsampling operations. Further, to constrain the increase in parameters and FLOPs we limit our choices to the configurations in Table 7, where we also report the corresponding results. We start off with the standard PatchStem from [34] ( $\mathrm{Conv}^{4,4}$ , L), with L indicating LayerNorm and G a GELU layer. In ( $\mathrm{Conv}^{7,4}$ , L, G), we increase the kernel size from PatchStem which results in mixing of feature maps,
<sup>3</sup>https://github.com/facebookresearch/ConvNeXt
Table 7: ConvStem design for ConvNeXt. We progressively modify the stem of ConvNeXt and track its effect on the robustness in seen and unseen threat models. The stem design is represented by the sequence of convolutional layers (Conv $^{x,y}$ , i.e. kernel size x and stride y), LayerNorm (L), and GELU (G).
Stem design	FLOPs (G)	Adversarial Tr. w.r.t. ℓ∞
clean	ℓ∞	ℓ2	ℓ1
Conv4,4, L [34]	4.49	71.0	46.5	38.1	14.9
Conv7,4, L, G	4.50	69.8	46.6	38.4	17.6
Conv7,4, L, G, Conv3,1, L, G	4.63	67.5	43.6	37.9	15.7
Conv7,2, L, G, Conv3,2, L, G	4.66	69.6	47.6	38.8	18.7
Conv5,2, L, G, Conv3,2, L, G	4.62	70.2	47.4	44.2	21.2
Conv3,2, L, G, Conv3,2, L, G	4.59	69.5	46.2	47.8	27.7
but this change does not yield any dividends in any of the metrics. Now we add another layer in our convolution stem. From  $(\mathrm{Conv}^{7,2},\mathrm{L,G - Conv}^{3,2},\mathrm{L,G})$  we keep the kernel size in the first layer 7 and vary the stride in second layer depending on the amount of downsampling required to match the standard ConvNeXt. For all these settings we do not see any improvement in  $\ell_2$  and  $\ell_{1}$  robustness but there are some gains in  $\ell_{\infty}$  robustness. This increase in  $\ell_{\infty}$  can be attributed to an increase in FLOPs. However, when trained with a good clean trained models as initialization, all these setting have a very similar  $\ell_{\infty}$  robustness. As next step, we reduce the kernel size in the first layer to 5 in configuration  $(\mathrm{Conv}^{5,2},\mathrm{L,G - Conv}^{3,2},\mathrm{L,G})$ , which leads to gains across all metrics, and reducing the kernel size to 3 yields further improvements in unseen threat models  $(\ell_{2}$  and  $\ell_{1})$ . Since these last two configurations look very promising, we trained clean models with such architectures to a similar level of clean performance and used them as initialization for adversarial training: we observed negligible difference across all metrics. Given  $(\mathrm{Conv}^{3,2},\mathrm{L,G - Conv}^{3,2},\mathrm{L,G})$  has lower FLOPs of the two we select this configuration as main design for ConvStem. From Table 14, this configuration only leads to  $2.9\%$  and  $0.1\%$  increase in FLOPs and parameters respectively.
Additionally, for ConvNeXt-B we use the same stem as ConvNeXt-T and it slightly improved  $\ell_2$  and  $\ell_1$  robustness without improving  $\ell_{\infty}$  robustness. We also found that adding an extra layer to this stem improved the  $\ell_2$  and  $\ell_1$  numbers (see Table 8) with a negligible increase in FLOPs and parameters. As this is a large model, an ablation similar to the one for ConvNeXt-T was not feasible. In the end, we use  $C_2$  (from Table 8) as ConvStem for our ConvNeXt-B models. From Table 14, this ConvStem increases the FLOPs by around  $4\%$  and the parameters by only  $0.2\%$ .
B.2 ViT ConvStem
We here study various designs of ConvStem on ViT-S (to keep the computational load low). We recall that for ViT the standard PatchStem generates 16x16 patches, which means we have a higher degree of freedom as to how the ConvStem can be designed compared to ConvNeXt (while keeping the increase in FLOPs from the standard model minimal).
In Table 9, we start off with the ConvStem as proposed by [62]: this configuration  $(\mathbf{C}_1)$  results in a  $8.1\%$  increase in FLOPs but the improvement in all metrics tested is quite significant. Now we limit the increase in FLOPs to this value and try several other configurations within this limit. Two
Table 8: ConvStem configurations for ConvNeXt-B. Each layer in configuration column is followed by LayerNorm and GELU activation. * indicates layers with stride 2 (one downsampling operation), and N (= 64) is the number of output channels of the convolution layer. The increase/decrease in numbers from the PatchStem are shown in green/red respectively.
Stem design	Adversarial Tr. w.r.t. ℓ∞
clean	ℓ∞	ℓ2	ℓ1
P1: PatchStem	75.6	54.3	48.5	23.7
C1: N*-2N*	75.2 -0.4	54.2 -0.1	48.9 +0.4	24.0 +0.3
C2: N*-1.5N*-2N	75.3 -0.4	54.4 +0.1	50.1 +1.6	24.7 +1.0
Table 9: ViT-S ConvStem ablation. N, 2N, 4N, 8N are the number of output channels in each convolution layer, where N=48, similar to that of [62]. Each layer is followed by LayerNorm and GELU activation. *: kernel-size 3 and stride 2 (one donwsampling in the layer), **: kernel-size 7 and stride 4 (double donwsampling in the layer).
Stem design	FLOPs (G)	Adversarial Training
clean	\( \ell_{\infty} \)	\( \ell_2 \)	\( \ell_1 \)
\( P_1 \): PatchStem	4.61	65.1	39.2	37.4	16.5
\( C_1 \): N*-2N*-4N*-8N*	4.99	68.0	43.3	46.3	27.0
\( C_2 \): N*-N*-N*-8N*	4.71	64.8	41.1	45.3	28.3
\( C_3 \): N**-8N**	4.78	64.5	40.3	46.2	32.3
\( C_4 \): N**-2N*-4N*-8N	4.84	64.7	40.4	46.4	32.6
\( C_5 \): N**-2N**-4N-8N	4.81	64.6	39.3	45.8	33.2
ways to change the configuration are most trivially feasible here: 1) reducing the number of output channels in each convolution layer and 2) changing the level of downsampling done in each layer. As the combination of these two changes most readily leads to the change in FLOPs, we only apply these two kinds of changes in our ablation study. Starting from  $C_1$ , we first reduce the number of output channels in each layer, since the input to the transformer block is 8N (the last layer's output channels cannot be changed). Both  $C_2$  and  $C_3$  lead to significant reduction in  $\ell_{\infty}$  numbers, hence we shift towards changing the amount of downsampling in each layer. Moving more downsampling to earlier layers further reduces  $\ell_{\infty}$  robustness compared to  $C_1$ . This small ablation leads us to fix the ConvStem for ViT-S to configuration  $C_1$ . From Table 14, the increase in FLOPs is only 8.1% and parameters increase by 3.3%.
We use the same ConvStem as ViT-S with final layers output channels scaled to  $16N$  to meet the input channel for transformer block in ViT-B. This only increases the FLOPs by  $2\%$  and parameters by a mere  $0.6\%$ . Similarly for our ViT-M, we again use the same ConvStem as ViT-S with last layer's output size set to 512.
Improvements in robustness with ConvStem. Our experiments throughout the paper, validate that a minimal change in the network architectures, adding the ConvStem provides consistent improvements in robustness of the adversarially trained models. Fig. 4 summarises this and shows how across network types the classifiers with ConvStem achieve better robustness in both seen  $(\ell_{\infty})$  and unseen  $(\ell_{2})$  threat models.
Figure 4: ConvStem yields consistent improvements across architectures, model sizes in seen and unseen robust accuracy. We show for 50 epochs of AT with pre-training and heavy augmentation, that adding ConvStem yields for ViT and ConvNeXt robustness improvements both in seen  $\ell_{\infty}$  (left subplot) and the unseen  $\ell_{2}$  (right subplot) threat models.
Table 10: Clean accuracy of standard trained ConvStem models. For all the main models in this work, we show their clean performance on RobustBench 5k validation set. The difference to non-ConvStem models with timm initialization is shown in red.
Architecture	Epochs	Clean acc.
ConvNeXt-T + ConvStem	100	80.9	-0.5
Isotropic ConvNeXt + ConvStem	100	79.2	-0.5
ViT-S + ConvStem	100	79.3	-1.0
ViT-M + ConvStem	50	81.8	-0.6
ConvNeXt-S + ConvStem	50	81.6	-1.0
ViT-B + ConvStem	50	81.6	-1.1
ConvNeXt-B + ConvStem	50	82.6	-0.8
B.3 Clean performance of ConvStem models
Our ConvStem models from Sec. 4 onward always use a standard classifier as initialization for adversarial training. This clean training was done either for 50 or 100 epochs depending on the size of the model, see Table 10. The non-ConvStem part of the model (ConvNeXt stages and all transformer blocks) were initialized with timm ImageNet-1k pre-trained weights and the ConvStem was randomly initialized. In Table 10, we show that for all models we were able to reach within  $1\%$  of clean performance of standard models on the RobustBench validation set. Certainly, initialization of non-ConvStem part helps in this fast and efficient learning. The good clean accuracy via short training also corroborates the improved training hypothesis of [62] when using ConvStem instead of a PatchStem without much hyperparameter tuning.
Table 11: Comparison to [14] with respect to training time. Both our ConvNeXt and the ConvNeXt + ConvStem attain higher robustness in the  $\ell_{\infty}$  threat model than the ConvNeXt of [14] while using less forward/backward passes per image (see discussion at the beginning of Sec. 5).
Model	Epochs	Attack Steps	Passes/Image	\( \ell_{\infty} \)
ConvNeXt-T [14]	110	1	220	44.4
ConvNeXt-T	50	2	150	46.5
ConvNeXt-T + ConvStem	50	1	100	46.4
ConvNeXt-T + ConvStem	50	2	150	47.5
ConvNeXt-T + ConvStem	300	2	900	49.5
ConvNeXt-T + ConvStem	300	3	1200	50.2
C Design choices, cost/size comparisons and new architectures
In this section, we answer (i) How the total training cost of our training regime compares to existing works (ii) What is the effect of using different optimizer for adversarial training (iii) Do the improvements as seen by adding ConvStem hold for the most recent architectures (iv) How does the model size change on adding ConvStem.
C.1 Training cost comparison
Table 11 shows the robustness in the seen  $\ell_{\infty}$  threat model of ConvNeXt-T model trained with strong clean pre-trained initialization and heavy augmentations when varying training cost. In particular, the training scheme cost is approximated by the number of passes of the network for each training image (counting the number of forward+backward passes per image combines the number of attack steps with training epochs used). First, we show that our ConvNeXt-T with 150 passes/images attains  $2.1\%$  higher robustness ( $46.1\%$  vs  $44.4\%$ ) than the same model trained with 220 passes/image by [14]. Moreover, adding the ConvStem achieves a similar result with only 100 passes/image (third row), and  $47.5\%$  robust accuracy with 150 passes/image (fourth row). Finally, increasing the number of epochs and training steps, which incurs in more expensive training, can further improve robustness.
Table 12: PGD vs APGD. Comparing the difference in robustness on using PGD instead of APGD (used in this work) trained for 2 steps each on a ConvNeXt-T. We evaluate clean performance and robustness.
Optimizer	clean	l∞	l2	l1
PGD	72.8	45.4	39.1	16.5
APGD	71.0	46.5	38.1	14.9
Table 13: Effect of replacing ConvStem with PatchStem in InternImage. We train the InternImage (original with ConvStem) and InternImage with PatchStem (PS) for 50 epochs in our heavy augmentation setup.
Architecture	clean	l∞	l2	l1
InternImage-T	72.9	47.0	48.5	25.9
InternImage-T + PS	72.7	47.0	47.2	24.5
C.2 APGD vs. PGD training
The optimizer for adversarial training is also a design choice to be considered while training robust models. Prior works [14, 46] mostly use either PGD [36] or APGD [8]. We use APGD since it has been shown to generally outperform PGD and does not need parameter tuning when changing radius or attack steps [8]. Additionally, in Table 12 we train a ConvNeXt-T with 2-step PGD (clean pre-training, heavy augmentations, 50 epochs), comparable to the classifier using 2-step APGD from Tables 1 and 2. In this regime APGD yields higher robustness in the seen  $(\ell_{\infty})$  threat model which is our main goal, although with a small loss in clean accuracy.
C.3 Testing more architectures
We further test the effect of different stem designs on InternImage [57], a modern architecture that uses deformable convolutions and has a ConvStem as its stem. In Table 13, we replace the ConvStem with a PatchStem and evaluate robustness for the seen  $(\ell_{\infty})$  and unseen  $(\ell_2,\ell_1)$  threat models. The structure of the PatchStem (PS) is the same as for ConvNeXt. For the PS model, we initialize the PS randomly and do clean pre-training for 50 epochs to reach  $82.8\%$  clean accuracy (the original InternImage with ConvStem has  $83.5\%$ ). Similar to ViT and ConvNeXt, we see that robustness for the unseen threat models goes down when replacing the ConvStem with the PatchStem. The models were adversarially (2-step APGD) trained for 50 epochs with ImageNet-1k clean trained model<sup>4</sup> as initialization and with the heavy augmentation regime, while other parameter settings were kept same as heavy augmentations in App. A.2.
Table 14: FLOP and parameter count. Here we list the FLOPS (Giga), Parameters (Millions) of all the main models and their ConvStem versions. The percentage increase in values on adding ConvStem are shown in red.
Architecture	FLOPs (G)	Params (M)
ConvNext-T	4.47	28.59
+ ConvStem	4.60	+2.9%	28.63	+0.1%
Isotropic ConvNeXt-S	4.29	22.31
+ ConvStem	4.67	+8.8%	23.04	+3.3%
ViT-S-16	4.61	22.05
+ ConvStem	4.99	+8.1%	22.78	+3.3%
ViT-M-16	8.01	38.85
+ ConvStem	8.38	+4.7%	39.5	+1.7%
ConvNeXt-S	8.70	50.10
+ ConvStem	8.79	+1.7%	50.33	+0.5%
ViT-B-16	17.58	86.57
+ ConvStem	17.93	+2.0%	87.14	+0.6%
ConvNeXt-B	15.38	88.59
+ ConvStem	15.97	+3.8%	88.75	+0.2%
4https://github.com/OpenGVLab/InternalImage
Table 15: Effect of using 21k-pre-trained models. Across the two best models discovered we see that there is no improvement in the main seen threat on initializing with ImageNet-21k pretrained weights from timm. The increase/decrease from ImageNet-1k counterparts of the same models are shown in green/red colors respectively.
Architecture	Epochs	clean	l∞
ConvNeXt-T	50	71.6	+0.6	46.7	+0.2
ViT-B-16	50	73.5	+0.2	49.6	-0.4
ConvNeXt-T	300	72.2	+0.2	48.3	-0.3
C.4 Increase in paramaters/FLOPs
In Table 14, we list the number of parameters and FLOPs of all models, with the percentage increase on adding the ConvStem shown in red: for all models adding ConvStem results in a marginal amount of extra parameters. For FLOPs, smaller ViTs have a bigger increase as compared to ConvNeXt. This can be attributed to the bigger size of ConvStem we use for ViT as we need to downsample before the first transformer block more severely than for ConvNeXt.
D More on Robust ImageNet Models
As the main aim of our work is to study and improve robustness of models trained for ImageNet to seen and unseen threat models, in this section we consider ablations (effect of clean pre-trainingig, data augmentations, weight-decay and label smoothing) related to this. Moreover, we provide more details about using robust ImageNet for fine-tuning to other datasets.
D.1 Using even better pre-training
The most robust models in our work use ImageNet-1k pre-trained weights of the respective architectures from timm. A natural hypothesis is that a better pre-trained model, namely an ImageNet-21k pre-trained model fine-tuned on ImageNet-1k, might lead to higher robustness. timm makes available for architectures considered in this work (ViT, ConvNeXt) 21k-pretrained, 1k-fine-tuned weights as well. In Table 15, we see the effect of using 21k-pretrained weights as initialization for adversarial training. The increase and decrease in clean accuracy and  $\ell_{\infty}$  robust accuracy as compared to the 1k-pre-trained initialization of the same models is shown in green and red respectively.