SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Longer training. Adding heavy data augmentation increases the complexity of the learning problem, which requires longer training, in particular for isotropic architectures. Thus we extend the training time from 50 to 300 epochs. As expected we see the strongest improvements for the isotropic architectures but even for ConvNeXt one observes consistent gains: this is in contrast to the claim of [38] that longer training does not help to improve robustness. The ConvNeXt-T + ConvStem, with  $49.5\%$  robust accuracy, outperforms all classifiers reported in [14] and [42], including those with up to 4 times more parameters.
5 Comparison with SOTA Models
In Table 2 we compare our classifiers with models from prior works trained for adversarial robustness w.r.t.  $\ell_{\infty}$  at  $\epsilon_{\infty} = 4 / 255$  on ImageNet. For each model we report number of parameters and FLOPs,
Table 2: Comparison to SOTA  $\ell_{\infty}$ -robust models on ImageNet. For each model we report the number of parameters, FLOPs, number of epochs of adversarial training (AT), the number of PGD steps in AT, clean and  $\ell_{\infty}, \ell_{2}, \ell_{1}$ -robust accuracy with  $\epsilon_{\infty} = 4/255, \epsilon_{2} = 2, \epsilon_{1} = 75$  (AutoAttack). Models of similar size are sorted in increasing  $\ell_{\infty}$ -robustness. Our ConvNeXt + ConvStem improves by  $5.8\%$  for small,  $7.2\%$  for medium, and  $2.8\%$  for large models over SOTA.
Architecture	Params (M)	FLOPs (G)	Source	Ep.	Adv. Steps	Adversarial	Tr. wrt ℓ∞
clean	ℓ∞	ℓ2	ℓ1
ResNet-50	25.0	4.1	[46]	100	3	65.88	33.18	18.88	3.82
ResNet-50	25.0	4.1	[2]	100	1	67.44	35.54	18.16	3.90
ViT-S	22.1	4.6	[2]	100	1	66.62	36.56	41.40	21.82
ViT-S	22.1	4.6	[14]	110	1	66.78	37.88	-	-
ViT-S	22.1	4.6	[37]	90	3	65.9	39.24	32.18	10.54
XCiT-S12	26.0	4.8	[14]	110	1	72.34	41.78	46.20	22.72
ViT-S	22.1	4.6	ours	300	2	69.22	44.04	37.52	15.12
RobArch-S	26.1	6.3	[42]	110	3	70.58	44.12	39.88	15.46
Isotropic-CN-S	22.3	4.3	ours	300	2	69.04	44.22	36.64	14.88
ConvNeXt-T	28.6	4.5	[14]	110	1	71.60	44.40	45.32	21.76
Isotropic-CN-S + ConvStem	23.0	4.7	ours	300	2	70.02	45.90	49.24	27.84
ViT-S + ConvStem	22.8	5.0	ours	300	2	72.56	48.08	50.40	26.68
ConvNeXt-T	28.6	4.5	ours	300	2	72.40	48.60	38.02	14.88
ConvNeXt-T + ConvStem	28.6	4.6	ours	300	2	72.72	49.46	48.42	24.52
ConvNeXt-T + ConvStem	28.6	4.6	ours	300	3	72.70	50.16	49.00	24.16
Wide-ResNet-50-2	68.9	11.4	[46]	100	3	68.82	38.12	22.08	4.48
ResNet-101	44.5	7.9	[37]	90	3	69.52	41.02	25.62	6.56
XCiT-M12	46.0	8.5	[14]	110	1	74.04	45.24	48.18	22.72
ViT-M	38.8	8.0	ours	50	2	71.72	47.24	49.02	29.20
ViT-M + ConvStem	39.5	8.4	ours	50	2	72.40	48.80	50.56	28.12
ConvNeXt-S	50.1	8.7	ours	50	2	74.10	52.32	43.84	19.52
ConvNeXt-S + ConvStem	50.3	8.8	ours	50	2	74.10	52.42	50.88	25.64
ViT-B	86.6	17.6	[37]	90	3	70.42	43.02	47.26	27.08
XCiT-L12	104.0	19.0	[14]	110	1	73.76	47.60	49.38	23.74
Swin-B	87.7	15.5	[37]	90	3	74.76	48.10	44.42	18.04
RobArch-L	104.0	25.7	[42]	100	3	73.46	48.92	39.48	14.74
ViT-B	86.6	17.6	ours	50	2	73.32	50.02	52.14	33.12
ViT-B + ConvStem	87.1	17.9	ours	50	2	74.38	52.58	54.38	31.20
ViT-B	86.6	17.6	[44]	300	2	76.62	53.50	-	-
ConvNeXt-B	88.6	15.4	ours	50	2	75.62	54.34	48.52	23.70
ConvNeXt-B + ConvStem	88.8	16.0	ours	50	2	75.32	54.38	50.06	24.76
ViT-B + ConvStem	87.1	17.9	ours	250	2	76.30	54.66	56.30	32.06
ConvNeXt-B	88.6	15.4	[32]*	300	3	76.02	55.82	44.68	21.23
ConvNeXt-B + ConvStem	88.8	16.0	ours	250	2	75.90	56.14	49.12	23.34
Swin-B	87.7	15.5	[32]*	300	3	76.16	56.16	47.86	23.91
ConvNeXt-B + ConvStem	88.8	16.0	ours	250	3	75.18	56.28	49.40	23.60
* concurrent work
the number of training epochs, the number of steps in the inner maximization, and clean and robust accuracy (AutoAttack on RobustBench validation set) in the three threat models. We group the models according to size for fair comparison, distinguishing small (< 30M parameters), medium ( $\approx 50\mathrm{M}$  params) and large models ( $\geq 80\mathrm{M}$  params).
1-Step vs 2-Step Adversarial Training. Current works use one to three steps of PGD to solve the inner problem, see column "Adv. Steps" in Table 2. It is an important question if our gains, e.g. compared to works like [14] who use a single step, are just a result of the stronger attack at training time. We test this by training a ConvNeXt-T + ConvStem model with strong pre-training and heavy augmentation with 1-step APGD adversarial training for 50 epochs. We get a  $\ell_{\infty}$ -robust accuracy
of  $46.4\%$  (clean acc. is  $71.0\%$ ) compared to  $47.5\%$  with 2-step APGD (see Table 1) which is just a difference of  $1.1\%$ . Thus our gains are only marginally influenced by stronger adversarial training. Moreover, our ConvNeXt + ConvStem models with 50 epochs of adversarial training outperform all existing SOTA models even though the computational cost of training them is lower (50 epochs of 2-step AT corresponds roughly to 75 epochs of 1-step AT and 37.5 epochs of 3-step AT). For our best small and large model we also check the other direction and train 3-step AT models, which improve by  $0.7\%$  for ConvNeXt-T + ConvStem and  $0.14\%$  for ConvNeXt-B + ConvStem in  $\ell_{\infty}$ -robustness. This suggests that 2-step AT is a good compromise between achieved robustness and training time. In App. C.1 we additionally show how our training scheme yields more robust models than that of [14], even with less than half the effective training cost.
Scaling to larger models. We apply our training scheme introduced in the previous sections to architectures with significantly more parameters and FLOPs. We focus on ConvNeXt-S/B and ViT-M/B: in particular, for ConvNeXt-B we adapt the convolutional stem used for ConvNeXt-T by adding an extra convolutional layer, while for the ViTs we just adjust the number of output channels to match the feature dimension of the transformer blocks (more details in App. B). We train the medium models for 50 epochs and the large models for 50 and 250 epochs (with ConvStem only).
Comparison with SOTA methods. Among small models, all three architectures equipped with ConvStem outperform in terms of  $\ell_{\infty}$ -robustness all competitors of similar size with up to  $5.8\%$  improvement upon the ConvNeXt-T of [14]. Moreover, our ConvNeXt-T + ConvStem is with  $50.2\%$  robust accuracy  $1.2\%$  better than the RobArch-L [42] (the previously second most robust model) which has  $3.6\times$  more parameters and FLOPs. The ConvStem positively impacts generalization of robustness to unseen attacks of all three architectures, with improvements between  $2.8\%$  and  $4.2\%$  for  $\ell_2$  and  $1.4\%$  and  $5.2\%$  for  $\ell_1$  (with respect to the XCiT-S12). For medium models the picture is the same: our ViT-M+ConvStem and the ConvNeXt-S+ConvStem outperform both in seen and unseen threat models their non-ConvStem counterparts and the best existing model (XCiT-M12 of [14]).
For large models we compare to the previously most robust model, the ViT-B of [44], with  $53.5\%$ $\ell_{\infty}$ -robust accuracy (the model is not available so we cannot evaluate  $\ell_{1}$  and  $\ell_{2}$ -robustness). It is remarkable that our ConvNeXt-B+ConvStem with only 50 epochs of training outperforms this ViT-B trained for 300 epochs already by  $0.9\%$  and with 250 epochs training even by  $2.5\%$ . An interesting observation is that our ViT-B+ConvStem with 250 epochs training is with  $54.7\%$  just  $1.3\%$  worse than our ConvNeXt-B+ConvStem but has significantly better generalization to unseen attacks:  $56.3\%$  vs.  $49.1\%$  for  $\ell_{2}$  and  $32.1\%$  vs  $23.3\%$  for  $\ell_{1}$ . Regarding the concurrent work of [32], we note that our ConvNeXt-B+ConvStem trained for 250 epochs with 2-step AT is already better than their ConvNeXt-B 3-step AT model trained for 300 epochs ( $\approx 40\%$  longer training time), while having significantly better generalization to the unseen  $\ell_{1}$  and  $\ell_{2}$ -attacks. Our 3-step AT ConvNeXt-B+ConvStem outperforms their best model, a Swin-B, even though it is trained for  $20\%$  less epochs, and has better generalization to  $\ell_{2}$ . Finally, we note that our ViT-B+ConvStem significantly outperforms their Swin-B in terms of robustness to unseen attacks.
Additional experiments. InternImage [57], a recent non-isotropic model, already uses a ConvStem. Replacing it with a PatchStem preserves the robustness in  $\ell_{\infty}$  (seen threat model) but decreases the robustness to unseen  $\ell_{1}$ - and  $\ell_{2}$ -attacks (App. C.3), further validating the effectiveness of ConvStem.
Table 3: Optimal test time image resolution. For our small and large models trained at  $224 \times 224$  with 300 or 250 epochs AT, we provide the optimal test time resolution and the resulting improvements (green) in clean and  $\ell_{\infty}$ -robust accuracy ( $\epsilon = 4/255$ ) compared to the training resolution.
Architecture	Res.	clean	\( \ell_{\infty} \)
ConvNeXt-T	256	72.4	+0.0	49.4	+0.8
ConvNeXt-T + CvSt	288	74.8	+1.9	50.8	+1.3
Iso-CN-S	288	71.2	+2.2	46.4	+2.2
Iso-CN-S + CvSt	288	72.3	+2.1	48.1	+2.2
ViT-S	256	69.9	+0.7	44.2	+0.2
ViT-S + CvSt	288	73.7	+1.2	49.3	+1.2
ViT-B + CvSt	256	76.6	+0.3	55.8	+1.1
ConvNeXt-B + CvSt	256	76.9	+1.0	57.3	+1.2
Table 4: ImageNet:  $\ell_{\infty}$ -robust accuracy at  $\epsilon_{\infty} = 8/255$ . While models from the literature are trained from scratch for  $\epsilon_{\infty} = 8/255$ , we fine-tune our 4/255-robust models for only 25 epochs to the larger radius.
Architecture	Source	clean	\( \ell_{\infty} \)
ResNet-50	[46]	54.9	19.7
XCiT-S	[14]	63.5	25.0
ViT-S + CvSt	ours	67.5	28.2
XCiT-L	[14]	69.2	28.7
CN-T + CvSt	ours	68.6	29.6
CN-B + CvSt	ours	71.7	33.2
Figure 3: (Robust) accuracy improves with increasing test-time image resolution. Evaluation of clean (left) and  $l_{\infty}$ -robust accuracy (right, evaluation using  $\mathrm{APGD}_{\mathrm{T - DLR}}$ ) at  $\epsilon_{\infty} = 4 / 255$  for the top models from Table 1 for varying test input resolution.
6 Influence of Resolution and Fine-Tuning
In the following we analyze how robustness behaves under variation of test time resolution, and robust fine-tuning to either a larger radius on ImageNet or other datasets.
Improving robustness via increasing test-time image resolution. Unlike small datasets like CIFAR-10, for ImageNet the input images are resized before inference. Using higher resolution at test time can lead to improved clean accuracy for standard ImageNet models [52, 17]. On the other hand increasing dimension for fixed radius makes the  $\ell_{\infty}$ -threat model more powerful. To study which effect prevails, we test clean and  $\ell_{\infty}$ -robust accuracy for each of our 2-step AT models from 'small' (300 epochs) and 'large' (250 epochs) in Table 2 when varying test time input resolution between 192 and 384. Note that all models have been trained on  $224 \times 224$  images only, and no fine-tuning to different resolutions is done. Also, for ViTs we have to adapt the models via interpolation of the positional embedding as suggested by [4]. Fig. 3 shows that all classifiers obtain their highest robustness for image resolutions of 256 or 288, higher than the training one. For faster evaluation we use for this plot  $\mathrm{APGD}_{\mathrm{T - DLR}}$  with 40 iterations and 3 target classes. Then, we select for each model the best input resolution, and evaluate  $\ell_{\infty}$ -robustness with AutoAttack in Table 3. Increasing input resolution significantly improves clean and  $\ell_{\infty}$ -robust accuracy. Our ConvNeXt-B + ConvStem attains SOTA  $76.9\%$  and  $57.3\%$  clean and  $\ell_{\infty}$ -robust accuracy, respectively, at resolution 256. Additional results on the role of the input resolution can be found in App. D.5, including a discussion of what it means for threat models other than  $\ell_{\infty}$ .
Varied perturbation strength. In Table 5, we test the effect of varying the test time resolution on the robustness in  $\ell_{\infty}$  for radii other than  $4/255$  used for training (we use our ConvNeXt-B + ConvStem, with robustness evaluation via AutoAttack on the RobustBench ImageNet set). For all radii  $\epsilon_{\infty}$ , the best resolution is either 256 or 288, that is larger than the training resolution of 224, which supports
Table 5: Increased resolution across perturbation strengths. For ConvNeXt-B+ConvStem, we see that both best clean and  $\ell_{\infty}$  robust accuracies are attained a higher resolution than the one trained for (224) across all values of  $\epsilon_{\infty}$ . The difference from base value at the resolution of 224 is shown in color and the best result for each perturbation strength (row) is highlighted.
ε∞	Input resolution
192	224*	256	288	320
clean	74.1	-1.8	75.9	76.9	+1.0	77.7	+1.8	77.2	+1.3
2/255	64.6	-2.3	66.9	67.9	+1.0	68.6	+1.7	68.4	+1.5
4/255	53.0	-3.2	56.1	57.3	+1.2	57.2	+1.1	56.6	+0.5
6/255	41.0	-2.8	43.8	44.4	+0.6	44.5	+0.7	43.0	-0.8
8/255	29.5	-0.9	30.4	31.0	+0.6	29.8	-0.6	27.9	-2.5
Table 6:  $\ell_{\infty}$ -Robust accuracy at radius  $\epsilon_{\infty} = 8/255$ . We examine fine-tuning our ConvNeXt + ConvStem models robust to  $\epsilon_{\infty} = 8/255$  from Table 4 to other datasets. We compare to the XCiT models of [14] when available.
Architecture	Flowers-102	CIFAR-100	CIFAR-10
clean	\( \ell_{\infty} \)	clean	\( \ell_{\infty} \)	clean	\( \ell_{\infty} \)
XCiT-S	82.86	47.91	67.34	32.19	90.06	56.14
CN-T + CvSt	80.42	50.89	72.32	32.30	92.72	55.97
XCiT-L	-	-	70.76	35.08	91.73	57.58
CN-B + CvSt	83.98	54.77	74.83	34.24	93.55	58.17
the results above. Unlike for the smaller radii, at  $\epsilon_{\infty} = 8 / 255$  the robust accuracy at both resolutions 288 and 320 is worse than at 224. This suggests that there are diminishing returns for higher radii. The attack strength increases more quickly with the perturbation budget at higher resolution than the improvements in clean accuracy. While for the  $\ell_{\infty}$ -threat model the comparison across different image resolutions can be done directly, the comparison for the  $\ell_{2}$ -threat model is more subtle and we discuss potential ways to do this in App. D.5. If one keeps the perturbation budget per pixel constant, that means increasing  $\epsilon_{2}$  linearly with the image resolution, then the gains in  $\ell_{2}$ -robustness are marginal.
Fine-tuning to larger radius. While  $\epsilon = 4 / 255$  is the standard radius for the  $\ell_{\infty}$ -threat model on ImageNet, even larger perturbations might still be imperceptible. [14] train XCiTs with  $\epsilon = 8 / 255$  from scratch for 110 epochs. However, given the success of using a warm start for adversarial training (see Sec. 4.1), we propose to fine-tune the model fully trained for robustness at  $\epsilon = 4 / 255$  with adversarial training with the larger radius. Such classifiers already have non-trivial robustness at  $\epsilon = 8 / 255$ , and we expect they require only relatively small changes to adapt to the larger radius. We use our models from Table 2 to fine-tune for 25 epochs (details in App. A.5), and report the results in Table 4: our ConvNeXt-B + ConvStem improves significantly over the previous best result both in terms of clean and robust accuracy, even though we just use relatively short fine-tuning.
Fine-tuning to other datasets. We fine-tune our models trained to be robust at radius  $8/255$  from Table 4 to other datasets and compare to the results of [14] in Table 6. For Flowers-102 (resolution  $224 \times 224$ ) the improvement over XCiT in robust accuracy is similar to the difference of the models in Table 4 and shows that the benefits of more robust models transfer to smaller datasets. For CIFAR (resolution  $32 \times 32$ ) our models tend to achieve similar robustness as the XCiT models but at a much better clean accuracy. We expect that here even stronger robustness gains are possible by further optimizing the fine-tuning. For details on the fine-tuning setup see App. D.6.
We have examined two extreme representatives of today's modern architectures for image classification, ViT and ConvNeXt, regarding adversarial robustness. We have shown that ConvNeXt + ConvStem yields SOTA  $\ell_{\infty}$ -robust accuracy as well as good generalization to unseen threat models such as  $\ell_1$  and  $\ell_2$  across a range of parameters and FLOPs. Given that for smaller model sizes the ConvNeXt + ConvStem is much easier to train than the ViT, it has the potential to become similarly popular for studying adversarial robustness on ImageNet like (Wide)-ResNets for CIFAR-10. On the other hand ViT + ConvStem excels in generalization to unseen threat models. Thus there is no clear winner between ViT and ConvNeXt. Given that ImageNet backbones are used in several other tasks, we think that our robust ImageNet models could boost robustness also in these domains. One such example is our recent work in robust semantic segmentation [11]. Another interesting open question is why the ConvStem leads to such a boost in generalization to  $\ell_1$ - and  $\ell_2$ -threat models.
Limitations. Given the extensive set of experiments already presented in this work and our limited computational resources, we were unable to include other modern architectures. Hence, even though we cover quite diverse architectures, we do not cover all modern architectures for ImageNet.
Broader Impact. As this work studies the robustness of deep neural networks to adversaries, we do not see any potential negative impact of our work. Instead, our work could help make modern deep models robust and safer to deploy.
The authors acknowledge support from the DFG Cluster of Excellence "Machine Learning - New Perspectives for Science", EXC 2064/1, project number 390727645 and Open Philanthropy. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Naman D Singh.
[1] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In ECCV, 2020.
[2] Yutong Bai, Jieru Mei, Alan Yuille, and Cihang Xie. Are transformers more robust than CNNs? In NeurIPS, 2021.
[3] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In NeurIPS, 2019.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[5] Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning. In CVPR, 2020.
[6] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In NeurIPS Datasets and Benchmarks Track, 2021.
[7] Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In ICML, 2020.
[8] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
[9] Francesco Croce and Matthias Hein. Mind the box:  $l_{1}$ -apgd for sparse adversarial attacks on image classifiers. In ICML, 2021.
[10] Francesco Croce and Matthias Hein. Adversarial robustness against multiple and single  $l_p$ -threat models via quick fine-tuning of robust classifiers. In ICML, 2022.
[11] Francesco Croce, Naman D Singh, and Matthias Hein. Robust semantic segmentation: Strong adversarial attacks and fast training of robust models. arXiv preprint, arXiv:2306.12941, 2023.
[12] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR, 2020.
[13] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. In NeurIPS, 2021.
[14] Edoardo Debenedetti, Vikash Sehwag, and Prateek Mittal. A light recipe to train robust vision transformers. In First IEEE Conference on Secure and Trustworthy Machine Learning, 2023.
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[17] Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. In NeurIPS, 2021.
[39] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
[40] Tianyu Pang, Min Lin, Xiao Yang, Junyi Zhu, and Shuicheng Yan. Robustness and accuracy could be reconcilable by (proper) definition. In ICML, 2022.
[41] Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. In ICLR, 2021.
[42] ShengYun Peng, Weilin Xu, Cory Cornelius, Kevin Li, Rahul Duggal, Duen Horng Chau, and Jason Martin. Robarch: Designing robust architectures against adversarial attacks. arXiv preprint arXiv:2301.03110, 2023.
[43] Rahul Rade and Seyed-Mohsen Moosavi-Dezfooli. Reducing excessive margin to achieve a better accuracy vs. robustness trade-off. In ICLR, 2022.
[44] Sylvestre-Alvise Rebuffi, Francesco Croce, and Sven Gowal. Revisiting adapters with adversarial training. In ICLR, 2023.
[45] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A. Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Data augmentation can improve robustness. In NeurIPS, 2021.
[46] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In NeurIPS, 2020.
[47] Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In ICML, 2021.
[48] Shiyu Tang, Siyuan Liang, Ruihao Gong, Aishan Liu, Xianglong Liu, and Dacheng Tao. Exploring the relationship between architecture and adversarially robust generalization. arXiv preprint arXiv:2209.14105, 2022.
[49] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlx architecture for vision. In NeurIPS, 2021.
[50] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[51] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. In ECCV, 2022.
[52] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. In NeurIPS, 2019.
[53] Florian Tramér and Dan Boneh. Adversarial training and robustness for multiple perturbations. In NeurIPS, 2019.
[54] Asher Trockman and J Zico Kolter. Patches are all you need? Transactions on Machine Learning Research, 2023.
[55] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, 2022.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[57] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In CVPR, 2023.
[58] Ross Wightman. Pytorch image models. https://github.com/rwrightman/pytorch-image-models, 2019.