Can CNNs Be More Robust Than Transformers?
Authors: Zeyu Wang
Yutong Bai
Yuyin Zhou
Cihang Xie
Venue: ICLR 2023 poster (2023)
Keywords: CNNs
Transformers
Out-of-Distribution Robustness

Abstract: The recent success of Vision Transformers is shaking the long dominance of Convolutional Neural Networks (CNNs) in image recognition for a decade. Specifically, in terms of robustness on out-of-distribution samples, recent research finds that Transformers are inherently more robust than CNNs, regardless of different training setups. Moreover, it is believed that such superiority of Transformers should largely be credited to their \emph{self-attention-like architectures per se}. In this paper, we question that belief by closely examining the design of Transformers. Our findings lead to three highly effective architecture designs for boosting robustness, yet simple enough to be implemented in several lines of code, namely a) patchifying input images, b) enlarging kernel size, and c) reducing activation layers and normalization layers. Bringing these components together, we are able to build pure CNN architectures without any attention-like operations that are as robust as, or even more robust than, Transformers. We hope this work can help the community better understand the design of robust neural architectures. The code is publicly available at https://github.com/UCSC-VLAA/RobustCNN.

Published as a conference paper at ICLR 2023
CAN CNNS BE MORE ROBUST THAN TRANSFORMERS?
Zeyu Wang' Yutong Bai Yuyin Zhou' Cihang Xie UC Santa Cruz PJohns Hopkins University
ABSTRACT
The recent success of Vision Transformers is shaking the long dominance of Con- volutional Neural Networks (CNNs) in image recognition for a decade. Specifi- cally, in terms of robustness on out-of-distributioo samples, recent research finds that Transformers are inherently more robust than CNNs, regardless different training setups. Moreover, it is believed that such superiority of Transformers should largely be credited to their self-attention paper, we question that belief by closely examining the design of Transformers. Our findings lead to three highly effective architecture designs for boosting ro- bustness, yet simple enough to be implemented in several lines of code, namely a) patchifying input images, b) enlarging kernel size, and c) reducing activation layers and normalization layers. Bringing these components together, we are able to build pure CNN architectures without any attention-like like operations that are as robust as, or even more robust than, Transformers. We hope this work can help the community better understand the design of robust neural architectures. The code is publicly available at http://ghhhbcccoootcccccAAAAAAA
architectures per se. In this
INTRODUCTION
The success of deep learning in computer vision largely driven by Convolutional Neural Networks (CNNs). Starting from the milestone work AlexNet (Krizhevsky et al., 2012), CNNs keep pushing the frontier of computer vision (Simonyan & Zisserman, 2015; He et al., 2016; Tan Le, 2019). Interestingly, the recently emerged Vision Transformer (ViT) (Dosovitskiy et al., 2020) challenges the leading position of CNNs. ViT offers a completely different roadmap- by applying the pure self-attention-bert based architecture to sequences of image patches, ViTs are able to attain competitive performance on a wide range of visual benchmarks compared to CNNs.
The recent studies on -uttof-distribution robustness (Bai et al., 2021; Zhang al., 2022; Paul & Chen, 2022) further heat up the debate between CNNs and Transformers. Unlike the standard vi- sual benchmarks where both models are closely matched, Transformers
much more robust than CNNs when testing out of the box. Moreover, Bai et al. (2021) argue that, rather than being benefited by the advanced training recipe provided in (Touvron al., 2021 a), such strong out-of-distribution robustness comes inherently with the Transformer's 's self-attention-lit like architecture. For example, simply "upgrading" a pure CNN to a hybrid architecture (i.e., with both CNN blocks and Trans- former blocks) can effectively improve out-of-distribution robustness (Bai et al., 2021).
Though is generally believed that the architecture difference the key factor that leads to the robustness gap between Transformers and CNNs, existing works fail to answer which architectural elements Transformer should be attributed such stronger robustness. The most relevant analysis works point out that Transformer blocks,
is provided in (Bai et al., 2021; Shao et al., where self-attention operation is the pivot unit, are critical for robustness. Nonetheless, given a) the Transformer block itself is already compound design, and b) Transformer also contains many other layers (e.g., patch embedding layer), the relationship between robustness and Transformer's architectural elements remains confounding. In this work, take closer look at the architecture design of Transformers. More importantly, we aim to explore, with the help of the architectural elements from Transformers, whether CNNs be robust learners well.
Our diagnose delivers three key messages for improving out offtooffrriibttion robustness, from the per- spective of neural architecture design. Firstly, patchifying images into non-overlapped patches can

---

Published as a conference paper at ICLR 2023
Stylzed Imagehett
ImageNal-C(
Imagehlet-RR R (1)
InageNeet-keichh (1)
$12
IECE
41.
42.5
27.7
Rasleedo Cat-dmall Rolunt--astee
Dert Ae Rolkur Recharde
DefiSraal Roausk-aath
Cen-Sral Rabutt
Figure 1: Comparison of out-of-distribution robustness among ResNet, DeiT, and our enhanced ResNet (dubbed Robust-RexNet). Though Deï'--S largely outperforms the vanilla ResNet, it per- forms worse than Ressst-ResNet on these robustness benchmarks.
substantially contribute to out-of-distribution robustness; more interestingly, regarding the choice of patch size, we find the larger the better. Secondly, despite applying small convolutional kernels is a popular design recipe, we observe adopting a much larger convolutional kernel size (e.g.. from 3 x 3 to 7 x 7, even to 11 11) is necessary for securing model robustness samples. Lastly, as inspired by the recent work (Liu et al., 2022), we note aggressively reducing the number of normalization layers and activation functions is beneficial for out-of-diitrrbuttio bustmess; meanwhile, as a byproduct, the training speed could be accelerated by up to ~23, due to fewer normalization layers being used (Gitman & Ginsburg. 2017; Brock al., 2021).
-uitr--distribution
Our experiments verify that all these three architectural elements consistently and effectively im- prove -ii-t---iitiitttionn robustness on a set of CNN architectures. The largest improvement is reported by integrating all of them into CNNs' architecture design- as shown in Fig. 1, without ap- plying any self-attention-lik components, our enhanced ResNet (dubbed Kobust-ResNer is able to outperform similar-scale Transformer, DeiT-S, by 2.4% on 0.5% on ImageNet-C (42.8% vs. 42.3%), 4.0% on ImageNet-R (41.9% vs. 45.9%) and 3.9% on ImageNet-Skette Skeech (29. 1% 3..0%). We hope this work help the community better understand the underlying principle of designing robust neural architectures.
(16.2% 18.6%),
2 RELATED WORKS
Transformers (Vaswani et al., 2017), which apply self-attention to enable global interactions between input elements, underpins the success of building foundation models in natural language processing (Devlin et al., 2019; Yang et al., 2019; Dai et al., 2019; Radford et al., 2018; 2019; Brown et al., 2020; Bommasani al., 2021). Recently, Dosovitskiy el al. (Dosovitskiy et al., 2020) show Transformer attains competitive performance sification task. Later works keep pushing the potential of Transformer on a variety of visual tasks, in both supervised learning (Touvron al., 2021a; Yuan et al., 2021; Liu et al., 2021; Wang al., 2021; Yuan et al., 2021; Zhai et al., 2021; Touvron et al., 2021b; Xue et al., 2021) and self-supervised learning (Caron et al., 2021; Chen al., 2021; Bao et al., 2022; Zhou al., 2022; Xie al., 2021; He et al., 2022), showing seemly inevitable trend on replacing CNNs in computer vision.
Vision Transformers.
the challenging ImageNet clas-
CNNs striking back. There is a recent surge in works that aim at retaking the position of CNNs as the favored architecture. Wightman et al. (Wightman et al., 2021) demonstrate that, with the ad- vanced training setup, the canonical ResNet-50 able boost its performance by 4% on ImageNet. In addition, prior works show modernizing CNNs' architecture design is essential either sim- ply adopting Transformer's patchify layer (Trockman Kolter, 2022) or aggressively bringing in every possible component from Transformer (Liu et al., 2022), CNNs are able to attain competitive performance w.r.t. Transformers across a range of visual benchmarks.
Our work is closely related to ConvNeXt (Liu et al., 2022), while shifting the study focus from stan- dard accuracy to robustness. Moreover, rather than specifically offering a unique neural architecture as in (Liu et al., 2022), this paper aims to provide set of useful architectural elements that allows CNNs to be able to match, or even outperform, Transformers when measuring robustness.
Out-of-distribution robustness. Dealing with data from shifted distributions is commonly en- countered problem when deploying models in the real world. To simulate such challenges, sev- eral distt---istiibuttin benchmarks have been established, including measuring model performance
2

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of various baseline models. We replace the ResNet Bottleneck block with four
different stronger building blocks, and keep the computational cost of bascline models on par with
DeiT counterparts. We observe significant gain in terms of clean image accuracy, but our CNN
baseline models are still less robust than DeiT.
Architecture
FLOPs
(f)
S-IN (
(N)
IN-R (1)
IN-SK (T)
ResNet50
4.1G
78.4
9.9
51.2
39.2
27.7
DeiT-S
4.6G
79.8
16.2
42.8
41.9
29.1
ResNet-DW
4.8G
79.7
10.6
50.1
41.3
29.1
ResNet-Inverte-DDW
4.6G
80.0
10.8
47.7
41.9
29.4
ResNee-UUp-Invereed--
4.7G
79.7
12.9
47.9
42.9
30.8
ResNet-Dowa-Lnerteed--
4.5G
79.6
12.1
49.0
42.5
29.5
images with common corruptions (Hendrycks & Dietterich, 2018) with various renditions (Hendrycks et al., 2021). Recently, a set of works (Bai et al., 2021; Zhang et al., 2022; Paul & Chen, 2022) find that Transformers are inherently much more robust than CNNs out-of-distribution samples. Our work is direct follow-up of Bai et al. (2021), whereas we stand on the opposite side-we show CNNs can in turn outperform Transformers in out-of-distribution robustness.
SETTINGS
Following the setup in Bai et al. (2021), in this paper, the robustness is thoroughly compared between CNNs and Transformers using ResNet (He et al., 2016) and ViT (Dosovitskiy et al., 2020).
CNN Block Instantiations. We consider four different block instantiations, as shown in Fig. 2. The first block is Depth-wise ResNet Bottleneck block (Xie et al., 2017), in which the xx3 vanilla convolution layer is replaced with a 3x3 depth-wise convolution layer. The second block is the Inverted Depth-wise ResNet Bottleneck block, in which the hidden dimension is four times the input dimension (Sandler et al., 2018). The third block is based on the second block, with the depth- wise convolution layer moved up in position in ConvNeXT (Liu et al., 2022). Similarly, based on the second block, the fourth block moves down the position of the depthwise convolution layer. We replace the bottleneck building block in the original ResNet architecture with these four blocks, and refer to the resulting models as ResNet-DW, Resseet-nnvered-DD DW, ResNet- -n-er-eredd DDW, and respectively.
1x1, 96-3384
1x1, 96-2384
ReLU
BN, ReLU
BN, ReLUU
BN, ReLU
d3x3, 96-396
d3x3. 384-3884
1xl, 96-3884
1x1, 384-396
BN, ReLU
BN,
BN,
1x1, 96-3984
1x1, 384-296
1x1, 384-296
d3x3, 96-396
BN
BN
BN
BN
ReLU
ReLU
ReLU
ReLU
(a) ResNes-DW
Jb) DesNae-Inverted-DWW
to Restic-Jj--imeeradttW
Figure 2: Architectures of four different instantiations CNN block.
Computational Cost. this work, we use FLOPS to measure model size. Here we note that directly replacing the ResNet bottleneck block with the above four instantiated blocks will signif- icantly reduce the total FLOPs of the model, due to the usage of the depth-wise convolution. To mitigate the computational cost loss and to increase model performance, following the spirit of ResNeXT (Xie et al., 2017), we change the channel setting of each stage from (64, 128, 256, 512) to (96, 192, 384, 768). We then tune the block number in stage 3 to keep the total FLOPs of baseline models roughly the same as D-i The final FLOPs of our baseline models are shown in Tab. 1. Unless otherwise stated, all models considered in this work of similar scale DeiT-S.
3

---

Published as a conference paper at ICLR 2023
Robustness Benchmarks. In this work, extensively evaluate the model performance out-of- distribution robustness using the following benchmarks: 1) Iaaell-mageeeeeeiiii et al., 2018) which contains synthesized images with shape-texture conflicting cues; 2) ImageNet-C(Henddiie & Dietterich, 2018), with various common image corruptions; 3) ImageNet-R (Hendrycks et al., 2021), which contains natural renditions of ImageNet object classes with different textures and local image statistics; 4) ImageNet-Sken Skeeet (Wang et al., 2019), which includes sketch images of the same ImageNet classes collected online.
Training Recipe. CNNs may achieve better robustness by simply tuning the training recipe (Bai et al., 2021; Wightman et al., 2021). Therefore, in this work, we deliberately apply the standard 300- epoch DeiT training recipe (Touvron et al., 2021a) for all models unless otherwise mentioned, so that the performance difference between models can only be attributed to the architecture difference.
Baseline Results. The results are shown in Tab. 1. For simplicity, we use "IN", "S-IN", "IN- C", "IN-R", and "IN-SK" as abbreviations for "ImageNet", "Stylized-ImageNer" "ImageNet-C", "ImageNet-R", and "ImageNet-Sketch". Same as the conclusion from (Bai et al., 2021), we can observe that DeiT-S shows much stronger robustness generalization than ResNet50. Moreover, note that, even by equipping with stronger depth- wise convolution layers, the ResNet architecture is only able achieve comparable accuracy clean images while remaining significantly less robust than its DeiT' counterpart. This finding indicates that the key to the impressive robustness of Vision Transformer lies in its architecture design.
COMPONENT DIAGNOSIS
In this section, we present three effective architectural designs that draw inspiration from Trans- formers, but work surprisingly well on CNNs as well. These designs are as follows: 1) patchifying input images (Sec. 4.1), b) enlarging the kernel size (Sec. 4.2), and finally, 3) reducing the number of activation layers and normalization layers (Sec. 4.3).
4.1 PATCHIFY STEM
A CNN or a Transformer typically down-samples the input image at the beginning of the network to obtain a proper feature map size. A standard ResNet architecture achieves this by using a 7x7 convolution layer with stride 2, followed by 3x3 max pooling with stride 2, resulting in 4x resolution reduction. On the other hand, ViT adopts much more aggressive down-sampling strategy by partitioning the input image into pxp non-overlapping patches and projects each patch with linear layer. In practice, this is implemented by a convolution layer with kernel size p and stride p. where p is typically set to 16. Here the layers preceding the typical ResNet block or the self-attention block in ViT are referred to as the stem. While previous works have investigated the importance of stem setup in CNNs (Trockman & Kolter, 2022) and Transformers (Xiao et al., 2021), none have examined this module from the perspective of robustness.
To investigate this further, replace the ResNet-style stem in the baseline models with the VIT- style patchify stem. Specifically, we use convolution layer with kernel size p and stride p. where varies from 4 16. We keep the total stride of the model fixed, so that a 224x224 input image will always lead to a 7x7 feature map before the final global pooling layer. In particular, the original ResNet sets stride=2 for the first block in stages 3, and 4. We set stride=1 for the first block in stage 2 when employing the 8x8 patchify stem, and set stride=1 for the first block in stages 2 and 3 when employing the 16x 16 patchify stem. To ensure a fair comparison, we add extra blocks in stage 3 to maintain similar FL.OPs as before.
In Tab. 2, we observe that increasing the patch size of the ViT-style patchify stem leads to increased performance on robustness benchmarks, albeit potentially at the cost of clean accuracy. Specifically, for all baseline models, when the patch size is set to 8, the performance on all robustness benchmarks is boosted by at least 0.6%. When the patch size is increased to 16, the performance on all robustness benchmarks is boosted by at least 1.2%, with the most significant improvement being 6.6% on Stylized-ImageNer ImageNet. With these results, contributes to the strong robustness of VIT as observed in (Bai et al., 2021), and meanwhile, can play vital role in closing the robustness gap between CNNs and Transformers.
conclude that this simple patchify operation largely
4

---

Published as a conference paper at ICLR 2023
The performance of different baseline models equipped with ViT-style patchify stem. "PX" refers to the model having a ViT-style patchify stem with the patch size set to X. The block number stage 3 increased when the patch size is larger. With larger patch size, the models attain better robustness but worse clean accuracy.
Table 2:
Architecture
FLOPs
IN (1)
S-IN (1)
IN-C (1)
IN-R (1)
IN-SK (T)
ResNe150 DeïT-S
4.1G
78.4
9.9
51.2
39.2
27.7
4.6G
79.8
16.2
42.8
41.9
29.1
ResNet-DW
4.8G
79.7
10.6
50.1
41.3
29.1
P4
4.6G
79.6
11.4
51.2
40.5
28.3
P8
4.6G
79.9
12.7
47.6
42.5
30.1
+ P16
4.6G
78.5
17.0
44.3
44.8
32.0
ResNet-Inverte-DDW
4.6G
80.0
10.8
47.7
41.9
29.4
4.5G
80.2
11.2
48.8
41.5
29.8
P8
4.6G
79.8
14.8
45.4
44.0
31.8
+ P16
4.6G
77.9
17.2
42.5
44.9
32.4
ResNet- ResNet-Up-lnverted
4.7G
79.7
12.9
47.9
42.9
30.8
P4
4.5G
80.0
13.5
48.9
42.3
30.7
P8
4.7G
79.8
15.7
46.3
44.0
31.4
+ P16
4.5G
77.9
17.0
43.9
44.6
32.0
DesNet-Dow--Inerrte---
4.5G
79.6
12.1
49.0
42.5
29.5
4.4G
79.8
12.4
48.6
41.8
29.4
P8
4.5G
79.6
15.2
46.2
43.5
30.7
+ P16
4.6G
78.0
16.2
43.8
43.8
30.6
We also conduct experiments with more advanced patchify stems. Surprisingly, while these stems improve the corresponding clean image accuracy, we find that they contribute little to out-of- distribution robustness. Further details can be found in Appendix A.2. This observation suggests that clean accuracy and nntoof-distribution robustness do not always exhibit a positive correlation. In other words, designs that enhance clean accuracy may not necessarily result in better robust- ness. highlighting the importance of exploring methods that can improve robustness in addition to enhancing clean accuracy.
4.2 LARGE KERNEL SIZE
One critical property that distinguishes the self-attention operation from the classic convolution op- cration is its ability to operate on the entire input image or feature map, resulting in global receptive field. The importance of capturing long-range dependencies has been demonstrated for CNNs even before the advent Vision Transformer. A notable example is Non-local Neural Network, which has been shown to be highly effective for both static and sequential image recognition, even when equipped with only one non-local block (Wang et al., 2018). However, the most commonly used ap- proach in CNN is still stack multiple 3x3 convolution layers to gradually increase the receptive field of the network as it goes deeper.
In this section, we aim mimic the behavior of the self-attention block, by increasing the kernel size of the depth-wise convolution layer. We experiment with various kernel sizes, including 5, 7, 9, 11, and 13, and evaluate their performance different robustness benchmarks, are shown in Fig. 3. Our findings suggest that larger kernel sizes generally lead to better clean accuracy and stronger robustness. Nevertheless, we also observe that the performance gain gradually saturates when the kernel size becomes too large.
noteworthy that using (standard) convolutions with larger kernels will result in a significant increase in computation. For example, if we directly change the kernel size in ResNet50 from to 5, the total FLOPs of the resulting model would be 7.4G, which considerably larger than its Transformer counterpart. However, with the usage of depth-wise convolution layer, increasing the kernel size from 3 to 13 typically only increases the FLOPs by 0.3G, which is relatively small compared to the FLOPs of DeiT-S (4.6G). The only exceptional case here is due to the large channel dimension in its Inverted Bottleneck design, increasing the kernel size from 3 to 13 results in total increase of 1.4G FLOPS, making it unfair comparison some extent.

---

Published as a conference paper at ICLR 2023
ImageNet (1) DNV
Stylized Beylized-ImagnNet (1)
ImageNat-CC (1)
01.5
15.0
12.9
42.0
00.0
11.0
10.0
44.0
11
13
Kernel Bize
Kernel
Kernel Size
ImageNet-R (1)
ImegeNet-Sketthh
20
51
42
20
10
Robustness evaluation of model with varying kernel size. Different colors and shapes denote different models.
Figure 3:
Table 3:
The performance of models with differently removed activation layers. We use / to denote keeping an activation layer and to denote removing an activation layer.
ReLU 2nd
Architecture
Fl.OPs
3rd
IN N
S-IN
IN-C
IN-R
IN-SK 1
ResNet50
4.1G
78.4
9.9
51.2
39.2
27.7
DeiT-$
4.6G
79.8
16.2
42.8
41.9
29.1
/
/
79.7
10.6
50.1
41.3
29.1
/
X
x
74.8
8.3
58.6
37.1
24.5
ResNet-DW
4.8G
/
X
73.2
8.6
60.5
36.2
23.3
X
79.4
11.7
50.5
42.8
29.8
/
V
80.0
10.8
47.7
41.9
29.4
/
x
X
80.7
12.3
46.4
45.3
31.9
ResNet-Inverted-De
4.6G
1
X
80.3
11.2
48.5
44.0
30.9
X
72.5
9.7
59.5
37.5
24.7
/
79.7
12.9
47.9
42.9
30.8
x
x
67.8
9.4
65.0
33.0
19.7
ResNettUp-Inveereed.
4.7G
/
x
80.1
13.0
47.3
44.2
30.0
x
68.7
8.9
64.4
33.4
20.2
/
79.6
12.1
49.0
42.5
29.5
x
x
80.7
13.0
46.6
45.9
32.8
ResNee-Doww-Ineetted-DWW 4.5G
/
x
70.1
9.9
63.3
35.2
21.8
X
/
68.8
8.3
65.1
33.7
21.9
As a side note, the extra computational cost incurred by a large kernel size can be mitigated by employing a patchify stem with a large patch size. As result, our final models will still be on the same scale as DeiT-S. For models with multiple proposed designs, please refer to Sec. 5.
4.3 REDUCING ACTIVATION AND NORMALIZATION LAYERS
In comparison to a ResNet block, typical Vision Transformer block has fewer activation and nor- malization layers (Dosovitskiy al., 2020; Liu et al., 2021). This architecture design choice has also been found to be effective improving the performance ConvNeXT (Liu et al., 2022). In- spired by this, adopt the idea of reducing the number of activation and normalization layers in

---

Published as a conference paper at ICLR 2023
Table 4: The performance of models with differently removed activation and normalization layer. Note that the results are obtained with the best choice of only one activation layer in block based on the results Tab. 3. We use denote keeping normalization layer and X to denote removing normalization layer. It is observed that the highlighted architectural design choices consistently lead to improved robustness upon the baselines.
Architecture
FLOPPs
IN t
S-IN
IN-C 4
IN-R
IN-SK
1s
2nd
3rd
ResNe150
4.1G
78.4
9.9
51.2
39.2
27.7
DeïT-S
4.6G
79.8
42.8
41.9
29.1
/
/
79.7
10.6
50.1
41.3
29.1
/
X
x
79.4
11.6
49.3
43.3
29.9
ResNet-DW
4.8G
X
79.5
10.6
49.5
43.1
29.3
79.3
11.1
50.7
42.5
30.2
/
80.0
10.8
47.7
41.9
29.4
x
x
80.4
12.9
47.0
45.8
32.4
ResNet-Inverted-De
4.6G
/
x
80.0
11.2
49.3
44.1
31.2
x
80.1
13.0
46.8
44.3
30.8
/
/
79.7
12.9
47.9
42.9
30.8
x
x
80.5
14.3
45.0
47.2
33.1
ResNet-UpphIverredd-WW
4.7G
/
x
80.4
12.9
47.0
45.8
32.4
x
/
80.1
13.0
46.8
44.3
30.8
79.6
12.1
49.0
42.5
29.5
X
x
80.5
13.6
46.4
45.9
33.3
4.5G
/
x
80.7
13.1
46.1
45.5
31.7
X
80.8
12.7
46.5
46.1
33.1
1x1, 384->96
1x1, 96-2384
d3x3, 96-296
1x1, 96--384
BN
BN, ReLU
BN
BN, ReLU
03x3, 96-396
d3x3, 384->384
1x1, 96 96-2384
1xl, 384-296
ReLU
1x1, 96-23B4
1x1, 384 996
1x1, 384 384-396
d3x3, 96-396
ReLU
DoW
RpsNic-Up Inverted DW
Down Inw
Figure 4: The optimal position of normalization and activation layers for different block architec- tures.
all four block instantiations for exploring its effects robustness generalization. Specifically, a ResNet block typically contains one normalization layer and one activation layer following each convolution layer, resulting in total of three normalization and activation layers in one block. In our implementation, we remove two normalization layers and activation layers from each block, resulting in only one normalization layer and activation layer.
We conduct experiments with different combinations of removing activation and normalization lay- ers and empirically find that the best results are achieved by leaving only one activation layer after the convolution layer where the channel dimension expands (i.e.. the number of output channels is larger than that of the input channels) and leaving one normalization layer after the first convolution layer. The optimal configuration is visualized in Fig. 4. The results of the models with differently re- moved layers are shown in Tab. 3 and Tab. 4. For example, for ResNet-Up-Inverted DW, we observe considerable improvements of 1.4% on Stylized-Imaget (45.0% vs. 47.9%), 4.3% on ImageNet-R (47.2% vs. 42.9%). and 2.3% on ImageNet ImageNet-Sketell Sketch (33.1% 30.8%). Moreover, reducing the number of normalization layers results in lower GPU memory usage and faster training (Gitman & Ginsburg, 2017: Brock et al., 2021), with up to 23% speed-up achieved by simply removing few normalization layers.
(14.3% 12.9%), 2.9% on ImageNet-C

---

Published as a conference paper at ICLR 2023
Table 5: The performance of different baseline models equipped with ViT-style patchify stem, large kernel size, and fewer activation and normalization layers. The model with the suffix "PX" refers that has a ViT-style patchify stem with patch size set to X. "KX" refers that the model uses blocks with depth-wist Septh-wise convolution layer with kernel size X. "NormXActY" refers to that the model has only one normalization layer after the Xth convolution layer and one activation layer after the Yth convolution layer in the block.
Architecture
FLOPs
IN (t)
S-IN (1)
IN-C (1)
IN-R (t)
IN-SK (T)
ResNet50
4.1G
78.4
9.9
51.2
39.2
27.7
DeiT-S
4.6G
79.8
16.2
42.8
41.9
29.1
ResNet-DW
4.8G
79.7
10.6
50.1
41.3
29.1
P16 K11 Norm Act3
4.5G
79.4
18.6
42.3
45.9
33.0
ResNet-Inverted-DW
4.6G
80.0
10.8
47.7
41.9
29.4
P16 K7 Norm Act1
4.6G
79.0
19.5
42.1
45.9
32.8
ResNet-Up-luverted DW
4.7G
79.7
12.9
47.9
42.9
30.8
+P16 + K11 Nor2
4.4G
79.2
20.2
40.9
48.7
35.2
ResNe--Dwwn-Inveeed-DWY
4.5G
79.6
12.1
49.0
42.5
29.5
P16 + K11 + NormlActl
4.6G
79.9
19.3
41.6
46.0
32.8
Table 6: The robustness generalization of different models when DeiT-S serves as the teacher model. The results DeiT-S here are included for reference; trained by the default recipe without knowledge distillation.
Student Architecture
FLOPs
IN (f)
S-IN
IN-C (L)
IN-R
IN-SK
DeiT-S
4.6G
79.8
16.2
42.8
41.9
29.1
ResNet-DW
4.8G
79.5
10.2
50.8
40.9
28.9
Resustt-
4.5G
79.3
20.1
41.2
46.5
33.2
ResNee-Inneeted-DDW
4.6G
80.1
11.0
47.9
42.4
30.4
Rolnss-ResWee--meeteedDDWW
4.6G
79.1
19.4
42.4
45.9
33.0
ResNee-UpplneeteedDDWW
4.7G
79.9
14.1
48.4
43.6
31.0
Robus--ReNeer//p--weenecDDD
4.4G
79.3
19.6
41.1
49.3
35.4
ResNecc-DewninneeeeddWWWW
4.5G
79.7
12.3
48.5
42.3
30.4
Sahust-ReeWeeeDDDon---eeeeeeer
4.6G
79.9
18.9
41.0
46.4
33.5
COMPONENTS COMBINATION
In this section, we explore the impact of combining all the proposed components on the model's performance. Specifically, we adopt 16x 16 patchify stem and an 11x11 kernel size, along with the corresponding optimal position for placing the normalization and activation layer for all ar- chitectures. An exception here is ResNeet-Iverred-DW for which we use 7x7 kernel size, as we empirically found that using too-large kernel size (e.g., 11x11) can lead to unstable training
As shown in Tab. 5, and Tab. 9, Tab. 10, Tab. 11 in Appendix A.3, we can see these simple de- signs not only work well when individually applied to ResNet, but work even better when used together. Moreover, by employing all three designs, ResNet outperforms DeiT all four out- of-distribution benchmarks. These results confirm the effectiveness of our proposed architecture designs and suggest that a pure CNN without any self-attention-like block can achieve robustness, as good as VIT.
KNOWLEDGE DISTILLATION
Knowledge Distillation is technique used for training weaker student model with lower capac- ity by transferring the knowledge of stronger teacher model. Typically, the student model can achieve similar better performance than the teacher model through knowledge distillation.
8

---

Published as a conference paper at ICLR 2023
However, applying knowledge distillation directly let ResNet-50 (the student model) learn from DeiT-S (the teacher model) has been found to be less effective in enhancing robustness as shown in (Bai et al., 2021). Surprisingly, when the model roles switched, the student model DeiT-S remarkably outperforms the teacher model ResNer-50 on a range robustness benchmarks, leading to the conclusion that the key achieving good robustness of DeiT lies in its architecture, which therefore cannot be transferred to ResNet through knowledge distillation.
To investigate this further, repeat these experiments with models that combine all three proposed architectural designs as the student models and with DeïT-S as the teacher model. As shown in Tab. 6, we observe that with the help of our proposed architectural elements from ViT, our resulting Robust-ResNer family now can consistently perform better than DeiT on out-of-distribution sam- ples. In contrast, the baseline models, despite achieving good performance on clean ImageNet, are not as robust as the teacher model DeiT--S.
7 LARGER MODELS
To demonstrate the effectiveness of our proposed models on larger scales, we conduct experiments to match the total FLOOSs of DeiT-Base. Specifically, increase the base channel dimensions (128, 256, 512, and 1024) and add over 20 blocks stage 3 of the network, while using the ConvNeXT- recipe for training (Liu et al., 2022) !. We compare the performance of the adjusted models to DeîT-B, as shown in Tab. 7. Our results indicate that our proposed Robust-ResNet family performs favorably against DeiT-B even on larger scales, suggesting that our method has great potential for scaling up models.
Table 7: Comparing the robustness of Robust-ResNet and DeiT at a larger scale.
Architecture
FLOPs
(1)
S-IN (t)
IN-C (
IN-R (t)
IN-SK (*)
ResNet-200
15.1G
81.7
13.8
42.2
45.1
34.5
DeiT-B
17.6G
81.8
19.6
37.9
44.7
32.0
Aebust-ResNer-DW...
17.2G
80.5
19.2
38.8
46.6
34.2
Robust ReseNee-Invertedd-W DW-B
17.1G
81.6
22.1
35.5
50.2
36.7
17.1G
81.9
23.1
35.1
50.5
37.1
Nesetst--Reee-t-o--nneeeeee -WW-B
17.3G
81.3
22.2
35.6
49.7
37.8
CONCLUSION
The recent study by (Bai al., 2021) claims that Transformers outperform CNNs out-of- distribution samples, with the self-attention-lik architectures being the main contributing factor. In contrast, this paper examines the Transformer architecture more closely and identifies several beneficial designs beyond the self-attention block. By incorporating these designs into ResNet, have developed a CNN architecture that can match even surpass the robustness of Vision Transformer model of comparable size. We hope our findings prompt researchers to reevaluate the robustness comparison between Transformers and CNNs, and inspire further investigations into developing more resilient architecture designs.
ACKNOWLEDGEMENT
This work is supported by a gift from Open Philanthropy, TPU Research Cloud (TRC) program, and Google Cloud Research Credits program.
REFERENCES
Yutong Bai, Jieru Mei, Alan Yuille, and Cihang Xie. Are transformers more robust than cnns? In NeurIPS, 2021.
We have also experimented with the DeiT training recipe, but found that it did not improve performance significantly in this case.

---

Published as a conference paper at ICLR 2023
Hangbo Bao, Li Dong, and Furu Wei. BETT: BERT pre-training of image transformers. In ICLR, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint
2021.
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. ICML, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training coff-superviee vision transformers. In ICCV, 2021.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical data aug- mentation with no separate search. In NeurIPS, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. ACL, 2019.
Jacob Devlin, Ming-Woy Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Xiaohan Ding. Xiangyu Zhang, Ningning Ma. Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vtg-style convnets great again. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 13733-13742, 2021.
Xiaohan Ding, Honghao Chen, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Repmlpnet: Hi- crarchical vision mlp with re parameterized locality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 578-587, 2022a.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kemee design in In Proceedings the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11963-11975, 2022b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. In ICLR, 2018.
Igor Gitman and Boris Ginsburg- Comparison of batch normalization and weight normalization algorithms for the large-scale image classification. arXiv preprint 45X1v:1709.08145, 2017.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer convnet's clothing for faster inference. In ICCV, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. In ICLR, 2018.
10

---

Published as a conference paper at ICLR 2023
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In CVPR, 2020.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Alex Krizhevsky, Nya Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. In CVPR, 2022.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Sayak Paul and Pin- Yu Chen. Vision transformers are robust learners. In AAAI, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. https: /ss--sswssssstt2 amazonaws com/ openai- apenaai-assets/eeeeae wwwpeeetteeeeeeaeh--heeaeeeeeeee wwwecccchovvsseeeeeeeeeeeeeeeeee understtanding_paeer 2018.018.
1anguageg
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pun-Yu Chen, and Cho-Jui Hsieh. On the adversarial robust- ness of visual transformers. arXiv preprint arXiv:2 /5670, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. ICLR, 2015.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In ICML, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021b.
Asher Trockman and Zico Kolter. Patches are all you need? arXiv preprint arXiv:2201.09792 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. NeurIPS, 2019.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
11

---

Published as a conference paper at ICLR 2023
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. arXiv preprint arXiv:2100.0046,, 2021.
Tete Xiao, Piotr Dollar, Mannat Singh. Eric Mintun, Trevor Darrell, and Ross Girshick. Early convolutions help transformers see better. NeurlPS, 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self- supervised learning with swin transformers. arXiv preprint wxXi::2105.04533, 2021.
Fuzhao Xue, Ziji Shi, Yuxuan Lou, Yong Liu, and Yang You. Go wider instead of deeper. arXiv preprint urXiv:2107. 1117, 2021.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeuriPS, 2019.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujum Shi, Zihang Jiang. Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint (00.220066045600 2021.
Chongzhi Zhang, Mingyuan Zhang, Shanghang Zhang, Daisheng Jin, Qiang Zhou, Zhongang Cai, Haiyu Zhao, Shuai Yi, Xianglong Liu, and Ziwei Liu. Delving deep into the generalization of vision transformers under distribution shifts. In CVPR, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In AAAI, 2020.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022.
A APPENDIX
A.1 TRAINING RECIPE DETAILS
We follow the standard 300-epoch DeiT training recipe (Touvron et al., 2021a) in this work. Specif- ically, we train all models using AdamW optimizer (Loshchilov & Hutter, 2019). We set the initial decrease
base learning rate 5e-4, and apply the cosine learning rate scheduler weight decay, we additionally adopt six data augmentation & regularization strategies (i.e., Ran- dAug (Cubuk et al., 2020), MixUp (Zhang et al., 2018), CutMix (Yun al., 2019), Random Erasing (Zhong et al., 2020), Stochastic Depth (Huang et al., 2016), and Repeated Augmentation (Hoffer al., 2020)) to regularize training.
Besides
A.2 MORE STEM EXPERIMENTS
Recent studies suggest replacing the ViT-style patchify stem with small number of stacked 2-stride 3x3 convolution layers can greatly ease optimization and thus boost the clean accuracy (Graham et al., 2021; Xiao et al., 2021). To verify its effectiveness implemented the convolution stem for ViT-S in (Xiao et al., 2021). with a stack of 4 3x3 convolution layers with stride 2. The results are presented in Tab. 8. Surprisingly, though the use of convolutional stem does attain higher clean accuracy, appears to be not as helpful as ViT-style patchify stem on out-of-distribution robustness.
robustness benchmarks, we have also
12

---

Published as a conference paper at ICLR 2023
Table 8: The performance of different baseline models equipped with ViT-style patchify stem and convolution stem. "PX" refers to that the model has a ViT-style patchify stem with patch size set to X: "C" refers that has convolutional stem.
Architecture
Fl.OPs
IN
S-IN (t)
IN-C (
IN-R (1)
IN-SK ())
ResNet50 DeïT-S
4.1G
78.4
9.9
51.2
39.2
27.7
4.6G
79.8
16.2
42.8
41.9
29.1
ResNet-DW
4.8G
79.7
10.6
50.1
41.3
29.1
P16
4.6G
78.5
17.0
44.3
44.8
32.0
4.5G
79.6
16.2
48.4
43.0
30.2
ResNet-Inverte--DW
4.6G
80.0
10.8
47.7
41.9
29.4
+
4.6G
78.1
17.4
42.9
44.7
32.4
+C
4.6G
79.9
16.9
46.2
43.2
31.2
ResNet- Lnnieet-Up--vetteddeeee
4.7G
12.9
47.9
42.9
30.8
+ P16
4.5G
77.9
17.0
43.9
44.6
32.0
+
4.5G
79.1
16.6
48.4
43.2
30.3
4.5G
79.6
12.1
49.0
42.5
29.5
+ P16
4.6G
78.0
16.2
43.8
43.8
30.6
C
4.6G
79.3
16.4
49.0
42.4
29.2
Table 9: The performance of different baseline models equipped with ViT-style patchify stem and large kernel size. The models with the suffix "PX" refers to that it has ViT-style patchify stem with patch size set to X. "KX" refers to that the model uses blocks with depth wise convolution layer with kernel size X.
Architecture
FLOPs
IN (t)
S-IN (1)
IN-C (L)
IN-R (1)
IN-SK (t)
ResNe150 DeïT-S
4.1G
78.4
9.9
51.2
39.2
27.7
4.6G
79.8
16.2
42.8
41.9
29.1
ResNet-DW
4.8G
79.7
10.6
50.1
41.3
29.1
P16 + K11
4.5G
79.0
18.8
42.5
45.2
32.4
4.6G
80.0
10.8
47.7
41.9
29.4
P16 + K7
4.6G
79.2
19.8
41.1
46.4
33.7
ResNet-Upp Ses----veeeee---- -WW P16 KII
4.7G
79.7
12.9
47.9
42.9
30.8
4.4G
78.0
17.2
43.7
47.1
31.2
ResNet-Down. Doanee-DDww---eertted Inverted-DW P16 KIl
4.5G
79.6
12.1
49.0
42.5
29.5
4.6G
78.1
18.9
43.7
42.9
29.6
A.3 MORE COMPONENTS COMBINATION EXPERIMENTS
Here show more results of combining different components proposed in this paper in Tab. 9, Tab. 10, and Tab. 11.
A.4 MORE COMPARISON WITH OTHER MODELS
Besides DeiT, here we also evaluate our proposed Robust-ResNer against two state-of-the-art mod- els, ConvNeXt (Liu al., 2022) and Wwin-Transformer (Liu al., 2021), in terms of out-of- distribution robustness. As shown in Tab. 12, all four of our models perform similarly to, or better than ConvNeXt or Swin- Woin-Trrasformer all -ntto-ddistriuttionn tests.
A.5 REPEATING EXPERIMENTS
To demonstrate the statistical significance of the robustness improvements achieved by our proposed components, conduct three runs with different random seeds and report the mean and standard deviation Tab. 13. observe only small variations across the three runs, which confirms the consistent and reliable performance gains achieved by our proposed models.
13

---

Published as a conference paper at ICLR 2023
Table 10: The performance of different baseline models equipped with ViT-style patchify stem and fewer activation and normalization layers. The model with the suffix "PX" refers to that it has a ViT- style patchify stem with patch size set to X. "NormXActY" refers that the model has only one normalization layer after the Xth convolution layer and one activation layer after the Yth convolution layer in the block.
Architecture
FLOPs
(1)
S-IN (t)
())
IN-R (t)
IN-SK (T)
ResNet50 DeïT-S
4.1G
78.4
9.9
$1.2
39.2
27.7
4.6G
79.8
16.2
42.8
41.9
29.1
ResNet-DW P16 + Norm Act3
4.8G
79.7
10.6
50.1
41.3
29.1
4.6G
78.7
16.2
43.0
46.3
33.1
ResNet-Inverted-D DW
4.6G
80.0
10.8
47.7
41.9
29.4
P16 + Norm Act!
4.6G
79.3
17.5
41.7
46.1
32.9
ResNet- DesNet-UUpltweee Up-Inverted DW
4.7G
12.9
47.9
42.9
30.8
+ P16 + NormlAct.
4.5G
79.1
18.6
41.5
48.0
33.8
SeaNet-Down-Inerteee
4.5G
12.1
49.0
42.5
29.5
+ P16 + NormlActl
4.6G
79.7
18.0
40.7
47.1
33.4
Table 11: The performance of different baseline models equipped with large kernel size and fewer activation and normalization layers. "KX" refers to that the model uses blocks with depth-wise convolution layer with kernel size X. "NormXActY" refers to the model has only normalization layer after the Xth convolution layer and one activation layer after the Yth convolution layer in
block.
Architecture
FLOPs
(t)
S-IN (t)
(N)
IN-R (t)
IN-SK (T)
ResNet50
4.1G
78.4
9.9
$1.2
39.2
27.7
DeiT-S
4.6G
79.8
16.2
42.8
41.9
29.1
ResNet-DW
4.8G
79.7
10.6
50.1
41.3
29.1
+ K11 + NormlAct Act3
5.0G
13.3
45.2
43.9
30.8
ResNet-InvertedD --W
4.6G
80.0
10.8
47.7
41.9
29.4
+ K7 Norm Act1
5.0G
13.1
46.7
45.7
32.1
ResNet- Unn-et--p--nvertedd -DW
4.7G
12.9
47.9
42.9
30.8
+ K1l + NormlAct
4.9G
81.4
15.8
42.9
48.3
33.8
BeaNet-Down-nverrtee
4.5G
79.6
12.1
49.0
42.5
29.5
+ KIl + AAct
4.6G
81.3
14.2
43.6
46.1
32.8
Table 12:
Robustness comparison against two state-of-the-art models, ConvNeXT and Swin-
Transformer.
Architecture
FLOPs
IN (1)
S-IN (1)
IN-C (1)
IN-R (t)
IN-SK (1)
ConvNeXt-T
4.5G
82.1
18.9
41.6
47.2
33.9
Swin-T
4.5G
81.4
15.5
48.4
41.3
29.3
Robust-ResNet-DW
4.5G
79.4
42.3
45.9
33.0
Robust-ResNNe--leetee..DD
4.6G
79.0
19.5
42.1
45.9
32.8
Robust- Robust-ResNet- -oww-eee--
4.4G
79.2
20.2
40.9
48.7
35.2
4.6G
79.9
19.3
41.6
46.0
32.8
A.6 IMAGENET-A EVALUATION
The ImageNet-A dataset comprises a set of natural adversarial samples that have considerable negative impact on the performance machine learning models. In Tab. 14, we compare the per- formance of Robust-ResNV models and DeiT the ImageNet-A dataset. Notably, while the Robust-ResNer models do not perform as well as DeiT with an input resolution of 224, increasing the input resolution (e.g., to 320) significantly narrows the gap between Robust-ResNet and Dei'T
14

---

Published as a conference paper at ICLR 2023
Table 13: The results of repeating experiments three times with different random seeds. It can be observed that all Robust-ResNet variants bring statistically stable robustness improvement.
Architecture
FL.OPs
IN (1)
S-IN (
(N-C(D)
IN-R (1)
IN-SK (*)
ResNet50
4.1G
78.4
9.9
51.2
39.2
27.7
DeiT-S
4.6G
79.8
16.2
42.8
41.9
29.1
ResNet-DW
4.8G
79.7
10.6
50.1
41.3
29.1
Robust-ReSNet-DW
4.5G
79.3=
19.2:0.66
41.7=20.6
46.0::0.1
3..6:10.3
ResNat-Inverted-DDW
4.6G
80.0
10.8
47.7
41.9
29.4
4.6G
79.2+0022
19.8+0.6
41.9
45.8+
02.6+0.1
ResNet-Up-lnverree
4.7G
79.7
12.9
47.9
42.9
30.8
Robust-ResNet RobusttResNeet-Uppllveeeeee DW
4.4G
79.3=1.11
19.6:10.4
41.2:10.4
48.740.1
35.0
DeeNNet-Dow--Livereed-----
4.5G
42.5
29.5
Robust- Reve
4.6G
79.7=0.1
19.5+0.2
41.8+0.2
46.0+0.1
33.0+H.1
Table 14: Robustness comparison ImageNet-A.
Resolution
Model
224
320
ResNet50
7.0
15.5
DeiT-S
19.9
23.6
Robust-ResNet-DD
12.3
19.9
Robust-ResNNc RessNet-Inverted-DD
11.8
22.3
Robust- ReeuuttReNeet-UUe -nneeee-e-----ede-------
12.5
19.0
Robust ResNNet-DonnInvereee DW
14.0
21.2
on ImageNet-A. We conjecture this is because objects of interest in ImageNet-A tend to be smaller than in standard ImageNet.
A.7 STRUCTURAL RE-PARRAMETERIZATIOO
A line of recent works promotes the idea of training-time multi-branch but inference-tine plain model architecture via structural re-parameterization (Ding et al., 2021; 2022a;b). RepLKNet (Ding al., 2022h), in particular, has shown that re-parameterization using small kernels alleviate the optimization issue associated with large-kernel convolution layers, without incurring additional inference costs. Given that the Robust-ResNet models also use large kernels, here we experiment with the idea of structural ce-parameterization and leverage training-time multi-branch block ar- chitecture to further enhance model performance. The block architectures are detailed in Fig. 5. The results of two different model scales, shown in Tab. 15 and Tab. 16, demonstrate generally improved performance with this re- ce-paaametrrization approach. One exception may be Sebbust-Resee--UD Up- Inverted-DW, which occasionally exhibits slightly worse robustness with re-parameterization. No- tably, with the re-parameterization technique, we are able to train the Robust- Robust-ResNet ResNee-Innerred--D model using kernel size of 11.
15

---

Published as a conference paper at ICLR 2023
1x1, 384 :96
1x4, 384-46
16 96-3884
1xl. 96 :384
BN
BN RoLU
dlIIiL 96-396
611xl) 364-5094
BN
BN
1xl. 96-3391
1xL, 96-3884
1cl, 384-596
3xl. 384-065
00 ReatenDDW (Taart
IM Ramhee-2 (Two
Rested resttd DWN (Toint
1 Restec inverted DWN (Tes))
1el, 96-484
ellvil
HB6
BIN
BNLReLUU
1xl, 384.46
ReLu
d11x11 96
d3x3, 96
1xL, 286
Tel, 16k-66
BNN
ditxni 96-266
iwattedo
nestedioo
Figure 5: Architectures of four robust CNN block instantiations with structural reparameterization.
Table 15:
The performance of Robust-ResNet models trained with structural re-parameterization
technique at the scale of DeiT-S.
Architecture
FLOPs
IN (t)
S-IN (t)
IN-C (
IN-R (t)
IN-SK (t)
ResNet50 DeiT-S
4.1G
78.4
9.9
51.2
39.2
27.7
4.6G
79.8
16.2
42.8
41.9
29.1
Robust-ResNet-DW
4.5G
79.4
18.6
42.3
45.9
33.0
+ Re-param
4.5G
79.6
19.0
41.1
47.8
34.2
Robust ResNet-Inverted-DV -D
4.6G
79.0
19.5
42.1
45.9
32.8
+ Re-param
4.6G
79.2
20.1
42.0
47.2
34.4
Robust CresNNet-UpInneertte --W DW
4.4G
79.2
20.2
40.9
48.7
35.2
Re-param
4.4G
79.4
18.5
41.4
48.7
34.6
Robust-ResNet DeesNee-Down-Inveee Inverted-DW -DW
4.6G
79.9
19.3
41.6
46.0
32.8
+ Re-param
4.6G
80.7
21.2
38.1
49.0
34.9
Table 16: The performance of Robust-ResNet models trained with structural re-parameterization technique at the scale of DeiT-B.
Architecture
FLOPs
IN (f)
S-IN (t)
IN-C (L)
IN-R (f)
IN-SK (1)
ResNet-200 DeřT-B
15.1G
81.7
13.8
42.2
45.1
34.5
17.6G
81.8
19.6
37.9
44.7
32.0
Robust- Resbust-Rs Re-param
17.2G
80.5
19.2
38.8
46.6
34.2
17.2G
80.8
20.1
38.6
47.5
35.4
Robust Inverre- -DW-B
17.1G
81.6
22.1
35.5
50.2
36.7
+ Re- Re-param
17.1G
82.1
23.1
35.4
50.7
37.4
Robust-ResNet Up- Selp-Inveeree-DD DW-B
17.1G
81.9
23.1
35.1
50.5
37.1
+ Re-param
17.1G
82.2
22.9
34.5
50.6
37.8
Robust ResNel-Dower Lowonn-nveeted--ee DW-B
17.3G
81.3
22.2
35.6
49.7
37.8
+ Re-param
17.3G
81.6
22.0
34.9
51.1
38.1
16


Reviews:
[{"review_id": "idQOImc-vk", "reviewer": "Reviewer_ntqh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper seeks to understand architecture changes to traditional CNNs that render them as, or more, robust as Transformers when applied to out of distribution data. The main conclusions of the paper are that employing (a) an initial partition of the image into non-overlapping patches; (b) larger kernel sizes in a resnet block; and (c) having fewer non-linear layers (i.e., with normalizations and activations), can improve robustness.", "summary_of_the_review": "Overall, this is an interesting empirical analysis of how architecture design choices can affect robustness, and specifically, of the elements of the Transformer architecture that can be 'back-ported' to CNNs to increase the latter's robustness.\n\nHowever, since the main contribution is this empirical analysis, there is more of a burden for it to be complete, and as mentioned above, it is lacking in several respects.\n\n### Post Rebuttal\n\nThe authors have provided a detailed response with several additional experiments that address all my concerns. I believe the paper should be accepted (I'd give it a 7 instead of 8, but since that's no longer an option, I believe it's closer to 8 than 6).", "strength_and_weaknesses": "**Strengths**\n\n- The paper conducts a thorough empirical evaluation in terms of different combination of the main design choices.\n- It is interesting to look at the effect of architecture choices not just on accuracy but also robustness.\n\n**Weaknesses**\n\n- Most of the underlying design choices themselves have appeared in previous papers. As authors note, their work is very close to the ConvNeXT work, with the main difference being that their focus is on robustness.\n- I missed an evaluation of ConvNeXT itself in the various tables. While the ConvNeXT paper did not focus on robustness, is it the case that the ConvNeXT models are also just as robust as the best models proposed in the paper? If so, then this paper's analysis would still be useful, but the narrative would need to be re-cast as understanding the robustness of ConvNeXT vs ViT.\n- The paper lists accuracy numbers but without error-bars. It is difficult to gauge if the differences above are statistically significant. It would be worth repeating each experiment 3-5 times (at least for the -S models) and report both mean and standard deviation.\n- The tables compare models in terms of FLOPs, but not in terms of number of parameters. Do the proposed models have a larger or smaller number of parameters? How does the accuracy compare when we compare to models with equivalent numbers of parameters? Again, it is reasonable to want models with equivalent FLOPs, but the #parameters analysis could reveal an alternate hypothesis for what makes the architectures robust (viz, number of parameters).\n- The paper also misses an evaluation of adversarial robustness --- in terms of adversarial perturbations and to Imagenet-A (the so called natural adversarial examples). Given that the paper's contribution is largely empirical, these comparisons are needed for completeness. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "PZ9u5PV9nGW", "reviewer": "Reviewer_rxfk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed three modifications for convolutional network to make it more robust: 1) split the input image into non-overlapping image patches; 2) use large kernel size; 3) remove some BN and ReLU layer. Numerical experiments on several ImageNet variant datasets validate the robustness improvement with slight degradation of the accuracy on clean datasets.\n", "summary_of_the_review": "This work lacks novelty in theory. The proposed method is not universal and sometimes even degrade the performance dramatically.", "strength_and_weaknesses": "[Strength]\n\n* The proposed method is able to improve the robustness of CNN models and make them comparable to DeiT-S in terms of robustness.\n* The proposed method is validated on four popular ImageNet variants for robustness evaluation\n* Experiments are reported in great detail, with sufficient ablation studies.\n\n[Weakness]\n\n* The paper is mostly empirical and lacks theoretical motivation or guarantee. The authors simply report the accuracy increment or decrement without further investigation. So it is a good-to-know fact but cannot inspire in-depth understanding of the robustness of CNN.\n\n* The numerical experiments suggest that, the proposed three modifications, are not universal. Sometimes they can improve accuracies on ImageNet variants but sometimes dramatic performance drop can happen. What is worse, the patterns of good / bad cases are random. So the proposed method cannot transfer well. For a new given dataset, it is hard to tell whether we should apply the proposed method, or how we should choose the pattern in BN/ReLU removal.\n\n* The proposed method will hurt the performance on clean datasets most of the time. This is a hard trade-off in practice.\n\n* The robustness improvement over DeiT is not very significant for some networks and some datasets. Sometimes it is worse than DeiT (Table 1).\n\n* Only DeiT is selected for comparison. Why DeiT? What about Swin-Transformer or other robust VIT models?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6xm9EWeIWA", "reviewer": "Reviewer_DpNa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "* **Motivation**: This paper takes a deep dive into why Vision Transformer (ViT) networks have empirically been found to be more robust than Convolutional Neural Networks (as evaluated by performance on out-of-distribution samples).\n\n* **Approach**: They isolate some of the key components in DeiT that is missing in CNNs (besides the obvious attention mechanism), and do A/B experiments to figure out which components contribute the most to the differences in robustness. The components they evaluate are:\n1. Patchifying Stem: DeiT partitions images into PxP non-overlapping patches. The authors replicate this in CNNs by applying non-overlapping convolutions.\n2. Large Kernel Size: DeiT has better global receptive field. The authors replicate this in CNNs by increasing kernel size.\n3. Reducing Activation and Normalization Layers: DeiT has fewer activation and normalization layers than a typical ResNet block. The authors replicate this by reducing #activation/normalization layers in the ResNet block.\nThe authors call the model with above 3 modifications a **Robust-ResNet**.\n\n* **Findings**:\n1. Patchifying has the most noticeable impact in improving robustness\n2. Robust-ResNet is very similar in terms of #FLOPs and clean accuracy to both a DeiT and the (unmodified) ResNet. However, whereas the (unmodified) ResNet had noticeably worse robustness performance compared to DeiT, Robust-ResNet is able to achieve robustness results that are even slightly better than DeiT.\n3. Whereas ResNet students (in model distillation) struggle in terms of robustness when the teacher is a ViT, Robust-ResNets seem to outperform the ViT teacher in terms of robustness.\n4. Improved robustness seems to typically come at a cost to (clean) accuracy - E.g. Table 2", "summary_of_the_review": "I recommend for this paper to be accepted because:\n1. The strengths outlined above outweigh the weaknesses\n2. I think the findings & experiments would be interesting to the community\n3. This type of paper (A/B comparison of different components) helps in shining some light on the underlying workings of neural networks so that we are not just random searching for better architectures.", "strength_and_weaknesses": "**Strengths**\n1. Well written with an easy to follow flow\n2. I like the overall thought behind the paper of trying to figure out which components in ViTs have what amount of importance in getting better robustness. This type of paper shines some light on making better architectural decisions and understanding the effects of each component. Well done!\n3. Experiments are well organized and easy to extract information from.\n\n**Weaknesses**\n1. Table 7 seems incomplete without results from unmodified ResNet.\n2. The paper shows what matters through empirical evidence (and the experiments are interesting on their own), but it doesn't seem to provide any explanation/potential hypothesis about why patchifying/large kernel size/reducing #activation/normalization layers could improve robustness.\n\n**Questions**\n1. Table 4 description: What is meant by \"It is observed that the optimal choice always lead to best robustness.\" I don't see that pattern.\n2. All tables: How did you decide which row to highlight?\n3. Is patchifying stem (i.e. non-overlapping convolution) only done in the first convolution operation of Robust-ResNet?\n4. page 6 last paragraph: why such a drastic speed up from removing activation/normaliaztion layers?\n\n**Typos**\n1. page 2, paragraph 1: due to less normalization layers are used -> due to less normalization layers being used\n2. page 3, paragraph 1: Transformers are inherently much more robustness than CNNs -> Transformers are inherently much more robust than CNNs \n3. page 3, paragraph 1: last sentence was hard to read (\"Our work is a direct follow-up...\")\n4. page 3, \"Settings\" section,  paragraph 2: in whichthe hidden -> in which the hidden \n5. page 3, last paragraph: FLOating Point operations per second -> FLoating-point Operations Per Second\n6. page 6, last paragraph: speep up training -> speed up training\n7. page 7, table 3 description: block is keepped -> block is kept\n8. page 7, table 4: always lead to -> always leads to\n9. page 8, \"Components Combination\" section:  16×16 patchify stem with patch size ->  16×16 patchify stem with patch size 16\n10. page 9, section 7: still performs favorably -> still perform favorably", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster