SlowGuess/NeurIPS_2023_OCR huggingface dataset:
though it has a very low FLOPs (0.22B), its GPU latency is 7.83, which is even larger than our VanillaNet-13 with a 11.9B FLOPs. In fact, the inference speed in this setting is highly related to the complexity and number of layers. We can compare the inference speed of ShuffleNetV2x1.5 and ShuffleNetV2x2. In fact, their difference only lies in the number of channels. Therefore, although their number of parameters and FLOPs differs a lot. (0.3B v.s. 0.6B), their inference speed is nearly the same (7.23 and 7.84). We can also find in Table 1 that the straightforward architecture including ResNet, VGGNet and our VanillaNet without extra branch and complex blocks (e.g., squeeze and excitation block or densely connects) achieves the highest inference speed.
To this end, we propose the VanillaNet, which is simple and has few convolutional layers without any branch (even without shortcut). We set different number of layers in VanillaNets to construct a series of networks. As shown in Table 1, the VanillaNet-9 achieves a  $79.87\%$  accuracy with only a 2.91ms inference speed in GPU, which is over  $50\%$  faster than the ResNet-50 and ConvNextV2-P with similar performance. The surprising result demonstrate the potential of VanillaNet in real-time processing over the existing deep networks. We also scale the number of channels and the pooling size to obtain the proposed VanillaNet-13-1.5  $\times \dagger$ , which achieves an  $83.11\%$  Top-1 accuracy on ImageNet, which suggests that the proposed vanilla neural network still have power to obtain such a high performance on large scale image classification task. It is suggested that we may not need deep and complex networks on image classification since scaling up VanillaNets can achieve similar performance with deep networks. Although the proposed VanillaNet requires much more FLOPs and parameters, its inference speed in GPUs is much faster. In fact, the inference speed in GPUs and the number of parameters or FLOPs are two different aspects to measure the performance of neural networks. According to the "no free lunch" rule, it is nearly impossible to be satisfactory to both sides. Therefore, the proposed VanillaNet aims to the inference speed in GPUs and achieve much more higher speed then existing SOTA architectures.
The Figure 2 shows the depth and inference speed of different architectures. The inference speed with batch size 1 is highly related to the depth of the network instead of the number of parameters, which suggest that simple and shallow networks have huge potential in real-time processing. It can be easily find that the proposed VanillaNet achieve the best speed-accuracy trade-off among all these architectures with low GPU latency, which demonstrates the superiority of the proposed VanillaNet when the computing power is sufficient.
In this section, we conduct ablation study to investigate the effectiveness of proposed modules, including the series activation function and the deep training technique. Besides, we analyze the influence of adding shortcuts in the proposed VanillaNet.
Influence of number of series in activation function. In the above section, we propose a series activation function to enhance the performance of plain activation function and enable global information exchange in feature maps. Table 2 shows the performance of the proposed VanillaNet using different number of  $n$  in Equation 6. When  $n = 0$ , the activation function degenerate into the plain ReLU activation function. Although the inference speed of this network is
higher than using the series activation function, the network can only achieve a  $60.53\%$  top-1 accuracy on the ImageNet dataset, which cannot be applied in real-world applications. It proves that the poor non-linearity of activation function results in poor performance of vanilla networks.
To this end, we propose the series activation function. When  $n = 1$ , the network can achieve a  $74.53\%$  accuracy, which is a huge improvement compared with  $60.53\%$ . The result demonstrates the effectiveness of the proposed activation function. When the number of  $n$  increases, the performance of the network brings up. We find that  $n = 3$  is a good balance in the top-1 accuracy and latency. Therefore, we use  $n = 3$  for the rest experiments. It should be noted that the FLOPs of the proposed activation function is very small compared with the original network, which is the same as the conclusion we derive in Equation 9.
Influence of deep training. As the VanillaNet is very shallow, we propose to increase the training nonlinearity to bring up its performance. We then analyze the effectiveness of the proposed deep training technique. Table 3 shows the results on using deep training technique with VanillaNet-6. As a result, the original VanillaNet achieves a  $75.23\%$  top-1 accuracy, which is the baseline. By using the deep training technique, the proposed VanillaNet can achieve a  $76.36\%$  accuracy. The results demonstrate that the proposed deep training technique is useful for the shallow network.
Table 3: Ablation study on different networks.
Network	Deep train.	Series act.	Top-1 (%)
VanillaNet-6	✓	59.58
60.53
✓	75.23
✓	✓	76.36
AlexNet	✓	57.52
59.09
✓	61.12
✓	✓	63.59
ResNet-50	✓	76.13
76.16
✓	76.30
✓	✓	76.27
Moreover, we further apply the deep
training and series activation function in other networks to show the generalization ability of the two techniques. Table 3 reports the results of AlexNet and ResNet-50, which are two classical deep neural networks, on the ImageNet dataset. The original AlexNet can only achieve a  $57.52\%$  accuracy with 12 layers. By applying the proposed deep training and series activation function, the performance of AlexNet can be largely brought up by about  $6\%$ , which demonstrates that the proposed technique is highly effective for shallow networks. When it turns to ResNet-50 whose architecture are relatively complex, the performance gain is little. This results suggests the deep and complex networks already have enough non-linearity without the proposed techniques.
Influence of shortcuts. In deep neural networks, a common sense is that adding shortcut can largely ease the training procedure and improve the performance [23]. To this end, we investigate whether shortcut would benefit the performance of shallow and simple network. We propose to use two kinds of location of shortcut, i.e., shortcut after the activation function and shortcut before the activation function, which are proposed in the original ResNet [23] and PreAct-ResNet [24], respectively. Since the number
Table 4: Ablation on adding shortcuts.
Type	Top-1 (%)
no shortcut	76.36
shortcut before act	75.92
shortcut after act	75.72
of channels is large and the original convolution is with kernel size  $1 \times 1$  in VanillaNet, adding a shortcut (even with  $1 \times 1$  kernel size) would largely increase the FLOPs. Therefore, we use the
parameter-free shortcut. It should be noted that if the stride is 2, the parameter-free shortcut will use an average pooling to decrease the size of feature maps and if the number of channel is increasing, the parameter-free shortcut utilizes padding for the extra channels following the original setting in [23].
Table 4 shows the ablation study on adding shortcuts. We surprisingly find that using shortcuts, in spite of any type of shortcuts, has little improvement on the performance of the proposed VanillaNet. We suggest that the bottleneck of shallow networks is not the identity mapping, but the weak non-linearity. As reported in ResNet [23], the error rate of 18 layers' plain network (without shortcut) is  $27.94\%$ , which is similar to 18 layers' ResNet  $(27.88\%)$ . This result also suggests that shortcuts are not necessary for shallow networks. In fact, the shortcut is useless for bringing up the non-linearity and may decrease non-linearity since the shortcut skips the activation function to decrease the depth of the vanilla network, therefore results in lower performance.
Performance of transfer learning. To further demonstrate the generalization ability of the proposed VanillaNet, we fine-tune the proposed VanillaNet (pre-trained on ImageNet) on the well-known long-tailed classification dataset: iNaturalist 2018 [49] and follow the setting in Deit [48]. As shown in the Table 5, the performance of VanillaNet in pretraining scenarios is similar with other methods, which demonstrates its universality.
Table 5: Accuracy  $(\%)$  on ImageNet and iNaturalist 2018.
Model	ImageNet	iNat18
ResNet-50	79.6	69.8
VanillaNet-9	72.5	72.5
DeiT-B	81.8	73.2
VanillaNet-12	81.6	73.6
4.3 Experiments on COCO
Table 6: Performance on COCO detection and segmentation. FLOPs are calculated with image size (1280, 800)on Nvidia A100 GPU.
Framework	Backbone	FLOPs	Params	FPS	\( AP^b \)	\( AP_{50}^b \)	\( AP_{75}^b \)	\( AP^m \)	\( AP_{50}^m \)	\( AP_{75}^m \)
RetinaNet [34]	Swin-T [36]	245G	38.5M	27.5	41.5	62.1	44.2	-	-	-
VanillaNet-13-D	397G	75.4M	29.8	43.0	62.8	44.3	-	-	-
Mask RCNN [21]	Swin-T [36]	267G	47.8M	28.2	42.7	65.2	46.8	39.3	62.2	42.2
ConvNextV2-N [53]	221G	35.2M	34.4	42.9	65.5	46.9	39.6	62.5	42.2
VanillaNet-13-D	421G	77.1M	32.6	42.9	65.5	46.9	39.6	62.5	42.2
To further demonstrate the effectiveness of the proposed VanillaNet on downstream tasks, we conduct evaluation in the COCO dataset [35]. We use RetinaNet [34] and Mask-RCNN [21] as the framework to evaluate the proposed method. FPS is measured on Nvidia A100 GPU. Details can be found in supplementary materials.
Table 6 shows the performance of the proposed VanillaNet on COCO detection and segmentation. The proposed VanillaNet can successfully achieve similar performance with the ConvNext and the Swin backbone. Although the FLOPs and Parameters of VanillaNet is much higher than Swin and ConvNext, it has much higher FPS, which demonstrates the effectiveness of vanilla architectures on object detection and instance segmentation tasks.
This paper fully investigates the feasibility of establishing neural networks with high performance but without complex architectures such as shortcut, high depth and attention layers, which embodies the paradigm shift towards simplicity and elegance in design. We present a deep training strategy and the series activation function for VanillaNets to enhance its non-linearity during both the training and testing procedures and bring up its performance. Experimental results on large scale image classification datasets reveal that VanillaNet performs on par with well-known deep neural networks and vision transformers, thus highlighting the potential of minimalism in deep learning. We will further explore better parameter allocation for efficient VanillaNet architectures with high performance. In summary, we prove that it is possible to achieve comparable performance with the state-of-the-art deep networks and vision transformers using a very concise architecture, which will unlock the potential of vanilla convolutional network in the future.
[58] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019.
[59] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[60] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104-12113, 2022.
[61] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
[62] Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond. International Journal of Computer Vision, 2023.
A Network Architectures
The detailed architecture for VanillaNet with 7-13 layers can be found in Table 7, where each convolutional layer is followed with an activation function. For the VanillaNet-13-1.5×, the number of channels are multiplied with 1.5. For the VanillaNet-13-1.5×†, we further use adaptive pooling for stage 2,3 and 4 with feature shape  $40 \times 40$ ,  $20 \times 20$  and  $10 \times 10$ , respectively.
Input	VanillaNet-5	VanillaNet-6	VanillaNet-7/8/9/10/11/12/13
stem	224×224	4×4, 512, stride 4
stage1	56×56	[1×1, 1024]×1 MaxPool 2×2	[1×1, 1024]×1 MaxPool 2×2	[1×1, 1024]×2 MaxPool 2×2
stage2	28×28	[1×1, 2048]×1 MaxPool 2×2	[1×1, 2048]×1 MaxPool 2×2	[1×1, 2048]×1 MaxPool 2×2
stage3	14×14	[1×1, 4096]×1 MaxPool 2×2	[1×1, 4096]×1 MaxPool 2×2	[1×1, 4096]×1/2/3/4/5/6/7 MaxPool 2×2
stage4	7×7	-	[1×1, 4096]×1	[1×1, 4096]×1
classifier	7×7	AvgPool 7×7 1×1, 1000
For classification on ImageNet, we train the VanillaNets for 300 epochs utilizing the cosine learning rate decay [39]. The  $\lambda$  in Equ. 1 is linearly decayed from 1 to 0 on epoch 0 and 100, respectively. The training details can be found in Table 8. For the VanillaNet-11, since the training difficulty is relative large, we use the pre-trained weight from the VanillaNet-10 as its initialization. The same technique is adopted for VanillaNet-12/13.
For detection and segmentation on COCO, we train all the networks using 12 epochs, multi-scale training augmentation and a linear learning rate decay for fair comparison. Following ConvNextV2 [53] which utilize self-supervised training, we use the ImageNet pre-trained weight using knowledge distillation [20] with  $n = 4$  for a higher receptive field. We train the VanillaNet-13 using the Adamw optimizer with a batch size of 32, an initial learning rate of 8e-5 for RetinaNet and 1.3e-4 for Mask RCNN, an 0.05 weight decay and an 0.6 layer wise decay.
Table 7: Detailed architecture specifications.
Training Config	VanillaNet-{5/6/7/8/9/10/11/12/13}
weight init	trunc. normal (0.2)
optimizer	LAMB [58]
loss function	BCE loss
base learning rate	3.5e-3{5,8-13}/4.8e-3{6-7}
weight decay	0.35/0.35/0.35/0.3/0.3/0.25/0.3/0.3/0.3
optimizer momentum	β1, β2=0.9, 0.999
batch size	1024
training epochs	300
learning rate schedule	cosine decay
warmup epochs	5
warmup schedule	linear
dropout	0.05
layer-wise lr decay [6, 1]	0{5,8-12}/0.8{6-7,13}
randaugment [7]	(7, 0.5)
mixup [61]	0.1/0.15/0.4/0.4/0.4/0.4/0.8/0.8/0.8
cutmix [59]	1.0
color jitter	0.4
label smoothing [46]	0.1
exp. mov. avg. (EMA) [41]	0.999996{5-10}/0.999992{11-13}
test crop ratio	0.875{5-11}/0.95{12-13}
Table 8: ImageNet-1K training settings.
(a)Mis-classified by ResNet-50-TNR
(c)Mis-classified by VanillaNet-9
Figure 3: Visualization of attention maps of the classified samples by ResNet-50 and VanillaNet-9. We show the attention maps of their mis-classified samples and correctly classified samples for comparison.
(b)Correctly classified by ResNet-50-TNR
(d)Correctly classified by VanillaNet-9
C Visualization of Attention
To have a better understanding of the proposed VanillaNet, we further visualize the features using GradCam++ [4], which utilizes a weighted combination of the positive partial derivatives of the feature maps generated by the last convolutional layer with respect to the specific class to generate a good visual explanation.
Figure 3 shows the visualization results for VanillaNet-9 and ResNets-50-TNR [52] with similar performance. The red color denotes that there are high activation in this region while the blue color denotes the weak activation for the predicted class. We can find that these two networks have different attention maps for different samples. It can be easily found that for ResNet-50, the area of active region is smaller. For the VanillaNet with only 9 depth, the active region is much larger than that of deep networks. We suggest that VanillaNet may be strong in extract all relative activations in the input images and thoroughly extract their information by using large number of parameters and FLOPs. In contrast, VanillaNet may be weak on analyzing part of the useful region since the non-linearity is relatively low.