VanillaNet: the Power of Minimalism in Deep Learning
Authors: Hanting Chen
Yunhe Wang
Jianyuan Guo
Dacheng Tao
Venue: NeurIPS 2023 poster (2023)
Keywords: computer vision
foundation models.

Abstract: At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at https://github.com/huawei-noah/VanillaNet and https://gitee.com/mindspore/models/tree/master/research/cv/vanillanet

VanillaNet: the Power of Minimalism in Deep Learning
Hanting Chen', Yunhe Wang'; Jianyuan Guo Dacheng Tao²
Huawei Noah's Ark Lab. School of Computer Science, University of Sydney.
{chenhanting yunhe. wang jianyuan. Saarruuannggu)ooaaee guo):Ouaveii.cmmmmmm com, dacheng. wawdsydeey.ay....ooooooooooo edu,
Abstract
At the heart of foundation models is the philosophy of "more is different", emplified by the astonishing success computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for paradigm shift towards simplicity. In this study, we introduce VanillaNet, neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self- attention, VanillaNet refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource- constrained environments. Its to--dderaadd and highly simplified architec- ture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant poten- tial to redefine the landscape and challenge the status quo of foundation model, setting new path for elegant and effective model design. Pre-trained models and codes are available at attppp::/ggihhhb...om/mseseeeeea and httphhts://// //te::////eeee com hww.a//nwwereoooaaa//eeee//arraa vanillanet
Introduction
Over the past few decades, artificial neural networks have made remarkable progress, driven by the idea that increasing network complexity leads improved performance. These networks, which consist of numerous layers with a large number of neurons or transformer blocks [50] 36), are capable of performing variety of human-like tasks, such as face recognition 30) speech recognition (9) object detection 43
17 [8], natural language processing [50], and content generation 3 The
impressive computational power of modern hardware allows neural networks to complete these tasks with both high accuracy and efficiency. As result, Al-embedded devices are becoming increasingly prevalent in our lives, including smartphones, AI cameras, voice assistants, and autonomous vehicles.
Admittedly, one notable breakthrough in this field is the development of AlexNet [29], which consists of 12 layers and achieves state-of-the-ar performance on the large-scale image recognition benchmark [8]. Building on this success, ResNet [23] introduces identity mappings through shortcut connections, enabling the training of deep neural networks with high performance across wide range of computer vision applications, such image classification [45), object detection 43), and
semantic segmentation [8). The incorporation of human-designed modules in these models, as well as the continued increase in network complexity, has undeniably enhanced the representational
ccoreepponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

224
58
224
56
14
512
1024
2048
4096
4096
1000
Figure The architecture of VanillaNet-6 model, which consists of only convolutional layers. which are very easily to be employed on any modern hardwares. The size of input features are downsampled while the channels are doubled in each stage, which borrows from the design of classical neural networks such as AlexNet (29 and VGGNet 45
capabilities of deep neural networks, leading to a surge of research on how to train networks with more complex architectures [8 24 54 to achieve even higher performance.
Apart from convolutional architectures, Dosovitskiy et al. 14] have introduced the transformer architecture to image recognition tasks, demonstrating its potential for leveraging large-scale training data. Zhai al. [60] investigated the scaling laws of vision transformer architectures, achieving an impressive 90.45% top-1 accuracy on the ImageNet dataset, which indicates that deeper transformer architectures, like convolutional networks, tend to exhibit better performance. Wang et al. [51 further proposed scaling the depth of transformers to I, 1,000 layers for even higher accuracy. Liu et al. 37 revisited the design space of neural networks and introduced ConvNext, achieving similar performance to state-of-the-arr transformer architectures.
Although well-optimized deep and complex neural networks achieve satisfying performance, their increasing complexity poses challenges for deployment. For example, shortcut operations in ResNets consume significant off-chip memory traffic as they merge features from different layers [32]. Furthermore, complicated operations such as axial shift in AS-MLP [33] and shift window self- attention in Swin Transformer [36] require sophisticated engineering implementation, including rewriting CUDA codes.
These challenges call for a paradigm shift towards simplicity in neural network design. However, the development of ResNet has seemingly led to the abandonment of neural architectures with pure convolutional layers (without extra modules such shortcuts). This is mainly due to the performance enhancement achieved by adding convolutional layers not meeting expectations. As discussed plain networks without shorteuts suffer from gradient vanishing, causing 34-layer plain network to perform worse than an 18 18-layer one. Moreover, the performance of simpler networks like AlexNet 29 and VGGNet (45) has been largely outpaced by deep and complex networks, such as ResNets 23 and VIT [8]. Consequently, less attention has been paid to the design and optimization of neural networks with simple architectures. Addressing this issue and developing concise models with high performance would be of great value.
23
To this end, we propose VanillaNet, novel neural network architecture emphasizing the elegance and simplicity of design while retaining remarkable performance in computer vision tasks. VanillaNet achieves this by eschewing excessive depth, shortcuts, and intricate operations such as self-attention, leading to series of streamlined networks that address the issue of inherent complexity and are well- suited for seoource-limitee environments. To train our proposed VanillaNets, we conduct comprehensive analysis of the challenges associated with their simplified architectures and devise "deep training" strategy. This approach starts with several layers containing non-linear activation functions. As the training proceeds, we progressively eliminate these non-linear layers, allowing for easy merging while preserving inference speed. To augment the networks' non-linearity, we put forward an efficient, series-based activation function incorporating multiple Icarnable affine transfor-

---

mations. Applying these techniques has been demonstrated to significantly boost the performance of less complex neural networks. As illustrated in Figure VanillaNet surpasses contemporary networks with elaborate architectures concerning both efficiency and accuracy, highlighting the potential of minimalist approach in deep learning. This pioneering investigation VanillaNet paves the way for new direction in neural network design, challenging the established norms of foundation models and establishing a new trajectory for refined and effective model creation.
A Vanilla Neural Architecture
Over the past decades, researchers have reach some consensus in the basic design of neural networks. Most of the state-of-the-ar image classification network architectures should consist of three parts: a stem block to transform the input images from 3 channels into multiple channels with downsampling. a main body to learn useful information, a fully connect layer for classification outputs. The main body usually have four stages, where each stage is derived by stacking same blocks. After each stage, the channels of features will expand while the height and width will decrease. Different networks utilize and stack different kinds of blocks construct deep models.
Despite the success of existing deep networks, they utilize large number of complex layers to extract high-level features for the following tasks. For example, the well-known ResNet [23] requires 34 or 50 layers with shortcuts for achieving over 70% top-1 accuracy on ImageNet. The base version of VIT [4] consists of 62 layers since the query, key and value in self-attention require multiple layers to calculate.
With the growing of AI chips, the bottleneck of inference speed of neural networks would not be FLOPs or parameters, since modern GPUs can easily do parallel calculation with strong computing power. In contrast, their complex designs and large depths block their speed. To this end, we propose the vanilla network, Le., VanillaNet, whose architecture shown in Figure D We follow the popular design of neural network with the stem, main body and fully connect layer. Different with existing deep networks, we only employ one layer in each stage to establish a extremely simple network with as few layer as possible.
Here we show the architecture of the VanillaNet in details, which takes 6 layers as an example. For the stem, we utilize a 4 x 4 x 3 x C convolutional layer with stride 4 following the popular settings in [23 36 37 map the images with 3 channels features with C' channels. At stage 1, 2 and 3, maxpooling layer with stride 2 is used to decrease the size and feature map and the number of channels is increased by At stage 4, we not increasing the number of channels as it follows an average pooling layer. The last layer is fully connected layer to output the classification results. The kernel size of each convolutional layer is X 1, since we aim to use minimal calculation cost for each layer while keep the information of feature maps. The activation function is applied after each 1 x convolutional layer. To ease the training procedure of the network, batch normalization is also added after each layer. For the VanillaNet with different number of layers, we add blocks in each stage, which will be detailed in the supplementary material. should be noted that the VanillaNet has shortcut, since empirically find adding shorteut shows little performance improvement. This also gives another benefit that the proposed architecture is extremely easy to implemented since there are no branch and extra blocks such as squeeze and excitation block 277.
Although the architecture of VanillaNet is simple and relatively shallow, its weak non-linearity caused limit the performance, Therefore, we propose series of techniques to solve the problem.
3 Training of Vanilla Networks
is common in deep learning to enhance the performance of models by introducing stronger capacity in the training phase [7). To this end, propose to utilize deep training technique to bring up the ability during training in the proposed VanillaNet, since deep network have stronger non- non-linearity than shallow network.
3.1 Deep Training Strategy
The main idea of deep training strategy train two convolutional layers with an activation function instead of single convolution layer in the beginning of training procedure. The activation function is
3

---

gradually reduce to an identity mapping with the increasing number of training epochs. At the end of training. two convolutions can be easily merged into the one convolution to reduce the inference time, We then describe how to conduct this
This kind of idea is also widely used in CNNs I1 13 I0 12
technique in detail.
For an activation function A(x) (which can be the usual functions such ReLU and Tanh), we combine it with an identity mapping, which can be formulated as:
(x = (1 A)) Ar.
where À is a Denote the current epoch and the number of deep training epochs and E, respectively. We set A == Therefore, at the beginning of training (e = 0), A' = A(r), which means the network have strong non-linearity. When the training converged, we have A'(x) = x, which means the two convolutional layers have no activation functions in the middle. We further demonstrate how to merge these two convolutional layers.
to balance the non-linearity the modified activation function (x))).
We first convert every batch normalization layer and its preceding convolution into a single convo- lution, which is a commonly used technique. We denote W E Cout (C., xkxk), B E IR Rout as the weight and bias matrices of convolutional kernel with Can input channels, Cout output channels and kernel size k. The scale, shift, mean and variance in batch normalization are represented as y, B, p. a RCoue, respectively. The merged weight and bias matrices are:
W 74 W. B (B, 8,
1.)m d;
+ 8r,
where subscript i E 2,, ) denotes the value in i-th output ehannels.
After merging the convolution with batch normalization, we begin to merge the two 1 x convolutions. Denote x E RC HxW and y E R Cous x HI' xW' as the input and output features, the convolution can be formulated as:
W x W- im2col(x) W. XX,
(3)
where * denotes the convolution operation, - denotes the matrix multiplication and X E ( XXX is derived from the im2col operation to transform the input into a matrix corresponding the kernel shape. Fortunately, for convolution, we find that the im2col operation becomes simple reshape operation since there are no need for sliding kernels with overlap. Therefore, denote the weight matrix of two convolution layers as W1 and W2, the two convolution without activation function is formulated as:
y W (W2 x) W W im2col(2) (x) (W1 W2) X.
(4)
Therefore, x 1 convolution can merged without increasing the inference speed. It worth noticed that although the deep training technique will increase the training FLOPs, it would not affect the inference cost, which is much more important when implementation.
3.2 Series Informed Activation Function
There have been proposed several different activation functions for deep neural networks, including the most popular Rectified Linear Unit (ReLU) and its variants (PReLU [22), GeLU (25) and
Swish 42.. They focus on bring up the performance of deep and complex networks using different
activation functions. However, as theoretically proved by the existing works 40) 13 47), the limited
power of simple and shallow network are mainly caused by the poor non-linearity which different with deep and complex networks and thus has not been fully investigated.
In fact, there are two ways improve the non-linearity of neural network: stacking the non-linear activation layers or increase the non-linearity of each activation layer, while the trend of existing networks choose the former one, which results in high latency when the parallel computation ability is excess.
One straight forward idea to improve non-linearity of activation layer is stacking. The serially stacking of activation function is the key idea of deep networks. contrast, we turn to concurrently stacking the activation function. Denote single activation function for input in neural network as

---

A(z). which can be the usual functions such ReLU and Tanh. The concurrenlyy stacking of A(z) can be formulated as:
a. Q4A(r + b.), Ax(x) Σ
where n denotes the number of stacked activation function and a,, b, are the scale and bias (which are learned parameters) of each activation avoid simple accumulation. The non-linearity of the activation function can be largely enhanced by concurrently stacking, Equation can be regarded as series in mathematics, which is the operation of adding many quantities.
To further enrich the approximation ability of the series, we enable the series based function to learn the global information by varying the inputs from their neighbors, which is similar with BNET [56]. Specifically, given a input feature & E R #xxC where H, W and C' are the number of its width, height and channel, the activation function is formulated as:
Ag (Thhmme))) =
E
O.j.cA(Titt.ttu +be
(6)
(.j((-nn)
where h E (1, H). E 21, W) and (1,2...., C). It is easy to see that when n O, the series based activation function (x))) degenerates to the plain activation function A(x), which means that the proposed method can be regarded as general extension of existing activation functions, We use ReLU as the basic activation function to construct our series since it is efficient for inference in GPUs.
We further analyze the computation complexity the proposed activation function compared with its corresponding convolutional layer. For convolutional layer with K kernel size, Cin input channels and Cout output channels, the computational complexity
(COONV) = H x W x Cin x Cout x k2
while computation cost of its series activation layer is:
(SS)) = X W x Cout X (2n + 1)2.
(8)
Therefore, we have:
(COONV) H X W Cin Cost x K2 Cin k²
(9)
(SS)) x W x Cout x (2n - 1)2 (2n + 1)
Taking the 4th stage in VanillaNet-B as an example, where Cm = 2048, k = 1 and n 3, the ratio about 84. In conclusion, the computation cost of the proposed activation function is still much lower than the convolutional layers. More experimental complexity analysis will be shown in the following section.
is noted that the proposed SIAF is significantly different with BNET [66.. The motivation of BNET is to improve BN's flexibility of fitting complex data distributions, which is resolved by introducing linear transformation module with considering each neuron's neighborhood. In contrast, the motivation of the proposed SIAF is to enhance the non-linearity of shallow networks, which is implemented by combining multiple activation functions with different weights and biases. Empirically, shown in Table in paper, the SIAF only works for shallow networks yet with huge improvements (>4.0% on ImageNet), while BNET is general method with relatively little improvements 0.5% ImageNet).
4 Experiments
In this section, we conduct experiments to verify the performance of the proposed VanillaNet on large scale image classification. Ablation study is provided investigate effectiveness of each component of the proposed VanillaNet. We also visualize the feature of VanillaNet to further study how the proposed network learns from images.
4.1 Comparison with SOTA architectures
To illustrate the effectiveness of the proposed method, we conduct experiments on the ImageNet dataset, which consists of 224 x 224 pixel RGB color images. The ImageNet dataset contains 1.28M
5

---

Table I: Comparison ImageNet. The depth denotes the number of non-linear layers. The Reall Acc denotes the accuracy with the ImageNet Real label [2]. Latency is tested on Nvidia A GPU with batch size of 1. When comparing with other baselines, also apply the batch norm folding for fair comparison.
Model
Params (M)
FLOPs (B)
Depth
Latency (ms)
Ace (%)
Real Acc (%)
Mobile Small
26
2.5
0.06
6.65
67.67
74.33
MobilcNenV3-Large
26
5.5
0.22
7.83
74.04
80.01
ShuffleNe\V21.55
44
3.5
0.30
7.23
73.00
80.19
ShuffleNetV2x2 26
7.4
0.58
51
7.84
76.23
82.72
RepVGG-A0
13
8,1
1.36
3.22
72.41
79.33
RepVGG-A1
13
12.8
2.37
3.24
74.46
81.02
RepVGG-BO
R3
14.3
3.1
29
3.88
75.14
81.74
RepVGG-B3
110.9
26.2
29
4.21
80.50
86.44
VITAE-T
55
4.8
1.5
67
13.37
75.3
82.9
VITAE-S
23.6
5.6
116
22.13
82.0
87.0
ViTAEV2-S
62
19.2
130
24.53
82.6
87.6
ConwNextV2-A
531
3.7
6.07
82.79
ConwNextV2-F
5.2
0.78
41
6.17
78.0
84.08
ConvNextV2-PP
53
9.1
1.37
41
6.29
79.7
85.60
ConvNextV2-N
53
15.6
2.45
47
6.85
81.2
ConwNextV2-TT
28.6
4.47
8,40
82.5
ConvNextV2-B
[33
88.7
15.4
113
15.41
84.3
Swin-T
36
28.3
10.51
81.18
86.64
Swin-S
36
49.6
8.7
96
20.25
83.21
87.60
CMT-T 19
9.7
1.3
13.71
79.0
85.4
CMT-S
19
26.26
4.05
135
23.09
83.4
88.2
ResNet-18-TNR 52
11.7
1.8
3.12
70.6
79.4
ResNet-34-TNR
52
21.8
3.7
34
5,57
75.5
83.4
ResNet-50-TNN
52
25.6
4.1
50
7.64
79.8
85.7
VanillaNet-5
15.5
5.2
1.61
72.49
79.66
VanillaNct-6
32.5
6.0
2.01
76.36
82.86
VanillaNet-7
32.8
2.27
77.98
84.16
VanillaNet-8
37.1
7.7
2.56
79.13
85.14
VanillaNet-9
41.4
8.6
2.91
79.87
85.66
VanillaNet-11
45.7
9.4
10
3,24
80.57
86.25
VanillaNet-11
50.0
10.3
3.59
81.08
86.54
VanillaNet-12 12
54.3
11.1
12
3,82
81.55
86.81
VanillaNet-13
58.6
11.9
13
4.26
82.05
87.15
1aillaNNe-13-1155-
127.8
26.5
13
7.83
82.53
87.48
VanillaNet- 13-1.5x
127.8
48.9
9.72
83.11
87.85
training images and 50K validation images with 1000 categories. We utilize strong regularization since the proposed VanillaNet has large number of parameters in each layer to capture useful information from the images with limited non-linearity. We also report ImageNet Real results where the labels are refined. The latency is tested on Nvidia A 100 GPU.
We propose architecture for VanillaNet with different number of layers. Table Sableeshooe shows the classifi- cation results on the ImageNet dataset using different networks. We list the number of parameters, FLOPs, depth, GPU latency and accuracy for comparison. In the past decades, researchers focus on minimize the FLOPs or the latency in ARM/CPU for portable networks since they assume that the computing power in edge devices limited. As the development of modern AI chips, several mobile devices such as driverless vehicle [31] and robots [16] are required and able to carry multiple GPUs with huge computing power for seeking real-time feedback of external inputs. Therefore, we test the GPU latency with batch size 1, which means that the Al chip has enough computing power to calculate each network. Under this situation, we find that the inference speed has little relationship with the number of FLOPs and parameters. Taking MobileNetV3-Lart
example,

---

Vanillaict 13..5
Swir- VITAIIS
VandieNes-S11.0
Sais
Dy-ilaieee331111 Srx
VITIN2
ReyVEO 88
Rwtr-
Snin/
Racket SPITN Atr.
VantiaaNot
YanilaNar11 10, RepAGo
Roshoe (O SOTTNR
Accuracy (%)
ParNect
Accurasy (%)
OancNodi22
RitoteriNen BO ConehwaaV2
Vardieleet
VanilaNee
P
Sarlict
PrekGr-3177N
STAAT
VaniiaNer
Reckee
MIBAE
Replic Do
ViriliNNen5
Rop Al
VIGGI-ON
XobieNett2
VIDG23
DesNin TNR
10
Depth (in log scale)
GPU Latency with 1s=1 (ms)
(a) Accuracy v.s. depth
(b) Accuracy v.s. inference speed
Figure 2: Top-1 Accuracy on ImageNet v.s. inference speed Nvidia A100 GPU with batch size 1. Size of the circle is related to the depth and parameters of each architecture in (a) and (b), respectively. VanillaNet achieves comparable performance with deep neural networks while with much smaller depth and latency.
though it has very low FLOPs (0.22B), its GPU latency is 7.83, which is even larger than our VanillaNet-1 13 with a 11.9B FLOPs. In fact, the inference speed in this setting is highly related to the complexity and number of layers. We can compare the inference speed of ShufileNerV2x1 1.5 and Shuff In fact, their difference only lies in the number of channels. Therefore, although their number of parameters and FLOPSs differs lot. (0.3B v.s. 0.6B), their inference speed is nearly the same (7.23 and 7.84). We can also find in Table | that the straightforward architecture including ResNet, VGGNet and our VanillaNet without extra branch and complex blocks (e.g., squeeze and excitation block or densely connects) achieves the highest inference speed.
To this end, propose the VanillaNet, which is simple and has few convolutional layers without any branch (even without shortcut). We set different number of layers in VanillaNets to construct a series of networks. As shown in Table the VanillaNet-9 9 achieves a 79.87% accuracy with only 2.91ms inference speed in GPU, which iS over 50% faster than the ResNet-50 and ConyNextVV2PP with similar performance. The surprising result demonstrate the potential of VanillaNet in real-time processing over the existing deep networks. We also scale the number of channels and the pooling size to obtain the proposed -anillaNet-13-115xx ImageNet, which suggests that the proposed vanilla neural network still have power to obtain such high performance on large scale image classification task. It is suggested that we may not need deep and complex networks on image classification since scaling up VanillaNets can achieve similar performance with deep networks. Although the proposed VanillaNet requires much more FLOPs and parameters, its inference speed in GPUs is much faster. In fact, the inference speed in GPUs and the number of parameters or FLOPs are two different aspects to measure the performance of neural networks. According to the "no free lunch" rule, it is nearly impossible to be satisfactory to both sides. Therefore, the proposed VanillaNet aims the inference speed in GPUs and achieve much more higher speed then existing SOTA architectures.
which achieves an 83.1 15.11% % Top-1 accuracy on
The Figure shows the depth and inference speed of different architectures. The inference speed with batch size is highly related to the depth of the network instead of the number of parameters, which suggest that simple and shallow networks have huge potential in real-time processing. It can be easily find that the proposed VanillaNet achieve the best speed-accuras trade-off among all these architectures with low GPU latency, which demonstrates the superiority of the proposed VanillaNet when the computing power is sufficient.
4.2 Ablation Study
In this section, we conduct ablation study to investigate the effectiveness of proposed modules, including the series activation function and the deep training technique. Besides, we analyze the influence of adding shortcuts in the proposed VanillaNet.
7

---

Influence of number of series in activation function. In the above section, we propose series activation function to enhance the perfor- mance of plain activation function and enable global information exchange in feature maps. Table showss the performance the proposed
Table 2: Ablation study on the number of series.
FLOPS (B)
Lateney (ms)
Top-1 (%)
0
5.83
1.96
60.53
1
5.86
1.97
74.53
2
5.91
1.99
75.62
VanillaNet using different number of n in Equa-
3
5.99
2.01
76.36
When ni O, the activation function de- generate into the plain ReLU activation function. Although the inference speed of this network is higher than using the series activation function, the network can only achieve 60.53% top-1 accuracy on the ImageNet dataset, which cannot be applied real-world applications. It proves that the poor non-linearity of activation function results in poor performance of vanilla networks.
tion 6
4
6.10
2.18
76.43
To this end, we propose the series activation function. When n 1, the network can achieve 74.53% accuracy, which is a huge improvement compared with 60.53%. The result demonstrate the effectiveness of the proposed activation function. When the number of n increases, the performance of the network brings up. We find that = 3 is good balance in the top-1 accuracy and latency, Therefore, we use n = 3 for the rest experiments. It should be noted that the FLOPs of the proposed activation function very small compared with the original network, which is the same as the conclusion we derive in Equation
Influence of deep training. As the VanillaNet very shallow, we pro- pose to increase the training non- linearity to bring up its performance. We then analyze the effectiveness of the proposed deep training tech- nique. Table shows the results on using deep training technique with VanillaNet-6. As result, the original VanillaNet achieves a 75.23% top-1 accuracy, which is the baseline. By using the deep training technique, the proposed VanillaNet 76.36% accuracy. The results demon- strate that the proposed deep training technique useful for the shallow network.
Table 3: Ablation study different networks.
Network
Deep train. Series act.
Top-1 (%)
59.58
60.53
VanillaNet-6
75.23
76.36
57.52
59.09
AlexNet
V
61.12
63.59
achieve
76.13
76.16
ResNet-50
76.30
76.27
Moreover, we further apply the deep
training and series activation function in other networks show the generalization ability of the two techniques. Table 3 the results of AlexNet and ResNet-50, which are two classical deep neural networks, the ImageNet dataset. The original AlexNet only acheive 57.52% accuracy with 12 layers. By applying the proposed deep training and series activation function, the performance of AlexNet can be largely brought up by about 6%, which demonstrates that the proposed technique is highly effective for shallow networks. When it turns to ResNet-50 whose architecture are relatively complex, the performance gain is little. This results suggests the deep and complex networks already have enough non-linearity without the proposed techniques.
Influence of shortcuts. In deep neural networks, mon sense is that adding shortcut can largely case the
training procedure and improve the performance 23 To
Table 4: Ablation on adding shortcuts.
this end, we investigate whether shortcut would benefit the performance of shallow and simple network. We propose to use two kinds of location of shortcut, i.e., shortcut after the activation function and shorteut before the activation function, which are proposed in the original ResNet 23 and PreAct-ResNer (24), respectively. Since the number of channels is large and the original convolution is with kernel size x 1 in VanillaNet, adding shortcut (even with 1 kernel size) would largely increase the FLOPs. Therefore, we use the
Type
Top-1 (%)
no shortcut
76.36
shortcut before act
75.92
shortcut after act
75.72

---

parameter-free shortcut. It should be noted that if the stride is 2, the parameter- freumeeee--feeeeeeee shortcut will use an average pooling to decrease the size of feature maps and the number of channel is increasing. the parameter-free shortcut utilizes padding for the extra channels following the original setting in 23
Table 4 shows the ablation study adding shortcuts. We surprisingly find that using shortcuts, in spite of any type of shortcuts, has little improvement on the performance of the proposed VanillaNet. We suggest that the bottleneck of shallow networks not the identity mapping, but the weak non- linearity. As reported in ResNet [23], the error rate of 18 layers' plain network (without shortcut) is 27. 27.94%, which is similar 18 layers' ResNet (27. (27.88%). This result also suggests that shortcuts are not necessary for shallow networks. In fact, the shortcut is useless for bringing up the non-lineane linearity and may decrease non- con-linearity since the shorteut skips the activation function decrease the depth of the vanilla network, therefore results in lower performance.
Performance of transfer learning. To further demonstrate the generalization ability of the proposed VanillaNet, we fine-tune the proposed VanillaNet (pre-trained on ImageNet) on the well-known long- tailed classification dataset: iNaturalist 2018 49] and follow the setting in Deit [48). As shown in
Table 5: Accuracy (%) on ImageNet and iNat- uralist 2018.
Model
ImageNet
iNat18
ResNet-50
79.6
69.8
VanillaNet-9
72.5
72.5
the Table [ the performance of VanillaNet in pre- training scenarios is similar with other methods, which demonstrates its universality.
Deit-B
81.8
73,2
VanillaNet-12
81.6
73.6
4.3 Experiments COCO
Table 6: Performance on COCO detection and segmentation. FLOPs are calculated with image size
(1280, 800)on Nvidia A100 GPU.
Framework
Backbone
FLOPs
Params
FPS
APh
AP
APis
APM
AP
AP
Swin-T [6
245G
38.5M
27.5
41.5
62.1
44.2
RetinaNet
3
VanillaNet-13-D
397G
75.4M
29.8
43.0
62.8
44.3
Swin-T T 36
267G
47.8M
28.2
42.7
65.2
46.8
39.3
62.2
42.2
Mask RCNN 21
ConvNextV2-N 53
221G
35.2M
34.4
42.9
65.5
46.9
39.6
62.5
42.2
VanillaNet-13-D
421G
77.1M
32.6
42.9
65.5
46.9
39.6
62.5
42.2
To further demonstrate the effectiveness of the proposed VanillaNet on downstream tasks, we conduct evaluation in the COCO dataset [35]. We use RetinaNet (34 and Mask-RCNN 21 as the framework to evaluate the proposed method. FPS is measured on Nvidia A100 GPU. Details can be found in supplementary materials.
Table shows the performance of the proposed VanillaNet on COCO detection and segmentation. The proposed VanillaNet can successfully achieve similar performance with the ConvNext and the Swin backbone. Although the FL.OPs and Parameters of VanillaNet is much higher than Swin and ConvNext, it has much higher FPS, which demonstrates the effectiveness of vanilla architectures on object detection and instance segmentation tasks.
5 Conclusion
This paper fully investigates the feasibility of establishing neural networks with high performance but without complex architectures such shortcut, high depth and attention layers. which embodies the paradigm shift towards simplicity and elegance in design. We present deep training strategy and the series activation function for VanillaNets to enhance its non-linearity during both the training and testing procedures and bring up its performance. Experimental results on large scale image classification datasets reveal that VanillaNet performs on par with well-known deep neural networks and vision transformers, thus highlighting the potential of minimalism in deep learning. We will further explore better parameter allocation for efficient VanillaNet architectures with high performance. In summary, we prove that it is possible to achieve comparable performance with the state-of-the-an deep networks and vision transformers using a very concise architecture, which will unlock the potential of vanilla convolutiaonal network in the future.

---

References
[1] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprint wrXiv:2/06.0224, 2021.
[2] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159, 2020.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:18777 2020.
[4] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient-basee based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV), pages 839-847. 847. IEEE, 2018
[5] Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, Chunjing Xu, and Tong Zhang. Universal adder neural networks. arXiv preprint 205.::2205..440 2021.
[6] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint (tti::2003.10555: 2020.
[7] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-28882 255. Ieee, 2009.
[9] Li Deng, Geoffrey Hinton, and Brian Kingsbury. New types of deep neural network learning for speech recognition and related applications: An overview. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 8599-8603 IEEE, 2013.
[10] Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful enn via asymmetric convolution blocks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1911-1920, 2019.
[11] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Diverse branch block: Building a convolution as an inception-like unit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10886-10895, 2021.
[12] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11963 1975, 2022.
[13] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repygg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13733-13742, 2021.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[15] Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Conference learning theory, pages 907-940. PMLR, 2016.
[16] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. 2017 IEEE international conference on robotics and automation (ICRA), pages 3389-3396. IEFE, 2017.
[17] Jianyuan Guo, Kai Han, Yunhe Wang. Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2154-2166 21,, 2021.
[18] Jianyuan Guo, Kai Han, Yunhe Wang, Chao Zhang, Zhaohui Yang, Han Wu, Xinghao Chen, and Chang Xu. Hit-detector: Hierarchical trinity architecture search for object detection. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 1 11405-11414, 2020.
10

---

[19] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12175-12185, 2022.
[20] Zhiwei Hao, Jianyuan Guo, Kai Han, Han Hu, Chang Xu, and Yunhe Wang. Vanillakd: Revisit the power of vanilla knowledge distillation from small scale to large scale. arXiv preprint WwXiv:2305.15781 2023.
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 630-645. Springer, 2016.
[25] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 2016.
[26] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, al. Searching for mobilenetv3, In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314-1324, 2019.
[27] Jie Hu, Li Shen, and Gang Sun. Squeeze and-excitation networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018
the
(28) Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[29] Alex Krizhevsky, Πla Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097-1105, 2012.
[30] Steve Lawrence, C Lee Giles, Ah Chung Tsoi, and Andrew D Back. Face recognition: A convolutional neural-network approach. IEEE transactions on neural networks, 8(1):98-113, 1997.
[31] Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Soeren Kammel, J Zico Kolter, Dirk Langer, Oliver Pink, Vaughan Pratt, al. Towards fully autonomous driving: Systems and algorithms. In 2011 IEEE intelligent vehicles symposium (IV), pages 163-168. IEEE, 2011.
[32] Guilin Li, Junlei Zhang, Yunhe Wang, Chuanjian Liu, Matthias Tan, Yunfeng Lin, Wei Zhang, Jiashi Feng, and Tong Zhang. Residual distillation: Towards portable deep neural networks without shortcuts. Advances in Neural Information Processing Systems, 33:8935-8944 2020.
[33] Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. As-mlp: An axial shifted mlp architec- ture for vision. arXiv preprint arXiv:2107.08336 2021.
[34] Tsung-Yt Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[35] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[36] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
11

---

[37] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1976--1 1986, 2022.
[38] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, pages 3431-3440, 2015.
[39] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 1608. 03983, 2016.
[40] Hrushikesh N Mhaskar and Tomaso Poggio. Deep vs. shallow networks: An approximation theory perspective. Analysis and Applications, 14((6))829-844 2016.
[41] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging.
SIAM journal on control and optimization, 30(4):838-855, 1992.
[42] Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions, arXiv preprint
2017.
[43] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.
[44] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mobilenetv2: Inverted residuals and linear bottlneekss. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 45 4510-4520, 2018.
[45] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXiv:1409.15555 2014.
[46] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 1826222, 2016.
[47] Matus Telgarsky. Benefits of depth neural networks, In Conference on learning theory, pages 1517-1539. PMLR, 2016.
[48] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[49] Grant Van Horn. Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 9769-8778. 2018.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[51] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00553 2022.
[52] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv: 10.00046, 2021.
[53] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. arXiv preprint arXiv:2301
2023.
[54] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[55] Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances in Neural Information Processing Systems, 2021.
[56] Yuhui Xu, Lingxi Xie, Cihang Xie, Wenrui Dai, Jieru Mei, Siyuan Qiao, Wei Shen, Hongkai Xiong, and Alan Yuille. Bnet: Batch normalization with enhanced linear transformation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[57] Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quantization networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattrnn Recognition, pages 7308-7316, 2019.
12

---

[58] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv: /90v: 1et.i:994440., 2019.
[59] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032. 2019.
[60] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12104-12113, 2022.
[61] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv:1710.09412, 2017.
[62] Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. Vitaev2: Vision transformer ad- vanced by exploring inductive bias for image recognition and beyond. International Journal of Computer Vision, 2023.
13

---

A Network Architectures
The detailed architecture for VanillaNet with 7-13 layers can be found in Table 7 where each convolutional layer is followed with an activation function. For the VanilaNeet-33115x* the number of channels are multiplied with 1.5. For the VanillaNet-13-
1 we further use adaptive pooling for stage 2,3 and 4 with feature shape 40x x0 and x0x 10, respectively.
Input
VanillaNet-5
VanillaNet-ó
YamillaNaa77////1011111111
stem
224
4x4, 512, stride
[1x1, x MaxPool 2x2
[1x1, x024]x MaxPool 2x2
[1x1, 1024]x2 MaxPool 2x2
stagel
56x56
2048J x
11x1,
[1x1. 2048Jx
stage2
28 x88
MaxPool 2x2
MaxPool 2x2
MaxPool 2x2
11x1, x MaxPool 2x2
x096]
[1x1, 4096))
stage3
14x
MaxPool 2x2
MaxPool 2x2
stago4
7x7
[1x1, x096]
[1x1, x096]
AvgPool 7x7
classifier
7x
1 1000
Table 7: Detailed architecture specifications.
B
Training Details
For classification on ImageNet, we train the VanillaNets for 300 epochs utilizing the cosine learning rate decay [39]. The A in Equ. i linearly decayed from on epoch 0 and 100, respectively. The training details can be fould in Table 8 For the VanillaNet- I, since the training difficulty is relative large, we use the pre-trained weight from the VanillaNet-10 as its initialization. The same technique is adopt for VanillaNet-12/1
For detection and segmentation COCO, train all the networks using 12 epochs, multi-scale train- ing augmentation and a linear learning rate decay for fair comparison. Following ConvNextV2 [53] which utilize self-supervised training,
the ImageNet pre-trained weight using knowledge distillation (20] with n = for a higher receptive field. We train the VanillaNet-13 using the Adamw optimizer with batch size of 32, an initial learning rate of 8e-5 for RetinaNet and 1.3e-4 for Mask RCNN, an 0.05 weight decay and an 0.6 layer wise decay.
Training Config weight init optimizer
VanillaNct- 1
trunc. normal (0.2)
LAMB 58
loss function
BCE loss
base learning rate
3.5e-3 (5.8-13) /4.8e-3 [6-7]
weight decay
0..30..5.0.300033330005550.....0
optimizer momentum
81, =0.9., 0.999
batch size training epochs
1024
300
learning rate schedule
cosine decay 5 linear 0.05
warmup epochs warmup schedule dropout
layer-wise Ir decay 6
0 15.8- 12) /0.8 (6-7.13)
randaugment 7
(7. 0.5)
mixup [61]
1...50..4400400.40040........ 1.0
cutmix (59
color jitter
0.4
label smoothing
46
0.1
exp. mov. avg. (EMA) 41 test crop ratio
0.999996 (5- 10) 99.99992 (11-13)
0.875 [5-11] /0.95 (12-13)
Table 8: ImageNet- IK training settings.
14

---

(aasss-cl by ResNet-50-TNR
(b)Conrectly classified by ResNet-50-TNR
Misssllssiiie by VanillaNet-9
(d)Correctly classified by VanillaNet-9
Figure 3: Visualization of attention maps of the classified samples by ResNet-50 and VanillaNet-9, We show the attention maps of their mis-classified samples and correctly classified samples for comparison.
C Visualization of Attention
To have a better understanding of the proposed VanillaNet, we further visualize the features using CradCam+ [.. which utilizes weighted combination of the positive partial derivatives of the feature maps generated by the last convolutional layer with respect to the specific class to generate good visual explanation.
Figure shows the visualization results for VanillaNet-9 and ResNets-50-TNR 52 with similar performance. The red color denotes that there high activation in this region while the blue color denotes the weak activation for the predicted class. We can find that these two networks have different attention maps for different samples. It can be easily found that for ResNet-50, the of active region smaller. For the VanillaNet with only 9 depth, the active region much larger than that of deep networks. We suggest that VanillaNet may be strong in extract all relative activations in the input images and thoroughly extract their information by using large number of parameters and FLOOP. In contrast, VanillaNet may be weak on analyzing part of the useful region since the non-linearity is relatively low.
15


Reviews:
[{"review_id": "Jl60YwUtV7", "reviewer": "Reviewer_qK4N", "summary": "The authors propose a minimalist neural network backbone architecture as an alternative to current ResNets and ViTs. The minimalism is focussed on faster inference, avoiding complex operations with higher time complexity when implemented on modern GPUs. In their proposed architecture, individual operations are limited to convolutions, activation functions and linear projections, avoiding more time-complex operations like skip connections or self-attention. The resulting 6-layer network surpass latest CNN and ViT backbones at a fraction of inference latency on ImageNet top-1 accuracy metrics. A novel deep training strategy (merge two consecutive layers post-training) and series activation functions (stacked activation functions for more complex non-linearity under lower time complexity) are introduced to facilitate better learning by these minimalist shallow networks. In addition to the base architecture, a series of scaled-up versions provide increasing performance, still at minimal inference latency.", "questions": "See 1 in weaknesses", "limitations": "No clear discussion of limitations.  ", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1) Most of the paper is extremely well written with good logical flow, strong motivation for each proposal, and clear explanations of methodology. \n2) Proposed deep training and series activation functions appear novel, well-motivated, and interesting. \n3) Thorough ablations and discussion clearly demonstrate motivation for each design choice\n4) Strong results over SOTA backbones for classification, detections, and segmentation \n5) Detailed training details (in supp.) supports reproducibility of results", "weaknesses": "1) Missing details in series activation function: it appears scale and bias terms (Eq 5, 6) are learned parameters - please explicitly mention this or otherwise explain how these values are selected. Assuming former, how are these values initialized? Does the initialization affect final outcome (maybe conduct some ablation)? What are the final values for these parameters that are learned across runs (do they collapse in any occasion? do they convey any information about data distribution)? Can these parameters be fixed at start instead of learning? \n2) Writing (minor): grammatical errors in numerous places across sections 3 & 4, captions of figures not descriptive enough or self-contained (e.g. Table 4 - explain terms like depth, real-acc vs acc in caption - reader needs to skim through main text to understand headers of table).    \n3) More evaluations could strengthen the paper (and convince about universality of new backbone): maybe include results on some long-tailed classification datasets, more than one detection and segmentation dataset.    \n4) Section 4.2 & Figure 2 (attention map) does not appear to convey anything useful in current form. Consider updating description / main text to discuss one of the examples in detail (or maybe move to supplementary).  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Missing details in series activation function.\n\n**A1:** Sorry for the unclear claim. The scale and bias terms are learned parameters. We initialize them using Kaiming initialization [1]. We also test other initialization. As shown in the Table below, the initialization affect little final outcome (<0.2%).\n\n| VanillaNet-5 | Kaiming init[1] | Xavier init[2] | Gaussian init | \n|:---:|:---:|:---:|:---:|\n| Accuracy | 72.49 | 72.45 | 72.36 | \n\nThe final values differs across runs. The reason is that deep network is highly non-convex and the converged values rely on the initialized values (which is different across runs). Therefore, these parameters cannot be fixed at start.\n\n*[1] Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ICCV 2015.*\n\n*[2] Understanding the difficulty of training deep feedforward neural networks. AISTATS 2010.*\n\n**Q2:** Writing.\n\n**A2:** Sorry for the mistake. We will fix the grammatical errors and enrich the caption of tables to make them self-contained.\n\n**Q3:** More evaluations.\n\n**A3:** Thanks for the suggestion. We conduct experiments on the well-known long-tailed classification dataset: iNaturalist 2018 [3] and follow the setting in Deit [4].\n\n| Model | ImageNet Acc. | iNat18 Acc. | \n|:---:|:---:|:---:|\n| ResNet-50 | 79.6 | 69.8 | \n| VanillaNet-9 | 72.49 | 72.45 | \n| Deit-B | 81.8 | 73.2 | \n| VanillaNet-12 | 81.6 | 73.6 | \n\nAs shown in the Table above, the performance of VanillaNet in long-tailed dataset is similar with that in classical dataset compared with other methods, which demonstrates its universality. We will include these results in the final version.\n\n*[3] The inaturalist challenge 2018 dataset. Arxiv 2018.*\n\n*[4] Training data-efficient image transformers & distillation through attention. PMLR 2021.*\n\n**Q4:** Attention map.\n\n**A4:** Thanks for the suggestion. We will move them to supplementary and include more discussion in the final version.\n"}, {"review_id": "5F27O9S8DH", "reviewer": "Reviewer_prVa", "summary": "This paper introduces VanillaNet, a neural network architecture that emphasizes simplicity in design. By discarding commonly used ingredients, e.g., high depth, shortcuts, and self-attention mechanism, VanillaNet is refreshingly concise. VanillaNet series have only 5-13 layers. The authors devise the “deep training” strategy, in which nonlinear activation functions are pruned after training to restore the original architecture. They also leverage series-based activation function to incorporate multiple learnable affine transformations. The experiments on classification and detection demonstrate the effectiveness of the proposed VanillaNet.", "questions": "Inference speed: The authors report the latency on GPU and pytorch. However, in more scenarios (or platform), such as tensorRT, will the gap between VanillaNet and other networks become smaller?\n\nSeries Informed Activation Function: What’s the difference between SIAF and BNET? I think these two methods should be explained in more details.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Motivation: This paper is different from the previous research direction of the backbone architecture, abandoning the exploration of the residual connection or deepen the network. It shows the potential of the shallow network. The proposed VanillaNet avoids complex operations and make the structure more compact and easy to understand.\n\nQuality: The paper is clear and easy to follow.\n\nOriginality: The “deep training technique” proposed in this paper is well motivated and effective. The activation function is gradually reduced to an identity mapping during training. And finally two convolutions can be easily merged into one convolution. I like this idea. In addition, the “series activation function” which enables global information exchange also seems effective.\n\nExperiment: The authors conduct sufficient experiments and ablation studies to demonstrate the effectiveness of the proposed VanillaNet. VanillaNet can achieve comparable performance to current SOTA models on several computer vision tasks with faster inference speed\n\nOverall, the motivation challenges the “more is different” foundational model philosophy, opening up new possibilities for efficient deployment of deep learning models.", "weaknesses": "About the application of “deep training technique”. It seems that this technique can only be apllied to BN. If the normalization layer is LN, it cannot be merged.\n\n\nAbout the number of parameters. Although the inference speed of VanillaNet is superior to other methods, the heavy number of paramters and “deep train” strategy may increase the training cost of the network. According to my understanding, the amount of network parameters in training stage should be much larger than the paratmers shown in Table 4 (in inferece stage).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The number of parameters.\n\n**A1:** Thanks for the suggestion. Although the deep training technique will increase the training cost, it would not affect the inference cost, which is much more important when implementation.\n\n**Q2:** Inference speed:\n\n**A2:** Thanks for the nice concern. We test the speeds using tensorRT. The results show that the proposed VanillaNets achieve higher speed with similar performance.\n\n| name | Acc(%) | Latency(ms) using TRT on A100 |\n|:---:|:---:|:---:|\n| Swin-T | 81.18 | 1.41 |\n| ResNet-18 |70.6 | 0.41 | \n| ResNet-34 |75.5|0.77|\n| ResNet-50 |79.8|0.80|\n| ResNet-101 |81.3|1.58 |\n| ResNet-152 |81.8|2.30 |\n| **VanillaNet-5** | 72.49 |0.33 |\n| **VanillaNet-6** | 76.36 |0.40|\n| **VanillaNet-9** | 79.87 |0.58|\n| **VanillaNet-12** | 81.55 |0.75|\n| **VanillaNet-13** | 82.05|0.82|\n\n**Q3:** Difference between SIAF and BNET.\n\n**A3:** The motivation of BNET is to improve BN's flexibility of fitting complex data distributions, which is resolved by introducing a linear transformation module with considering each neuron's neighborhood. In contrast, the motivation of SIAF is to enhance the non-linearity of shallow networks, which is implemented by combining multiple activation functions with different weights and biases. Empirically, as shown in Table 2 in our paper, the SIAF only works for shallow networks yet with huge improvements (>4.0% on ImageNet), while BNET is a general method with relatively little improvements (~0.5% on ImageNet). Moreover, SIAF can combined with BNET to further improve the networks’ performance. We will include more analysis in the final version.\n"}, {"review_id": "Fl5FxkXlKK", "reviewer": "Reviewer_xYue", "summary": "The paper introduces VanillaNet, a neural network architecture that emphasizes simplicity in design while maintaining remarkable performance in computer vision tasks. VanillaNet achieves this by avoiding excessive depth, shortcuts, and intricate operations like self-attention. The authors propose a \"deep training\" strategy to train VanillaNets, which starts with several layers containing non-linear activation functions and progressively eliminates them as training proceeds. Additionally, an efficient, series-based activation function incorporating multiple learnable affine transformations is introduced to augment the networks' non-linearity. The experimental results show that VanillaNet achieves comparable or even better performance than state-of-the-art deep neural networks and vision transformers on various computer vision tasks while being more efficient and resource-friendly.\n", "questions": "1. Table 3 shows that shortcuts are useless for the proposed VanillaNets. As the shortcuts are one of the most popular components in modern networks, it would be better to provide more explainations about this phenomenon. \n2. As shown in the suppmentary material, the VanillaNet-13 is achieved by adding convolutions on the 3rd stage of VanillaNet-7, what is the designed logic behind?\n", "limitations": "yes. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper is well-written and organized, with clear explanations of the proposed VanillaNet architecture. The paper also provides detailed experimental setups and results, making it easy to understand and reproduce the experiments.\n2. The proposed VanillaNet is novel. By avoiding excessive depth, shortcuts, and intricate operations like self-attention, VanillaNet offers a streamlined and efficient design. This simplicity not only makes VanillaNet well-suited for resource-limited environments but also challenges the prevailing notion that increasing network complexity leads to improved performance, which may have implications for future research in the field.\n3. The experimental results are convincing. Despite its simplicity and low depth, VanillaNet delivers performance on par with renowned deep neural networks and vision transformers. For example, a 13 layers' VanillaNet achieves an 83% top-1 accuracy on ImageNet. \n", "weaknesses": "1. While VanillaNet's simplicity and low depth make it an attractive option for efficient deployment in GPUs, one weakness of the proposed method is that it has significantly more parameters and FLOPs than some existing methods. It is noted that VanillaNet has a large number of parameters in each layer to capture useful information from the images with limited non-linearity. This can make it more challenging to train and deploy VanillaNet on certain devices with limited computational resources. \n2. The proposed deep training strategy is used to enhance the performance of VanillaNet by introducing stronger capacity during training, which suggests that the proposed method requires more training FLOPs. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** More parameters and FLOPS.\n\n**A1:** Thanks for the nice concern. The inference speed in GPUs and the number of parameters or FLOPs are two different aspects to measure the performance of neural networks. According to the “no free lunch” rule, it is nearly impossible to be satisfactory to both sides. Therefore, the proposed VanillaNet aims to the inference speed in GPUs and achieve much more higher speed then existing SOTA architectures.\n\n**Q2:** More training FLOPs.\n\n**A2:** Thanks for the suggestion. Although the deep training technique will increase the training FLOPs, it would not affect the inference cost, which is much more important when implementation.\n\n**Q3:** Explanation on shortcuts.\n\n**A3:** We suggest that the bottleneck of vanilla networks is not the identity mapping, but the weak non-linearity. The shortcut is useless for bringing up the non-linearity. As reported in ResNet [1], the error rate of 18 layers’ plain network is 27.94%, which is similar to 18 layers’ ResNet (27.88%). This result also suggests that shortcuts are not necessary for shallow networks.\n\n*[1] Deep Residual Learning for Image Recognition. CVPR 2016.*\n\n**Q4:** Designed logic.\n\n**A4:** Thanks for the nice concern. We follow the designed logic in ResNet [1] to assign extra layers in the 3rd stages for simplify, which can be further optimized in the future work. \n"}, {"review_id": "vuCm53HrTz", "reviewer": "Reviewer_jvnA", "summary": "The paper presents VanillaNet, a neural network architecture that emphasizes simplicity by avoiding high depth, shortcuts, and complex operations like self-attention. The proposed VanillaNet performs comparably to well-known deep neural networks and vision transformers.", "questions": "- Since pre-training significantly impacts the model's performance, it would be better to see how the proposed model architecture performs in pre-training scenarios. ", "limitations": "the authors adequately addressed the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: To the best of my knowledge, the proposed VanillaNet is novel.\n\n**Quality**:     This paper is of high quality since it proposed a new model architecture and conducted extensive experiments on ImageNet and ablation studies to demonstrate its effectiveness. \n\n**Clarity**: The paper is well-structured and clear, providing detailed explanations of the VanillaNet architecture and the proposed techniques.\n\n**Significance**: The paper is significant as it paves the way for a new direction in neural network design. ", "weaknesses": "-  Although the proposed VanillaNet reduced the latency a lot, the required Params and FLOPs increased a lot. e.g. FOR SWIN-S and VannillaNet that have comparable accuracy on ImageNet, the proposed vanillaNet needs 127.8 M and 48.9 FLOPs while Swin-S only needs 49.6 M and 8.7B FLOPs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** More parameters and FLOPs.\n\n**A1:** Thanks for the nice concern. The inference speed in GPUs and the number of parameters or FLOPs are two different aspects to measure the performance of neural networks. According to the “no free lunch” rule, it is nearly impossible to be satisfactory to both sides. Therefore, the proposed VanillaNet aims to the inference speed in GPUs and achieve much more higher speed then existing SOTA architectures.\n\n**Q2:** Performance in pre-training scenarios.\n\n**A2:** We fine-tune the proposed VanillaNet (pre-trained on ImageNet) on the well-known long-tailed classification dataset: iNaturalist 2018 [1] and follow the setting in Deit [2].\n\n| Model | ImageNet Acc. | iNat18 Acc. | \n|:---:|:---:|:---:|\n| ResNet-50 | 79.6 | 69.8 | \n| VanillaNet-9 | 72.49 | 72.45 | \n| Deit-B | 81.8 | 73.2 | \n| VanillaNet-12 | 81.6 | 73.6 | \n\nAs shown in the Table above, the performance of VanillaNet in pre-training scenarios is similar with other methods, which demonstrates its universality. We will include these results in the final version.\n\n*[1] The inaturalist challenge 2018 dataset. Arxiv 2018.*\n\n*[2] Training data-efficient image transformers & distillation through attention. PMLR 2021.*\n"}, {"review_id": "D3vr9YedBp", "reviewer": "Reviewer_R2Cr", "summary": "This paper aims to minimize the complex design of modern deep models. The authors propose an architecture, VanillaNet, which eliminates high depth, residual connections, non-linear activations, self-attentions, and normalization layers, but during training, the model still use some components, such as activations and normalizations, to help model training. They also propose an activation function series to boost the capacity and performance of the models. Due to the simplicity, the model is ideal for resource-constrained environments and has lower latency than complex deep models. Their experiments show that the proposed model has similar performance than modern deep models.", "questions": "The main concerns are listed above. In addition, is the proposed activation function series used in inference? If so, then the model still has activations and batch normalization (although the later one is folded into convolution), and compared ResNet, only the residual connections are removed. If not, does this mean all convolutions in each stage can be merged into a single convolutions? \n\nThis is an important question since the price of VanillaNet is that the parameter count and FLOPs need to be significantly higher to achieve similar performance compared to baselines.", "limitations": "There is no discussion on limitations. You should include a small limitation section. For example, you could include a discussion about the need for higher parameter count and FLOPs.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed model does not have high depth, residual connection, and normalization layers, which is beneficial for latency. \n2. Despite the lack of complex design, the performance of the proposed idea is comparable to modern deep models. \n3. The proposed activation function series offers performance boost with slightly more FLOPs and parameters.", "weaknesses": "1. It would be better to have a figure indicating where the merging operation describe in section 3.1 occurs in the model, so it is difficult to see where the merging opportunities are.  \n2. The idea of merging batch normalization with convolution describe between line 132 and line 137 is an already proposed inference acceleration called Batch Normalization Folding. Please make a clear distinction between what design are novel in this paper and what design are coming from existing work. \n3. The content between lines 177-183 calculates the complexity (which directly tie to FLOPs) of the activation function series and claims the cost is very small, but in lines 90-91, the authors claims FLOPs is not important, so they are contradicting. Rather, it would make more sense to compare the latency of proposed activation and convolution. \n4. The latency (profiled using a batch of size 1) in Table 4 could be misleading. In many inference cases, multiple instances can be batched together to improve throughput and make better usage of available hardwares, so it would be much better if throughput or latency (measured using reasonably large batch size) is also provided. \n5. Code is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Merging operation.\n\n**A1:** Sorry for the unclear statement. The inference architecture of VanillaNet can be found in the supplementary material. During the training time, each convolution layer is doubled into two convolutions with an activation between them. After training, the two convolutions will be merged into one convolution layer. We will provide a figure to explain this operation in detail in the final version.\n\n**Q2:** Clear distinction of designs.\n\n**A2:** Sorry for the misleading. The Batch Normalization Folding is coming from existing work and the others (deep training techniques and series informed activation functions) are designed in this paper. We will claim it in the final version.\n\n**Q3:** Compare the latency of proposed activation and convolution.\n\n**A3:** Thanks for the nice suggestion. As shown in the Table below, the latency of VanillaNet-6 using the proposed activation is 2.01, which is similar with that without the activation (1.96). Therefore, the latency cost of the activation is also very small, which is the same as the analysis by using FLOPs. \n\n| VanillaNet-6 | Acc(%) | Latency(ms) |\n|:---:|:---:|:---:|\n| with proposed activation | 76.36 | 2.01 |\n| without proposed activation | 60.53 | 1.96 |\n\n\n**Q4:** Provide latency measured using reasonably large batch size.\n\n**A4:** Thanks for the nice concern. We test the speeds using a 256 batchsize. The results show that the proposed VanillaNets can still achieve higher speed with similar performance in this setting.\n\n| name | Acc(%) | Latency(ms) on A100 with bs 256 |\n|:---:|:---:|:---:|\n| Swin-T | 81.18 | 96.69 |\n| ResNet-18 |70.6 | 31.24 | \n| ResNet-34 |75.5|54.06|\n| ResNet-50 |79.8|74.83|\n| ConvNextV2-P | 79.7|53.92|\n| ConvNextV2-N | 81.2| 70.41|\n| **VanillaNet-5** | 72.49 |22.64 |\n| **VanillaNet-6** | 76.36 |25.24|\n| **VanillaNet-9** | 79.87 |32.75|\n| **VanillaNet-12** | 81.55 |40.35|\n\n**Q5:** Code is not provided.\n\n**A5:** Thanks for the nice suggestion. We will provide code link in the final version. \n\n**Q6:** Is the proposed activation function series used in inference?\n\n**A6:** Yes. The model still has activations and batch normalization and all convolutions in each stage cannot be merged into a single convolutions. Compared with ResNet, we introduce the series informed activation function and the deep training technique. As a result, the proposed VanillaNet achieves much higher inference speed compared with ResNet.\n"}]

Author rebuttal:

Meta-review: In this paper, the authors present a simple convolutional neural network architecture for computer vision problems that matches many modern architectures advancements in the last few years. The goal of this architecture is to aim for faster inference, and avoid complex operations such as skip connections, higher depths, and self-attention. The resulting architecture is demonstrated on image classification with ImageNet, and object detection with COCO achieving favorable performance in light of inference speed. The authors achieve this through a training strategy in which they slowly adapt a nonlinearity in between two layers towards an identity function. At the end of training, the two subsequent convolutions may be folded into a single linear operation for fast inference.

The reviewers commented positively on the simplicity of the method, the overall performance both in terms of latency and predictive performance, and the overall novelty. The reviewers also indicated concerns about the lack of methodological details, the increased number of model parameters relative to baseline models, and the limitation of the method to Batch Normalization (and not LayerNorm). Given that there remained no consensus in the reviewer scores, I considered this paper to be borderline acceptance. The paper shows strong performance and results but I am concerned that the scope of impact for these results may be limited due to the fact that these models must be CNN's with BatchNorm. The research community has generally moved away from this architecture style in order to provide additional functionality not available in standard CNN architectures. That said, given the simplicity and strength of these results and in spite of these limitations, I have decided to accept this paper to conference as I think it would be a great reminder that “older” architecture designs still have strong merit and should not be abandoned.
Decision: Accept (poster)