SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanting Chen $^{1}$ , Yunhe Wang $^{1*}$ , Jianyuan Guo $^{1}$ , Dacheng Tao $^{2}$
<sup>1</sup> Huawei Noah's Ark Lab. <sup>2</sup> School of Computer Science, University of Sydney. {chenhanting,yunhe.wang, jianyuan.guo}@huawei.com, dacheng.tao@sydney.edu.au
At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at https://github.com/huawei-noah/VanillaNet and https://gitee.com/mindspore/models/tree/master/research/cv/vanillanet.
Over the past few decades, artificial neural networks have made remarkable progress, driven by the idea that increasing network complexity leads to improved performance. These networks, which consist of numerous layers with a large number of neurons or transformer blocks [50, 36], are capable of performing a variety of human-like tasks, such as face recognition [30], speech recognition [9], object detection [43, 17, 18], natural language processing [50], and content generation [3]. The impressive computational power of modern hardware allows neural networks to complete these tasks with both high accuracy and efficiency. As a result, AI-embedded devices are becoming increasingly prevalent in our lives, including smartphones, AI cameras, voice assistants, and autonomous vehicles.
Admittedly, one notable breakthrough in this field is the development of AlexNet [29], which consists of 12 layers and achieves state-of-the-art performance on the large-scale image recognition benchmark [8]. Building on this success, ResNet [23] introduces identity mappings through shortcut connections, enabling the training of deep neural networks with high performance across a wide range of computer vision applications, such as image classification [45], object detection [43], and semantic segmentation [38]. The incorporation of human-designed modules in these models, as well as the continued increase in network complexity, has undeniably enhanced the representational
Figure 1: The architecture of VanillaNet-6 model, which consists of only 6 convolutional layers, which are very easily to be employed on any modern hardwares. The size of input features are downsampled while the channels are doubled in each stage, which borrows from the design of classical neural networks such as AlexNet [29] and VGGNet [45].
capabilities of deep neural networks, leading to a surge of research on how to train networks with more complex architectures [28, 24, 54] to achieve even higher performance.
Apart from convolutional architectures, Dosovitskiy et al.[14] have introduced the transformer architecture to image recognition tasks, demonstrating its potential for leveraging large-scale training data. Zhai et al.[60] investigated the scaling laws of vision transformer architectures, achieving an impressive  $90.45\%$  top-1 accuracy on the ImageNet dataset, which indicates that deeper transformer architectures, like convolutional networks, tend to exhibit better performance. Wang et al.[51] further proposed scaling the depth of transformers to 1,000 layers for even higher accuracy. Liu et al.[37] revisited the design space of neural networks and introduced ConvNext, achieving similar performance to state-of-the-art transformer architectures.
Although well-optimized deep and complex neural networks achieve satisfying performance, their increasing complexity poses challenges for deployment. For example, shortcut operations in ResNets consume significant off-chip memory traffic as they merge features from different layers [32]. Furthermore, complicated operations such as axial shift in AS-MLP [33] and shift window self-attention in Swin Transformer [36] require sophisticated engineering implementation, including rewriting CUDA codes.
These challenges call for a paradigm shift towards simplicity in neural network design. However, the development of ResNet has seemingly led to the abandonment of neural architectures with pure convolutional layers (without extra modules such as shortcuts). This is mainly due to the performance enhancement achieved by adding convolutional layers not meeting expectations. As discussed in [23], plain networks without shortcuts suffer from gradient vanishing, causing a 34-layer plain network to perform worse than an 18-layer one. Moreover, the performance of simpler networks like AlexNet [29] and VGGNet [45] has been largely outpaced by deep and complex networks, such as ResNets [23] and ViT [8]. Consequently, less attention has been paid to the design and optimization of neural networks with simple architectures. Addressing this issue and developing concise models with high performance would be of great value.
To this end, we propose VanillaNet, a novel neural network architecture emphasizing the elegance and simplicity of design while retaining remarkable performance in computer vision tasks. VanillaNet achieves this by eschewing excessive depth, shortcuts, and intricate operations such as self-attention, leading to a series of streamlined networks that address the issue of inherent complexity and are well-suited for resource-limited environments. To train our proposed VanillaNets, we conduct a comprehensive analysis of the challenges associated with their simplified architectures and devise a "deep training" strategy. This approach starts with several layers containing non-linear activation functions. As the training proceeds, we progressively eliminate these non-linear layers, allowing for easy merging while preserving inference speed. To augment the networks' non-linearity, we put forward an efficient, series-based activation function incorporating multiple learnable affine transfor
mations. Applying these techniques has been demonstrated to significantly boost the performance of less complex neural networks. As illustrated in Figure 2, VanillaNet surpasses contemporary networks with elaborate architectures concerning both efficiency and accuracy, highlighting the potential of a minimalist approach in deep learning. This pioneering investigation of VanillaNet paves the way for a new direction in neural network design, challenging the established norms of foundation models and establishing a new trajectory for refined and effective model creation.
2 A Vanilla Neural Architecture
Over the past decades, researchers have reach some consensus in the basic design of neural networks. Most of the state-of-the-art image classification network architectures should consist of three parts: a stem block to transform the input images from 3 channels into multiple channels with downsampling, a main body to learn useful information, a fully connect layer for classification outputs. The main body usually have four stages, where each stage is derived by stacking same blocks. After each stage, the channels of features will expand while the height and width will decrease. Different networks utilize and stack different kinds of blocks to construct deep models.
Despite the success of existing deep networks, they utilize large number of complex layers to extract high-level features for the following tasks. For example, the well-known ResNet [23] requires 34 or 50 layers with shortcuts for achieving over  $70\%$  top-1 accuracy on ImageNet. The base version of ViT [14] consists of 62 layers since the query, key and value in self-attention require multiple layers to calculate.
With the growing of AI chips, the bottleneck of inference speed of neural networks would not be FLOPs or parameters, since modern GPUs can easily do parallel calculation with strong computing power. In contrast, their complex designs and large depths block their speed. To this end, we propose the vanilla network, i.e., VanillaNet, whose architecture is shown in Figure 1. We follow the popular design of neural network with the stem, main body and fully connect layer. Different with existing deep networks, we only employ one layer in each stage to establish a extremely simple network with as few layers as possible.
Here we show the architecture of the VanillaNet in details, which takes 6 layers as an example. For the stem, we utilize a  $4 \times 4 \times 3 \times C$  convolutional layer with stride 4 following the popular settings in [23, 36, 37] to map the images with 3 channels to features with  $C$  channels. At stage 1, 2 and 3, a maxpooling layer with stride 2 is used to decrease the size and feature map and the number of channels is increased by 2. At stage 4, we do not increasing the number of channels as it follows an average pooling layer. The last layer is a fully connected layer to output the classification results. The kernel size of each convolutional layer is  $1 \times 1$ , since we aim to use minimal calculation cost for each layer while keep the information of feature maps. The activation function is applied after each  $1 \times 1$  convolutional layer. To ease the training procedure of the network, batch normalization is also added after each layer. For the VanillaNet with different number of layers, we add blocks in each stage, which will be detailed in the supplementary material. It should be noted that the VanillaNet has no shortcut, since we empirically find adding shortcut shows little performance improvement. This also gives another benefit that the proposed architecture is extremely easy to implemented since there are no branch and extra blocks such as squeeze and excitation block [27].
Although the architecture of VanillaNet is simple and relatively shallow, its weak non-linearity caused limit the performance, Therefore, we propose a series of techniques to solve the problem.
3 Training of Vanilla Networks
It is common in deep learning to enhance the performance of models by introducing stronger capacity in the training phase [5, 57]. To this end, we propose to utilize a deep training technique to bring up the ability during training in the proposed VanillaNet, since deep network have stronger non-linearity than shallow network.
3.1 Deep Training Strategy
The main idea of deep training strategy is to train two convolutional layers with an activation function instead of a single convolution layer in the beginning of training procedure. The activation function is
gradually reduce to an identity mapping with the increasing number of training epochs. At the end of training, two convolutions can be easily merged into the one convolution to reduce the inference time. This kind of idea is also widely used in CNNs [11, 13, 10, 12]. We then describe how to conduct this technique in detail.
For an activation function  $A(x)$  (which can be the usual functions such ReLU and Tanh), we combine it with an identity mapping, which can be formulated as:
$$
A ^ {\prime} (x) = (1 - \lambda) A (x) + \lambda x, \tag {1}
$$
where  $\lambda$  is a hyper-parameter to balance the non-linearity of the modified activation function  $A'(x)$ . Denote the current epoch and the number of deep training epochs as  $e$  and  $E$ , respectively. We set  $\lambda = \frac{e}{E}$ . Therefore, at the beginning of training ( $e = 0$ ),  $A'(x) = A(x)$ , which means the network has strong non-linearity. When the training converged, we have  $A'(x) = x$ , which means the two convolutional layers have no activation functions in the middle. We further demonstrate how to merge these two convolutional layers.
We first convert every batch normalization layer and its preceding convolution into a single convolution, which is a commonly used technique. We denote  $W \in \mathbb{R}^{C_{out} \times (C_{in} \times k \times k)}$ ,  $B \in \mathbb{R}^{C_{out}}$  as the weight and bias matrices of convolutional kernel with  $C_{in}$  input channels,  $C_{out}$  output channels and kernel size  $k$ . The scale, shift, mean and variance in batch normalization are represented as  $\gamma, \beta, \mu, \sigma \in \mathbb{R}^{C_{out}}$ , respectively. The merged weight and bias matrices are:
$$
W _ {i} ^ {\prime} = \frac {\gamma_ {i}}{\sigma_ {i}} W _ {i}, B _ {i} ^ {\prime} = \frac {(B _ {i} - \mu_ {i}) \gamma_ {i}}{\sigma_ {i}} + \beta_ {i}, \tag {2}
$$
where subscript  $i \in \{1, 2, \dots, C_{out}\}$  denotes the value in  $i$ -th output channels.
After merging the convolution with batch normalization, we begin to merge the two  $1 \times 1$  convolutions. Denote  $x \in \mathbb{R}^{C_{in} \times H \times W}$  and  $y \in \mathbb{R}^{C_{out} \times H' \times W'}$  as the input and output features, the convolution can be formulated as:
$$
y = W * x = W \cdot \operatorname {i m} 2 \operatorname {c o l} (x) = W \cdot X, \tag {3}
$$
where  $*$  denotes the convolution operation,  $\cdot$  denotes the matrix multiplication and  $X\in \mathbb{R}^{(C_{in}\times 1\times 1)\times (H^{\prime}\times W^{\prime})}$  is derived from the im2col operation to transform the input into a matrix corresponding to the kernel shape. Fortunately, for  $1\times 1$  convolution, we find that the im2col operation becomes a simple reshape operation since there are no need for sliding kernels with overlap. Therefore, denote the weight matrix of two convolution layers as  $W^{1}$  and  $W^{2}$ , the two convolution without activation function is formulated as:
$$
y = W ^ {1} * \left(W ^ {2} * x\right) = W ^ {1} \cdot W ^ {2} \cdot \operatorname {i m 2 c o l} (x) = \left(W ^ {1} \cdot W ^ {2}\right) * X, \tag {4}
$$
Therefore,  $1 \times 1$  convolution can be merged without increasing the inference speed. It is worth noticing that although the deep training technique will increase the training FLOPs, it would not affect the inference cost, which is much more important when implementation.
3.2 Series Informed Activation Function
There have been proposed several different activation functions for deep neural networks, including the most popular Rectified Linear Unit (ReLU) and its variants (PReLU [22], GeLU [25] and Swish [42]). They focus on bring up the performance of deep and complex networks using different activation functions. However, as theoretically proved by the existing works [40, 15, 47], the limited power of simple and shallow network are mainly caused by the poor non-linearity, which is different with deep and complex networks and thus has not been fully investigated.
In fact, there are two ways to improve the non-linearity of a neural network: stacking the non-linear activation layers or increase the non-linearity of each activation layer, while the trend of existing networks choose the former one, which results in high latency when the parallel computation ability is excess.
One straight forward idea to improve non-linearity of activation layer is stacking. The serially stacking of activation function is the key idea of deep networks. In contrast, we turn to concurrently stacking the activation function. Denote a single activation function for input  $x$  in neural network as
$A(x)$ , which can be the usual functions such ReLU and Tanh. The concurrently stacking of  $A(x)$  can be formulated as:
$$
A _ {s} (x) = \sum_ {i = 1} ^ {n} a _ {i} A \left(x + b _ {i}\right), \tag {5}
$$
where  $n$  denotes the number of stacked activation function and  $a_{i}, b_{i}$  are the scale and bias (which are learned parameters) of each activation to avoid simple accumulation. The non-linearity of the activation function can be largely enhanced by concurrently stacking. Equation 5 can be regarded as a series in mathematics, which is the operation of adding many quantities.
To further enrich the approximation ability of the series, we enable the series based function to learn the global information by varying the inputs from their neighbors, which is similar with BNET [56]. Specifically, given a input feature  $x \in \mathbb{R}^{H \times W \times C}$ , where  $H$ ,  $W$  and  $C$  are the number of its width, height and channel, the activation function is formulated as:
$$
A _ {s} \left(x _ {h, w, c}\right) = \sum_ {i, j \in \{- n, n \}} a _ {i, j, c} A \left(x _ {i + h, j + w, c}\right) + b _ {c}, \tag {6}
$$
where  $h \in \{1,2,\dots,H\}$ ,  $w \in \{1,2,\dots,W\}$  and  $c \in \{1,2,\dots,C\}$ . It is easy to see that when  $n = 0$ , the series based activation function  $A_{s}(x)$  degenerates to the plain activation function  $A(x)$ , which means that the proposed method can be regarded as a general extension of existing activation functions. We use ReLU as the basic activation function to construct our series since it is efficient for inference in GPUs.
We further analyze the computation complexity of the proposed activation function compared with its corresponding convolutional layer. For a convolutional layer with  $K$  kernel size,  $C_{in}$  input channels and  $C_{out}$  output channels, the computational complexity is:
$$
\mathcal {O} (\text {C O N V}) = H \times W \times C _ {\text {i n}} \times C _ {\text {o u t}} \times k ^ {2}, \tag {7}
$$
while computation cost of its series activation layer is:
$$
\mathcal {O} (\mathrm {S A}) = H \times W \times C _ {\text {o u t}} \times (2 n + 1) ^ {2}. \tag {8}
$$
$$
\frac {\mathcal {O} (\mathrm {C O N V})}{\mathcal {O} (\mathrm {S A})} = \frac {H \times W \times C _ {i n} \times C _ {o u t} \times K ^ {2}}{H \times W \times C _ {o u t} \times (2 n + 1) ^ {2}} = \frac {C _ {i n} \times k ^ {2}}{(2 n + 1) ^ {2}}. \tag {9}
$$
Taking the 4th stage in VanillaNet-B as an example, where  $C_{in} = 2048$ ,  $k = 1$  and  $n = 3$ , the ratio is about 84. In conclusion, the computation cost of the proposed activation function is still much lower than the convolutional layers. More experimental complexity analysis will be shown in the following section.
It is noted that the proposed SIAF is significantly different with BNET [56]. The motivation of BNET is to improve BN's flexibility of fitting complex data distributions, which is resolved by introducing a linear transformation module with considering each neuron's neighborhood. In contrast, the motivation of the proposed SIAF is to enhance the non-linearity of shallow networks, which is implemented by combining multiple activation functions with different weights and biases. Empirically, as shown in Table 3 in our paper, the SIAF only works for shallow networks yet with huge improvements ( $>4.0\%$  on ImageNet), while BNET is a general method with relatively little improvements ( $0.5\%$  on ImageNet).
In this section, we conduct experiments to verify the performance of the proposed VanillaNet on large scale image classification. Ablation study is provided to investigate effectiveness of each component of the proposed VanillaNet. We also visualize the feature of VanillaNet to further study how the proposed network learns from images.
4.1 Comparison with SOTA architectures
To illustrate the effectiveness of the proposed method, we conduct experiments on the ImageNet [8] dataset, which consists of  $224 \times 224$  pixel RGB color images. The ImageNet dataset contains 1.28M
Table 1: Comparison on ImageNet. The depth denotes the number of non-linear layers. The Reall Acc denotes the accuracy with the ImageNet Real label [2]. Latency is tested on Nvidia A100 GPU with batch size of 1. When comparing with other baselines, we also apply the batch norm folding for fair comparison.
Model	Params (M)	FLOPs (B)	Depth	Latency (ms)	Acc (%)	Real Acc (%)
MobileNetV3-Small [26]	2.5	0.06	48	6.65	67.67	74.33
MobileNetV3-Large [26]	5.5	0.22	48	7.83	74.04	80.01
ShuffleNetV2x1.5 [44]	3.5	0.30	51	7.23	73.00	80.19
ShuffleNetV2x2 [26]	7.4	0.58	51	7.84	76.23	82.72
RepVGG-A0 [13]	8.1	1.36	23	3.22	72.41	79.33
RepVGG-A1 [13]	12.8	2.37	23	3.24	74.46	81.02
RepVGG-B0 [13]	14.3	3.1	29	3.88	75.14	81.74
RepVGG-B3 [13]	110.9	26.2	29	4.21	80.50	86.44
ViTAE-T [55]	4.8	1.5	67	13.37	75.3	82.9
ViTAE-S [55]	23.6	5.6	116	22.13	82.0	87.0
ViTAEV2-S [62]	19.2	5.7	130	24.53	82.6	87.6
ConvNextV2-A [53]	3.7	0.55	41	6.07	76.2	82.79
ConvNextV2-F [53]	5.2	0.78	41	6.17	78.0	84.08
ConvNextV2-P [53]	9.1	1.37	41	6.29	79.7	85.60
ConvNextV2-N [53]	15.6	2.45	47	6.85	81.2	-
ConvNextV2-T [53]	28.6	4.47	59	8.40	82.5	-
ConvNextV2-B [53]	88.7	15.4	113	15.41	84.3	-
Swin-T [36]	28.3	4.5	48	10.51	81.18	86.64
Swin-S [36]	49.6	8.7	96	20.25	83.21	87.60
CMT-T [19]	9.7	1.3	90	13.71	79.0	85.4
CMT-S [19]	26.26	4.05	135	23.09	83.4	88.2
ResNet-18-TNR [52]	11.7	1.8	18	3.12	70.6	79.4
ResNet-34-TNR [52]	21.8	3.7	34	5.57	75.5	83.4
ResNet-50-TNR [52]	25.6	4.1	50	7.64	79.8	85.7
VanillaNet-5	15.5	5.2	5	1.61	72.49	79.66
VanillaNet-6	32.5	6.0	6	2.01	76.36	82.86
VanillaNet-7	32.8	6.9	7	2.27	77.98	84.16
VanillaNet-8	37.1	7.7	8	2.56	79.13	85.14
VanillaNet-9	41.4	8.6	9	2.91	79.87	85.66
VanillaNet-10	45.7	9.4	10	3.24	80.57	86.25
VanillaNet-11	50.0	10.3	11	3.59	81.08	86.54
VanillaNet-12	54.3	11.1	12	3.82	81.55	86.81
VanillaNet-13	58.6	11.9	13	4.26	82.05	87.15
VanillaNet-13-1.5×	127.8	26.5	13	7.83	82.53	87.48
VanillaNet-13-1.5×†	127.8	48.9	13	9.72	83.11	87.85
training images and 50K validation images with 1000 categories. We utilize strong regularization since the proposed VanillaNet has large number of parameters in each layer to capture useful information from the images with limited non-linearity. We also report the ImageNet Real results where the labels are refined. The latency is tested on Nvidia A100 GPU.
We propose architecture for VanillaNet with different number of layers. Table 1 shows the classification results on the ImageNet dataset using different networks. We list the number of parameters, FLOPs, depth, GPU latency and accuracy for comparison. In the past decades, researchers focus on minimize the FLOPs or the latency in ARM/CPU for portable networks since they assume that the computing power in edge devices is limited. As the development of modern AI chips, several mobile devices such as driverless vehicle [31] and robots [16] are required and able to carry multiple GPUs with huge computing power for seeking real-time feedback of external inputs. Therefore, we test the GPU latency with batch size 1, which means that the AI chip has enough computing power to calculate each network. Under this situation, we find that the inference speed has little relationship with the number of FLOPs and parameters. Taking MobileNetV3-Large a an example,
(a) Accuracy v.s. depth
(b) Accuracy v.s. inference speed
Figure 2: Top-1 Accuracy on ImageNet v.s. inference speed on Nvidia A100 GPU with batch size 1. Size of the circle is related to the depth and parameters of each architecture in (a) and (b), respectively. VanillaNet achieves comparable performance with deep neural networks while with much smaller depth and latency.