Title: VanillaNet: the Power of Minimalism in Deep Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: computer vision

foundation models.
Abstract: At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at https://github.com/huawei-noah/VanillaNet and https://gitee.com/mindspore/models/tree/master/research/cv/vanillanet

VanillaNet: the Power of Minimalism in Deep Learning

Abstract

At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like selfattention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resourceconstrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at https://github.com/huawei-noah/VanillaNet and https://gitee.com/mindspore/models/tree/master/research/cv/ vanillanet .

1 Introduction

Over the past few decades, artificial neural networks have made remarkable progress, driven by the idea that increasing network complexity leads to improved performance. These networks, which consist of numerous layers with a large number of neurons or transformer blocks [50, 36], are capable of performing a variety of human-like tasks, such as face recognition [30], speech recognition [9], object detection [43, 17, 18], natural language processing [50], and content generation [3]. The impressive computational power of modern hardware allows neural networks to complete these tasks with both high accuracy and efficiency. As a result, AI-embedded devices are becoming increasingly prevalent in our lives, including smartphones, AI cameras, voice assistants, and autonomous vehicles.

Admittedly, one notable breakthrough in this field is the development of AlexNet [29], which consists of 12 layers and achieves state-of-the-art performance on the large-scale image recognition benchmark [8]. Building on this success, ResNet [23] introduces identity mappings through shortcut connections, enabling the training of deep neural networks with high performance across a wide range of computer vision applications, such as image classification [45], object detection [43], and semantic segmentation [38]. The incorporation of human-designed modules in these models, as well as the continued increase in network complexity, has undeniably enhanced the representational

56

28

224

56

512

1024

2048

4096

3

4096

1000

capabilities of deep neural networks, leading to a surge of research on how to train networks with more complex architectures [28, 24, 54] to achieve even higher performance.

Apart from convolutional architectures, Dosovitskiy et al.[14] have introduced the transformer architecture to image recognition tasks, demonstrating its potential for leveraging large-scale training data. Zhai et al.[60] investigated the scaling laws of vision transformer architectures, achieving an impressive 90.45% top-1 accuracy on the ImageNet dataset, which indicates that deeper transformer architectures, like convolutional networks, tend to exhibit better performance. Wang et al.[51] further proposed scaling the depth of transformers to 1,000 layers for even higher accuracy. Liu et al.[37] revisited the design space of neural networks and introduced ConvNext, achieving similar performance to state-of-the-art transformer architectures.

Although well-optimized deep and complex neural networks achieve satisfying performance, their increasing complexity poses challenges for deployment. For example, shortcut operations in ResNets consume significant off-chip memory traffic as they merge features from different layers [32]. Furthermore, complicated operations such as axial shift in AS-MLP [33] and shift window selfattention in Swin Transformer [36] require sophisticated engineering implementation, including rewriting CUDA codes.

These challenges call for a paradigm shift towards simplicity in neural network design. However, the development of ResNet has seemingly led to the abandonment of neural architectures with pure convolutional layers (without extra modules such as shortcuts). This is mainly due to the performance enhancement achieved by adding convolutional layers not meeting expectations. As discussed in [23], plain networks without shortcuts suffer from gradient vanishing, causing a 34-layer plain network to perform worse than an 18-layer one. Moreover, the performance of simpler networks like AlexNet [29] and VGGNet [45] has been largely outpaced by deep and complex networks, such as ResNets [23] and ViT [8]. Consequently, less attention has been paid to the design and optimization of neural networks with simple architectures. Addressing this issue and developing concise models with high performance would be of great value.

To this end, we propose VanillaNet, a novel neural network architecture emphasizing the elegance and simplicity of design while retaining remarkable performance in computer vision tasks. VanillaNet achieves this by eschewing excessive depth, shortcuts, and intricate operations such as self-attention, leading to a series of streamlined networks that address the issue of inherent complexity and are well-suited for resource-limited environments. To train our proposed VanillaNets, we conduct a comprehensive analysis of the challenges associated with their simplified architectures and devise a "deep training" strategy. This approach starts with several layers containing non-linear activation functions. As the training proceeds, we progressively eliminate these non-linear layers, allowing for easy merging while preserving inference speed. To augment the networks' non-linearity, we put forward an efficient, series-based activation function incorporating multiple learnable affine transfor-

mations. Applying these techniques has been demonstrated to significantly boost the performance of less complex neural networks. As illustrated in Figure 2, VanillaNet surpasses contemporary networks with elaborate architectures concerning both efficiency and accuracy, highlighting the potential of a minimalist approach in deep learning. This pioneering investigation of VanillaNet paves the way for a new direction in neural network design, challenging the established norms of foundation models and establishing a new trajectory for refined and effective model creation.

2 A Vanilla Neural Architecture

Over the past decades, researchers have reach some consensus in the basic design of neural networks. Most of the state-of-the-art image classification network architectures should consist of three parts: a stem block to transform the input images from 3 channels into multiple channels with downsampling, a main body to learn useful information, a fully connect layer for classification outputs. The main body usually have four stages, where each stage is derived by stacking same blocks. After each stage, the channels of features will expand while the height and width will decrease. Different networks utilize and stack different kinds of blocks to construct deep models.

Despite the success of existing deep networks, they utilize large number of complex layers to extract high-level features for the following tasks. For example, the well-known ResNet [23] requires 34 or 50 layers with shortcuts for achieving over 70% top-1 accuracy on ImageNet. The base version of ViT [14] consists of 62 layers since the query, key and value in self-attention require multiple layers to calculate.

With the growing of AI chips, the bottleneck of inference speed of neural networks would not be FLOPs or parameters, since modern GPUs can easily do parallel calculation with strong computing power. In contrast, their complex designs and large depths block their speed. To this end, we propose the vanilla network, i.e., VanillaNet, whose architecture is shown in Figure 1. We follow the popular design of neural network with the stem, main body and fully connect layer. Different with existing deep networks, we only employ one layer in each stage to establish a extremely simple network with as few layer as possible.

Here we show the architecture of the VanillaNet in details, which takes 6 layers as an example. For the stem, we utilize a 4 × 4 × 3 × C convolutional layer with stride 4 following the popular settings in [23, 36, 37] to map the images with 3 channels to features with C channels. At stage 1, 2 and 3, a maxpooling layer with stride 2 is used to decrease the size and feature map and the number of channels is increased by 2. At stage 4, we do not increasing the number of channels as it follows an average pooling layer. The last layer is a fully connected layer to output the classification results. The kernel size of each convolutional layer is 1 × 1 , since we aim to use minimal calculation cost for each layer while keep the information of feature maps. The activation function is applied after each 1 × 1 convolutional layer. To ease the training procedure of the network, batch normalization is also added after each layer. For the VanillaNet with different number of layers, we add blocks in each stage, which will be detailed in the supplementary material. It should be noted that the VanillaNet has no shortcut, since we empirically find adding shortcut shows little performance improvement. This also gives another benefit that the proposed architecture is extremely easy to implemented since there are no branch and extra blocks such as squeeze and excitation block [27].

Although the architecture of VanillaNet is simple and relatively shallow, its weak non-linearity caused limit the performance, Therefore, we propose a series of techniques to solve the problem.

3 Training of Vanilla Networks

It is common in deep learning to enhance the performance of models by introducing stronger capacity in the training phase [5, 57]. To this end, we propose to utilize a deep training technique to bring up the ability during training in the proposed VanillaNet, since deep network have stronger non-linearity than shallow network.

3.1 Deep Training Strategy

The main idea of deep training strategy is to train two convolutional layers with an activation function instead of a single convolution layer in the beginning of training procedure. The activation function is

gradually reduce to an identity mapping with the increasing number of training epochs. At the end of training, two convolutions can be easily merged into the one convolution to reduce the inference time. This kind of idea is also widely used in CNNs [11, 13, 10, 12]. We then describe how to conduct this technique in detail.

For an activation function A ( x ) (which can be the usual functions such ReLU and Tanh), we combine it with an identity mapping, which can be formulated as:

where λ is a hyper-parameter to balance the non-linearity of the modified activation function A ′ ( x ) . Denote the current epoch and the number of deep training epochs as e and E , respectively. We set λ = e E . Therefore, at the beginning of training ( e = 0 ), A ′ ( x ) = A ( x ) , which means the network have strong non-linearity. When the training converged, we have A ′ ( x ) = x , which means the two convolutional layers have no activation functions in the middle. We further demonstrate how to merge these two convolutional layers.

We first convert every batch normalization layer and its preceding convolution into a single convolution, which is a commonly used technique. We denote W ∈ R C out × ( C in × k × k ) , B ∈ R C out as the weight and bias matrices of convolutional kernel with C in input channels, C out output channels and kernel size k . The scale, shift, mean and variance in batch normalization are represented as γ, β, µ, σ ∈ R C out , respectively. The merged weight and bias matrices are:

where subscript i ∈ { 1 , 2 , ..., C out } denotes the value in i -th output channels.

After merging the convolution with batch normalization, we begin to merge the two 1 × 1 convolutions. Denote x ∈ R C in × H × W and y ∈ R C out × H ′ × W ′ as the input and output features, the convolution can be formulated as:

where ∗ denotes the convolution operation, · denotes the matrix multiplication and X ∈ R ( C in × 1 × 1) × ( H ′ × W ′ ) is derived from the im2col operation to transform the input into a matrix corresponding to the kernel shape. Fortunately, for 1 × 1 convolution, we find that the im2col operation becomes a simple reshape operation since there are no need for sliding kernels with overlap. Therefore, denote the weight matrix of two convolution layers as W 1 and W 2 , the two convolution without activation function is formulated as:

Therefore, 1 × 1 convolution can merged without increasing the inference speed. It is worth noticed that although the deep training technique will increase the training FLOPs, it would not affect the inference cost, which is much more important when implementation.

3.2 Series Informed Activation Function

There have been proposed several different activation functions for deep neural networks, including the most popular Rectified Linear Unit (ReLU) and its variants (PReLU [22], GeLU [25] and Swish [42]). They focus on bring up the performance of deep and complex networks using different activation functions. However, as theoretically proved by the existing works [40, 15, 47], the limited power of simple and shallow network are mainly caused by the poor non-linearity, which is different with deep and complex networks and thus has not been fully investigated.

In fact, there are two ways to improve the non-linearity of a neural network: stacking the non-linear activation layers or increase the non-linearity of each activation layer, while the trend of existing networks choose the former one, which results in high latency when the parallel computation ability is excess.

One straight forward idea to improve non-linearity of activation layer is stacking. The serially stacking of activation function is the key idea of deep networks. In contrast, we turn to concurrently stacking the activation function. Denote a single activation function for input x in neural network as

A ( x ) , which can be the usual functions such ReLU and Tanh. The concurrently stacking of A ( x ) can be formulated as:

where n denotes the number of stacked activation function and a i , b i are the scale and bias (which are learned parameters) of each activation to avoid simple accumulation. The non-linearity of the activation function can be largely enhanced by concurrently stacking. Equation 5 can be regarded as a series in mathematics, which is the operation of adding many quantities.

To further enrich the approximation ability of the series, we enable the series based function to learn the global information by varying the inputs from their neighbors, which is similar with BNET [56]. Specifically, given a input feature x ∈ R H × W × C , where H , W and C are the number of its width, height and channel, the activation function is formulated as:

where h ∈ { 1 , 2 , ..., H } , w ∈ { 1 , 2 , ..., W } and c ∈ { 1 , 2 , ..., C } . It is easy to see that when n = 0 , the series based activation function A s ( x ) degenerates to the plain activation function A ( x ) , which means that the proposed method can be regarded as a general extension of existing activation functions. We use ReLU as the basic activation function to construct our series since it is efficient for inference in GPUs.

We further analyze the computation complexity of the proposed activation function compared with its corresponding convolutional layer. For a convolutional layer with K kernel size, C in input channels and C out output channels, the computational complexity is:

while computation cost of its series activation layer is:

Therefore, we have:

Taking the 4th stage in VanillaNet-B as an example, where C in = 2048 , k = 1 and n = 3 , the ratio is about 84 . In conclusion, the computation cost of the proposed activation function is still much lower than the convolutional layers. More experimental complexity analysis will be shown in the following section.

It is noted that the proposed SIAF is significantly different with BNET [56]. The motivation of BNET is to improve BN's flexibility of fitting complex data distributions, which is resolved by introducing a linear transformation module with considering each neuron's neighborhood. In contrast, the motivation of the proposed SIAF is to enhance the non-linearity of shallow networks, which is implemented by combining multiple activation functions with different weights and biases. Empirically, as shown in Table 3 in our paper, the SIAF only works for shallow networks yet with huge improvements (>4.0% on ImageNet), while BNET is a general method with relatively little improvements ( 0.5% on ImageNet).

4 Experiments

In this section, we conduct experiments to verify the performance of the proposed VanillaNet on large scale image classification. Ablation study is provided to investigate effectiveness of each component of the proposed VanillaNet. We also visualize the feature of VanillaNet to further study how the proposed network learns from images.

4.1 Comparison with SOTA architectures

To illustrate the effectiveness of the proposed method, we conduct experiments on the ImageNet [8] dataset, which consists of 224 × 224 pixel RGB color images. The ImageNet dataset contains 1.28M

training images and 50K validation images with 1000 categories. We utilize strong regularization since the proposed VanillaNet has large number of parameters in each layer to capture useful information from the images with limited non-linearity. We also report the ImageNet Real results where the labels are refined. The latency is tested on Nvidia A100 GPU.

We propose architecture for VanillaNet with different number of layers. Table 1 shows the classification results on the ImageNet dataset using different networks. We list the number of parameters, FLOPs, depth, GPU latency and accuracy for comparison. In the past decades, researchers focus on minimize the FLOPs or the latency in ARM/CPU for portable networks since they assume that the computing power in edge devices is limited. As the development of modern AI chips, several mobile devices such as driverless vehicle [31] and robots [16] are required and able to carry multiple GPUs with huge computing power for seeking real-time feedback of external inputs. Therefore, we test the GPU latency with batch size 1, which means that the AI chip has enough computing power to calculate each network. Under this situation, we find that the inference speed has little relationship with the number of FLOPs and parameters. Taking MobileNetV3-Large a an example,

5

10

20

40

80

160

Depth (in log scale)

70

72

74

76

78

80

82

Accuracy (%)

VanillaNet-5

VanillaNet-11

VanillaNet-13-1.5x

ResNet-18-TNR

ResNet-34-TNR

ResNet-50-TNR

DenseNet-121

ParNet-S

ParNet-M

ParNet-L

ParNet-XL

MobileNetV2

RegNet-04

RegNet-08

MobileNetV3-L

VGG11-BN

VGG13-BN

VGG16-BN

RepVGG-A0

RepVGG-A1

RepVGG-B0

RepVGG-B3

ConvNeXtV2-F

ConvNeXtV2-P

ConvNeXtV2-T

Swin-T

Swin-S

EfficientNet-B0

ViT-L

DenseNet-161

ViTAE-T

ViTAE-S

ViTAEV2-S

2

4

8

16

28

72

74

76

78

80

82

Accuracy (%)

VanillaNet-5

VanillaNet-10

VanillaNet-13

ResNet-34-TNR

ResNet-50-TNR

MobileNetV3-L

RepVGG-A0

RepVGG-A1

RepVGG-B0

RepVGG-B3

ConvNeXtV2-F

ConvNeXtV2-P

ConvNeXtV2-N

ConvNeXtV2-T

Swin-T

Swin-S

ViTAE-T

ViTAE-S

ViTAEV2-S

VanillaNet-13-1.5x

VanillaNet-13-1.5x

(b) Accuracy v.s. inference speed

though it has a very low FLOPs (0.22B), its GPU latency is 7.83, which is even larger than our VanillaNet-13 with a 11.9B FLOPs. In fact, the inference speed in this setting is highly related to the complexity and number of layers. We can compare the inference speed of ShuffleNetV2x1.5 and ShuffleNetV2x2. In fact, their difference only lies in the number of channels. Therefore, although their number of parameters and FLOPs differs a lot. (0.3B v.s. 0.6B), their inference speed is nearly the same (7.23 and 7.84). We can also find in Table 1 that the straightforward architecture including ResNet, VGGNet and our VanillaNet without extra branch and complex blocks ( e.g., squeeze and excitation block or densely connects) achieves the highest inference speed.

To this end, we propose the VanillaNet, which is simple and has few convolutional layers without any branch (even without shortcut). We set different number of layers in VanillaNets to construct a series of networks. As shown in Table 1, the VanillaNet-9 achieves a 79.87% accuracy with only a 2.91ms inference speed in GPU, which is over 50% faster than the ResNet-50 and ConvNextV2-P with similar performance. The surprising result demonstrate the potential of VanillaNet in real-time processing over the existing deep networks. We also scale the number of channels and the pooling size to obtain the proposed VanillaNet-13-1.5 × † , which achieves an 83.11% Top-1 accuracy on ImageNet, which suggests that the proposed vanilla neural network still have power to obtain such a high performance on large scale image classification task. It is suggested that we may not need deep and complex networks on image classification since scaling up VanillaNets can achieve similar performance with deep networks. Although the proposed VanillaNet requires much more FLOPs and parameters, its inference speed in GPUs is much faster. In fact, the inference speed in GPUs and the number of parameters or FLOPs are two different aspects to measure the performance of neural networks. According to the 'no free lunch' rule, it is nearly impossible to be satisfactory to both sides. Therefore, the proposed VanillaNet aims to the inference speed in GPUs and achieve much more higher speed then existing SOTA architectures.

The Figure 2 shows the depth and inference speed of different architectures. The inference speed with batch size 1 is highly related to the depth of the network instead of the number of parameters, which suggest that simple and shallow networks have huge potential in real-time processing. It can be easily find that the proposed VanillaNet achieve the best speed-accuracy trade-off among all these architectures with low GPU latency, which demonstrates the superiority of the proposed VanillaNet when the computing power is sufficient.

4.2 Ablation Study

In this section, we conduct ablation study to investigate the effectiveness of proposed modules, including the series activation function and the deep training technique. Besides, we analyze the influence of adding shortcuts in the proposed VanillaNet.

Influence of number of series in activation function. In the above section, we propose a series activation function to enhance the performance of plain activation function and enable global information exchange in feature maps. Table 2 shows the performance of the proposed VanillaNet using different number of n in Equation 6. When n = 0 , the activation function degenerate into the plain ReLU activation function. Although the inference speed of this network is

higher than using the series activation function, the network can only achieve a 60.53% top-1 accuracy on the ImageNet dataset, which cannot be applied in real-world applications. It proves that the poor non-linearity of activation function results in poor performance of vanilla networks.

To this end, we propose the series activation function. When n = 1 , the network can achieve a 74.53% accuracy, which is a huge improvement compared with 60.53%. The result demonstrate the effectiveness of the proposed activation function. When the number of n increases, the performance of the network brings up. We find that n = 3 is a good balance in the top-1 accuracy and latency. Therefore, we use n = 3 for the rest experiments. It should be noted that the FLOPs of the proposed activation function is very small compared with the original network, which is the same as the conclusion we derive in Equation 9.

Influence of deep training. As the VanillaNet is very shallow, we propose to increase the training nonlinearity to bring up its performance. We then analyze the effectiveness of the proposed deep training technique. Table 3 shows the results on using deep training technique with VanillaNet-6. As a result, the original VanillaNet achieves a 75.23% top-1 accuracy, which is the baseline. By using the deep training technique, the proposed VanillaNet can achieve a 76.36% accuracy. The results demonstrate that the proposed deep training technique is useful for the shallow network.

Moreover, we further apply the deep training and series activation function in other networks to show the generalization ability of the two techniques. Table 3 reports the results of AlexNet and ResNet-50, which are two classical deep neural networks, on the ImageNet dataset. The original AlexNet can only acheive a 57.52% accuracy with 12 layers. By applying the proposed deep training and series activation function, the performance of AlexNet can be largely brought up by about 6%, which demonstrates that the proposed technique is highly effective for shallow networks. When it turns to ResNet-50 whose architecture are relatively complex, the performance gain is little. This results suggests the deep and complex networks already have enough non-linearity without the proposed techniques.

Influence of shortcuts. In deep neural networks, a common sense is that adding shortcut can largely ease the training procedure and improve the performance [23]. To this end, we investigate whether shortcut would benefit the performance of shallow and simple network. We propose to use two kinds of location of shortcut, i.e., shortcut after the activation function and shortcut before the activation function, which are proposed in the original ResNet [23] and PreAct-ResNet [24], respectively. Since the number

of channels is large and the original convolution is with kernel size 1 × 1 in VanillaNet, adding a shortcut (even with 1 × 1 kernel size) would largely increase the FLOPs. Therefore, we use the

parameter-free shortcut. It should be noted that if the stride is 2, the parameter-free shortcut will use an average pooling to decrease the size of feature maps and if the number of channel is increasing, the parameter-free shortcut utilizes padding for the extra channels following the original setting in [23].

Table 4 shows the ablation study on adding shortcuts. We surprisingly find that using shortcuts, in spite of any type of shortcuts, has little improvement on the performance of the proposed VanillaNet. We suggest that the bottleneck of shallow networks is not the identity mapping, but the weak nonlinearity. As reported in ResNet [23], the error rate of 18 layers' plain network (without shortcut) is 27.94%, which is similar to 18 layers' ResNet (27.88%). This result also suggests that shortcuts are not necessary for shallow networks. In fact, the shortcut is useless for bringing up the non-linearity and may decrease non-linearity since the shortcut skips the activation function to decrease the depth of the vanilla network, therefore results in lower performance.

Performance of transfer learning. To further demonstrate the generalization ability of the proposed VanillaNet, we fine-tune the proposed VanillaNet (pre-trained on ImageNet) on the well-known longtailed classification dataset: iNaturalist 2018 [49] and follow the setting in Deit [48]. As shown in the Table 5, the performance of VanillaNet in pretraining scenarios is similar with other methods, which demonstrates its universality.

4.3 Experiments on COCO

To further demonstrate the effectiveness of the proposed VanillaNet on downstream tasks, we conduct evaluation in the COCO dataset [35]. We use RetinaNet [34] and Mask-RCNN [21] as the framework to evaluate the proposed method. FPS is measured on Nvidia A100 GPU. Details can be found in supplementary materials.

Table 6 shows the performance of the proposed VanillaNet on COCO detection and segmentation. The proposed VanillaNet can successfully achieve similar performance with the ConvNext and the Swin backbone. Although the FLOPs and Parameters of VanillaNet is much higher than Swin and ConvNext, it has much higher FPS, which demonstrates the effectiveness of vanilla architectures on object detection and instance segmentation tasks.

5 Conclusion

This paper fully investigates the feasibility of establishing neural networks with high performance but without complex architectures such as shortcut, high depth and attention layers, which embodies the paradigm shift towards simplicity and elegance in design. We present a deep training strategy and the series activation function for VanillaNets to enhance its non-linearity during both the training and testing procedures and bring up its performance. Experimental results on large scale image classification datasets reveal that VanillaNet performs on par with well-known deep neural networks and vision transformers, thus highlighting the potential of minimalism in deep learning. We will further explore better parameter allocation for efficient VanillaNet architectures with high performance. In summary, we prove that it is possible to achieve comparable performance with the state-of-the-art deep networks and vision transformers using a very concise architecture, which will unlock the potential of vanilla convolutiaonal network in the future.

References

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159 , 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV) , pages 839-847. IEEE, 2018.

Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, Chunjing Xu, and Tong Zhang. Universal adder neural networks. arXiv preprint arXiv:2105.14202 , 2021.

Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 , 2020.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 702-703, 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Li Deng, Geoffrey Hinton, and Brian Kingsbury. New types of deep neural network learning for speech recognition and related applications: An overview. In 2013 IEEE international conference on acoustics, speech and signal processing , pages 8599-8603. IEEE, 2013.

Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1911-1920, 2019.

Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Diverse branch block: Building a convolution as an inception-like unit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10886-10895, 2021.

Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11963-11975, 2022.

Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13733-13742, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Conference on learning theory , pages 907-940. PMLR, 2016.

Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA) , pages 3389-3396. IEEE, 2017.

Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2154-2164, 2021.

Jianyuan Guo, Kai Han, Yunhe Wang, Chao Zhang, Zhaohui Yang, Han Wu, Xinghao Chen, and Chang Xu. Hit-detector: Hierarchical trinity architecture search for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11405-11414, 2020.

Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12175-12185, 2022.

Zhiwei Hao, Jianyuan Guo, Kai Han, Han Hu, Chang Xu, and Yunhe Wang. Vanillakd: Revisit the power of vanilla knowledge distillation from small scale to large scale. arXiv preprint arXiv:2305.15781 , 2023.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 630-645. Springer, 2016.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1314-1324, 2019.

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7132-7141, 2018.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. PMLR, 2015.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS , pages 1097-1105, 2012.

Steve Lawrence, C Lee Giles, Ah Chung Tsoi, and Andrew D Back. Face recognition: A convolutional neural-network approach. IEEE transactions on neural networks , 8(1):98-113, 1997.

Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Soeren Kammel, J Zico Kolter, Dirk Langer, Oliver Pink, Vaughan Pratt, et al. Towards fully autonomous driving: Systems and algorithms. In 2011 IEEE intelligent vehicles symposium (IV) , pages 163-168. IEEE, 2011.

Guilin Li, Junlei Zhang, Yunhe Wang, Chuanjian Liu, Matthias Tan, Yunfeng Lin, Wei Zhang, Jiashi Feng, and Tong Zhang. Residual distillation: Towards portable deep neural networks without shortcuts. Advances in Neural Information Processing Systems , 33:8935-8946, 2020.

Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. As-mlp: An axial shifted mlp architecture for vision. arXiv preprint arXiv:2107.08391 , 2021.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012-10022, 2021.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11976-11986, 2022.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR , pages 3431-3440, 2015.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.

Hrushikesh N Mhaskar and Tomaso Poggio. Deep vs. shallow networks: An approximation theory perspective. Analysis and Applications , 14(06):829-848, 2016.

Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization , 30(4):838-855, 1992.

Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941 , 2017.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS , pages 91-99, 2015.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Matus Telgarsky. Benefits of depth in neural networks. In Conference on learning theory , pages 1517-1539. PMLR, 2016.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8769-8778, 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00555 , 2022.

Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476 , 2021.

Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. arXiv preprint arXiv:2301.00808 , 2023.

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1492-1500, 2017.

Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances in Neural Information Processing Systems , 2021.

Yuhui Xu, Lingxi Xie, Cihang Xie, Wenrui Dai, Jieru Mei, Siyuan Qiao, Wei Shen, Hongkai Xiong, and Alan Yuille. Bnet: Batch normalization with enhanced linear transformation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quantization networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7308-7316, 2019.

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962 , 2019.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6023-6032, 2019.

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12104-12113, 2022.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond. International Journal of Computer Vision , 2023.

A Network Architectures

The detailed architecture for VanillaNet with 7-13 layers can be found in Table 7, where each convolutional layer is followed with an activation function. For the VanillaNet-13-1.5 × , the number of channels are multiplied with 1.5. For the VanillaNet-13-1.5 × † , we further use adaptive pooling for stage 2,3 and 4 with feature shape 40 × 40, 20 × 20 and 10 × 10, respectively.

B Training Details

For classification on ImageNet, we train the VanillaNets for 300 epochs utilizing the cosine learning rate decay [39]. The λ in Equ. 1 is linearly decayed from 1 to 0 on epoch 0 and 100, respectively. The training details can be fould in Table 8. For the VanillaNet-11, since the training difficulty is relative large, we use the pre-trained weight from the VanillaNet-10 as its initialization. The same technique is adopt for VanillaNet-12/13.

For detection and segmentation on COCO, we train all the networks using 12 epochs, multi-scale training augmentation and a linear learning rate decay for fair comparison. Following ConvNextV2 [53] which utilize self-supervised training, we use the ImageNet pre-trained weight using knowledge distillation [20] with n = 4 for a higher receptive field. We train the VanillaNet-13 using the Adamw optimizer with a batch size of 32, an initial learning rate of 8e-5 for RetinaNet and 1.3e-4 for Mask RCNN, an 0.05 weight decay and an 0.6 layer wise decay.

(a)Mis-classified by ResNet-50-TNR

(c)Mis-classified by VanillaNet-9

(d)Correctly classified by VanillaNet-9

C Visualization of Attention

To have a better understanding of the proposed VanillaNet, we further visualize the features using GradCam++ [4], which utilizes a weighted combination of the positive partial derivatives of the feature maps generated by the last convolutional layer with respect to the specific class to generate a good visual explanation.

Figure 3 shows the visualization results for VanillaNet-9 and ResNets-50-TNR [52] with similar performance. The red color denotes that there are high activation in this region while the blue color denotes the weak activation for the predicted class. We can find that these two networks have different attention maps for different samples. It can be easily found that for ResNet-50, the area of active region is smaller. For the VanillaNet with only 9 depth, the active region is much larger than that of deep networks. We suggest that VanillaNet may be strong in extract all relative activations in the input images and thoroughly extract their information by using large number of parameters and FLOPs. In contrast, VanillaNet may be weak on analyzing part of the useful region since the non-linearity is relatively low.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to minimize the complex design of modern deep models. The authors propose an architecture, VanillaNet, which eliminates high depth, residual connections, non-linear activations, self-attentions, and normalization layers, but during training, the model still use some components, such as activations and normalizations, to help model training. They also propose an activation function series to boost the capacity and performance of the models. Due to the simplicity, the model is ideal for resource-constrained environments and has lower latency than complex deep models. Their experiments show that the proposed model has similar performance than modern deep models.\n\nSTRENGTHS:\n1. The proposed model does not have high depth, residual connection, and normalization layers, which is beneficial for latency. \n2. Despite the lack of complex design, the performance of the proposed idea is comparable to modern deep models. \n3. The proposed activation function series offers performance boost with slightly more FLOPs and parameters.\n\nWEAKNESSES:\n1. It would be better to have a figure indicating where the merging operation describe in section 3.1 occurs in the model, so it is difficult to see where the merging opportunities are.  \n2. The idea of merging batch normalization with convolution describe between line 132 and line 137 is an already proposed inference acceleration called Batch Normalization Folding. Please make a clear distinction between what design are novel in this paper and what design are coming from existing work. \n3. The content between lines 177-183 calculates the complexity (which directly tie to FLOPs) of the activation function series and claims the cost is very small, but in lines 90-91, the authors claims FLOPs is not important, so they are contradicting. Rather, it would make more sense to compare the latency of proposed activation and convolution. \n4. The latency (profiled using a batch of size 1) in Table 4 could be misleading. In many inference cases, multiple instances can be batched together to improve throughput and make better usage of available hardwares, so it would be much better if throughput or latency (measured using reasonably large batch size) is also provided. \n5. Code is not provided.\n\nQUESTIONS:\nThe main concerns are listed above. In addition, is the proposed activation function series used in inference? If so, then the model still has activations and batch normalization (although the later one is folded into convolution), and compared ResNet, only the residual connections are removed. If not, does this mean all convolutions in each stage can be merged into a single convolutions? \n\nThis is an important question since the price of VanillaNet is that the parameter count and FLOPs need to be significantly higher to achieve similar performance compared to baselines.\n\nLIMITATIONS:\nThere is no discussion on limitations. You should include a small limitation section. For example, you could include a discussion about the need for higher parameter count and FLOPs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper presents VanillaNet, a neural network architecture that emphasizes simplicity by avoiding high depth, shortcuts, and complex operations like self-attention. The proposed VanillaNet performs comparably to well-known deep neural networks and vision transformers.

STRENGTHS:
**Originality**: To the best of my knowledge, the proposed VanillaNet is novel.

**Quality**:     This paper is of high quality since it proposed a new model architecture and conducted extensive experiments on ImageNet and ablation studies to demonstrate its effectiveness. 

**Clarity**: The paper is well-structured and clear, providing detailed explanations of the VanillaNet architecture and the proposed techniques.

**Significance**: The paper is significant as it paves the way for a new direction in neural network design.

WEAKNESSES:
-  Although the proposed VanillaNet reduced the latency a lot, the required Params and FLOPs increased a lot. e.g. FOR SWIN-S and VannillaNet that have comparable accuracy on ImageNet, the proposed vanillaNet needs 127.8 M and 48.9 FLOPs while Swin-S only needs 49.6 M and 8.7B FLOPs.

QUESTIONS:
- Since pre-training significantly impacts the model's performance, it would be better to see how the proposed model architecture performs in pre-training scenarios.

LIMITATIONS:
the authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces VanillaNet, a neural network architecture that emphasizes simplicity in design while maintaining remarkable performance in computer vision tasks. VanillaNet achieves this by avoiding excessive depth, shortcuts, and intricate operations like self-attention. The authors propose a \"deep training\" strategy to train VanillaNets, which starts with several layers containing non-linear activation functions and progressively eliminates them as training proceeds. Additionally, an efficient, series-based activation function incorporating multiple learnable affine transformations is introduced to augment the networks' non-linearity. The experimental results show that VanillaNet achieves comparable or even better performance than state-of-the-art deep neural networks and vision transformers on various computer vision tasks while being more efficient and resource-friendly.\n\nSTRENGTHS:\n1. This paper is well-written and organized, with clear explanations of the proposed VanillaNet architecture. The paper also provides detailed experimental setups and results, making it easy to understand and reproduce the experiments.\n2. The proposed VanillaNet is novel. By avoiding excessive depth, shortcuts, and intricate operations like self-attention, VanillaNet offers a streamlined and efficient design. This simplicity not only makes VanillaNet well-suited for resource-limited environments but also challenges the prevailing notion that increasing network complexity leads to improved performance, which may have implications for future research in the field.\n3. The experimental results are convincing. Despite its simplicity and low depth, VanillaNet delivers performance on par with renowned deep neural networks and vision transformers. For example, a 13 layers' VanillaNet achieves an 83% top-1 accuracy on ImageNet.\n\nWEAKNESSES:\n1. While VanillaNet's simplicity and low depth make it an attractive option for efficient deployment in GPUs, one weakness of the proposed method is that it has significantly more parameters and FLOPs than some existing methods. It is noted that VanillaNet has a large number of parameters in each layer to capture useful information from the images with limited non-linearity. This can make it more challenging to train and deploy VanillaNet on certain devices with limited computational resources. \n2. The proposed deep training strategy is used to enhance the performance of VanillaNet by introducing stronger capacity during training, which suggests that the proposed method requires more training FLOPs.\n\nQUESTIONS:\n1. Table 3 shows that shortcuts are useless for the proposed VanillaNets. As the shortcuts are one of the most popular components in modern networks, it would be better to provide more explainations about this phenomenon. \n2. As shown in the suppmentary material, the VanillaNet-13 is achieved by adding convolutions on the 3rd stage of VanillaNet-7, what is the designed logic behind?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VanillaNet, a neural network architecture that emphasizes simplicity in design. By discarding commonly used ingredients, e.g., high depth, shortcuts, and self-attention mechanism, VanillaNet is refreshingly concise. VanillaNet series have only 5-13 layers. The authors devise the “deep training” strategy, in which nonlinear activation functions are pruned after training to restore the original architecture. They also leverage series-based activation function to incorporate multiple learnable affine transformations. The experiments on classification and detection demonstrate the effectiveness of the proposed VanillaNet.\n\nSTRENGTHS:\nMotivation: This paper is different from the previous research direction of the backbone architecture, abandoning the exploration of the residual connection or deepen the network. It shows the potential of the shallow network. The proposed VanillaNet avoids complex operations and make the structure more compact and easy to understand.\n\nQuality: The paper is clear and easy to follow.\n\nOriginality: The “deep training technique” proposed in this paper is well motivated and effective. The activation function is gradually reduced to an identity mapping during training. And finally two convolutions can be easily merged into one convolution. I like this idea. In addition, the “series activation function” which enables global information exchange also seems effective.\n\nExperiment: The authors conduct sufficient experiments and ablation studies to demonstrate the effectiveness of the proposed VanillaNet. VanillaNet can achieve comparable performance to current SOTA models on several computer vision tasks with faster inference speed\n\nOverall, the motivation challenges the “more is different” foundational model philosophy, opening up new possibilities for efficient deployment of deep learning models.\n\nWEAKNESSES:\nAbout the application of “deep training technique”. It seems that this technique can only be apllied to BN. If the normalization layer is LN, it cannot be merged.\n\n\nAbout the number of parameters. Although the inference speed of VanillaNet is superior to other methods, the heavy number of paramters and “deep train” strategy may increase the training cost of the network. According to my understanding, the amount of network parameters in training stage should be much larger than the paratmers shown in Table 4 (in inferece stage).\n\nQUESTIONS:\nInference speed: The authors report the latency on GPU and pytorch. However, in more scenarios (or platform), such as tensorRT, will the gap between VanillaNet and other networks become smaller?\n\nSeries Informed Activation Function: What’s the difference between SIAF and BNET? I think these two methods should be explained in more details.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a minimalist neural network backbone architecture as an alternative to current ResNets and ViTs. The minimalism is focussed on faster inference, avoiding complex operations with higher time complexity when implemented on modern GPUs. In their proposed architecture, individual operations are limited to convolutions, activation functions and linear projections, avoiding more time-complex operations like skip connections or self-attention. The resulting 6-layer network surpass latest CNN and ViT backbones at a fraction of inference latency on ImageNet top-1 accuracy metrics. A novel deep training strategy (merge two consecutive layers post-training) and series activation functions (stacked activation functions for more complex non-linearity under lower time complexity) are introduced to facilitate better learning by these minimalist shallow networks. In addition to the base architecture, a series of scaled-up versions provide increasing performance, still at minimal inference latency.\n\nSTRENGTHS:\n1) Most of the paper is extremely well written with good logical flow, strong motivation for each proposal, and clear explanations of methodology. \n2) Proposed deep training and series activation functions appear novel, well-motivated, and interesting. \n3) Thorough ablations and discussion clearly demonstrate motivation for each design choice\n4) Strong results over SOTA backbones for classification, detections, and segmentation \n5) Detailed training details (in supp.) supports reproducibility of results\n\nWEAKNESSES:\n1) Missing details in series activation function: it appears scale and bias terms (Eq 5, 6) are learned parameters - please explicitly mention this or otherwise explain how these values are selected. Assuming former, how are these values initialized? Does the initialization affect final outcome (maybe conduct some ablation)? What are the final values for these parameters that are learned across runs (do they collapse in any occasion? do they convey any information about data distribution)? Can these parameters be fixed at start instead of learning? \n2) Writing (minor): grammatical errors in numerous places across sections 3 & 4, captions of figures not descriptive enough or self-contained (e.g. Table 4 - explain terms like depth, real-acc vs acc in caption - reader needs to skim through main text to understand headers of table).    \n3) More evaluations could strengthen the paper (and convince about universality of new backbone): maybe include results on some long-tailed classification datasets, more than one detection and segmentation dataset.    \n4) Section 4.2 & Figure 2 (attention map) does not appear to convey anything useful in current form. Consider updating description / main text to discuss one of the examples in detail (or maybe move to supplementary).\n\nQUESTIONS:\nSee 1 in weaknesses\n\nLIMITATIONS:\nNo clear discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-15"}